Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004396 IMPLEMENTATION COMPLETION AND RESULTS REPORT IDA 47790 ON A CREDIT IN THE AMOUNT OF SDR 66.2 MILLION (US$100 MILLION EQUIVALENT) TO THE SOCIALIST REPUBLIC OF VIETNAM FOR THE PROJECT PREPARATION TECHNICAL ASSISTANCE FACILITY PROJECT ( P118610 ) {April 20, 2018} Transport & Digital Development Global Practice East Asia And Pacific Region CURRENCY EQUIVALENTS (Exchange Rate Effective Apr 20, 2018) Vietnamese Dong Currency Unit = (VND) US$1 = VND 22,772.00 US$1 = SDR 0.69 FISCAL YEAR July 1 - June 30 Regional Vice President: Victoria Kwakwa Country Director: Ousmane Dione Senior Global Practice Director: Jose Luis Irigoyen Practice Manager: Almud Weitz Task Team Leader(s): Nga Thi Quynh Dang ICR Main Contributor: Rukshan Rodriguez, Cia Sjetnan ABBREVIATIONS AND ACRONYMS CPF Country Partnership Framework CPS Country Partnership Strategy DO Development Outcome DPL Development Policy Loan EAPDE East Asian and Pacific Development Effectiveness EMCC Vietnam Economic Management and Competitiveness Credit FERD Foreign Economic Relations Department FM Financial Management GDP Gross Domestic Product GoV Government of Vietnam HCMC Ho Chi Minh City IA Implementing Agency ICR Implementation Completion and Results Report INT Institutional Integrity IP Implementation Progress ISR Implementation Status and Results Report M&E Monitoring and Evaluation MCDP Medium Cities Development Project MOF Ministry of Finance MPI Ministry of Planning and Investment MTR Mid-Term Review NMUP Northern Mountains Urban Development Program ODA Official Development Assistance OOG Office of Government (equivalent to Prime Minister’s Office) PAD Project Appraisal Document PCN Project Concept Note PCU Project Coordination Unit PDO Project Development Objective PforR Program-for-Results PHRD Policy and Human Resources Development PMU Project Management Unit PPA Public Procurement Agency PPF Project Preparation Facility PPTAF Project Preparation Technical Assistance Facility SBV State Bank of Vietnam SEDP Socio-Economic Development Plan SEDS Socio-Economic Development Strategy SME Small- and Medium-Size Enterprises TA Technical Assistance TTL Task Team Leader TABLE OF CONTENTS DATA SHEET ........................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 6 A. CONTEXT AT APPRAISAL .........................................................................................................6 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) .......................................9 II. OUTCOME .................................................................................................................... 11 A. RELEVANCE OF PDOs ............................................................................................................ 11 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 13 C. EFFICIENCY ........................................................................................................................... 20 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 22 E. OTHER OUTCOMES AND IMPACTS (IF ANY) ............................................................................ 22 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 22 A. KEY FACTORS DURING PREPARATION ................................................................................... 22 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 23 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 24 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 24 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 26 C. BANK PERFORMANCE ........................................................................................................... 27 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 28 V. LESSONS AND RECOMMENDATIONS ............................................................................. 29 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 31 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 37 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 39 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 40 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 41 ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 42 The World Bank Project Preparation Technical Assistance Facility Project (P118610) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P118610 Project Preparation Technical Assitance Facility Project Country Financing Instrument Vietnam Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Organizations Borrower Implementing Agency Socialist Republic of Vietnam MPI Project Development Objective (PDO) Original PDO The Project Development Objective is that Government agencies have increased capacity to plan and prepare public investments efficiently and to international quality standards. Project outcomes will include: (a) improved quality at entry, (b) shortened preparation periods and time to realize project benefits, (c) increased capacity within government agencies and the national consulting industry to produce project preparation documents of good quality, to an acceptable schedule. Page 1 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 100,000,000 58,142,356 52,797,321 IDA-47790 Total 100,000,000 58,142,356 52,797,321 Non-World Bank Financing Borrower 0 0 0 Total 0 0 0 Total Project Cost 100,000,000 58,142,356 52,797,321 KEY DATES FIN_TABLE_DAT Approval Effectiveness MTR Review Original Closing Actual Closing 24-Jun-2010 13-Oct-2010 15-Apr-2013 31-Dec-2015 31-Oct-2017 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 12-May-2014 21.88 28-Dec-2015 41.09 Change in Results Framework Change in Loan Closing Date(s) Cancellation of Financing 23-Mar-2017 50.85 Change in Results Framework Change in Components and Cost Cancellation of Financing Reallocation between Disbursement Categories KEY RATINGS Outcome Bank Performance M&E Quality Moderately Unsatisfactory Moderately Satisfactory Modest Page 2 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 28-Jun-2011 Satisfactory Moderately Satisfactory 5.00 Moderately 02 14-Apr-2012 Unsatisfactory 5.37 Unsatisfactory Moderately 03 01-Dec-2012 Moderately Unsatisfactory 5.37 Unsatisfactory Moderately 04 20-Jun-2013 Moderately Satisfactory 11.48 Unsatisfactory Moderately 05 12-Jan-2014 Moderately Unsatisfactory 19.14 Unsatisfactory Moderately 06 25-Jun-2014 Moderately Unsatisfactory 21.88 Unsatisfactory Moderately 07 24-Sep-2014 Moderately Unsatisfactory 25.38 Unsatisfactory Moderately 08 01-Apr-2015 Moderately Unsatisfactory 30.81 Unsatisfactory Moderately 09 20-Oct-2015 Moderately Unsatisfactory 38.07 Unsatisfactory 10 15-Apr-2016 Moderately Satisfactory Moderately Satisfactory 46.31 11 17-Nov-2016 Moderately Satisfactory Moderately Satisfactory 50.85 12 03-Jun-2017 Moderately Satisfactory Moderately Satisfactory 50.85 13 12-Oct-2017 Moderately Satisfactory Moderately Satisfactory 51.63 SECTORS AND THEMES Sectors Major Sector/Sector (%) Agriculture, Fishing and Forestry 14 Other Agriculture, Fishing and Forestry 14 Page 3 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) Education 6 Other Education 6 Energy and Extractives 15 Other Energy and Extractives 15 Transportation 38 Urban Transport 38 Water, Sanitation and Waste Management 27 Other Water Supply, Sanitation and Waste 27 Management Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Private Sector Development 77 Jobs 7 Job Creation 7 Public Private Partnerships 10 Regional Integration 60 Urban and Rural Development 34 Urban Development 27 Urban Infrastructure and Service Delivery 27 Rural Development 7 Rural Infrastructure and service delivery 7 ADM STAFF Role At Approval At ICR Regional Vice President: James W. Adams Victoria Kwakwa Page 4 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) Country Director: Victoria Kwakwa Ousmane Dione Senior Global Practice Director: Hoonae Kim Jose Luis Irigoyen Practice Manager: Hoonae Kim Almud Weitz Task Team Leader(s): Paul Vallely Nga Thi Quynh Dang ICR Contributing Author: Joseph Rukshan Rodriguez Page 5 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. Vietnam has experienced remarkable economic growth, human development improvements, and poverty reduction over the last thirty years. A large and impressive public investment program has been one of the main contributing factors to these significant achievements. The county’s 2006-2010 Socio-Economic Development Plan (SEDP), under implementation at the time of the preparation of the Project Preparation Technical Assistance Facility Project (PPTAF), called for a large increase in investment, particularly for infrastructure development. Official Development Assistance (ODA) sources were estimated to account for approximately $11 billion of such planned investment. This level of investment was expected to continue into the following SEDP cycle (2011 to 2015). 2. Despite high levels of public investment, it was acknowledged that the efficiency and speed of investments could be significantly improved—to maximize economic and social benefits. Delays in preparation and implementation of public investment generally—and infrastructure investment in particular—led the government, in 2009, to establish an ODA Inter-Ministerial Task Force. The objective of this task force was to address delays in the implementation of ODA financed investments (the stock of undisbursed ODA funds totaling about $11 billion at the end of 2008). The World Bank supported this Task Force and its work—as a member of the Six Banks group and through the Vietnam Economic Management and Competitiveness Credit (EMCC) for public investment reform (a $500M Development Policy Loan (DPL) covering project selection, project implementation, financial management, and project oversight). 3. World Bank projects were also experiencing delays in implementation and resources for project preparation were diminishing. There were significant time lags between Board approval of new World Bank lending projects and effectiveness, and disbursement was usually slow in the first years of implementation. Project readiness was identified as one of the reasons for these delays, where critical studies, including feasibility studies, and procurement processes, were usually not initiated until after Board approval—for lack of resources. In parallel, as a result of the 2008 global financial crisis, there was a decrease in the availability of financing through trust funds (including the closure of the Japan Policy and Human Resources Development (PHRD) grant) available to complement World Bank budget in project preparation—further accentuating the problem. Slow processing of resources from Project Preparation Facilities (PPF) mobilized for Vietnam meant PPFs often came late in the preparation process. The 2011 World Bank Country Partnership Strategy (CPS) and 2015 Performance and Learning Review (PLR) for Vietnam, as well as portfolio reviews held, highlight these issues. 4. The PPTAF, approved by the World Bank Group (WBG) Board of Directors on June 24, 2010, was designed to support the Government of Vietnam’s (GoV) work to improve efficiency and quality of public investments, including implementation of World Bank projects. Drawing on the ODA Task Force, a GoV-Six Banks joint action plan, and the DPL reform agenda, the PPTAF aimed, through technical assistance (TA) and a project preparation financing facility, to strengthen capacity for project preparation and planning for critical investments, including concessional financing for key infrastructure during the last two International Development Association (IDA) cycles (Vietnam graduated from IDA on June 30, Page 6 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) 2017. All IDA available to Vietnam was committed at the end of IDA16 and IDA17). The facility could be used to finance feasibility studies, detailed designs, and procurement for projects across all development sectors and associated policy support. The PPTAF aimed to strengthen government capacity—across agencies and at all levels—in project preparation and planning—ultimately leading to improvements in efficiency and quality of public investments. This included bringing international standards into planning and preparation processes. As project implementation advanced, the PPTAF also intended to strengthen the capacity of national consulting companies. 5. The PPTAF was a new and unique way of supporting project preparation and planning. GoV and other national stakeholders had no previous experience implementing a project such as the PPTAF. Within the World Bank, there was limited, but somewhat dated related experience. The project was designed to address specific constraints in the preparation and planning of critical investment projects. The project was implemented as a “learning by doing” experience. The project evolved throughout implementation, with a series of adjustments made along the way. The project was intended to support a varied range of investments, in infrastructure and social sectors, with decentralized agencies responsible for preparation and implementation, and the Ministry of Public Investment (MPI) in charge of overall coordination. Theory of Change (Results Chain) 6. The PPTAF aimed to strengthen government capacity at national and sub-national levels to more efficiently plan and prepare public investment projects, increasingly adhering to and applying international quality standards. Public investment planning and preparation at national and sub-national levels was to be improved through (i) direct support, including capacity support, to plan and prepare public investment projects more efficiently and in accordance with international standards; and (ii) capacity support to strengthen institutions, systems, and policies, including policies linked to public investment planning and preparation. The project was designed based on findings from various studies undertaken by the GoV and development partners in Vietnam that showed slow project start-up and readiness caused project delays. It was estimated that these delays could be reduced by financing critical project preparation steps, including feasibility studies, from a specific facility which would allow access to preparation funds before approval of investment project financing. The project was prepared at a time when public investments in Vietnam soared, including investment financing from development partners (World Bank financing rose from $711.8 million in FY07 to $1,148.8 million in FY09). This put additional pressure on government and partners to find ways to make the investment process more efficient and improve quality. Figure 1 below shows the link between activities supported under the project, outputs, PDO outcomes, and anticipated longer term outcomes the project would yield. Page 7 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) Figure 1. PPTAF - Theory of Change PDO/ Long-Term Activities Outputs Outcomes Outcomes Project Preparation High quality feasibility PDO: Increase capacity of studies, procurement, government agencies to Planning and Facility - Financing and and safeguards plan and prepare public preparation of public capacity support to documents developed investments efficiently investment projects prepare sub-projects, and to international during project undertaken more feasibility studies, preparation; quick quality standards effeciently and in line procurement disbursing projects; with international documents, studies reduced time in realising Improve the preparation quality standards etc. project benefits and planning of investment lending projects to build a quality Capacity Support - Draft Planning Law; MPI portfolio Specialist support Circular and Manual on to key GoV agencies ODA planning and M&E; Institutional capacity central database for ODA Strengthen the for public investment to strengthen projects; staff received planning and institutional environment institutions, fiduciary training; PMU for planning and preparation improved systems, and Director Forum preparating public policies established investment projects Project Development Objectives (PDOs) 7. The PDO of the PPTAF, as stated in the Financing Agreement, was to “increase the capacity of the Recipient’s entities to plan and prepare public investments efficiently and to international quality standards.” Key Expected Outcomes and Outcome Indicators 8. The PPTAF was a technical assistance project aimed at strengthening capacity of government agencies. It was designed to complement and leverage new World Bank lending operations in Vietnam and to accompany the country’s increase in public investment spending. As per the original results framework, the project outcome indicator was to improve “project quality at entry through an increase in details and range of documentation ready by appraisal of the resulting investment loan, and more substantial completion of start-up activities prior to investment loan approval.” This high-level indicator (which had no associated targets, but was to be measured through various results indicators) was dropped and replaced at a May 2014 restructuring. These indicators were again revised in December 2015, and changed out through a March 2017 restructuring—only months before the project closed. The project’s original intermediate outcome indicators were also revised throughout implementation, as were the financing amount and the length of the project implementation period. Please see Annex 6, Table 8, for a complete overview of all project indicators and changes made through restructurings. Components 9. The PPTAF had three components: Page 8 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) Component A: Project Preparation Facility (original allocation: $94.7 million; revised allocation: $63.77 million). (a) Preparation of public investment projects to be financed through official development assistance, through the provision of subproject financing. (b) Provision of support to subproject financing beneficiaries with respect to the preparation of subproject proposals, administrative management and financial management (FM), procurement, and disbursement under sub-projects. Component B: Project Preparation Facility Support and Capacity Building (original allocation: $4.70 million; revised allocation: $4.82 million). (a) Provision of support to the Ministry of Planning and Investment (MPI) with respect to the review and appraisal of subproject proposals and the monitoring and evaluation (M&E) of sub-projects. (b) Capacity building of subproject financing beneficiaries and other relevant entities with respect to the technical aspects of subproject implementation, including environmental and social safeguards. Component C: Incremental Operating Costs and Project Management Expenses (original allocation: $3.60 million; revised allocation: $0.41 million). (a) Project management, implementation, and M&E, including administrative and FM, procurement, and disbursement. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revisions to the PDO, Outcome Targets, and Project Indicators 10. The PPTAF was subject to three restructurings, all approved by the Vietnam Country Director. There were no revisions to the PDO. See Annexes 1 and 6 (Table 8) for a list of all project Indicators and Table 2 below for key indicators. 11. The first restructuring, May 2014, was based on recommendations from the project’s Mid-Term Review (MTR) and aimed to accelerate implementation and disbursement, improve project performance, and allow for better measurement of achievements towards the PDO. About $21.9 million had been disbursed at the time of the restructure (equivalent to 22 percent of the original commitment)—well below the estimations after close to four years of implementation. To address implementation issues linked to demand for funds and processing procedures and to improve results monitoring, the restructure (i) amended the financing agreement related to the on-lending mechanism; (ii) updated the project’s results framework (revising key indicators and targets and removing the results framework from the project Financing Agreement); and (iii) modified references to the Procurement and Consultant Guidelines to introduce more flexibility in the application of such guidelines. The removal, upon request from the authorities, of the reference to the on-lending mechanism in the terms and conditions for the financing of sub-projects had a significant impact on project implementation procedures, allowing all implementing agencies (IAs), not just budgetary ones, to be allocated funds on a grant basis. Prior to this change, funds were on-lent to decentralized agencies. In allocating funds on a grant basis, it was expected that demand Page 9 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) for PPTAF support would increase—demand for funding from the PPTAF had, until then, been below expectations and it was understood that the cost of accessing funds when on-lent was limiting demand. 12. Through the second restructuring, December 2015, (i) the project closing date was extended by 22 months, from December 31, 2015 to October 31, 2017; (ii) IDA Project funds, to the order of $11.428 million (SDR 8.24 million as of December 22, 2015), were cancelled; and (iii) the project’s results framework was revised to reflect the new closing date and removal of indicators (see Table 2 below). The restructuring was undertaken to address continued delays in the up-take and implementation of sub- projects—seen to be a result of (as outlined in the restructuring paper): (i) weak capacity at the level of implementing agencies at national and sub-project level; (ii) overcommitted consultants involved in project implementation, not providing adequate support; and (iii) protracted approval processes on the part of designated authorities. The extended project closing date allowed the project more time to realize its objectives as activities under ongoing sub-projects could be completed, while the cancellation of funds brought the estimated disbursements to project closing in line with reality. It also allowed the funds to be reallocated to other Vietnam IDA-funded projects, hence optimizing use of IDA funds. The changes to the results framework were primarily to align target dates with the new project closing date. 13. The third restructuring, March 2017, reflected conclusions from 2016 implementation support missions and resulted in (i) the cancellation of an additional $19.57 million of IDA project funds (XDR 14.53 million equivalent, as of March 8, 2017); and (ii) the updating of the project’s results framework to better measure project achievements. The changes were made to enhance management and measurement of project results and to optimize the use of Vietnam’s available IDA resources, allowing for the cancelled amount to be re-allocated to other Vietnam projects before IDA graduation. As in previous restructurings, changes made aimed to allow for project achievements to be better captured. Through this last restructure, further emphasis was also put in the results framework on capacity development activities supported under the project, introducing indicators linked to the drafting of a new planning law, procurement training, and the approval of a circular and manual on ODA planning and M&E within the MPI. 14. There were no revisions to the focus or support provided under the three project components, though the financing amount was revised as part of restructurings. Fund cancellations came from component 1, reducing the financing available for sub-projects. See table 1 below for details regarding changes to the financing amount per component. Table 1. PPTAF-Financing per Component, Original and Revised (in $M) Original financing amount Revised financing amount Component Disbursed amount ($M) ($M) ($M) Component A 94.7 63.77 Component B 4.7 4.82 Component C 0.6 0.41 Total 100.0 69.0 52.8 Page 10 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) Rationale for Changes and Their Implication on the Original Theory of Change 15. Changes made to the PPTAF through the three restructurings aimed to improve project implementation and effectiveness and strengthen results monitoring and assessment. From early stages of implementation, the project suffered from implementation delays. By December 2012, four months prior to the Mid-Term Review (MTR), the project had only disbursed $5.37 million and was rated Moderately Unsatisfactory for both Development Objective (DO) and Implementation Progress (IP). The MTR identified two main challenges to implementation: (i) limited demand from line ministries for PPTAF funds—seen to be a result of excessive MPI processes and procedures in the preparation of sub-projects; and (ii) weak project implementation capacity. As has been observed for other World Bank technical assistance/capacity development projects, the PPTAF team struggled to identify appropriate and measurable indicators that would show progress and impact—particularly with regards to the objective of strengthening capacity and processes at various levels. Consequently, each successive restructuring included changes to the results framework, with revisions to indicators (in terms of targets and in definition). The changes did not impact the original theory of change—rather they aimed to facilitate achievement and assessment of the original project outcomes and objectives. II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 16. The ICR rates the relevance of the PDO as ‘High’. The project was strategically relevant and aligned with the country’s expressed development priorities and the World Bank’s key areas of engagement. The three Country Partnership Strategies/Frameworks that were in place during the preparation and implementation of the project all emphasized the importance of effective and efficient public investment planning and preparation, and the critical role that government agencies and staff play in doing so. During PPTAF implementation, the World Bank engaged in a number of activities—including technical support—that aimed to strengthen national processes, including policies, and capacity across agencies, levels, and sectors for improved public investment planning and preparation. There was also strong focus on measures and support that would strengthen disbursement and implementation of World Bank financed projects. 17. The PPTAF was strategically relevant and aligned with the government’s development priorities and strategy. The project was prepared and implemented during a period that saw large increases in Vietnam’s public investment budget—one of the ways the authorities has spurred continued strong economic growth and highlighted in the GoV’s Socio-Economic Development Strategy (SEDS). Analyses (GoV analysis, Vietnam 2035, Vietnam Systematic Country Diagnostic) show the need for further productive infrastructure and that sustained and further investment in infrastructure, in particular, is important for growth and job creation. This was underlined in the GoV’s 2016-20 SEDP, which estimated demand for infrastructure investment capital over the period of about $139 billion, equivalent to 10-12 percent of GDP. It was also confirmed in the Prime Minister’s expressed priorities for the new government leadership team that took office in 2016. The World Bank’s lending portfolio in Vietnam also increased Page 11 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) substantially during the project preparation and implementation period. The GoV and development partners had identified delays in implementation and disbursement of the stock of investment projects to be a significant problem, and access to financing for project preparation ahead of project approval had been found to help address this issue. 18. Preparation and implementation of the PPTAF covered three Country Partnership Framework (CPF) cycles, and project objectives remained relevant throughout these cycles. The PPTAF was aligned with the 2007–2011 Country Partnership Strategy (CPS), which called for a large increase in investment, particularly for infrastructure development, and for supporting better planning processes, more transparent procurement, and stronger financial management—all of which fell under the governance pillar of the CPS (Pillar Four). The most significant portion of the project was implemented during the FY12–FY16 CPS (the CPS was extended by six months through the May 2015 Performance and Learning Review). The FY12–FY17 CPS stated that “…while the quality of portfolio implementation continues to be satisfactory, … the country would benefit significantly if implementation of its investment program were accelerated and development objectives achieved at a much faster pace” (paragraph 75). The CPS confirmed that the World Bank would, over the CPS period, continue to work with MPI and development partners to address systemic ODA implementation issues. At project level, the CPS identified the PPTAF as a critical vehicle for improving project readiness and accelerating preparation, but highlighted the need for improvements for the project to make financing available in a timely manner. Additionally, the project was seen to contribute directly to three CPS outcomes: (i) outcome 1.2 - improved quality and efficiency of infrastructure services (competitiveness pillar); (ii) outcome 2.2 - strengthened environmental protection and management (sustainability pillar); and (iii) outcome 3.2 - improved basic infrastructure and public service delivery (opportunity pillar). 19. The PPTAF was aligned with the objectives of the World Bank’s current Country Partnership Framework. The FY18-22 CPF, in place at project closure, was adopted as Vietnam graduated from IDA— to become an IBRD-only country (with access to IBRD and IDA transition resources). The strategy lays out WBG support using the institution’s various instruments and tools, including extensive analytical and policy work, aiming to support Vietnam in the consolidation of its development achievements and transitioning to a more sustainable development model. Under a cross-cutting governance pillar, the World Bank aims to support comprehensive reforms in the areas of public financial and investment management and to strengthen capacity and institutions at national and sub-national levels for improved efficiency and effectiveness. Under objectives 3 - enhance trade competitiveness, multimodal transport connectivity, and logistics services - and 4 - improve planning, management, and delivery of infrastructure and land in cities - there are also provisions for targeted support to strengthen investment planning and implementation processes, with a view to ensuring greater impact from resources spent. As the PPTAF exits the World Bank portfolio, the projects that it helped prepare and implement will continue to contribute to the objectives and focus areas of the current CPF. This relates primarily to objectives 3 and 4 (both under Focus Area One – Enable inclusive growth and private sector participation) (see Annex 6 for a list of projects prepared under the PPTAF and the CPF Focus Areas they contribute to), as well as cross- cutting themes such as climate change, gender, and poverty reduction. Furthermore, the capacity building initiatives supported under the project (procurement training, planning law, M&E manual, and ODA database) will continue to yield results and contribute to other CPF objectives, including the cross-cutting governance pillar. Page 12 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 20. In assessing the achievement of the PDO, the ICR considers project contributions in strengthening the capacity of beneficiary entities to (i) more efficiently plan and prepare public investments; and (ii) plan and prepare public investments to international quality standards. 21. The project’s PDO indicators and intermediate indicators were changed during three restructurings. To strengthen results monitoring, the original outcome indicator was dropped and two new PDO indicators introduced at a May 2014 Project restructure. Two of the original results indicators were also dropped and six intermediate results indicators introduced—ensuring consistency with the new PDO indicators. In a December 2015 restructuring, the target values of the two PDO indicators and the six intermediate results indicators were revised, to align with the extended Project implementation period (to October 2017). Through a March 2017 project restructure, three PDO indicators were introduced (previously intermediate results indicators), the value of one PDO indicator changed (from number to percentage), and four intermediate indicators were dropped. This ICR considers achievements as measured by the original as well as the restructured results frameworks and indicators, coupled with a qualitative analysis of achievement of project objectives (as stated in the PDO). As such, some of the indicators discussed do not feature in the project’s final results framework (in the final ISR and also generated automatically for this ICR). (The December 2015 framework includes indicators from May 2014 with an updated time-frame.) See Table 2 and Annex 6 for an overview of project indicators, original and revised, and achievements at project close. 22. The repeated changes to the results framework and updates as to how to assess and measure results and impact are a testament to the continuous challenges of measuring and assessing impact and results from technical assistance projects, including the PPTAF. The PPTAF was the first time such a TA project was implemented in Vietnam, and there was limited relevant global experience available to draw lessons from. (There were similar TA projects from China, 1986, and from Russia, 1995.) Table 2. PPTAF Key Outcome and Intermediate Outcome Indicators 1 Source/ Indicator Achievement at Origin of (PDO: Project Development Outcome Indicator Baseline Target Project Close Indicator IO: Intermediate Outcome Indicator) Improve Planning and Preparation of Investment Projects Original (May 27 20 27 months IO: Average time from PCN to Board approval 2010) months months (not achieved) PDO: Increase in the number of projects prepared under a sub-project that have achieved (i) 5% of the readiness criteria of completion of Original (May the (i) 25% procurement ready to award contracts at time of n/a 2010) portfolio (ii) n/a financing approval for either (i) 30% of the value (ii) n/a of the total project cost; or (ii) the first 18 months of activities 1Table 2 includes key indicators from throughout the life of the project. Some of these indicators do not feature in the portal- generated results framework, as this is based on the indicators in the last ISR. Page 13 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) For projects receiving PPTAF support, the time required to achieve 10% disbursement is (i) 21.6 (i) 15 (i) 21.6 months2 Original (May reduced compared with (i) the situation before months months (not achieved) 2010) the facility came into existence; and (iI) projects (ii) n/a (ii) n/a (ii) n/a prepared without accessing the facility December PDO: Number of PPTAF-financed sub-projects 15 sub-projects3 2015 with 30% of detailed designs completed before 0 10 (fully achieved) restructuring Board approval December PDO: Number of PPTAF-financed sub-projects 27 2015 detailed design packages using international 0 18 (fully achieved) restructuring standards like environment and social safeguard PDO: Time taken to complete international March 2017 30 12 14.9 months procurement packages related to civil works in restructuring months months (partially achieved) PPTAF-financed activities PDO: Average time taken from effectiveness to March 2017 5.5 6.0 first disbursement for projects that have used 5.2 months4 restructuring months months PPTAF funds PDO: Percentage of PPTAF-financed sub-projects March 2017 94 with 30% of detailed designs completed before 0 41 restructuring (fully achieved) Board approval March 2017 IO: Number of PPTAF-financed sub-projects 22 0 22 restructuring completed (fully achieved) PDO: Percentage of PPTAF-financed sub-projects March 2017 945 with the “quality of detailed design activities 0 70 restructuring (fully achieved) meeting international standards” Increase Institutional Capacity March 2017 IO: Number of studies prepared as inputs to the 12 0 12 restructuring development of draft Planning Law (fully achieved) March 2017 Yes IO: Procurement capacity program delivered 6 No Yes restructuring (fully achieved) IO: Number of officials and projects staff under March 2017 457 PPTAF-financed sub-projects trained on World 10 300 restructuring (fully achieved) Bank fiduciary requirements Circular: No March 2017 IO: MPI’s circular and manual on ODA Planning No Yes (delivered by an restructuring and M&E approved7 MPI department 2 Data is relative and manually calculated - there is no exact cut-off date in the system for disbursement of 10% for projects. 3 Sub-project is defined as a sub-financing agreement signed between MPI and PMU in which PPTAF finances partially or fully the preparation of an investment project. Nine sub-projects were partially financed by non-PPTAF sources. 4 Issue with baseline and target figures as target is higher than the baseline. We observe a slight improvement compared to the baseline figure. 5 An impact study was conducted by a third-party consultancy company to assess achievement of this indicator. However, given time limitations in conducting the study, the findings were based on a review of four sub-projects that met the following criteria: (a) sub-projects that financed detailed design contracts (some sub-projects did not have detailed design contracts); (b) sub-projects representing different regions (North, Hanoi, and South); and (c) sub-projects that had large contracts. As such, the four sub-projects examined are representative of the portfolio of sub-projects. 6 Recalculated after the submission and approval of the project’s last ISR. Delivery therefore not captured in the October 2017 ISR. 7 Not reported as achieved in the project’s last ISR as the Circular was delivered by an MPI department that was not supported by the PPTAF. Page 14 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) not supported by the PPTAF) Manual: Yes (partially achieved) March 2017 IO: Central database for all ODA projects Yes No Yes restructuring established8 (fully achieved) March 2017 IO: PMU Director’s Forum launched and Yes No Yes restructuring maintained (fully achieved) 23. Support to project objectives was provided in the form of capacity and direct support to the planning and preparation of investment projects (project preparation facility) and capacity development activities aimed to strengthen the institutional environment. Over the course of project implementation, a large portion of the technical assistance and capacity development activities were linked to World Bank financed projects (preparation of sub-projects including technical, procurement, fiduciary, and safeguards compliance). It can be assumed that this was also the intent from project start, but this is not explicitly stated in the project documents, including the PDO which refers to public investments at large. Impact from support, however, can be measured in terms of improvements in overall capacity to plan and prepare public investment projects (financed through the national budget or by other development partners) as well as capacity to plan, prepare, and implement World Bank financed investment projects. 24. Strengthened capacity to more efficiently plan and prepare public investments. This part of the PDO was measured through a number of qualitative PDO indicators in the original and restructured results frameworks. None of the original indicators, which focused on reduced project preparation periods, increased number of projects that achieve procurement readiness by financing approval, and reduced time to achieve 10 percent disbursement were achieved. These indicators were also dropped at the May 2014 restructure—the second dropped primarily as a result of difficulties in obtaining data on the 10 percent disbursement cut-off date (the system does not have such a feature). Two new indicators were introduced in 2014 and re-affirmed (with extended target dates) in 2015, both of which are considered substantially achieved: (i) percentage of PPTAF-financed sub-projects with 30 percent of detailed designs completed before Board approval. At project close, 94 percent of PPTAF-financed sub-projects had 30 percent of the detailed designs completed before Board approval—against a target of 41 percent and up from zero percent at project start; and (ii) average time taken from effectiveness to first disbursement for projects that have used PPTAF funds. At project close, this figure had been reduced to 5.2 months—down from 5.5 months in 2010.9 As such, there was a slight improvement in the time it takes to realize project benefits—as measured by timely first disbursement. This ICR revisited the original indicator, ‘Average time taken to achieve 10 percent of total disbursements’–an indicator that aimed to capture disbursements beyond the initial disbursement amount. The original target for this indicator was to reduce the average time taken to achieve 10 percent of total disbursement from 21.6 months to 15 months. The target was 8 This indicator is not listed in the March 2017 results framework, but figures in the project’s last ISR. It is therefore included in the table. 9 Note that the December 2015 results framework states a baseline value of 21.6 months (also used in the Implementation Status and Results Report). This is a mistake—it refers to the baseline value before the restructuring, where the measurement was for the “time taken to achieve 10 percent of disbursements” and not the “time taken from effectiveness to first disbursement.” The revised baseline value was 5.5 months, which is misaligned with the target of 6 months (baseline value is an improvement on the target value). Page 15 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) not achieved at project close as the average time taken to reach 10 percent disbursement was around 21.6 months. 25. The indicator concerning the number of sub-projects with detailed designs completed by Board approval exceeded the target set in the December 2015 restructure (15 versus a target of 10). This illustrates the project’s success in allowing for parallel—as opposed to sequential—preparation of project implementation documents—considered a way to improve efficiency of project planning and preparation. One would expect these efforts during preparation to lead to a reduction in implementation delays (as more work was done at preparation stage, one would, however, not expect there to be a reduction in the preparation time). This is reflected in a slight reduction in the average time taken from effectiveness to first disbursement for these same projects (from 5.5 months in 2010 to 5.2 months in 2017). We do not, however, see a decrease in the time taken to disburse 10 percent of commitments for PPTAF sub-projects (one of the original indicators). One reason for the continued disbursement lags in PPTAF-financed sub- projects (beyond the first disbursement) is that at project start, there were only four state-owned banks serving World Bank projects. During implementation, however, several private commercial banks were added to the roster (a positive development), though they had limited (or no) prior experience working with the World Bank/a development partner. Further, the State Bank of Vietnam (SBV) took considerable time selecting a serving bank for new projects.10 Another reason for continued disbursement lags are national procurement rules that do not allow for procurement activities to start until project effectiveness. In terms of speed of implementation, we also observe, during project implementation, a reduction in the time taken to complete international procurement packages related to civil works in PPTAF-financed sub-projects—from 30 to 14.9 months, achieving 84 percent of the target introduced in the December 2015 restructuring.11 (The indicator was dropped in the March 2017 restructuring.) PPTAF disbursement is seen to have been hampered by (i) significant time taken for MPI and line ministries to define and establish implementation mechanisms linked to the facility (as this was a new way of operating); (ii) the decentralized nature of project implementation, with 22 decentralized PMUs implementing various sub-projects; (iii) difficulties in securing counterpart funding needed for implementation and disbursement; and (iv) MPI’s weak capacity to manage and implement the project. These are all issues that are seen to hamper disbursement in other World Bank projects in Vietnam. 26. The intermediate outcome indicators pertaining to the capacity to more efficiently plan and prepare public investments were substantially achieved, including the provision of substantive input to the preparation of a new Planning Law, with the preparation of 12 background studies; the approval, by MPI, of a new manual on ODA planning and M&E (Circular not delivered); the delivery of a large-scale procurement capacity building program; the sustainable launch of a series of PMU Director Forums; and the establishment of a central database for all ODA projects in MPI.12 Furthermore, with financing from the project’s component A, the Project Preparation facility, 22 sub-projects were completed by project close (corresponding to the target value introduced at the March 2017 restructuring), aiming to improve quality at entry and hence reduce implementation delays and time taken to realize project benefits. These sub-projects contributed the preparation of 13 IDA/IBRD investment projects, to a total cost of $1.7 billion 10 Significantsupport was provided by the World Bank financial management team to resolve this bottleneck, highlighting the importance of advance actions (PMU establishment, PMU staffing (appointing project director, chief accountant, and subordinates), PMU office, procurement plan and activities, budget and disbursement plan…); and the appointment of the commercial bank to serve projects for the PMU to open DA accounts immediately after Board approval. 11 The calculation includes 17 of the 22 sub-projects—the projects with the largest procurement contracts. 12 This indicator is not listed in the March 2017 results framework, but figures in the project’s last ISR and is therefore included in the ICR discussion. Page 16 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) (equivalent to 22 percent of the current World Bank portfolio in Vietnam).13 It is important to note that the characterization of what constituted a “sub-project” changed during implementation. In the PAD, a sub-project would lead to a World Bank-financed lending operation, whereas after the project MTR, a sub-project was defined as a Sub-Financing Agreement between MPI and a PMU. 27. From the 22 PPTAF-financed sub-projects, 13 World Bank investment projects were prepared . There are more sub-projects than investment lending operations because a number of projects, including the Medium Cities Development Project and the Northern Mountains Urban Development Program (NMUP) Program-for-Results (PforR), were supported by several PPTAF-financed sub-projects (three and eight sub-projects respectively). The PPTAF-financed sub-projects also supported capacity building at different levels of government. For example, the NMUP was implemented at the provincial level. As there was no central, coordinating PMU, the PPTAF provided a platform for these provincial-level PMUs to build capacity to prepare and implement World Bank projects. During implementation, the NMUP has maintained its original structure of eight PMUs, transforming the preparation unit into an implementation unit for this PforR operation. The transition from preparation unit to implementation unit has also happened among other PPTAF sub-projects. 28. Strengthened M&E capacity. Additionally, to strengthen M&E capacity and processes linked to public investment preparation, M&E training courses were developed and held; an M&E database was set up in the Informatics Center of the MPI—a vehicle for MPI to monitor ODA projects and to serve the ministry’s decision-making processes; and a loan registration system was established within the Ministry of Finance (MOF) to help monitor public debt. Through these wide-ranging capacity building and technical assistance activities, the project has strengthened skills, processes, and institutions at the national and sub-national levels for more efficient planning and preparation of public investment projects in general. Institutionally, sustainable impact can be seen, in particular, from the input into the new Planning Law; the approval, adoption, and application of the MPI Manual on ODA Planning and M&E; the establishment of an ODA project monitoring database within MPI, and the MOF’s new loan registration system—work that will enhance public investment planning and preparation in the medium to long term. 29. Capacity has been significantly strengthened at the level of preparation and implementation of World Bank-financed projects, where officials and Project Management Units (PMUs), charged with implementing PPTAF sub-projects, have benefited from English language, governance, and project management support and training, including on M&E and World Bank fiduciary requirements. A total of 457 officials (line ministries, local authorities, various agencies), and project staff under the PPTAF- financed sub-projects were trained—surpassing the target of 300 (see Annex 6, table 8 for further details). From interviews conducted as part of the preparation of this ICR, it is clear that staff and officials who took part in these trainings benefited considerably—they gained skills and knowledge to address immediate project-related as well as more general project implementation challenges. To address implementation delays and improve quality at entry, significant emphasis was put on strengthening procurement frameworks during project preparation—thereby improving implementation readiness. Originally, this was to be measured in terms of the increase in the number of projects prepared under a 13Ten projects are currently in the World Bank portfolio, one has closed, and two are part of the FY18 pipeline. Several investment projects that benefited from PPTAF support include a number of sub-projects (e.g. the Medium Cities Development Project and the Northern Mountains Urban Development Program PforR). Part of the preparation of these investment sub- projects was financed from the PPTAF. As a result, there were more PPTAF sub-projects than investment projects receiving Project support. Page 17 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) sub-project that had achieved the “readiness criteria of completion of procurement ready to award contracts at time of financing approval”. (The indicator was not expected to have any results until the third year of operation. As the indicator was dropped, no monitoring was done.) During the March 2017 restructuring, this indicator was replaced by “the percentage of PPTAF-financed sub-projects with 30 percent of detailed designs completed before Board approval”—where the target was fully achieved (at 94 percent against a target of 41). 30. The indicators linked to the efficiency and capacity dimensions of this part of the PDO are deemed insufficient to adequately assess impact—particularly as they changed throughout implementation. While the time to prepare projects and the time to disburse were monitored through changing indicators, a third efficiency dimension, namely the time to realize project benefits, was not. Furthermore, the capacity dimension linked to efficiency is not seen to have been adequately captured by results indicators—though there are clear signs the PPTAF did contribute to improved capacity within the Government of Vietnam to better plan and prepare investment projects. As such, this ICR concludes there were only modest gains in improving the capacity to more efficiently plan and prepare investment projects in Vietnam. 31. Shortened preparation time from Project Concept Note (PCN) to Board approval. The PPTAF intended to strengthen national capacity to produce project preparation documents efficiently and to an acceptable schedule. The related intermediate outcome indicator was ‘Average time taken from PCN to Board approval’, included in the original results framework. The indicator was removed following the midterm review (MTR) in 2013. The indicator was revisited for this ICR—to evaluate if project preparation times had decreased from the initial baseline of 27 months. Project data shows that the preparation time did not improve for PPTAF sub-projects. Furthermore, a comparison of the preparation time for non- PPTAF-financed projects for the same period was 19 months. Examining work under the PPTAF, however, we find that more preparatory work was included in the project preparation cycle, including procurement, environmental, and design studies—with a view that this would lead to improved project quality at entry and speed up implementation (which is also in the interest of the client, particularly as the country graduates to an IBRD only country). As such, one would expect that preparation times would increase rather than decrease. 32. Strengthened capacity to plan and prepare public investments to international quality standards. The second part of the PDO was measured through two PDO indicators— one introduced in the May 2014 restructuring (time perspective revised in the December 2015 restructuring with the project extension): “the number of PPTAF-financed sub-projects detailed design packages using international standards like environment and social safeguard” (27 out of a target of 18 achieved); and one that was introduced through the March 2017 restructuring: “time taken to complete international procurement packages related to civil works in PPTAF-financed activities (achievement 14.9 months, against a target of 12 months—down from 30 months in 2010). Both targets are considered partially achieved. 33. Performance of PPTAF-financed versus non-PPTAF-financed projects. In addition to evaluating the quality at entry and the processing speed of the PPTAF-financed projects, this ICR attempted to compare the current performance of the PPTAF-financed projects to non-PPTAF-financed projects in the current active portfolio as of December 2017. As can be inferred from Table 3 below, there is no significant difference between the performance of the PPTAF-financed projects and non-PPTAF-financed projects with regard to development outcome (DO) and implementation progress (IP) performance ratings. Page 18 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) Table 3: Performance of PPTAF vs. Non PPTAF-Financed Projects % of Moderately Satisfactory or Better Ratings DO IP Non-PPTAF 92 78 PPTAF 89 78 34. Throughout project implementation, five intermediate outcome indicators were used to assess progress in strengthening capacity to plan and prepare public investments to international quality standards: (i) Number of officials and project staff under PPTAF-financed sub-projects trained on World Bank fiduciary requirements14--achieved at 152 percent of target (introduced at May 2014 restructuring, revised in December 2015 and retained at March 2017 restructuring); (ii) Number of studies prepared as key inputs to drafting of the Planning Law15—achieved at 100 percent of target (introduced at March 2017 restructuring); (iii) Number of PPTAF-financed PMUs with experience on World Bank policies and guidelines before Board approval—target of 10 PMUs exceeded (at 22) (introduced at May 2014 restructuring, revised in December 2015); (iv) Quality of detailed design activities financed by the PPTAF that meet international standards—target of 70 percent exceeded at 94 percent achievement16 (introduced at May 2014 restructuring, revised in December 2015); (v) Time taken to complete international procurement packages related to civil works in PPTAF- financed activities—reduced from 30 to 14.9 months, close to the target of 12 months 17 (introduced at May 2014 restructuring, revised in December 2015). (See Table 2 above and Annexes 1 and 6 for further information relating to achievement of indicators.) All indicators are considered substantially achieved. 35. The PPTAF is seen to have contributed considerably towards integrating international standards into project planning and preparation and to provide opportunities for staff and officials to familiarize themselves with and apply such standards. The project provided technical assistance that integrated international standards into project planning and preparation. It also delivered training where international standards were integrated—in a variety of areas ranging from procurement/fiduciary to design. Support to the drafting of the new Planning Law included exposure and application of international best practice and lessons learned and will have lasting effect as the law continues to guide new investments. Procurement support, to the preparation of a new procurement law and to the procurement agency, and training of trainers on procurement, is also seen to have lasting and significant impact. PMU 14 The indicator is seen to contribute to both parts of the PDO and therefore figures also in the above discussion. 15 The indicator is seen to contribute to both parts of the PDO and therefore figures also in the above discussion. 16 An impact study was conducted by a third-party consultancy company to assess achievement of this indicator. However, given time limitations in conducting the study, the findings were based on a review of four sub-projects that met the following criteria: (a) sub-projects that financed detailed design contracts (some sub-projects did not have detailed design contracts); (b) sub-projects representing different regions (North, Hanoi, and South); and (c) sub-projects that had large contracts. As such, the four sub-projects examined are representative of the portfolio of sub-projects. 17 The calculation of this indicator includes 17 of the 22 sub-projects—the projects with the largest procurement contracts. Page 19 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) staff, other officials, and private sector representatives involved in the implementation of the PPTAF and resulting World Bank financed investment projects were trained in World Bank procedures and processes, gaining first-hand, practical experience and knowledge applying internationally recognized quality standards and processes. Such skills and knowledge are transferable and render the investments sustainable in the medium to long term. Support is also seen to have contributed to improved professionalism of involved agencies at national and sub-national levels. Box 1: PPTAF contributions to improve implementation of World Bank financed projects in Vietnam Because of PPTAF support during project preparation, World Bank staff see strong capacity within the Northern Mountains Urban Development Program and Medium Cities Development Project to prepare project implementation documents according to international standards and to accompany procurement processes. The frequency and magnitude of adjustments made to implementation documents, or variation during project implementation, is less than what is observed in other projects (which did not receive PPTAF support). The time spent by TTLs supporting PMUs in daily project management is also less for projects that received PPTAF support than for non-PPTAF-supported projects. Teams also document savings in bidding processes as TORs are clearer and more precise and PMU staff more experienced. Justification of Overall Efficacy Rating 36. The ICR rates the efficacy of the PPTAF as ‘ Modest’. This is based on the analysis and discussion above (see Table 2 above and Annexes 1 and 6). While capacity development gains generally can be observed and the PPTAF served as highly relevant support to Vietnam’s soaring public investments, it is not clear that the capacity to more efficiently plan and prepare investment projects substantially improved. The ICR therefore concludes that the project only partially achieved its intended outcomes. C. EFFICIENCY Assessment of Efficiency and Rating 37. Project efficiency is rated ‘Modest’. This is based on analysis of costs, including unit costs, and qualitative analysis regarding value for money and efficiency gains. Data is limited, making comparisons and analysis difficult. 38. As is customary for TA credits, there was no economic or financial analysis done at project appraisal stage. The PAD states the project “… will not have itself present quantifiable economic and financial benefits” (paragraph 47). The PAD does, however, state that the PPTAF is expected to contribute to earlier delivery of benefits in projects that receive PPTAF support. Consequently, the discussion around project efficiency focuses on value for money—that is whether the cost of delivering the anticipated results (strengthened capacity for project planning and preparation) were reasonable.18 The discussion A comparative analysis is not possible as data regarding the cost and quality of PPTAF vs non-PPTAF-financed preparatory 18 work is not available. Page 20 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) will also consider PPTAF implementation length, PPTAF disbursement, and how the PPTAF affected the pace of achievement of results in sub-projects. 39. Preparation and implementation timeframe. The PPTAF took seven years from approval to completion. The original project implementation period was 5.2 years. The project closing date was extended twice, in total by 22 months, to address initial slow implementation and complete planned activities. By project close, the discussion above shows that most anticipated results were achieved. It is, however, important to note the costs associated with such extended project cycles (client cost in managing the project, human resources, and opportunity cost). 40. Cancellation of funds. About 52.8 percent of the original loan amount of $100 million was disbursed. The project cost estimate, demand for funding, and implementation schedule at the time of appraisal proved to be too optimistic. At appraisal, it was expected the project would finance preparation of 22 sub-projects, ranging from large-scale infrastructure investments, requiring preparation budgets of up to $20 million, to smaller-scale social sector operations, requiring individual consultant inputs costing around $0.5 million or less. Actual cost of support provided under the PPTAF, however, ranged from $0.5 million to $5 million. The cancellations during the second and third restructurings were therefore partially a result of requests for smaller-than-expected preparation funds—as well as the overly ambitious demand predictions from project preparation. Table 4 below shows a breakdown of initial and revised allocations for each component of the project, as well as unit cost. As per this data, actual cost per unit (e.g. per sub- project) for capacity support and project management (components B and C) came in at about the estimated cost at project appraisal. Unit cost under component A, the preparation facility, was significantly lower than planned. The ICR considers actual project costs to be reasonable. Table 4. Component and Unit Cost ($M) Component Original Allocation Revised Allocation ($M) (22 sub-projects) ($M) (22 sub-projects) A. Project Preparation Facility 94.70 60.00 ($4.30 M/sub-project) ($2.72 M/sub-project) B. Project Preparation Facility Support and 4.70 4.82 Capacity Building ($0.21 M/sub-project) ($0.22 M/sub-project) C. Incremental Operating Costs and Project 3.60 3.40 Management Expenses ($0.16 M/sub-project) ($0.15 M/sub-project) 41. From the World Bank budget perspective, project preparation and implementation costs were close to the original budget estimates. The original planned Work Program Agreement (WPA) was $480,000 and the revised final WPA was $627,528, with a burn rate of 81 percent (equivalent to ($508,051). This shows that even though the project was extended by 22 months, the overall World Bank budget cost only increased by 5.84 percent compared to originally planned. 42. Cost effectiveness of trainings delivered. The project supported a high number of training activities delivered by international and national companies as well as World Bank staff across a wide range of issues, including World Bank management procedures, procurement, financial management, transparency and governance, and English language. Consulting services were selected based on competitive selection. Selection Based on Consultant’s Qualifications and Least Cost Selection were the Page 21 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) two most commonly applied selection methods under the project and complied with MoF’s standard cost norms. Unit costs of trainings were standard and reasonable. D. JUSTIFICATION OF OVERALL OUTCOME RATING 43. Overall outcome rating for the project is ‘Moderately Unsatisfactory’, with shortcomings in the operation’s achievement of its objectives, in its efficiency, or in its relevance. Justifications for relevance, efficacy, and efficiency ratings are elaborated above. See Table 5 below. Table 5. Project Ratings Relevance of PDO Efficacy Efficiency Overall Outcome Rating Moderately High Modest Modest Unsatisfactory E. OTHER OUTCOMES AND IMPACTS (IF ANY) 44. Gender. There were no specific gender-focused activities financed under the PPTAF. Through support to sub-projects and resulting IDA and IBRD operations, however, the PPTAF has had indirect impact on gender issues. These impacts and results are documented in reporting pertaining to these operations, including for the Central Highlands Poverty Reduction Project. 45. Institutional Strengthening. The Planning Law that was passed by the National Assembly in October 2017, the preparation of which was supported by the PPTAF, has helped improve the quality and efficiency of investment planning in Vietnam. The new law enables government to more effectively use investments in orienting and directing the socio-economic development of the country. For example, the new law has provided a legal framework for the World Bank financed Can Tho Resilience project in developing a master plan for the Mekong Delta area. Other aspects linked to institutional strengthening are covered in sections above. 46. Mobilizing Private Sector Financing – n/a. 47. Poverty Reduction and Shared Prosperity. There were no specific poverty reduction activities financed under the PPTAF. Through support to sub-projects and subsequent IDA and IBRD operations, however, the PPTAF has indirectly had significant impact in terms of poverty and shared prosperity. These impacts and results are documented in reporting pertaining to these operations, including for the Central Highlands Poverty Reduction Project, the NMUP, and the Vietnam School Readiness Promotion Project. 48. Other Unintended Outcomes and Impacts – n/a. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 49. Strong government commitment and rationale. From the start of Project preparation, there was strong government commitment to adopting a holistic and systemic approach to addressing project Page 22 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) implementation delays. The GoV also shared the view that a project preparation facility, coupled with a comprehensive and strong capacity building program, would help address implementation delays and improve return on investment projects. The project also had a clear rationale and problem statement. Despite this, the PPTAF experienced significant delays in implementation and weak demand for funding. 50. Extensive background analyses served to inform project design. Project design was built on conclusions and findings from studies done by government and development partners of causes for delayed project implementation. These studies showed that slow project start-up and readiness was a major cause in project implementation delays. It was believed these delays could be reduced by preparing projects through a separate facility before Investment Project Financing was in place. The PPTAF’s own implementation arrangements reflect these study findings. MPI and the World Bank considered making project funding available to non-World Bank-financed sub-projects. Given the innovative nature of the project, both parties agreed that it should initially focus on potential World Bank financed projects. 51. The PPTAF was a new way of engaging in Vietnam. As such, most of the agencies and institutions that were involved in the preparation and implementation of the project had no prior experience leading or undertaking the work at hand. The design was somewhat complicated, with MPI assuming a collaborative role, and with decentralized agencies taking on a key role in implementation. Neither MPI nor decentralized agencies had, at start of project preparation or implementation, the capacity needed to successfully execute their roles. 52. Realistic objectives – see the discussion below on the M&E framework. 53. Appropriate plan for monitoring – see the discussion below on the M&E framework. B. KEY FACTORS DURING IMPLEMENTATION 54. Complicated institutional set-up. Project implementation relied on a high level of coordination and collaboration, with MPI in the lead, having a role of bringing together line ministries at central level as well supporting and collaborating with decentralized agencies. MPI did not have prior experience managing and leading this type of work. Processes for accessing funding from the preparation facility were considered cumbersome and heavy, and roles and responsibilities were not clear or fully understood. Further, many decentralized agencies did not have experience or capacity to fulfill their roles and responsibilities from project start. 55. Regulatory changes posed issues. Changes in government regulations and procedures were a constraint to project implementation. Progress in implementing the project was slowed down as PPTAF documentation and manuals had to be revised to ensure compliance and consistency with national regulation—though much work was done at the level of the MPI and the World Bank to address issues in a systematic and timely manner. 56. Increasing national debt affected project disbursement. Vietnam’s total outstanding public debt (central government, publicly guaranteed, and local government) is estimated at about 62 percent of GDP, about 15 percentage points higher than in 2010, and quickly approaching the nationally established ceiling of 65 percent of GDP. Increased fiscal pressure has affected disbursement across the World Bank portfolio in Vietnam (as well as those of other development partners) as allocations through the national budget are limited and often not in line with a project’s annual budget estimations. For instance, the last four PPTAF sub-projects did not receive their annual budget allocation for 2017 until the PPTAF was closed. Page 23 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) Consequently, the IP ratings of some of the projects that received PPTAF preparation funding have been downgraded. 57. Regulatory change and stricter fiscal discipline led to responsibilities being devolved to local project owners. The country context in Vietnam evolved significantly from project start to close—in line with the country’s important development gains (economic and social). In 2010, access to central funding for project preparation was considered a major hurdle and constraint to ensuring timely and quality project implementation. By 2017, local project owners (e.g. provinces or other agencies in charge of project preparation and implementation) had become responsible for project preparation and implementation. To accompany preparation and implementation—and to ensure stronger fiscal management—central government has devolved responsibilities to local project owners, who have become responsible for making available human and financial resources. In parallel, capacity has been built at provincial and local levels for taking on this wider responsibility—including through the PPTAF. As these changes were implemented, the need for a central financing source for project preparation significantly diminished—partially explaining the excess project funds and needed fund cancellations (cancellations being used fully to address other development challenges through IDA recommitments). IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) Quality of M&E Design 58. The PPTAF Results Framework and M&E structure had shortcomings. Three successive restructurings, where the project’s results framework and its indicators were significantly revised, aimed to address these shortcomings (see Annex 6). These repetitive changes to the results framework illustrate issues linked to measuring and assessing results and impact from the project—often a problem in TA projects and observed across the World Bank. Repeated changes to the results framework, as well as changes late in project implementation (March 2017), raise the question of whether changes were also aimed to align the results framework with actual results and achievements. 59. The project’s theory of change and intended outcomes, as stated in the PAD, were clear, though the project is considered overly ambitious, with a high level PDO for which it was difficult to formulate monitorable outcome indicators. Therefore, monitoring relied on a set of (ever-changing) intermediate outcome and output indicators, which, in some cases, were narrowly related to project activities—as well as the PDO. Many indicators were output, rather than outcome, focused, making it hard to properly assess progress, particularly linked to the ambitious and high level PDO. Further, the link between the PDO and the project indicators was weak, and a number of indicators were based on implicit assumptions or assumptions that may not hold true (e.g. “average time from PCN to Board approval” – it is not clear that a reduction in project preparation time will lead to increased government capacity). 60. As evidenced by the successive restructurings, there was, throughout Project implementation, a constant struggle and preoccupation with improving the results framework, though none of the restructurings are seen to address the inherent issues in the framework—as this would likely have required changes to the PDO. The removal of the results framework from the project’s Financing Agreement was, however, a positive step, introducing more flexibility, though without solving the major Page 24 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) issues. The increased focus, from 2014, on the broader institutional and policy agenda, including preparations for the planning law, and the inclusion of new indicators in this area, was a successful development. 61. Support sought to strengthen the M&E framework. In May 2015, members of the East Asia and Pacific Development Effectiveness (EAPDE) unit reviewed the project. Some conclusions from this review were taken into account to strengthen project activities and the M&E framework. A key recommendation from the mission, to revise the project development objective in order to bring it more in line with the project’s results framework, was not deemed feasible. Two international M&E consultants were later hired, with the aim to strengthen the results framework. As such, considerable efforts were made to adjust and improve the M&E part of the project, though the nature of the project (TA/capacity development) meant it was difficult to find an appropriate, measurable, and robust set of indicators. Quality of M&E Implementation 62. The project Results Framework was the primary M&E tool to track progress toward achievement of the PDO. The M&E system was project-specific and not part of a wider institutional framework. 63. Strong reporting throughout implementation. The central Project Coordination Unit (PCU) (at MPI level) maintained and prepared a detailed quarterly monitoring report, which contained the details of Project outputs and activities. The quarterly report was linked to project budget and Procurement Plans. The PCU was responsible for collecting and consolidating data from different sources that could be used to assess achievement towards project objectives—as per the M&E framework. This work was done satisfactorily and submitted on time to the World Bank. The World Bank team maintained a detailed log of all sub-project activities, which was updated weekly. There was constant follow-up and contact between the central project management unit in the MPI, the PMUs at sub-project level, and the World Bank team, where bottlenecks and issues were identified and practical solutions sought on a just-in-time basis. The process became even more rigorous over the second half of project implementation, when the PCU enhanced its field visits and call support to PMUs. This intensive project management approach, which supported the effective monitoring system, significantly helped improve overall project implementation. Quality of M&E Utilization 64. Consistent data collection and monitoring. M&E data was collected by the PCU, from the sub- project PMUs, on a quarterly basis. This data served as the basis for quarterly project reporting from the PCU to the government and the World Bank. Quarterly reports were used to identify and address implementation shortcomings. Data was also used for internal World Bank purposes, including the preparation of Implementation Status and Results Reports (ISRs). World Bank TTLs throughout implementation were Vietnam-based and had systematic and deep engagement with project counterparts at national and decentralized levels. The M&E systems in place, though not perfect, served to strengthen project management and improve the overall performance of the project. As the M&E arrangements under the project were specific to the project—and not institutionalized—it is unlikely their use will go beyond the project completion date. 65. The beneficiary assessment done towards project close provides a comprehensive review of some sub-projects. However, the assessment only covers four sub-projects and was conducted late in Page 25 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) Project implementation. As such, results and finding do not represent all sub-projects and could not be used for Project adjustments. 66. M&E feedback loop. Shortcomings identified in the M&E system throughout implementation led to changes—through restructurings—to the project results framework. While, as discussed above, these restructurings may not have fully addressed all M&E-related issues, the M&E system was clearly used with the intention to improve project implementation, with increased emphasis on how to better capture and assess Project achievements (e.g. evaluation), and to ensure stronger links with anticipated project outcomes. Justification of Overall Rating of Quality of M&E 67. The M&E rating for the project is ‘Modest’, with significant shortcomings observed in the quality of the project’s M&E system design in particular. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 68. Supervision teams were generally proactive in following up implementation steps linked to fiduciary and safeguards management. ISR ratings in these areas were primarily ‘satisfactory’ or ‘moderately satisfactory’. Safeguards Compliance 69. Social and environmental safeguards compliance rated ‘satisfactory’. A TA project, providing capacity development support and finance to the preparation of sub-projects, there were no involuntary resettlement or ethnic minority issues incurred. During implementation, the PPTAF mobilized social and environmental safeguards specialists to provide trainings on social safeguards to relevant members of the sub-projects, to frequently monitor progress. A safeguard management framework was prepared to support the screening all proposed sub-projects for potential socio-environmental impacts. There were no reported noncompliance issues during implementation stage. The compliance rating of social and environmental safeguards is ‘Satisfactory’. Fiduciary Compliance 70. Financial management and procurement arrangements generally complied with the agreed processes and procedures and led to overall efficient use of funds. Throughout most of the project implementation period, accurate and timely information was provided, with the World Bank loan proceeds being used for the intended purposes and following World Bank procurement procedures. From the first supervision mission in November 2011, the project procurement and financial management ratings (later combined fiduciary ratings) were consistently ‘Moderately Satisfactory’. Reviews also recognized adequate fiduciary staff, systematic accounting, and internal control systems, up-to date maintenance of supporting documents in the Project, and implementation of auditor recommendations for annual audit. Nevertheless, in one instance, INT substantiated fraud and obstruction by one of the consultancies that had been selected to support one of the sub-projects financed by the project, leading to a debarment of the firm. Page 26 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) C. BANK PERFORMANCE Quality at Entry 71. The PPTAF was highly relevant—put in place to address specific constraints linked to public investment planning and preparation—and had strong government commitment. The project was prepared and processed efficiently. Appraisal and Board approval took place three and a half and six and a half months, respectively, after the project concept note (PCN) review. This is fast—globally and compared to project preparation times for the World Bank Vietnam portfolio, where the average project preparation time in FY10 was 24.7 months. The short preparation time may have adversely affected the quality of Project design and readiness for implementation. 72. Ambitious design and objectives. As has been the case with similar capacity building/TA operations across the World Bank, the original project design—as well as its over-arching objectives— were highly ambitious. The PPTAF, a $100 million multisector TA operation, was planned to be delivered in about five years, and to significantly and sustainably strengthen capacity at national and decentralized levels—making use of a new way of engaging (working across sectors, using a centrally managed funding mechanism to prepare projects with local project owners). And this was to be done within a context of relatively weak capacity. As such, we see that many of the operations planned for PPTAF support did not materialize—only one of the four sub-projects identified at appraisal (Small and Medium Cities Development) came into fruition. Furthermore, the institutional set-up, with MPI taking on a coordinating role, working with a large number of central and sub-national agencies, did not facilitate project implementation. 73. The M&E framework had serious flaws. The project suffered, throughout implementation, from a disconnect between the stated PDO, which was very high level, and the project indicators. Furthermore, the PDO related to building general capacity for public investment planning and preparation, while most of the activities and support under the project were linked to and targeted World Bank funded projects, resulting in an inherent disconnect between the PDO and the project indicators. The project’s M&E system also did not create an adequate framework for capturing outcomes and impact from investments. Further issues linked to the project’s M&E system are discussed above under the section on quality of monitoring and evaluation. 74. Implementation arrangements were cumbersome. The design of the sub-project application and approval processes became a barrier in the processing chain, which eventually was simplified and streamlined to increase demand for PPTAF support. The on-lending feature for funds was revised to on- granting, thereby making it more attractive for eligible projects to request and receive funds. Although this change allowed for the preparation of additional projects, it eliminated the prospect of establishing a sustainable facility. Quality of Supervision 75. Strong supervision and client engagement. Throughout its seven years of implementation, four World Bank staff served as Task Team Leader (TTL) for the PPTAF. There was reasonable overlap and continuity between the four TTLs and the change in TTLs is not seen to have negatively affected Project performance or implementation. The TTLs and team members made considerable effort to ensure proper implementation of the project. Nevertheless, the project’s highly ambitious objective, nature—a TA project aimed to strengthen capacity—and design rendered supervision difficult. Task Team members were Vietnam based, thereby facilitating communication and engagement between national stakeholders Page 27 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) and the World Bank. These opportunities were taken advantage of. From April 2012, when the project was rated ‘Moderately Unsatisfactory’, the pace and intensity of supervision increased. The team also sought to address project issues through the three consecutive restructurings. The increased focus, from 2014, on the broader institutional and policy agenda, is considered highly positive and the team is commended for this. 76. The need for a central project preparation fund changed over time. As Vietnam’s fiscal constraints increased and the GoV implemented measures to reign in investment borrowing, including responsibilization of local project owners, the need for a central project preparation mechanism diminished. This change was not foreseeable at project preparation stage. The task team is commended, however, for taking actionable and practicable measures to reflect these changes in the country context, including the cancellation of significant project IDA resources (which were recommitted). 77. The MTR (April 2013) – a missed opportunity? While the MTR resulted in and fed into a project restructure (May 2014), it can be argued that the MTR and the subsequent restructure did not go far enough to appropriately address critical design and M&E issues, including the project’s ambitions. Doing so, may have led to a change in the DO, further cancellation of project funds, and increased emphasis on the broader institutional/policy component. While acknowledging challenges and delays in implementation, the MTR concluded that the project’s objectives could still be met. 78. Candor and quality of performance reporting. World Bank ISRs reported candidly on issues linked to Project implementation and rated the project ‘Moderately Unsatisfactory’ for both IP and DO over a four-year period—from April 2012 to April 2016 when the project was upgraded to ‘Moderately Satisfactory’. 79. The supervision of the financial management (FM) and procurement aspects was satisfactorily conducted. Procurement bottlenecks were partially alleviated by (a) the changes to procurement and consultant guidelines which introduced more flexibility in the use of the guidelines, providing the borrower with an option to use either the May 2010 or the January 2011 version of the IBRD Procurement Guidelines; and (b) continuous training programs. Procurement post review supervision missions were conducted. FM specialists participated in all supervision missions and provided necessary oversight and training. Supervision of environmental and safeguards aspects was also satisfactory and a substantial amount of training was provided to the PCU and PMU to ensure due diligence in all assessments undertaken. PCU specialist staff also provided hands-on support to sub-projects, working with the PMUs. Justification of Overall Rating of Bank Performance 80. Based on the analysis above, World Bank performance is rated ‘Moderately Satisfactory’. D. RISK TO DEVELOPMENT OUTCOME 81. The PPTAF strengthened capacity among government agencies in the planning and preparation of public investments. International standards are increasingly used and adopted as part of the public investment planning process, including through the application of the new Planning Law. It is expected that Project gains will have long term effect in Vietnam—a country that has proven to effectively take on board and apply processes and skills that allow for the country to move forward. 82. Long term effect through integration of practical PPTAF experiences into the drafting of the new ODA management decree. More specifically, it can be highlighted that the Foreign Economic Relations Page 28 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) Department (FERD) (in charge of drafting the new ODA management Decree) and the Public Procurement Agency (PPA) (in charge of overseeing procurement works), through their direct experience implementing PPTAF funds, have gained in-depth knowledge as to the constraints represented by ODA Decree 16 on PMUs in implementing ODA-funded projects. As such, operational and practical insights and knowledge gained by the PPTAF team, which is also in charge of the drafting of the new Decree, will inform and be integrated into the forthcoming Decree. V. LESSONS AND RECOMMENDATIONS 83. TA/capacity development projects require conservatism in design and scope. The PPTAF had a clear rationale and justification, as well as problem statement. There was also strong commitment on the government and the World Bank side. Despite this, the project struggled throughout implementation, and demand for financing did not meet expectations. Using a portfolio development project to strengthen capacity of government agencies to plan and prepare investments can be a successful approach—and the idea is deemed a good one. Nevertheless, inherent and customary issues linked to TA/capacity development projects need to be carefully considered—both in design and implementation. Such projects are usually difficult to implement quickly and effectively, and should be designed conservatively—in size and scope. They also require a robust M&E framework and mechanisms and strong outreach to stakeholders to ensure familiarity with approach and processes. 84. Preparing a TA/capacity development project requires time and creative solutions. The PPTAF was prepared in a short time period. While short project preparation times can demonstrate strong commitment and responsiveness of the World Bank, it may not provide teams with enough time to properly think through the design. This may be particularly true for capacity development projects, which typically respond to and aim to address very specific (rather than generic) needs and issues. As such, approaches used in other projects may not be appropriate and new designs and approaches will need to be developed. And this takes time. 85. Defining outcome-oriented indicators to track progress in developing capacity at the institutional, organizational, and individual levels is challenging. Formulating and pitching the PDO at the right level is a complicated—but important—task for most project preparation teams. This is especially true for TA/capacity development operations. Revisions to the PDO should be seriously considered if the link between the PDO and the project’s activities and results framework is deemed weak. Furthermore, it is challenging to define easily monitorable outcome indicators that allow teams to assess progress in strengthening capacity at various levels. Such TA projects would also benefit from a detailed capacity development needs assessment, leading to a capacity development plan. 86. Strong supervision is critical for implementing TA/capacity development projects and may require task teams to be permanently on the ground. Field-based task teams may facilitate implementation of TA projects, allowing for frequent communication and engagement. MTRs should be used effectively and external views may be beneficial in identifying opportunities and options for change and adjustments that may strengthen implementation and impact. 87. Strong commitment from government and strong project management are factors of success that are even more crucial in the case of a TA/capacity development project than in other types of projects. Commitments need, however, to be translated throughout the implementation chain, to include Page 29 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) all levels and actors/agencies. Implementation processes also need to be easily navigable and reflect institutional realities. 88. The design of such TA/capacity development projects should reflect the reality of how government agencies and institutions work and their structure and account for any challenges linked to inter-agency or inter-ministerial coordination. Roles and responsibilities need to be clearly defined and understood. Smaller, sector-specific engagements may facilitate implementation of such projects and ensure simpler processing. Page 30 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Improve Planning and Preparation of Investment Projects Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Time taken to complete Months 30.00 12.00 12.00 14.90 international procurement packages related to civil 31-Mar-2010 31-Mar-2015 31-Oct-2017 19-May-2017 works in PPTAF financed activities Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Average time taken from Months 21.60 6.00 6.00 5.20 effectiveness to first disbursement for projects 31-Mar-2010 31-Mar-2015 31-Oct-2017 31-Oct-2017 that have used PPTAF funds Comments (achievements against targets): Issue with baseline and target figures, since the baseline following restructuring should have been 5.5 months Page 31 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) and not 21.6 months. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of PPTAF Percentage 0.00 45.00 41.00 94.00 financed Sub-projects with 30 percent of detailed designs 18-May-2010 31-Mar-2015 31-Oct-2017 19-May-2017 completed before Board approval Comments (achievements against targets): Recalculated after the submission and approval of the project’s last ISR. Delivery therefore not captured in the October 2017 ISR. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of PPTAF Percentage 0.00 81.00 70.00 94.00 financed sub-projects with the "quality of detailed 24-May-2010 31-Mar-2015 31-Oct-2017 31-Oct-2017 design activities meeting international standards" Comments (achievements against targets): An impact study was conducted by a third-party consultancy company to assess achievement of this indicator. However, given time limitations in conducting the study, the findings were based on a review of four subprojects that met the following criteria: (a) subprojects that financed detailed design contracts (some subprojects did not have detailed design contracts); (b) subprojects representing different regions (North, Hanoi, and South); and (c) subprojects that had large contracts. As such, the four subprojects examined are representative of the portfolio of sub-projects. Page 32 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) Objective/Outcome: Increase Institutional Capacity Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Time taken to complete Months 30.00 12.00 12.00 14.90 international procurement packages related to civil 31-Mar-2010 31-Mar-2015 31-Oct-2017 19-May-2017 works in PPTAF financed activities Comments (achievements against targets): A.2 Intermediate Results Indicators Component: Component 1: project preparation facility Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of PPTAF-financed Number 0.00 22.00 22.00 22.00 subprojects completed 18-Mar-2010 31-Oct-2017 31-Oct-2017 31-Oct-2017 Comments (achievements against targets): Component: Component 2: project preparation facility support and capacity building Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 33 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) Number of officials and Number 10.00 300.00 300.00 457.00 project staff under PPTAF financed sub-projects trained 18-May-2010 31-Oct-2017 31-Oct-2017 31-Oct-2017 on WB fiduciary requirements Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of studies prepared Number 0.00 12.00 12.00 12.00 as key inputs to drafting of the Planning Law 18-May-2010 31-Oct-2017 31-Oct-2017 31-Oct-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Procurement Capacity Yes/No N Y Y Y Building Program is delivered 18-Mar-2010 31-Oct-2017 31-Oct-2017 31-Oct-2017 Comments (achievements against targets): Achieved after the submission and approval of the Project’s last ISR. Delivery therefore not captured in the October 2017 ISR. Indicator Name Unit of Measure Baseline Original Target Formally Revised Actual Achieved at Page 34 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) Target Completion MPI's Circular and Manual on Yes/No N Y Y N ODA planning and M&E is approved 18-Mar-2010 31-Oct-2017 31-Oct-2017 31-Oct-2017 Comments (achievements against targets): The Circular was delivered by an MPI department that was not supported by the PPTAF, but the Manual was delivered under the PPTAF. Hence this indicator is partially achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion A central database for all Yes/No N Y Y Y ODA projects is established in the MPI 01-Jun-2012 31-Oct-2017 31-Oct-2017 31-Oct-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion PMU Director's Forum is Yes/No N Y Y Y launched and maintained 18-Mar-2010 31-Oct-2017 31-Oct-2017 31-Oct-2017 Comments (achievements against targets): Page 35 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1: Improve Planning and Preparation of Investment Projects 1. Number of PPTAF-financed sub-projects detailed design packages using international standards like environment and social safeguard Outcome Indicators 2. Number of PPTAF-financed sub-projects with 30% of detailed designs completed before Board approval 1. Percentage of PPTAF-financed sub-projects with the “quality of detailed design activities meeting international standards” 2. Percentage of PPTAF-financed sub-projects with 30% of detailed designs completed before Board approval Intermediate Results Indicators 3. Time taken to complete international procurement packages related to civil works in PPTAF-financed activities 4. Average time taken from effectiveness to first disbursement for projects that have used PPTAF funds 5. Average time from PCN to Board approval 1. Number of PPTAF-financed sub-projects completed Key Outputs by Component 2. For projects receiving PPTAF support, the time required to achieve 10% (linked to the achievement of the Objective/Outcome 1) disbursement Objective/Outcome 2: Increase Institutional Capacity 1. MPI’s circular and manual on ODA Planning and M&E approved Outcome Indicators 2. Procurement capacity program delivered 1. Number of studies prepared as inputs to the development of draft Planning Law Intermediate Results Indicators 2. PMU Director’s Forum launched and maintained 1. Number of officials and projects staff under PPTAF-financed sub-projects trained Key Outputs by Component on World Bank fiduciary requirements (linked to the achievement of the Objective/Outcome 2) 2. Central database for all ODA projects established Page 36 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Supervision/ICR Nga Thi Quynh Dang Task Team Leader(s) Kien Trung Tran Procurement Specialist(s) Ha Thuy Tran Financial Management Specialist Thu Thi Le Nguyen Environmental Safeguards Specialist Quang Nhat Nguyen Social Safeguards Specialist Minh Phan Team Member B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY10 18.638 174,224.42 FY11 0 0.00 Total 18.64 174,224.42 Supervision/ICR FY10 0 0.00 FY11 14.834 68,359.74 FY12 3.112 12,044.47 FY13 12.660 43,183.39 FY14 8.868 35,603.96 FY15 18.220 28,880.46 Page 37 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) FY16 21.690 44,029.64 FY17 28.600 57,968.33 FY18 22.200 84,772.08 Total 130.18 374,842.07 Page 38 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Component A: Project 0 63.77 0 Preparation Facility Component B: Project Preparation Support and 0 4.82 0 Capacity Building Component C: Incremental Operating Costs and Project 0 .41 0 Management Expenses Total 0.00 69.00 0.00 Page 39 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) ANNEX 4. EFFICIENCY ANALYSIS As is customary for TA credits, there was no economic or financial analysis done at project appraisal stage. Page 40 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS The borrower (MPI) was consulted during the preparation of the ICR and the following comments received: “As stated in the report, the Borrower through the MPI/PCU was responsible for collecting and consolidating data from different sources that could be used to assess achievement towards project objectives—as per the M&E framework and this work was done satisfactorily and submitted on time to the World Bank. During preparation of this ICR the PCU was closely cooperated with the Bank, so that all the results of the indicators presented in this report were agreed, the Borrower has no further comments on it. The project’s results framework and its indicators were significantly revised through three project restructurings aimed to address shortcomings and in fact, some adjustments in project activities have been made brought positive results to the project (e.g. the addition of capacity building activities for the Component B of the project). The Borrower agreed with the statement in the report that significant shortcomings are observed as to the quality of the Project’s M&E system design. If proper adjustments were made promptly during the project MTR phase, the results would be better achieved.” Comments received on behalf of MPI from Mr. Cao Manh Cuong, Deputy Director General, Director of PPTAF Foreign Economic Relations Dept., Ministry of Planning and Investment, Hanoi, Vietnam Page 41 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) Table 6. Projects prepared under the PPTAF and associated CPF Focus Areas Relevant CPF Focus Area Project Name Project Number (FY18-22 CPF) Medium Cities Development Project P116398 Focus Area 1: Da Nang-Quang Ngai Expressway Development Project P106235 Enable Danang Sustainable City Development Project P123384 inclusive Vietnam Road Asset Management Project P123961 growth and Central Highlands Poverty Reduction P128072 private Northern Mountains Urban Development Program P143596 sector participation Second Ho Chi Minh City Environmental Sanitation Project P127978 Ho Chi Minh City Green Transport Development Project P126507 Focus Area 2: Vietnam School Readiness Promotion Project P117393 Invest in Fostering Innovation through Research, Science and P117394 people and Technology knowledge Vietnam Inclusive Innovation Project P121643 Table 7. Overview of Key Trainings, Workshops, and Study-Tours Delivered and Supported under the PPTAF Training/Course Participants – representatives of Date Location ODA projects - central and decentralized PPTAF 2011 Hanoi PMUs - line ministries - PMUs for WB financed projects Project preparation and - central and decentralized PPTAF 2011, 2013 Hanoi, Da implementation for World Bank funded PMUs Nang, Ho Chi projects - line ministries Minh City - PMUs for WB financed projects (HCMC) Guidance on project implementation for - central and decentralized PPTAF 2011, 2012 Hanoi, Binh World Bank funded projects PMUs Minh, Ninh - line ministries Binh - PMUs for WB financed projects The World Bank's procedures and - central and decentralized PPTAF 2011 Hanoi, Vietnam Regulations on procurement PMUs management and financial - line ministries management - PMUs for WB financed projects Guidance on preparation and - central and decentralized PPTAF 2011, 2012 Da Nang implementation of PPTAF project PMUs Management, transparency, and anti- - central and decentralized PPTAF 2011, 2012, Hanoi, HCMC corruption in implementation of ODA PMUs 2014 projects in Vietnam - line ministries - PMUs for WB financed projects Page 42 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) The World Bank's procedures and - central and decentralized PPTAF 2012 Hanoi Vietnam regulations on bid PMUs management and financial - line ministries management - PMUs for WB financed projects Procurement procedures in preparation - central and decentralized PPTAF 2013 Dalat and implementation of World Bank PMUs funded projects in Vietnam - line ministries - PMUs for WB financed projects Financial management procedures in - central and decentralized PPTAF 2013 Quy Nhon project implementation for World Bank PMUs funded projects - line ministries - PMUs for WB financed projects English language training - decentralized PPTAF PMUs 2014, 2015 - PMUs for WB financed projects Under the planning law workplan: - government agencies 2015 Hanoi, HCMC, five consultation workshops on planning - parliamentary office Danang, Vinh laws - ministries Phuc, Quang - research institutes Ninh - universities Under the ODA workplan: - government agencies 2016, 2017 Hanoi, HCMC, (i) six training courses and one - provincial agencies Quang Ninh, workshop organized by the FERD, MPI, - ministries Vinh, Can Tho, on the institution and planning, M&E of - line agencies Dak Lak the medium term, and annual - implementation agencies investment plans for ODA and - PMUs for WB financed projects concessional loans (ii) two workshops and one training course organized by MOF on online application registration procedures of ODA and concessional loans Under the PPA workplan: - ministries 2017 Hanoi, Danang (i) two consultation workshops on - provincial authorities professional procurement training - provincial SMEs materials (ii) two training-of-trainer courses and three training workshops on professional procurement (iii) two workshops for SMEs Study tour to the UK and the - MPI, OOG, SBV, MOT, MOF 2014 Netherlands on project management (focus on infrastructure projects)19 Study tour to Canada on planning - National Assembly, MPI 2014 (linked to the new planning law) 19 The PPTAF stopped financing study-tours in 2014 as the value for money was not clear. Page 43 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) Table 8. PPTAF Outcome and Intermediate Indicators – Achievements PDO: Increase the capacity of government agencies to plan and prepare public investments efficiently and to international quality standards Project outcome indicators May 2014 restructure December 2015 restructure March 2017 restructure Original (May 2010): Improved May 2014: Original PDO December 2015: Target values March 2017: Three new PDO project quality at entry through indicator dropped. Replaced by: of PDO indicators revised: indicators introduced an increase in details and range (previously intermediate results of documentation ready by (A) Number of PPTAF-financed (A) Number of PPTAF-financed indicators). The value of one appraisal of the resulting sub-projects with 30% of sub-projects with 30% of PDO indicator changed from investment loan, and more detailed designs completed detailed designs completed number to percentage substantial completion of start- before Board approval. before Board approval. (indicator (C)). up activities prior to investment Baseline: 0 Baseline: 0 loan approval. Year 5 target: 10 Year 7 target: 10 (A) Time taken to complete Measured through results Contribution to strengthening Achieved: international procurement indicators capacity for more efficient Contribution to strengthening packages related to civil works planning and preparation capacity for more efficient in PPTAF-financed activities. planning and preparation Baseline: 30 months (B) Number of PPTAF-financed Target: 12 months sub-projects detailed design (B) Number of PPTAF-financed Achieved: 14.9 months (84% of packages using international sub-projects detailed design target) standards like environment and packages using international Contribution to strengthening social safeguard standards like environment and capacity for planning and Baseline: 0 social safeguard preparation to international Year 5 target: 18 Baseline: 0 quality standards Contribution to strengthening Year 7 target: 18 capacity for planning and Achieved: (B) Average time taken from preparation to international Contribution to strengthening effectiveness to first quality standards capacity for planning and disbursement for projects that preparation to international have used PPTAF funds. quality standards Baseline: 21.6 months Target: 6.0 months Page 44 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) Achieved: 6.1 months Contribution to strengthening capacity for more efficient planning and preparation (C) Percentage of PPTAF- financed sub-projects with 30% of detailed designs completed before Board approval. Baseline: 0% Target: 41% Achieved: 94% (293% of target) Contribution to strengthening capacity for more efficient planning and preparation (D) Percentage of PPTAF- financed sub-projects with the “quality of detailed design activities meeting international standards”. Baseline: 0% Target: 70% Achieved: 94% (134% of target) Contribution to strengthening capacity for planning and preparation to international quality standards Page 45 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) Intermediate outcomes Intermediate outcome Baseline Target Assessment of achievement indicators and contribution to PDO Original (May 2010): Increased Original (May 2010): capacity within participating (A) Reduced preparation period 27 months 20 months 27 months – achieved at 0% agencies and Vietnamese for projects compared with the (years 4, 5) consulting industry to produce situation before the project project preparation documents facility came into existence. of good quality, and to an Average time from PCN to acceptable schedule (to be Board approval measured through results indicators) (B) Increase in the number of (i) 5% of the 25% (years 3, projects prepared under a sub- portfolio 4, 5) project that have achieved the (ii) n/a readiness criteria of completion of procurement ready to award contracts at time of financing approval for either (i) 30% of the value of the total project cost; or (ii) the first 18 months of activities (C) For projects prepared using (i) 21.6 15 months 21.6% - achieved at 0% the Project, the time required (ii) n/a (years 4, 5) to achieve 10% disbursement is reduced compared with (i) the situation before the facility came into existence; and (ii) projects prepared without accessing the facility Page 46 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) May 2014: Two original results indicators dropped, six new intermediate results indicators introduced: (A) Time taken to achieve first 5.5 months 6 months disbursement from (year 5) Effectiveness for projects that have used PPTAF funds (B) Number of PPTAF-financed Not defined Not defined PMUs with experience on World Bank policies and guidelines before Board approval (C) Quality of Detailed Design 0 10 (year 5) activities financed by the PPTAF that meet international standards (D) Time taken to complete 30 months 12 months international procurement (year 5) packages related to civil works in PPTAF-financed activities (E) Number of officials and 10 300 (year 5) project staff under PPTAF- financed sub-projects trained on World Bank fiduciary requirements (F) Inputs to the development No Yes (year 5) Page 47 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) of draft Planning Law December 2015: All target values were revised to align with the new Project closing date (October 2017) (A) Time taken to achieve first 5.5 months 6 months 5.2 months – disbursement from (year 7) Contribution to strengthening effectiveness for projects that capacity for more efficient have used PPTAF funds planning and preparation (B) Number of PPTAF-financed 0 10 (year 7) N/A - PMUs with experience on World Contribution to strengthening Bank policies and guidelines capacity for more efficient before Board approval planning and preparation (C) Quality of Detailed Design 0% 70% 94% of detailed design activities activities financed by the PPTAF – achieved at 134% - that meet international Contribution to strengthening standards capacity for planning and preparation to international quality standards (D) Time taken to complete 30 months 12 months 14.9 months – achieved at 85% international procurement (year 7) Contribution to strengthening packages related to civil works capacity for planning and in PPTAF-financed activities preparation to international quality standards (E) Number of officials and 10 300 (year 7) 457 – achieved at 152% - project staff under PPTAF Contribution to strengthening Page 48 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) financed sub-projects trained capacity for planning and on World Bank fiduciary preparation to international requirements quality standards (F) Inputs to the development No Yes (years 5, 6, Yes – achieved at 100% - of draft Planning Law 7) Contribution to strengthening capacity for more efficient planning and preparation March 2017: Four intermediate indicators dropped/deleted (indicator (A), (B), (C), (D) above). (A) Number of officials and 10 300 457 – achieved at 152% - project staff under PPTAF- Contribution to strengthening financed sub-projects trained capacity for planning and on World Bank fiduciary preparation to international requirements (No change) quality standards (B) Number of studies prepared 0 12 12 – achieved at 100% - as key inputs to drafting of the Contribution to strengthening Planning Law (Revised) capacity for more efficient planning and preparation (C) Number of PPTAF-financed 0 22 22 – achieved at 100% - sub-projects completed (New) Contribution to strengthening capacity for more efficient planning and preparation (D) Procurement Capacity No Yes Yes - achieved at 100% - Building Program is delivered Contribution to strengthening (New) capacity for more efficient planning and preparation (E) MPI’s Circular and Manual No Yes Circular not delivered, manual Page 49 of 50 The World Bank Project Preparation Technical Assistance Facility Project (P118610) on ODA planning and M&E is approved – achieved at 50% - approved (New) Contribution to strengthening capacity for more efficient planning and preparation (F) PMU Director’s Forum is No Yes Yes - achieved at 100% - launched and maintained (New) Contribution to strengthening capacity for more efficient planning and preparation (G) Central database for all ODA No Yes Yes – achieved at 100% - projects established in MPI20 Contribution to strengthening capacity for more efficient planning and preparation 20 This indicator is not listed in the March 2017 results framework, but figures in the project’s last ISR. It is therefore included in the table and discussion. Page 50 of 50