ICRR 14272 Report Number : ICRR14272 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 12/09/2013 Country : Vietnam Project ID : P085080 Appraisal Actual Project Name : Road Safety Project US$M ): Project Costs (US$M): 34.92 27.64 L/C Number : C4073 Loan/ US$M ): Loan /Credit (US$M): 31.73 23.74 Sector Board : Transport US$M): Cofinancing (US$M ): Cofinanciers : Board Approval Date : 06/07/2005 Closing Date : 12/31/2009 12/31/2012 Sector (s): Central government administration (57%); Roads and highways (22%); Other social services (12%); Health (5%); General education sector (4%) Theme (s): Injuries and non-communicable diseases (67% - P); Other social development (33% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Peter Nigel Freeman Kristin Hallberg Soniya Carvalho IEGPS1 2. Project Objectives and Components: a. Objectives: The project development objective in the Credit Agreement (page 15), used in this review, is to reduce the rate of accidents, injury and death associated with road transport through physical improvement and institutional development to strengthen the management of road transport safety . The objective given in the PAD is virtually identical except that the word death is used instead of fatalities . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The ICR does not provide the cost at closure for each component, hence only the cost at appraisal per component is given below: Component A - Institutional and capacity building program (Appraisal project total US$ 7.84 million ) to provide technical assistance and project implementation support to strengthen the management and technical capacity of the National Traffic Safety Committee (NTSC) Executive Office, the Traffic Safety Project Management Unit (TS PMU), the sub-PMUs, and corridor supervision teams . This component was also to support the preparation of a performance-based national road safety strategy, including a financing plan . Component B . Road safety demonstration and awareness program (Appraisal project total US$ 20. 20 .89 million ) to develop and implement comprehensive, integrated safety programs along three high -risk national road corridors. The three demonstration corridors identified for inclusion under the project were : (i) National Highway 1 between Hanoi and Vinh (281 km); ( ii) National Highway 1 between Ho Chi Minh City and Can Tho City (1 5 1 km); and (3) Highway 51 from Dong Nai province to Vung Tau (80 km). The component supported: (i) a "Vietnam Register" to upgrade its national vehicle inspection system, computer and communications equipment, software development, and vehicle inspection equipment, including specialized vehicle inspection equipment at three primary national vehicle inspection stations (Hanoi, Ho Chi Minh City, and Vinh Long).; (ii) the Vietnam Road Administration (VRA) with the supply of equipment to improve the driver training centers at Chi Linh and in Dong Nai province and related facilities; (iii) consulting services for VRA to conduct road safety audits and blackspot or black route identification and treatment; (iv) the Traffic Police through the provision of vehicles and equipment to implement a general-deterrence road safety enforcement program in the three demonstration corridors; (v) a program of road user education and public awareness in the three demonstration corridors, using targeted social marketing techniques; (vi) the Ministry of Education and Training with the provision of school road safety education materials, equipment, and associated training for a program of primary, secondary, and tertiary road safety education in the three demonstration corridors; (vii) the Ministry of Health by providing materials, equipment, and training for an emergency medical services program in the three demonstration corridors ; (viii) operating costs and local consulting services. Component C . Road Safety Monitoring and Evaluation Program (Appraisal project total US$ 6.20 million ), to support the development of the National Road Accident Database System (NRADS) and a comprehensive program for project corridor monitoring and evaluation . The following component changes were introduced during implementation : A4.2: Reduction in overseas training from US$ 552,000 to US$270,000. B1: Some blackspots on Highway 51 and NH1 were covered under other projects (Highway 51 upgrading and NH1 widening). B5: Purchase of driving license printing system in lieu of equipment in the Dong Nai driver training center B7: Purchase of a camera system in lieu of cars and equipment for the police B4.3: Cancellation of sub-component B4.3 after agreement with the Government of Vietnam (GOV) that equipment for vehicle inspection centers would be provided through GOV funding B8,2: Reduction in the scope of work in education and awareness raising program under B 8.2. Instead of broad based awareness programs, these programs were redesigned to address specific target groups through selected themes. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project cost: The estimated cost of the project at appraisal was US$ 34.92 million (including contingencies). At closure the actual cost was US$ 27.64 million. Financing: An IDA credit of US$31.73 million was approved, but at closure only US$ 23.74 million had been spent. The US$6.9 million cancelled from the credit was due to (i) a reduction in the number of blackspots treated under the project since some were completed under other projects, (ii) contingencies amounting to approximately US$ 1.3 million that were not required, (iii) close management of consultants also generated savings of about US$ 1 million and (iv) exchange rate gains between SDR and USD were about US$ 1.2 million. Borrower contribution: The estimated Borrower contribution at appraisal was US$ 3.19 million. At closure the actual figure was US$3,90 million. Dates: The closing date was extended twice for a total of 36 months: first from December 31, 2009 to June 30, 2011 (18 months) to address the action plan to accelerate implementation that was set during the midterm review mission; and second for a further 18 months from June 30, 2011 to December 31, 2012 to allow for the conclusion of several on-going training activities and completion of substantial packages critical to the full achievement of the project development objectives. 3. Relevance of Objectives & Design: a. Relevance of Objectives: High Vietnam had a reported fatality rate of about 12 per 10,000 vehicles and an accident rate of about 20 per 10,000 vehicles in 2004, higher than in neighboring Thailand and Malaysia and significantly higher than the Organization for Economic Cooperation and Development (OECD) countries. By addressing the immediate need to reduce the rates of road accidents, injuries, and fatalities in the country, the project assisted the articulated priorities in the 2005 country planning documents, confirmed in the FY 2012-16 CPS and the 2011-2015 Government’s Socioeconomic Development Plan. Their objectives were (i) strengthening Vietnam's competitiveness in the regional and global economy, (ii) increasing the sustainability of its development, and (iii) broadening access to economic and social opportunity. Reducing road accident related health shocks was expected to improve the productivity of the work force, and address the administrative, legal, and governance arrangements required to improve road safety and build up cross-sector capacity towards a more sustainable development model . Lowering the health costs and economic harm resulting from road crashes would indirectly reduce the number of people in poverty . The Vietnam Road Safety Project was also consistent with the move towards the "Safe System" approach to road safety recommended by the OECD. The approach recognized that road systems could be designed to expect and accommodate human error, could address all elements of the road transport system in an integrated way, would require acceptance of shared overall responsibilities and accountability between system designers and road users, and through implementing innovative interventions and new partnerships, The project aimed to adopt a different approach by a developing country to improve road safety . Vietnam was seen as a good candidate to take on this challenge because of the GOV's continuing commitment to improve Road Safety . In 2011 the United Nations Decade of Action for Road Safety was launched in Vietnam and the Minister of Transport declared 2012 as the Year of Road Safety in Vietnam. b. Relevance of Design: Modest The design of the project took into consideration the findings of the Vietnam 2003 Transport Safety Strategy Review and was consistent in size and scope with objectives described in the Country Assistance Strategy Progress Report for Vietnam, 2004. The components, consisting of a demonstration program with capacity building and monitoring were logical to achieve the stated objectives . The results framework was based on the OECD road safety management pyramid, which links interventions to intermediate outcomes to primary outcomes in road safety . It was not clear, however, how well this approach would work or needed to be adapted in a developing country context where there was limited data and insufficient capacity . More seriously, the design was overly complex to support the objectives as formulated. 4. Achievement of Objectives (Efficacy): Modestly achieved The project development objective is to reduce the rate of accidents, injury and death associated with road transport through physical improvement and institutional development to strengthen the management of road transport safety . Regarding institutional capacity, the absence of formally established and fully functional sub -project implementing agencies was an unresolved issue since these agencies remained dependent on the availability of the staff that could be assigned to them. Despite several training activities for the procurement teams, procurement performance was less than satisfactory . Each project output contributed differently to addressing the many factors that cause road accidents along the corridors including the treatment of accident victims . The project delivered 34,000 safety helmets to 17 schools along the demonstration corridors; it removed 40 blackspots on the three demonstration corridors; trained and certified 150 road engineers as road safety auditors; it upgraded vehicle inspections, and designed a National Vehicle Testing Center. Under enforcement, the project commissioned a traffic enforcement camera system; upgraded equipment for driver licensing system nation -wide; upgraded driver training equipment and testing facilities at the Chi Linh Driving Training Center; and designed, installed and commissioned a new National Road Accident Database System. Finally, the project procured and delivered three ambulances as well as emergency equipment to 46 hospitals and clinics and completed first -aid responder training for traffic policemen and clinic medical staff . A performance based national road safety strategy, including a financing plan, was developed and approved by Government. Traffic speeds were reduced and valid licences are now held by 93 percent of drivers. However, two intermediate targets were only partially met, namely the preparation of annual reports on the effectiveness of corridor interventions and the percentage of vehicles passing a first time inspection . The project outcomes indicate that two of the identified outcomes were met i .e. the accident rate and fatality rate targets were achieved, (the project corridors showed reductions in accident rates from 39 to 25 accidents per 100 million vehicle- km. against a target of 29, and a reduction in the fatality rate from 13 to five fatalities per 100 million vehicle-km., against a target of six). The reduction in the injury rate, which was expected to reduce from 130 to 62 severe injuries per 100 million vehicle-km, was actually 107, falling well short of target. The ICR speculates (footnote, page 12) that this was likely due to under reporting of injuries prior to the project meaning that the baseline was too low, but this condition may have applied equally well to the accident and fatality rates too . 5. Efficiency: Substantial The Net Present Value and Economic Rate of Return (ERR) for the project show significant point values, but may be on the high side in terms of estimation . The methodology in Annex 3 of the ICR using the "willingness to pay" approach to accident costing as used in industrialized countries resulting in an ERR at closure of 77 percent. This review uses the more conservative "gross output method" recommended by the Asian Development Bank (ADB) for developing countries. The ADB approach gives an ERR of 59 percent. However, the ICR discussion on M&E (page 8) reports poor quality baseline data, which may have affected the results. It is unclear whether the three years of project delays were factored into the estimation . Nevertheless, even taking all these factors into account, the cost -benefit results were still likely strongly positive, showing what a good investment road safety projects can be . Coverage/scope at ICR was not available . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 34% 59% ICR estimate Yes 59% 0% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project had high relevance of objectives and modest relevance of design . The accident and fatality rates declined, meeting their targets, but the injury rate did not . Nevertheless, despite some methodological weaknesses, the rate of return was satisfactory . Human capacity was not improved sufficiently, however, and implementation issues at the start of the project meant that the deliverables were achieved three years later than originally intended . Because of the significant implementation delay at the start of the project, many project activities were implemented at a late stage, including the installation of cameras, the road safety awareness building campaign, and the National Road Accident Database System . There were thus some moderate shortcomings . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: Lack of funding, especially for the operation and maintenance on the blackspot interventions, is a threat that could undermine the project's achievements . Capacity could remain inadequate in the NTSC in the short to medium term, mainly due to inadequate staffing and limited technical knowledge in several areas of road safety . Moreover, the country’s institutional framework for road safety still lacks some critical elements found in good practice countries . For instance, without an operational /management level committee and sector working groups, the work of the NTSC will likely suffer from a lack of sustained and ongoing coordination between meetings . On the other hand, according to the ICR (page 15), a follow-on project is being discussed with the Bank, which would emphasize GOV ’s commitment on these issues. In 2011 the United Nations Decade of Action for Road Safety was launched in Vietnam and the Minister of Transport declared 2012 as the Year of Road Safety in Vietnam, so support is there at a high level - the challenge is for this high level commitment to filter down and result in additional resources at the operational level. According to the ICR (page 11),the Government has advised that it will allocate an annual budget to NTSC to sustain the road safety forum and trigger initiatives in road safety . The Ministry of Health is preparing a new master-plan for ambulance locations; the Ministry of Education and Training is rolling out school programs to university level, while the traffic police propose to roll out the installation of additional traffic camera enforcement equipment. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: The World Bank partnered with the World Health Organization in 2004 to produce the "World Report on Road Traffic Injury Prevention", demonstrating both organizations ’ recognition of road safety as a major world health issue. The Bank, based on this experience and commitment, and its substantial engagement in the road sector in Vietnam since 1994, considered it was well positioned to work with GOV to improve road safety . Conceptually, the project was path-breaking in that it sought to coordinate a holistic approach to road safety in Vietnam . However, the project exhibited serious weaknesses in two areas at entry : first, an overly complex design, which included 18 sub-components, which triggered the procurement of more than 60 packages, involved 63 cities and provinces all over Vietnam; and second the lack of clear and effective institutional arrangements, with an implementing agency (NTSC) which had neither adequate resources nor expertise to carry out its leadership role and direct participating agencies . In addition, TS PMU (under the authority of NTSC), which had procurement responsibilities for packages to be implemented by sub -PMUs, was not empowered to carry them out . This design made it extremely challenging to coordinate the preparation of all the components across agencies, and was a major factor behind implementation delays . The Borrower itself described the project design as unreasonable in its own ICR (page 36). No quality of entry assessment was made . More attention should have been devoted to setting up a rigorous M&E framework from the outset . Due to the low quality of the baseline figures estimated during preparation, the M&E activities had to rely on a different baseline . Many key results indicators were absent from the PAD for both component B and for institutional strengthening activities. at -Entry Rating : Quality -at- Moderately Unsatisfactory b. Quality of supervision: During the first two years of project implementation, (and despite the fact that the disbursement rate was below eight percent), Bank official supervision of the project remained limited . Prior to the project being flagged as having problem status in July 2008, only three official supervision missions had been conducted (December 2005, October 2006, and July 2007). Similarly, only three Implementation Status Reports had been completed - in January 2006, February 2007, and in June 2008. As a result, and also because of the complex institutional arrangements and the high number of packages in the project design, procurement problems were not thoroughly followed up and the closing date had to be extended twice . There was no quality of supervision assessment. However, following the 2009 midterm review, there was a visible turnaround, structured around a 19-target time bound action plan, and on which the extension of the project closing date was conditional . Feedback from GOV agencies showed that the much more proactive approach followed by the Bank supervision team played a strong role in the successful turnaround of the project during its last three years . The team in Hanoi in particular provided hands-on oversight of the project. According to the guidelines, where one dimension of performance is moderately unsatisfactory and one is moderately satisfactory, the overall performance is rated moderately unsatisfactory . Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: There were shortcomings in coordinating efforts across several line ministries in the early stage of implementation, although this gradually improved after the Mid -term Review. The NTSC was mandated as the key road safety planning and coordination body of the various ministries to facilitate implementation of the road safety initiatives. However, it did not have the authority, the staff nor the expertise to exercise its oversight . In effect, the NTSC only had advisory powers and was not positioned at the higher level of decision -making; its resources were markedly insufficient in terms of manpower, access to data, and for conducting studies and analyses, and its staff did not include many traffic safety specialists . The GOV has, nevertheless, shown commitment to and elevated the importance of road safety in the country, although this needs to be translated into more road resources in the sub -sector. In order to strengthen the capacity of NTSC, the GOV passed Decree No .32 to consolidate its structures and decision -making procedures with the Deputy Prime Minister installed as the Chair of NTSC . Also, a change of leadership was implemented at TS PMU, which had noticeable effects in terms of project ownership, clearer direction and faster delivery of procurement documents. Staffing resources were later strengthened in both NTSC and TSPMU . While this intervention came late in the project, it emphasized the sincerity and seriousness of the Government in the road safety field. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: Performance of the six agencies responsible for implementation varied from satisfactory to moderately unsatisfactory according to the ICR table (page 1). In most cases the work was completed but with delays largely due to inadequate capacity . The NTSC delegated much of their responsibilities to other agencies which resulted in delays and also to approvals of submissions from TS PMU . Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The M&E framework was designed based on the "4Es" (Engineering, Education, Enforcement, Emergency ) approach and the road safety management pyramid, which links interventions to intermediate outcomes to primary outcomes in road safety. The final outcomes to be measured were reductions in the number of accidents, injuries and fatalities occurring. Accidents were classified by severity levels of fatal, severe injury and slight injury . Intermediate outcomes (such as decrease in traffic exceeding posted speed limits, better maintained vehicles, increase in motor cycle riders wearing helmets or increase of safety belt usage ) were used to measure compliance with traffic law or standards . However, an opportunity was missed . Observers of the project have noted that there was no attempt to find out impact on those who benefitted from the distribution of the 34,000 safety helmets. However, due to a lack of reliable data at appraisal the M&E framework presented in the PAD omitted several important targets while some baseline values proved to be inaccurate . Issues with the quality of road safety data had been anticipated in the PAD, so to address the need for high quality data the project included the design, development and commissioning of a National Road Accident Database System, but this only became effective at a late stage. b. M&E Implementation: M&E implementation focused on the collection of data for the demonstration corridors, on the development of a methodology for monitoring road safety improvements and interventions, and on the training of local staff in methods and procedures. Six training courses were conducted on monitoring, evaluating and reporting of road safety projects and interventions, as well as reporting on primary and intermediate outcomes . Due to the low quality of the baseline figures used during preparation, difficulties were encountered in measuring changes against these baselines during implementation . Also, key results indicators for Component B (percentage of traffic exceeding posted speed limits; number of motorcycle riders wearing helmets; number of motorcycle riders licensed; commercial vehicle operators licensed; vehicles in demonstration corridors passing first time inspection ) were missing from the PAD and thus undermined the quality of the results framework . In the PAD baseline estimates for the total number of accidents and injuries were estimated based on factors applied to the number of fatalities as reported by the Ministry of Public Security (MOPS) - the factors applied were derived from experience in developed countries . This approach assumed (i) that MOPS data for number of fatalities were reliable, and (ii) that the factors applied were appropriate to Vietnam . Both of these assumptions proved to be flawed in that (i) the MOPS data underreported fatalities, and (ii) the factors used to determine the number of accidents and injuries were too low, resulting in further underreporting . To address these issues, a revised baseline was developed based on a more rigorous analysis of available data . In particular, greater use was made of Ministry of Health (MOH). For the purposes of the ICR, and to be able to compare the measurements with the original baseline, the M&E data reported at the end of the project was converted to be compatible with the baseline indicators from the PAD, using the factor of injuries reported by MOH to the amount of serious injuries reported by MOPS . This approach does not address underreporting of accidents and injuries in the PAD baselines, suggesting that the actual reduction in accidents and injuries was greater than that reported in this ICR . c. M&E Utilization: To improve the poor quality of the appraisal data the project included a contract for monitoring and evaluation along the demonstration corridors. Work under this contract confirmed the problem of significant underreporting of injuries and attempted to reconcile data from different sources . The results of this M&E work were only available at the end of the project and more attention should have been devoted from the outset to setting up a rigorous framework . M&E Quality Rating : Modest 11. Other Issues a. Safeguards: The project was classified as Category A and triggered the the Environmental Assessment (OP/BP 4.01) and Involuntary Resettlement (OP/BP 4.12) safeguard policies. An Environmental Management Plan (EMP) was prepared, providing a framework to prevent or mitigate potential environmental impacts. It also provided a monitoring plan to ensure EMP compliance, assessed potential environmental impacts and conducted a screening analysis of the 15 proposed blackspot improvements . The main potential impacts were found to relate to air quality, water quality and noise . It was considered unlikely that either the direct or indirect impacts of the engineering works would result in environmental conditions any worse than prevailed on the already existing roads, and any environmental impact that could not be prevented could be mitigated satisfactorily. The EMP covered five technical areas : materials supply (indirect impacts), soils, hydrology, air quality, and noise. The supervision consultants for black -treatment contracts provided adequate monitoring of the EMP implementation. No petitions on the environment issues were received from the public and the environmental safeguard policies were generally complied with . After investigation, no resettlement issues had emerged in the demonstration corridors . The review of potential social impacts, based on detailed designs for the corridors, confirmed there was no land acquisition, resettlement nor compensation involved for any of the packages or corridors, which were on existing alignments . b. Fiduciary Compliance: Despite minor inaccuracies, financial management was fairly satisfactory . Although the NTSC was the executing agency for the project, internal controls, quarterly financial reports, disbursement progress and follow up with audit issues were managed by the TS PMU . New accounting software was introduced in June 2010, but this did not prevent a pattern of minor mistakes in general accounting balances from continuing . Interim Financial Reports, expected by the Bank no later than 45 days after the end of each calendar quarter, were submitted on time and were generally of acceptable quality . The internal controls as regulated by the financial management manual were complied with, and bank, cash, and fixed assets were satisfactorily maintained . There were regular occurrences of ineligible expenditures, but these were swiftly refunded in accordance with the Project Credit Agreement . External audit reports were submitted to the Bank on time . They had unqualified audit opinions and acceptable quality . Procurement performance, however, was less than satisfactory . The lack of familiarity with Bank procurement guidelines and discrepancies in procedures between agencies substantially slowed down project implementation . The absence of formally established and fully functional sub -project implementing agencies, inadequate procurement capacity of the project implementing agencies and inadequate familiarity of the participating agencies with Bank procurement rules, together with the high -risk procurement environment were not sufficiently mitigated despite several training activities for the procurement teams . Also, appraisal procedures within GOV remained complex and lengthy, since the ministries retained their own procedures . c. Unintended Impacts (positive or negative): None d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately Note that IEG's rating for quality at Satisfactory Satisfactory entry was moderately unsatisfactory due to the project's overly complex design and the lack of clear and effective institutional arrangements. Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The following lessons are adapted from the ICR : A responsible lead organization with effective decision making powers and adequate capacity is a prerequisite to manage an effective road safety program . NTSC did not have any authority to streamline the implementation management systems in other agencies . Further, NTSC’s limited capacity prevented it from properly taking the role of an effective leading road safety agency . Simplicity of design is a key factor of success in environments where a multiplicity of stakeholders, sub-optimal administrative processes, weak institutional capacity, differences in appraisal procedures between ministries, and a wide geographical scope are bound to make implementation complex . The right mix of adequate funding and human resources for road safety interventions, institutional structures and processes is essential to establish and maintain national road safety strategies . A long term investment strategy, relying on multiple income streams to avoid over -dependency on a single source, would be ideal to implement the desired results. Some sources in other countries have included traffic fines, motor fuel, weight-distance charges, compulsory vehicle insurance, vehicle license fees, and toll fees . To ensure effective monitoring and evaluation a comprehensive road safety management information system is necessary, including a national road accident database . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: In general the ICR was well prepared and informative, giving appropriate coverage of the uneven features of this innovative project. The presentation of costs in the ICR could, however, have been more helpful . Ideally there should have been in Annex 1 costs per sub-component at appraisal and closure spent using the credit, not just a total for the project. Annex 2 should have had an additional column indicating costs per sub -component at closure. The figures given were for appraisal only. It would have been useful to have had a deeper discussion as to why the injury rate reduction was only partly achieved . a.Quality of ICR Rating : Satisfactory