Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-NORRED (P122629) Report Number: ICRR0022683 1. Project Data Project ID Project Name P122629 VN-NORRED Country Practice Area(Lead) Vietnam Health, Nutrition & Population L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-52590 31-Dec-2019 109,228,611.61 Bank Approval Date Closing Date (Actual) 31-May-2013 31-Dec-2020 IBRD/IDA (USD) Grants (USD) Original Commitment 150,000,000.00 0.00 Revised Commitment 119,395,733.12 0.00 Actual 109,228,611.61 0.00 Prepared by Reviewed by ICR Review Coordinator Group Judith Hahn Gaubatz Salim J. Habayeb Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Financing Agreement (page 4) and the Project Appraisal Document (PAD, page 7), the project objectives were as follows:  To increase the efficiency and equity in the use of hospital services in selected provinces of the Northeast (NE) and Red River Delta (RRD) regions. Page 1 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-NORRED (P122629) b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components (Note: The amounts below reflect IDA costs only) 1. Strengthening Capacity of Lower-level Hospitals to Deliver Quality Services (Appraisal: US$ 116.0 million; Actual: US$ 96.7 million): This component aimed to increase the capacity of the health system to provide more and better quality health services in the thirteen project provinces. Activities would draw from subproject proposals prepared by participating provinces, which identified areas in need of support at the provincial hospital and at least three district hospitals. Subproject proposals would be based on a needs analysis for developing hospital capacity, with a defined list of priority health services that the hospitals would be able to perform with project support and with "sponsorship" (training and technical supervision) from central hospital. Proposals would also include specific measurements of quality and management improvement. Activities also included support to the central Ministry of Health to provide quality technical support to the provinces, including the process of transferring knowledge and skills for health services from central to local levels. 2. Reducing the Financial Barriers to Access by the Economically Vulnerable (Appraisal: US$ 29.0 million; Actual: US$ 9.9 million): This component aimed to support the government's plan to achieve universal health coverage, particularly for the poor and near poor. Activities included: providing direct subsidies to support purchase of health insurance by the near poor (an additional 20% subsidy on top of the current 70% subsidy by the government) though conditional on whole family enrollment (rather than individuals); reimbursing hospitals for health expenditures above the insurance cap for catastrophic health expenditures; and conducting information and education campaigns to increase understanding of and enrollment in the voluntary health insurance program. 3. Project Management, Monitoring and Evaluation (Appraisal: US$ 5.0 million; Actual: US$ 2.7 million): This component supported operations of the project management units at central and provincial levels, including monitoring and evaluation. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost  The project cost at appraisal was US$ 154.0 million. This was later revised to US$ 119.4 million, of which US$ 113.4 million was actual.  The differences mainly derived from the savings from procurement of medical equipment, and funding being provided by the GOV for financial support to catastrophic health expenses. Other Page 2 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-NORRED (P122629) savings came from the use of hospitals’ own financial sources for continuous quality improvement activities as well as training costs that were lower that initially estimated.  The project subcomponent on funding catastrophic health expenditures was dropped following the Mid-Term Review. Financing  The project was financed by an IDA Credit of US$ 150.0 million, of which US$ 109.2 million disbursed.  US$24.0 million was cancelled from the Credit. The Public Debt Management Law that was passed in 2017 (a revision of the 2009 Law by the same name) affected all overseas development aid, including World Bank projects - any undisbursed funds (in the case of this project, those remaining after the subcomponent on catastrophic health expenditures was halted) were required to be returned to the Ministry of Finance, Borrower contribution  The appraised Borrower contribution was estimated at US$ 4.0 million. According to the PAD (page 14), the counterpart funding requirement was low because all participating provinces receive national government support for annual budgets. Counterpart funds were expected to be used for salary allowances for seconded government staff, pre-investment expenditures such as incremental operating costs, per diems for training, and initial consulting services. Dates  June 2018: The project was restructured to revise the results framework, replacing three of the four project outcome indicators with two revised indicators to ensure measurability.  May 2019: The project closing date was extended from December 2019 to December 2020, due to additional time needed for acquisition, installment, and operationalization of new techniques.  November 2020: US$24.0 million was cancelled from the Credit, due to procurement savings and unused funds from health insurance activities (some provinces utilized their own budgets to provide health insurance coverage for the near poor: US$7 million disbursed, compared to US$23.0 million planned) 3. Relevance of Objectives Rationale Vietnam has experienced significant economic growth and poverty reduction in the last couple decades, transforming itself into a lower middle income country. However, despite declines in the national poverty rate, there remains a significant population that hovers just above the official poverty line. There are also growing disparities in health outcomes and demographic changes due to an aging population and the increased prevalence of non-communicable diseases. To increase access to health services for vulnerable Page 3 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-NORRED (P122629) populations, the government adopted a national social health insurance program, which includes providing partial premium subsidies to encourage voluntary enrollment. While the officially classified "poor" are receiving complete subsidies, the "near poor," who comprise 10-30% of the population, are also in need of a near complete subsidy, as opposed to the current 70% subsidy from the government. According to the PAD (page 82), at the time of project appraisal, only about 25% of the near poor population had health insurance. Scaling up demand for health insurance also requires improving the availability and quality of health services. Hospital overcrowding is a significant challenge, particularly at the central and provincial level hospitals, due to several factors: inappropriate use of hospitals for basic health services, revenue enhancing incentives, low quality (actual and perceived) of health services at lower levels, draining of scarce human resources from lower levels, and inefficient referral systems. To address these challenges, the Government prepared a Masterplan on Reducing Hospital Overcrowding (issued in January 2013) which aims to increase capacity of the health system to provide better quality services at the provincial level. The Bank's Country Partnership Strategy (FY12-16), under the pillar of improved opportunity, identified improved public service delivery as a key outcome, with health insurance coverage of the poor and near poor as an outcome indicator. Hence, the project objectives are highly relevant to both the government priorities and Bank strategy. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To increase the efficiency in the use of hospital services in selected provinces of the Northeast (NE) and Red River Delta (RRD) regions. Rationale The theory of change for this objective was clear. Activities to increase the availability and quality of health services at lower-level facilities (district and provincial hospitals) were likely to lead to the intended outcome of reduced referrals and thus reduced use of expensive central hospital services. These activities would also likely reduce overcrowding, which is another indication of inefficiency in the hospital system. Project support to increase the supply of health services at lower levels included funding for medical equipment and training for hospital staff to provide services in five specialty areas (obstetrics/gynecology, pediatrics, cardiology, oncology, and trauma) and implementation of quality management approaches. Project activities were to be implemented in thirteen provinces in two regions, benefiting approximately 15.1 million people. These two regions were selected due to the fact that the other regions were already receiving Bank or Asian Development Bank support. As noted in the PAD (page 8), women and children were highly likely to benefit from the project, as the priority health services included those required by women and children Page 4 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-NORRED (P122629) (obstetrics/ gynecology, pediatrics) and the health insurance enrollment was conditional upon family enrollment, rather than individual. Outputs  Financing of sub-project proposals in the thirteen project provinces. 74 hospitals received seed money (approximately US$ 1.86 million) for quality improvement activities, with the provincial committees and hospitals providing additional own budget to implement the activities.  Construction and/or rehabilitation to improve infrastructure quality at 81 provincial and district hospitals, as well as the training centers in the seven central hospitals (target: 57).  Training of 4,805 health personnel (target: 4,050), of which 80% participated in technical training and 20% participated in quality management training. The ICR (page 18) noted that 48% of participants in capacity building activities organized by the central project management unit (training courses, workshops, and seminars on management and quality improvement) were females, with the proportion "consistently higher" at the provincial project management unit level.  Implementation of quality improvement and management approaches at participating health facilities. The ICR (page 18) reported that the project also contributed to the development of MOH national hospital quality criteria in the form of regulations, guidelines and training. Outcomes  The number of technical services transferred from central hospitals to provincial or district hospitals was 2,288. This achieved the target of 2,200. The ICR (page 42) noted that there were a few cases were planned transfer of services did not take place due to changes in personnel, shifts of health service needs in particular provinces, or mismatch of equipment types used during training vs. actual facilities - 89.9% of trained staff were able to apply the techniques in which they were trained.  The number of patients benefiting from new clinical techniques in five specialty areas (obstetrics/gynecology, pediatrics, cardiology, oncology, and trauma), which were services transferred from central to provincial hospitals, reached 12,824 patients at project closing. This achieved the target of 12,500 patients. The number of patients benefitting varied widely among the provinces, from 85 in Nam Dinh to 9,912 in Phu Tho.  The number of patients benefiting from new clinical techniques in three specialty areas (obstetrics/gynecology, pediatrics, and trauma), which were services transferred from provincial to district hospitals, reached 29,107 patients at project closing. This achieved the target of 29,000 patients.  The percentage of hospitals implementing Continuous Quality Improvement (CQI) or equivalent processes increased from 25.0% to 100% at project closing. This surpassed the target of 75%.  The average quality assessment score among the 14 provincial hospitals increased from 2.76 (out of 5) in 2014 to 3.38 (out of 5) in 2019. All 14 hospitals increased their scores, although to varying extents.  The percentage of patients satisfied with overall medical quality increased at both provincial hospitals (76.2% to 81.3%) and district hospitals (82.1% to 88.5%). Satisfaction with specific elements (waiting Page 5 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-NORRED (P122629) time, availability of treatment services, and availability of diagnostic facilities) all increased at both provincial and district hospitals. Improvements with regards to targets could not be evaluated as the targets were set lower than the baseline figures. The original key outcome indicators measuring "reduction in inpatients referred from the participating provincial hospitals to the central hospitals," "reduction in patients referred from participating subnational hospitals for three health conditions (deliveries, child pneumonia, appendicitis)," and "reduction in self-referral rate of insured patients from the participating provincial hospitals to the central hospitals" were dropped. The reasons provided by the Project Restructuring Paper, Section I, Rationale for restructuring, para, 2) were related to issues in data availability, data quality, and attribution. The data reported above demonstrate increases in access and utilization of hospital services (i.e., number of patients receiving services), strengthened lower level capacity and service availability (i.e. number of services now being delivered at provincial/district hospitals), and also improved quality (i.e. patient satisfaction). While the results do not include direct measures of utilization efficiency, the latter is expected to be achieved, or to have been achieved, because the increased availability and quality of health services that were satisfactorily realized at lower-level district and provincial hospitals would contribute to reducing the use of expensive central hospital services, including through self-referrals or unnecessary hospital referrals, as the needs for treatment are met at these lower-level hospitals, and as appropriately stipulated by the theory of change. Rating Substantial OBJECTIVE 2 Objective To increase equity in the use of hospital services in selected provinces of the Northeast (NE) and Red River Delta (RRD) regions. Rationale The theory of change for this objective was clear. Activities to increase affordability of health services for the poor and near poor, along with activities to increase demand for those services, were likely to lead to the intended outcome of increased access/utilization, contributing to increased equity in the use of services. Project support included expanding coverage of health insurance through direct subsidies and conducting of information campaigns to increase knowledge among the poor and near poor about health insurance. Outputs  Financing of health insurance premiums for over 1.3 million near poor.  Behavior change communication strategy for health insurance, including information materials and support from trained communication team.  The activity to reimburse hospitals for catastrophic health expenditures for the poor and near poor was dropped. This activity was deemed less effective, due to the government adding inclusion criteria for Page 6 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-NORRED (P122629) vulnerable groups that increased coverage of regular health insurance among the near poor and the complex administrative procedures for reimbursing different hospitals. Outcomes  Utilization of inpatient hospital services by the poor and near poor increased from 0.05 hospitalizations/person/year in 2015 to 0.15 in 2019, exceeding the target of 0.06 hospitalizations per person per year (ICR, p. 30).  Utilization of outpatient hospital services by the poor and near poor increased from 0.21 visits/person/year in 2015 to 1.22 in 2019, exceeding the target of 0.25 visits per person per year (ICR, p. 30).  Coverage of health insurance for the near poor reached 100% at project closing.  The number of people with access to a basic package of health, nutrition, and reproductive health services (number of people with a health insurance card) increased from 11,942,834 in 2015 to 14,703,163 in 2019. Achievement relative to the target could not be assessed as the original target was 10,882,272 (below actual baseline).  There was an increase in knowledge about health insurance among the near poor: o 58% of near poor knew at least three services covered by health insurance (increase from 16.4% in 2015, exceeded the target of 50%). o 52.6% of near poor knew the basic responsibilities for using the health insurance card (increase from 6.3% in 2015, exceeding the target of 50%). o 79.3% of near poor knew the basic procedures for using the health insurance card (increase from 3.2% in 2015 exceeding the target of 50%). o 51.3% of near poor knew their basic rights with the health insurance card (increase from 3.2% in 2015, exceeded the target of 50%). o Additional data in the ICR (page 43) demonstrated that increases in knowledge were greater among the near poor, compared to the general respondents (whose knowledge also increased, but to a lesser degree). Achievement is rated Substantial due to evidence among the near poor population of increased coverage of health insurance, increased utilization of services, and increased knowledge about health insurance. However, as noted in the ICR (page 19), important context to note is that during the project period, the poverty rate decreased significantly (45% in 2008 to 7% in 2018), which may confound to some extent the contribution of the project. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Page 7 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-NORRED (P122629) The two development objectives to increase the efficiency in the use of hospital services in selected provinces of the Northeast and Red River Delta regions; and to increase equity in the use of hospital services were almost fully achieved. Overall Efficacy is rated Substantial due to evidence of increased access and utilization of hospital services, strengthened lower- level capacity to deliver services, improved quality of services, and increased coverage and knowledge about health insurance among the near poor. However, there were evidence gaps in measuring increased efficiency in the use of hospital services. Overall Efficacy Rating Substantial 5. Efficiency At appraisal (PAD, Annex 5), there was no cost-benefit analysis or other quantitative estimate conducted due to "difficulties in assigning a monetary value to expected improvements in health outcomes." Instead, the PAD aimed to establish an economic case for the project investment by presenting the mechanisms through which cost savings would be achieved, including increased efficiencies in the production and consumption of health services and addressing of market failures in the health insurance market. At completion (ICR, Annex 4), the economic analysis discussed allocative efficiencies realized in the health sector, due to the project approach to shift basic service provision to the lower levels and to expand health insurance coverage for the near poor. Allocative efficiency was likely realized by investing in the proper technical services. Efficiency was also likely gained through increased capacity to manage and deliver services, implementation of continuous quality improvement processes, and reduction in unnecessary referrals (although data on all of the above were not quantified). The ICR (page 16) noted that there were possibly higher unit costs (data were not available) for health service delivery at the lagging provincial and district hospitals due to lower demand in these areas (i.e. number of patients), but that the overall impact on that community and the overall health system were still justifications for investment. Some inefficiency arose due to the lack of full optimization of medical services in some facilities, owing to few patients, lack of medical staff due to high turnover, lack of supporting medical equipment, and services not yet being included in health insurance benefit package. Nevertheless, most of the 2,288 new techniques that were transferred to lower- level facilities were highly used and only 341 techniques had not achieved optimal function by the end of 2019 (ICR, p. 53). (In the larger context, the ICR (page 54) noted that there was a significant annual increase in hospital costs, due to an increasingly sophisticated style of care, without corresponding improvements in health outcomes yet). There were also some implementation delays due to procurement delays for equipment particularly for radiation therapy and cardiological intervention, which contributed to a one- year project extension. However, all project activities were eventually completed, with savings realized from hospitals’ decision to use their own budgets for continuous quality improvement activities and also from lower unit costs of training than appraised (no data Page 8 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-NORRED (P122629) provided).The project resulted in procurement savings of US$6 million. There were also changes in local government policies to use their own local budgets to provide financial protection to their constituents resulting in undisbursed funds of US$14 million in subcomponent 2b for Supporting Catastrophic Health Care. The above savings and local financing changes led to the cancellation of US$24 million at project closing (ICR, page 17). Although there were some weaknesses in the PAD and ICR’s assessments of project efficiency, as many of the economic arguments provided in the PAD and ICR were more relevant to efficiency in the hospital sector rather than project efficiency, allocative and technical efficiencies under the project were substantially realized (ICR, pp. 16-17). Also, there were no indications from reported project findings that resources and inputs were not converted to results cost-effectively overall through the project’s accomplishment in transferring/developing capacities and new medical techniques to lower-level hospitals as planned and implemented, and with substantial savings. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Overall outcome is rated Satisfactory due to High Relevance, based on strong alignment with country conditions and Bank/government priorities, Substantial Efficacy as project objectives were almost fully achieved, and Substantial Efficiency, as the project was efficiently implemented overall, with substantial allocative and technical efficiencies realized, and with substantial procurement savings. a. Outcome Rating Satisfactory Page 9 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-NORRED (P122629) 7. Risk to Development Outcome Government policies and practices - both at the central and local levels - remain aligned with the project objectives. As reported in the ICR (page 15), by project closing, "most of the project provinces had committed to continue to support the near poor to purchase health insurance" thus supporting the sustainability of the objective to increase financial access for the poor and near poor and to increase equity. While there will continue to be recurrent costs for maintenance of the medical equipment at the provincial and district hospitals, these may be offset by the increased capacity of each facility to generate more direct revenue from delivering a wider range of medical services. The improved quality of hospital services is also likely to be continued due to the increased capacity of hospitals (through acquired equipment and staff training) to deliver services and implement quality management practices (including some resources allocated from their own budgets). 8. Assessment of Bank Performance a. Quality-at-Entry The project objectives and design were clearly aligned with government priorities and existing policies. The design drew upon extensive Bank experience in the health sector in Vietnam, including through regionally focused operations, particularly for expanding health insurance coverage and corresponding implementation mechanisms. The flexibility of the sub-project proposal approach allowed for the varying conditions and priorities among the thirteen project provinces, while also relying on local health systems needs assessments to ensure relevance and feasibility. Project preparation (and implementation) included building leadership commitment towards the project goals, at both the provincial government level and the hospital management level. Overall risk was assessed as Substantial due to low implementing agency capacity at the decentralized levels, therefore significant implementation support from the task team was expected. However, there were some moderate shortcomings in the project design; for example, the project planned to reimburse hospitals for catastrophic health care expenditures, but the reimbursement mechanism required complicated contractual procedures with hospitals. This, in addition to the fact that hospitals had to pay the treatment costs in advance, led to reluctance from hospitals to continue participating in the scheme. This activity was subsequently dropped. There were some shortcomings in M&E/results framework (feasibility in key outcome indicator data, accuracy of baseline and target figures) and data collection arrangements ( =data reliability). Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Page 10 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-NORRED (P122629) Supervision was proactive and intensive, relying on regular supervision missions (14 in total) and close monitoring of activities to assess progress and identify issues. The Bank team's proactive engagement with the local governments and hospital leaders helped to sustain commitment to project objectives and likelihood of activities continuing beyond the project period. The Bank team also provided strong fiduciary support, including for procurement of specialized medical equipment, to help overcome initial implementation delays. Project results framework was revised to account for changing circumstances and government directives, although the new indicators did not adequately measure all aspects of the stated project objectives (see Section 9 below). Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The original M&E framework was clearly linked to project design and objectives and drew upon existing monitoring systems (i.e. Social Services system). Evaluative activities were planned, including the beneficiary satisfaction surveys and hospital quality assessments. However, there were some shortcomings. There were inaccuracies in some baseline and target figures (number of people with access to basic services, patient satisfaction) and lack of feasibility/ reliability for collecting data for key outcome indicators (reduction in referral rates). The ICR did not report on why some target figures were set lower than the baseline figures. b. M&E Implementation The Vietnam Social Services department was identified at appraisal as the main source of data collection on referrals; however, there were significant challenges in obtaining accurate data, in part due to inconsistent and inaccurate coding practices. Although M&E staff were required to be hired/assigned at the central and provincial level, lack of experience and high turnover led to challenges in M&E implementation in the initial project period. There were significant revisions to the results framework: three of the original key outcome indicators (reduction in inpatients referred from the provincial hospitals to central level hospitals; reduction in patients referred from sub-national hospitals for three health conditions (deliveries, child pneumonia and appendicitis); and reduction in the self-referral rate of insured patients from the province hospitals to the central hospitals were replaced by two new indicators (number of patients benefiting new clinical techniques in five intervention areas, number of patients benefiting new clinical techniques in three intervention areas at district hospitals). The ICR (page 10) reported that these Page 11 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-NORRED (P122629) revisions were undertaken in part due to the introduction of hospital payment caps by the government that led to increased referrals from lower level hospitals and thus raised concerns about attributability of project activities, as well as issues of data availability and quality. Also, while the PAD (page 8) noted that improved efficiency would be achieved when the strengthened lower level capacity enabled the provincial and district hospitals to provide services and not refer patients to central hospitals, as well as increase willingness of patients to seek treatment at the local level, the revised key indicators did not directly measure increased efficiency. However, M&E implementation performance improved significantly after the Mid-term Review in 2017 (ICR, page 22). Also, according to the ICR, Provincial Project Management Units recruited full-time staff, and the Central Project Management Unit increased the intensity of supervision, support and efforts to increase the capacity of provincial units (ICR, page 22). c. M&E Utilization M&E data, including findings on beneficiary knowledge and satisfaction and on hospital quality, were used to inform project decisions during the project period, including reallocations of project financing within components and across hospitals. The M&E framework, as originally designed and subsequently revised, was generally adequate to assess project achievements. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as an Environmental Category "B" project due to medical waste issues and potential civil works, triggering the safeguard policy on Environmental Assessment (OP/BP 4.01). An Environmental and Social Management Framework was prepared to ensure compliance with Bank guidelines and national regulations. The safeguard policy on Indigenous Peoples (OP/BP 4.10) was also triggered due to the presence of ethnic minority groups in the project areas. A Social Assessment was conducted, which included identifying potential barriers for ethnic minority communities, and an Ethnic Minority Development Plan was prepared to ensure culturally appropriate interventions. Proper medical waste disposal equipment was provided by the project, along with training on medical waste management. According to the ICR (page 23), the results of M&E have shown that "the waste management and environmental protection at the health facilities under the Project improved over the years." The ICR did not report any problems with safeguard compliance. b. Fiduciary Compliance Page 12 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-NORRED (P122629) Financial management: The ICR (page 23) reported that financial management performance was Satisfactory, with no major problems. Delays in large procurements led to some slowdowns in disbursement, and the yearly budget allocation process also led to a one-off issue of several project provinces not receiving the donor funding amounts in their budget allocations. External audits were submitted regularly, although the ICR did not report on whether there were any qualified audits. Procurement: The ICR (page 23) reported that procurement performance was also Satisfactory with no major problems except some delays due to lack of experience and the complexity of medical equipment procurement. Given the significant role of new medical procedures in the project design, coordinating the procurement schedule for medical equipment and the training/ introduction of the relevant medical techniques was critical. Therefore, procurement risk ratings were initially High but adjusted to Moderate/Low during the project period as procurement capacity increased. As reported in the ICR (page 21), the Bank team provided intensive procurement training for CPMU/PPMUs and hands-on guidance and suggestions for the procurement of large value and technically complex medical equipment contracts c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory There were moderate Bank Performance Satisfactory Moderately Satisfactory shortcomings in the Quality at Entry, including in M&E design. Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons Lessons drawn from the ICR (pages 27-28) and additional lessons from IEG:  Certain elements of the project design can help to build ownership and thus contribute to achieving outcomes and ensuring sustainability. In the case of this project, the bottom-up planning mechanisms helped to ensure strong alignment of selected activities with local needs and priorities, and the requirement to use own budget to partially finance the quality Page 13 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-NORRED (P122629) improvement plans ensured continued funding and proactivity in promptly resolving implementation problems.  While a project's theory of change may be sound, an inadequate results framework can inhibit proper assessment of outcomes. In the case of this project, the original project indicators (reduced referral rates) were appropriate but were not feasible for collection by the participating hospitals, and the revised project indicators (number of patients benefitting from new hospital techniques) were not directly linked to the project objective to increase efficiency in the use of hospital services. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was concise and provided an informative narrative on project implementation experience. However, while the quality of the evidence is strong, there were some shortcomings in its presentation, as the actual data on achievements and analysis of the results chain is provided in the Annex (Annex 1A) rather than in the main body of the ICR (Efficacy, pages 14-15, primarily focused on Table 3 which listed the indicators that were achieved/ surpassed). Also, the analysis of Efficacy should have treated the project objectives separately - to increase efficiency and to increase equity. Although indicators may overlap in measuring aspects of both objectives (and can be reported as such), each objective is distinct and separate. Despite these shortcomings, the evidence was present to enable a proper assessment of the project and therefore the quality of the ICR is rated Substantial. a. Quality of ICR Rating Substantial Page 14 of 14