Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00006250 IMPLEMENTATION COMPLETION AND RESULTS REPORT IDA-54890 ON A CREDIT IN THE AMOUNT OF SDR 12.7 MILLION (US$19.5 MILLION EQUIVALENT) TO MONGOLIA FOR THE E-HEALTH PROJECT August 30, 2023 Health, Nutrition and Population Global Practice East Asia and Pacific Region CURRENCY EQUIVALENTS (Exchange Rate Effective April 24, 2023) Currency Unit = Mongolian Tugrik (MNT) MNT 3,483.83 = US$1 US$ 1.349860 = SDR 1 FISCAL YEAR July 1 – June 30 Regional Vice President: Manuela V. Ferro Country Director: Mara K. Warwick Regional Director: Cristian Aedo (acting) Practice Manager: Aparnaa Somanathan Task Team Leader(s): Kate Mandeville, Anna L Wielogorska, Pagma Genden ICR Main Contributor: Suzana Nagele de Campos Abbott, Kate Mandeville ABBREVIATIONS AND ACRONYMS CPF Country Partnership Framework CT Computerized Tomography DMEIA Department of Monitoring, Evaluation and Internal Audit EA Enterprise Architecture EHR Electronic Health Record EMR Electronic Medical Record FM Financial Management GAHI General Agency for Health Insurance GOM Government of Mongolia GPA General Procurement Agency HDS Health Data Standards HIEP Health Information Exchange Platform H-INFO Health Statistics Information System HIS Hospital Information System IBL Integrated Budget Law ICT Information and Communication Technology ISR Implementation Status and Results Report IPs Indigenous Peoples IRI Intermediate Results Indicator M&E Monitoring and Evaluation MDGs Millennium Development Goals MOF Ministry of Finance MOH Ministry of Health MRI Magnetic Resonance Imaging MTR Midterm Review NCD Noncommunicable Disease NCHD National Center for Health Development NDC National Data Center NEA National Enterprise Architecture NHIC National Health Information Center PACS Picture Archiving and Communication System PAD Project Appraisal Document PDO Project Development Objective PIU Project Implementation Unit RF Results Framework SHI Social Health Insurance SSIGO State Social Insurance General Office TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 6 A. CONTEXT AT APPRAISAL .........................................................................................................6 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION .............................................................. 14 II. OUTCOME .................................................................................................................... 16 A. RELEVANCE OF PDOs ......................................................................................................... 17 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 17 C. EFFICIENCY ........................................................................................................................... 21 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 22 E. OTHER OUTCOMES AND IMPACTS (IF ANY) ............................................................................ 22 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 23 A. KEY FACTORS DURING PREPARATION ................................................................................... 23 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 24 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 28 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 28 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 29 C. BANK PERFORMANCE ........................................................................................................... 31 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 33 V. LESSONS AND RECOMMENDATIONS ................................................................................ 33 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 36 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 44 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 46 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 47 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 50 ANNEX 6. ORIGINAL THEORY OF CHANGE ............................................................................ 51 The World Bank E-Health Project (P131290) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P131290 E-Health Project Country Financing Instrument Mongolia Investment Project Financing Original EA Category Revised EA Category Not Required (C) Not Required (C) Organizations Borrower Implementing Agency Mongolia Ministry of Health Project Development Objective (PDO) Original PDO To improve integration and utilization of health information and e-health solutions for better health service delivery in selected pilot sites. Page 1 of 52 The World Bank E-Health Project (P131290) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 19,500,000 19,500,000 7,971,809 IDA-54890 Total 19,500,000 19,500,000 7,971,809 Non-World Bank Financing 0 0 0 Borrower/Recipient 4,250,000 0 0 Total 4,250,000 0 0 Total Project Cost 23,750,000 19,500,000 7,971,809 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 06-Jun-2014 17-Aug-2015 18-May-2018 30-Sep-2019 30-Sep-2022 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 11-Jun-2015 0 Change in Implementing Agency Change in Results Framework Change in Loan Closing Date(s) Change in Legal Covenants 11-Sep-2018 1.16 Change in Results Framework Change in Components and Cost Reallocation between Disbursement Categories Change in Legal Covenants Change in Institutional Arrangements Change in Procurement 18-Jun-2020 3.58 Change in Loan Closing Date(s) Page 2 of 52 The World Bank E-Health Project (P131290) KEY RATINGS Outcome Bank Performance M&E Quality Unsatisfactory Moderately Unsatisfactory Modest RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 18-Oct-2014 Satisfactory Satisfactory 0 02 21-May-2015 Moderately Satisfactory Moderately Satisfactory 0 Moderately 03 15-Dec-2015 Moderately Satisfactory 0 Unsatisfactory 04 29-Jun-2016 Moderately Satisfactory Moderately Satisfactory 0 05 16-Dec-2016 Moderately Satisfactory Moderately Satisfactory .56 Moderately 06 07-Mar-2017 Unsatisfactory .56 Unsatisfactory Moderately 07 06-Jul-2017 Unsatisfactory 1.09 Unsatisfactory Moderately Moderately 08 26-Feb-2018 1.16 Unsatisfactory Unsatisfactory Moderately Moderately 09 04-Sep-2018 1.16 Unsatisfactory Unsatisfactory Moderately Moderately 10 21-Nov-2018 1.16 Unsatisfactory Unsatisfactory Moderately 11 20-Mar-2019 Moderately Satisfactory 1.73 Unsatisfactory Moderately 12 29-Oct-2019 Moderately Satisfactory 2.07 Unsatisfactory Moderately 13 04-May-2020 Moderately Satisfactory 2.77 Unsatisfactory Page 3 of 52 The World Bank E-Health Project (P131290) 14 12-Mar-2021 Unsatisfactory Unsatisfactory 5.02 Moderately Moderately 15 12-Dec-2021 7.71 Unsatisfactory Unsatisfactory Moderately 16 11-Jul-2022 Highly Unsatisfactory 8.88 Unsatisfactory SECTORS AND THEMES Sectors Major Sector/Sector (%) Health 100 Health 100 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Private Sector Development 20 Enterprise Development 20 MSME Development 20 Human Development and Gender 0 Disease Control 0 Pandemic Response 1 Health Systems and Policies 80 Health System Strengthening 80 Page 4 of 52 The World Bank E-Health Project (P131290) ADM STAFF Role At Approval At ICR Regional Vice President: Axel van Trotsenburg Manuela V. Ferro Country Director: Klaus Rohland Mara K. Warwick Director: Xiaoqing Yu Aparnaa Somanathan Practice Manager: Toomas Palu Aparnaa Somanathan Kate Mandeville, Anna L Task Team Leader(s): Aparnaa Somanathan Wielogorska, Pagma Genden Suzana Nagele de Campos ICR Contributing Author: Abbott Page 5 of 52 The World Bank E-Health Project (P131290) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. As summarized by the 2018 Systemic Country Diagnostic 1 , Mongolia is a unique country of extreme features. It is the most sparsely populated country in the world, with traditional nomadic herding gradually giving way to rapid urbanization (about half of the population live in the capital, Ulaanbaatar). It is a vibrant democracy landlocked between Russia and China, mining-dependent (approximately 90% of exports), dominated in trade with one neighbor (China accounts for approximately 90% of the country’s exports). Nearly half the population live in gers, with limited access to sanitation and heating in the face of an extreme climate, with very cold long winters and short hot summers. 2. Starting in 1990, Mongolia evolved into a multiparty democracy with a vibrant market economy. It adopted a new Constitution in 1992 and became a parliamentary republic. At the time of appraisal of the E-Health Project (the Project), the country was experiencing a second transformation fueled by a mineral resource boom. Poverty rates declined from 39 percent in 2010 to 27 percent in 2012, and several of its Millennium Development Goals (MDGs) had already been met. However, while economic growth and development have been impressive (highest GDP growth in the world in 2011 at over 17%), it has also been extremely volatile over the project timeline (flat growth in 2016 with a historic US$5.5 billion multilateral support package initiated in 2017). Recurring boom-and-bust cycles put at risk past gains in standards of living and poverty reduction, compounded by frequent political transition: between 1992 and 2017, fifteen Prime Ministers served the country with an average tenure of under 1.5 years. 3. The vast natural resources offered a unique opportunity—and challenge—to ensure that Mongolia’s mineral wealth was transformed into sustainable and equitable growth and contributed to poverty reduction. However, there was evidence of large disparities of the MDG indicators across socioeconomic groups and geographic regions. The extreme population sparseness in rural areas and overcrowding in urban and peri-urban areas implied significant challenges for ensuring equity in access to social services. 4. Mongolia had adopted several reforms to manage public resources sustainably, transparently, and in a decentralized manner to avoid boom-and-bust cycles and high volatility in public revenues resulting from reliance on mineral exports and exposure to commodity price swings. This volatility was particularly detrimental to social services that were highly dependent on public spending. The Government of Mongolia (GOM) had adopted a Fiscal Stability Framework in 2012 and introduced the Integrated Budget Law (IBL) in 2013, setting rules for budget planning and decentralization. As the IBL was implemented, public spending would be increasingly managed at the subnational level. Without good- quality data and integration among the national, historic allocation patterns and political interest at the 1 World Bank. 2018 Mongolia: Systematic Country Diagnostic. Washington, D.C.: World Bank Group Page 6 of 52 The World Bank E-Health Project (P131290) aimag (province equivalent) and soum (district equivalent) levels could supersede actual need to services, thus undermining equity in the distribution of revenues. Consistent and accurate data were seen as critical for achieving efficiency and equity as Mongolia moved away from input-based to performance-based financing. 5. The GOM had made significant investments in information and communication technology (ICT) to help improve public service delivery. Access networks were being extended nationwide, 200,000 internet connected points had been established, and internet users increased from under 200,000 in 2010 to over 657,000 in 2012, reaching almost 22 percent of the country’s population. The GOM’s e - Government program was enhancing the legal environment to develop e-Government and was supported and clearly stated in the national ICT Policy document that framed the vision of becoming a knowledge- based economy using ICT to accelerate development. Institutionally, the Information, Technology, Post and Telecommunications Authority (ITPTA) was responsible for the design and implementation of the ICT policy, supported technically by the National Information Technology Park (IT Park). The National Data Center (NDC) provided state-of-the-art facilities for secure storage of, and access to, government data. 6. Mongolia’s achievements in its health sector had been good relative to its income level, reflecting the GOM’s continued commitment to health, effective communicable disease control, and an extensive delivery infrastructure to provide services to its highly dispersed and largely rural population. Among the three levels of government, primary care services were provided at soum hospitals in rural areas and family group practices in the capital and aimag centers; the secondary level comprised aimag general hospitals and district hospitals; and the tertiary level comprised central hospitals and specialized institutions in Ulaanbaatar and five regional diagnostic and treatment centers. Social Health Insurance (SHI) was introduced in 1993 with substantial state budget funding to ensure coverage and financial protection. Total health spending accounted for about 5 percent of gross domestic product in 2011, of which the largest share (42 percent) was from the state budget; SHI financing accounted for 15.8 percent, and out-of-pocket payments constituted 40 percent of total health expenditures. 7. However, Mongolia was facing multiple health system challenges. First, adult mortality rates were rising, driven by the growth of noncommunicable diseases (NCDs) which were a major driver of health expenditures and poor health outcomes. Limited access to quality care, especially in rural areas, was resulting in under-diagnosis and delays in treatment of NCDs, requiring more expensive, acute care for NCD-related conditions. Second, health insurance coverage was declining, in terms of both enrollment rates and financial protection, as reflected in large socioeconomic differentials in health care use, particularly of secondary and tertiary hospitals. The existing delivery system was fragmented, with most patients bypassing primary care in favor of better quality secondary and tertiary services, leading to overcrowding at those levels. Finally, there was little coordination of care and follow-up of patients across the system’s different levels. 8. To provide better access to care in remote areas and a more integrated system, the Ministry of Health (MOH) formulated in 2009 an E-Health Strategy (2010–2014), however its implementation had faced challenges. The strategy recognized the critical role that e-Health solutions could play in addressing health care challenges and a more integrated system of financing and delivery, in line with the broader e- Government strategy. Its implementation, however, had resulted in the development of a disparate Page 7 of 52 The World Bank E-Health Project (P131290) number of information systems that contributed little to the exchange of clinical and health system data between providers and other agencies. Hospital Information Systems (HISs) were developed by at least two different vendors and existed only in tertiary hospitals and about half of the secondary hospitals. The GOM had secured financing from the Chinese government for developing entirely new HISs for all public hospitals, and while a centralized health statistics information system (H-Info) collected data on morbidity and mortality, utility statistics, and some quality indicators to process SHI claims, the H-Info did not function in practice. No health information systems were in place for primary health care facilities, and external donors had financed numerous independent telemedicine projects. 9. The health information system’s fragmentation exacerbated the high levels of fragmentation in financing and delivery, seriously undermining the health system’s performance. A system capable of facilitating electronic data exchange between providers’ HIS and the State Social Insurance General Office (SSIGO) would improve claims processing, support SSIGO in monitoring provider behavior, and contribute to improving efficiency and quality of SHI-financed services. Also, the coordination of care across different levels of the system was critical for addressing the rapidly rising burden of NCDs. Providing for information exchange between HISs at different levels of the system would help improve referrals and follow-up care. 10. In 2012, the GOM requested financing to address several shortcomings in the implementation of the E-Health Strategy: this was the first World Bank project in the health sector. These shortcomings included: (a) the lack of a robust Enterprise Architecture (EA) or roadmap for the development of e-Health activities; (b) inconsistent data formats, coding schemes, and timing of information delivery due to lack of health data standards (HDS); (c) lack of data interchange between different HISs to match information about patients and no link to the wider administrative and financing systems; (d) incomplete data in the existing health information system that limited the ability to create useful, consistent ‘maps’ for tracking diseases, especially NCDs; and (e) limited IT literacy and capacity and fragmented IT resources to manage a modern health information system. Theory of Change (Results Chain) 11. The revised Theory of Change is presented below, with new additions marked in bold, and the original Theory of Change is provided in Annex 6. Table 1. Revised Theory of Change Activities Outputs Outcomes Long- Term Outcome Component 1: • A suite of HDS, which • Foundations in place for Improved Build the e- would form the deployment of e-Health integration Health ‘common language’ solutions and prerequisites Improved and utilization Integration needed for diverse needed for greater health of health System computer systems to integration of health service information interoperate information between delivery and e-Health • Requirements for e- health facilities, the health solutions Health portal and viewer insurance agency, public Page 8 of 52 The World Bank E-Health Project (P131290) Activities Outputs Outcomes Long- Term Outcome developed: (a) the health agencies, private computer architecture, sector, and so on (b) a web-portal • Standardized data application and links, exchange permitted and (c) security between health care infrastructure to ensure providers and access for privacy and users to the HIEP initially confidentiality and links between the HIS • HIEP and the and other individual interconnections systems that could come between existing HIS, online in the future new HIS, other health • HIS providers able to information systems, an exchange data and, as the enterprise service bus e-Health system became for secure internet- fully functional, between based data exchange, health insurance and and gateway for use health services utilization authentication and reporting and handling of queries epidemiology designed and • The sharing of imaging constructed acquired in diagnostic • HIEP designed to departments with health include EHR, EMR for care professionals and primary health patients. The HIEP providers, e-referral infrastructure would be and other services used for image exchange • A central archiving and and distribution as the communication PACS exchange would environment for digital functionally and physically medical images be part of HIEP developed and selected facilities to use digitally acquired imagery (mini PACS) upgraded and local networks upgraded to share digital imagery • Servers required for the HIEP are housed and supported at the NDC • Clinical and imaging data exchange mechanisms tested under four pilots • Hardware provided for pilot sites and to fill Page 9 of 52 The World Bank E-Health Project (P131290) Activities Outputs Outcomes Long- Term Outcome gaps at the primary health care level Component 2: • Training provided to • Institutionalization of e- Institutional improve the computer Health in Mongolia, thus Strengthening literacy of health service assuring the long-term and Capacity providers sustainability of e-heath Building • IT technical training investments provided to improve IT • Strengthened human skills capacity for the successful Improved • Training center implementation and integration established institutionalization of e- and utilization • Facility connectivity Health in Mongolia of health Improved ensures ICT advances • Increased interoperability information health can be taken up of the system with and e-Health service • Legal framework for e- primary health care solutions delivery Health system facilities progressively Improved Improved developed connected integration health • Health facilities and utilization service progressively connected of health delivery to the high-speed information network and e-Health Component 3: • PIU established • Strengthened government solutions Project • Domain expert institutional capacity for Management consultation provided project management • Annual M&E implemented • Citizen engagement and beneficiary feedback surveyed Note: Text in bold indicates new elements from original Theory of Change. Project Development Objectives (PDOs) 12. The PDO was to improve integration and utilization of health information and e-Health solutions for better health service delivery in selected pilot sites. Key Expected Outcomes and Outcome Indicators 13. The key expected outcomes were the following: (a) for patients visiting facilities at selected pilot sites, that their records are accessed and referrals processed more expeditiously, follow-up care management is effective with reminders to the providers; (b) for service providers at pilot sites, that patients’ medical data are accessed from one place more expeditiously and administrative workload due to paperwork is reduced; and (c) for health care providers and national level administrators, that their IT skills and capacity are upgraded. Page 10 of 52 The World Bank E-Health Project (P131290) 14. The PDO (Outcome) indicators were as follows: • Percentage of patient episodes for which information, including medical image information is available for secure viewing at pilot facilities (Baseline: 0 percent; Target: 60 percent) • Percentage of statistical reports out of total report produced by pilot facilities that are transmitted electronically through the Health Information Exchange Platform (HIEP) to the National Center for Health Development (NCHD) (Baseline: 0 percent; Target: 75 percent) • Percentage of electronic referrals out of total referrals at pilot facilities (Baseline: 0 percent; Target 50 percent). 15. The Project’s Results Framework (RF) also included seven intermediate results indicators (IRIs) (see annex 1), of which three directly measured part of the PDO: • Percentage of pilot facilities which design, develop, and implement successful piloting of eHealth Portal, Viewer and HIEP (Baseline: 0 percent; Target 80 percent). • Percentage of digital images generated in pilot facilities which are transmitted to the central PACS (Baseline: 0 percent; Target 80 percent). • Improvements in IT literacy and health information management capacity among health professionals (Baseline: 0 percent; Target 200). There were five pilot facilities, two in the capital Ulaanbaatar, one in a district of Ulaanbaatar, and two in aimags: • Third State Central Hospital in Ulaanbaatar • National Center for Trauma and Orthopedics in Ulaanbaatar • Songinokhairkhan district General Hospital (with attached family health centers) • General Hospital of Khuvsgul aimag (with attached Family and Soum health centers) • General Hospital of Uvs aimag (with attached Family and Soum health centers) Components 16. The Project comprised five components as described below: Component 1: E-Health Foundational Activities (Total US$2.85 million; IDA USS$2.63 million) 17. This component aimed to lay the foundations for successful deployment of e-Health solutions and developing the prerequisites needed for greater integration of health information between health Page 11 of 52 The World Bank E-Health Project (P131290) facilities, the health insurance agency, public health agencies, private sector, and so on. Component 1 comprised two subcomponents. Subcomponent 1.1: Enterprise Architecture Development (Total US$1.93 million; IDA US$1.82 million) 18. Subcomponent 1.1 would support development of an EA for health, based on the National Enterprise Architecture (NEA) framework but adapted to the specifics of the health sector. A key activity under this subcomponent would be the analysis of business processes at key health facilities and subsequent development of all the required elements of the enterprise information system. Another key activity was the assessment of the legislative changes that were needed in order to make e-Health functional. Subcomponent 1.2: Health data standards (Total USS$0.92 million; IDA US$0.81 million) 19. Subcomponent 1.2 would establish a suite of HDS, which would form the ‘common language’ needed for diverse computer systems to interoperate. It would finance a combination of consultancy services and technical services for the development of the standards as well as Mongolia’s subscription to use one of the major health data exchanges standards, known as HL7. Component 2: Clinical Data Collection, Access and Sharing (Total US$16.06 million; IDA US$12.37 million) 20. Component 2 aimed to establish a solid basis for standardized data and medical image exchange between health care providers and users, develop the mechanism for such data exchange to take place, and pilot test this mechanism in selected sites. The development of the data and medical image exchange systems would be based on the detailed assessment of business processes carried out under Component 1. Component 2 comprised four subcomponents. Subcomponent 2.1: The e-Health portal and viewer (Total US$2.12 million; IDA US$1.12 million) 21. Subcomponent 2.1 aimed to establish a basis for standardized data exchange between health care providers and access for users to the HIEP initially and links between the HIS and other individual systems that could come online in the future. Investments comprise the development of (a) the computer architecture, (b) a web-portal application and links that would enable the e-Health system to become fully functional over time, and (c) security infrastructure for the portal to ensure appropriate levels of privacy and confidentiality. The existing National Identification Number would become the key patient identifier. Subcomponent 2.2: HIEP including enterprise service bus and secure gateway (Total US$7.23 million; IDA US$6.38 million) 22. Subcomponent 2.2 would enable different HISs from a range of providers to exchange data and, as the e-Health system became fully functional, between health insurance (for example, SSIGO) and health services utilization reporting and epidemiology (H-Info). The HIEP would be underpinned by the e-Health foundations and prerequisites developed under Component 1. Investments would include the design and construction of the HIEP and the interconnections between existing HISs, new HISs, other health Page 12 of 52 The World Bank E-Health Project (P131290) information systems, an enterprise service bus for secure internet-based data exchange, and gateway for use authentication and handling of queries. Subcomponent 2.3: Analysis and implementation of Picture Archiving and Communication Systems (PACS) (Total US$4.85 million; IDA US$4.00 million) 23. Subcomponent 2.3 would comprise the development of a central archiving and communication environment for digital medical images, the upgrading of selected facilities to be able to use digitally acquired imagery (mini PACS) and upgrading of local networks to be able to share digital imagery. The nationwide development of PACS would allow the sharing of imaging acquired in diagnostic departments with health care professionals and patients throughout the country. The HIEP infrastructure would be used for image exchange and distribution—the PACS exchange would functionally and physically be part of HIEP. Subcomponent 2.4: IT infrastructure, maintenance and support and pilot tests (Total US$1.86 million; IDA US$0.87 million) 24. Subcomponent 2.4 would finance the servers required for the HIEP (to be housed and supported at the NDC) and the pilot testing of the clinical and imaging data exchange mechanisms developed under Subcomponents 2.1, 2.2, and 2.3. Once the portal/viewer and HIEP had been developed and tested, four pilots would be carried out.2 The Project would finance consultancy and technical services for the design and implementation of the pilots and software. 25. Component 3: National Health Information Center (NHIC) (Total US$2.44 million; IDA US$2.23 million) would establish a center that oversees all aspects of e-Health in Mongolia, thus ensuring the long-term sustainability of e-heath investments. The Project would finance establishment costs of the NHIC, while the GOM would finance operating costs and office space. The NHIC would play a leading role in designing and implementing a change management strategy for e-Health with guidance and advice provided by the Change Management Advisor recruited under the Project Implementation Unit (PIU). Component 3 would finance two subcomponents. Subcomponent 3.1: NHIC standards and applications (Total US$0.77 million; IDA US$0.56 million) 26. Subcomponent 3.1 would finance (a) standards enforcement comprising the ongoing work of maintaining, disseminating, and enforcing the HDS access for all interested parties and stakeholders throughout implementation and (b) specialized e-Health application support in the areas of clinical practice, clinical care, and other processes relating to the health venues themselves. Subcomponent 3.2: Health Data Statistics (Total US$1.67 million; IDA US$1.67 million) 2The pilots were expected to be carried out in four different sub-systems: (i) Pilot 1: District Hospital to Family Group Practice, and to a Specialty Hospital and health insurance agency; (ii) Pilot 2: Aimag General Hospital to Soum Hospitals and to a tertiary hospital in Ulaanbaatar, and health insurance agency; (iii) Pilot 3: Aimag General Hospital to a tertiary hospital to/from a Specialty Hospital, and health insurance agency; and Pilot 4: A private hospital to/from one or more of the hospitals in Pilot 1, 2 or 3, and health insurance agency. Page 13 of 52 The World Bank E-Health Project (P131290) 27. The Health Information Dashboard system supports complex data analysis for policy development and health statistics. This subcomponent would finance two contracts: one contract encompassing hardware and data-base management software and a second contract financing analysis tools. 28. Component 4: Institutional Strengthening and Capacity Building (Total US$1.20 million; IDA US$1.20 million) would invest in the human capacity for the successful implementation and institutionalization of e-Health in Mongolia. It would finance (a) user training to improve the computer literacy of health service providers and (b) IT technical training to improve the IT skills and competence of NHIC staff. 29. Component 5: Project Management (Total US$1.20 million; IDA US$1.10 million) would finance the PIU, domain expert consultation, and annual monitoring and evaluation (M&E) of the Project. The PIU would comprise a Project Coordinator, and implementation and procurement specialist, a financial management (FM) specialist, EA specialist(s), a clinician to offer clinical support, and a change management specialist. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION 30. The Project was restructured thrice: (a) first restructuring in June 2015, (b) second restructuring in September 2018, and (c) third restructuring in June 2020. The focus of each of these is described below. Revised PDOs and Outcome Targets 31. The PDOs were not revised. However, it became apparent that Project’s reach was much larger than the pilot sites set out in the PDO, per se, considering that the benefits of e-Health integration would flow to the entire health system, similar to the benefits from capacity building and institutional strengthening which would benefit not only a limited number of individuals but would percolate across the entire health system. With this in mind, the second restructuring in September 2018, reinterpreted the scope of the ‘pilot’ in the PDO to be all geographical and operational activities that the Project financing sought to affect in delivering the goal of better integration and use of data. However, this was not reflected in the legal agreement. Still, the core of the HIEP system development would be tested at specific pilot sites which would receive the majority, but not all, of the hardware investments. Revised PDO Indicators 32. In view of the interpretation described above, the PDO indicators were expanded in the second restructuring to capture the impact and benefits flowing across all parts of the health system and, possibly, not uniquely to the narrow definition of ‘pilot sites’. To enable better tracking of the Project, new indicators were introduced, some of the PDO indicators were revised, and the IRIs were revised to reflect the change in the components’ structure and scope of activities. 33. Progress at the PDO level was to be tracked with two additional indicators reflecting the importance of connecting the primary care system to the e-Health system and the capacity building efforts to implement the e-Health system. The additional indicators were the following: Page 14 of 52 The World Bank E-Health Project (P131290) • Number of project area soums and family health centers transmitting monthly e-Health statistical reports (Baseline: 200, Target: 450) • Improvements in IT and health information management capacity among health professionals (Baseline: 0, Target: 2,000). 34. The PDO Indicator 1, ‘Percentage of patient episodes for which information, including medical image information is available for secure viewing at pilot facilities’, was dropped but continued to be monitored at the intermediate result level by the following indicator: ‘Percentage of digital images generated at pilot facilities that are transmitted to central Picture Archiving and Communication System (PACS)’. PDO Indicators 2 and 3 were maintained. 35. To reflect the revised component structure, the IRIs were also revised (see section II B and annex 1). Of these, two directly measured part of the PDO: • Percentage of pilot facilities which design, develop, and implement successful piloting of eHealth Portal, Viewer and HIEP (Baseline: 0 percent; Target 80 percent). • Percentage of digital images generated in pilot facilities which are transmitted to the central PACS (Baseline: 0 percent; Target 80 percent). Revised Components 36. To simplify implementation, the second restructuring adjusted and consolidated the Project’s five components into three components, as follows: 37. New Component 1 (Restructured Component 1 and 2)—Build the e-Health Integration System— was adjusted to focus on the practical implementation of HIEP and completion of the remaining foundational activities under the original Components 1 and 2. The subcomponents of the HIEP were defined to include the Electronic Health Record (EHR); Electronic Medical Record (EMR) for primary health care providers; e-referral services (referral and report/test result); and possibly other applicable e- services; an identification and authentication tool which might use the national e-services; network services for primary health care providers’ connection; statistical reporting based on data accessible through HIEP; and security privacy, quality, and IT management protocols. This new Component 1 would also finance additional hardware originally expected to be provided by the GOM through the NDC for pilot sites at National Center for Traumatology and Orthopedics; Uvs, Khuvsgul aimag hospitals, and Songinokhairkhan District hospital; and Shastin Central Hospital (State 3rd Central Hospital). In addition, hardware and software would be provided to fill gaps at the primary health care level that had been identified in the project-supported Feasibility Study of primary health care IT equipment needs and digital equipment for PACS. 38. New Component 2 (Restructured Component 3 and 4)—Institutional Strengthening and Capacity Building—was to focus on strengthening health professionals’ basic computer skills and HIEP literacy, training IT specialists on network administration and information security management, providing advanced training in EA and health information interoperability, establishing a training center, setting the building blocks for an e-Health system by ensuring facility connectivity, and developing the legal Page 15 of 52 The World Bank E-Health Project (P131290) framework for the system’s smooth functioning. Importantly, with the primary goal of establishing an interoperable system across the country, primary health care facilities would progressively be connected to the high-speed network to enable health information exchange. 39. New Component 3 (Original Component 5)—Project Management—was expanded to include a survey to be carried out by the PIU to ensure beneficiary feedback was included in the Project, as part of citizen engagement efforts Other Changes 40. The first restructuring extended the Closing Date by 12 months to September 30, 2020; changed the name of the implementing agency from Ministry of Health and Sports to Ministry of Health; and, in addition, extended (a) the dated covenant’s target date for establishment of the PIU by one year and (b) end-target dates in the RF by one year. In addition to the changes to the RF and project components described above, the second restructuring reallocated funds among disbursement categories, revised disbursement estimates, adjusted the legal covenant related to NHIC staffing, and modified institutional arrangements. The third restructuring extended the Closing Date by a further 24 months, to September 30, 2022. Rationale for Changes and Their Implication on the Original Theory of Change 41. The rationale for changes in the first restructuring was to reflect the delay in the signing of the financing agreement and project being approved by Parliament and made effective. The rationale for changes in the second restructuring was to simplify the Project’s design, activities, and implementation processes to improve efficiency and the likelihood of achieving key indicator targets. It aimed to do this by revising the Project’s implementation strategies, prioritizing activities that underpinned achievement of the PDO by consolidating and focusing on core objectives, revisiting financing required and reallocating funds, delineating roles of the various e-Health institutions clearly, and revising the RF to allow proactive monitoring of the Project’s progress. The revision of the legal covenant and institutional arrangements was that the GOM would establish an E-Health Division in the MOH with dedicated full-time staff with the mandate for defining standards, the e-Health policy, and overall policy governance. Since the mandate of the Center for Health Development already covered the technical role for data management, the establishment of the NHIC as a separate institution would have been redundant. The rationale for changes in the third restructuring was to provide the additional time necessary to complete implementation of Project activities and achieve the PDO in view of earlier delays, compounded by the COVID-19 pandemic and related travel embargos. Only the second restructuring affected the original Theory of Change, as presented in Table 1. II. OUTCOME 42. A split rating has not been prepared as achievement of the PDO was never considered Satisfactory. As a restructured project that was streamlined, including adjustments to the PDO indicators in its RF, the evaluation of the Project’s achievements should apply a disbursement-weighted methodology. However, at the time of its second restructuring in August 2018, the Project had achieved Page 16 of 52 The World Bank E-Health Project (P131290) very little (see section III B), its PDO rating was considered Moderately Unsatisfactory, and it had only disbursed 13 percent of the IDA Credit. Following the second restructuring, implementation improved only marginally, with the result that progress toward achievement of the PDO was never considered Satisfactory or even Moderately Satisfactory, after restructuring. As a result, Efficacy is rated Modest both before and after restructuring. Because of this, a split rating has not been prepared; efficacy is evaluated using the PDO indicators and IRIs following the Project restructuring, and Overall Outcome is assessed considering also Relevance and Efficiency ratings over the life of the Project, as per the ICR Guidelines. Even if the analysis were carried out, it would conclude that the split rating produced an Outcome rating of 2.74. A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 43. The Project continues to be aligned with the World Bank Group’s Country Partnership Framework (CPF) for Mongolia for the period FY21–FY25 discussed by the Board of Executive Directors on May 25, 2021.3 The CPF’s Focus Area 3 (Improving quality of life), Objective 3.1 (Enhancing quality of health services), mentions that Mongolia’s health sector faces challenges in delivering quality service, as health service delivery is characterized by large technical and allocative inefficiencies. Further, it highlights that the system is not equipped to meet the growing burden of NCDs, the leading cause of mortality and morbidity in Mongolia, and that Mongolia needs to transition to a high-value delivery system supported by the efficient use of public expenditures. By focusing on effective management and control of NCDs by facilitating the real-time exchange of patient information and providing accurate and regularly updated data to monitor NCDs, the PDO and design respond fully to these priorities. The PDO became even more relevant during implementation with the onset of the COVID-19 pandemic that further exposed weaknesses in Mongolia’s public health systems, highlighting the need to build and strengthen preparedness. The GOM has continued to highlight the urgent need for an integrated e-Health system, even after Project closure, particularly in light of recent financing reforms that require data from service providers. The relevance of PDOs is rated High. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 44. While the PDO mentions the long-term outcome of better health service delivery, this assessment assesses the more immediate and intertwined outcomes of improving integration and utilization of health information and e-Health solutions. The original PDO was To improve integration and utilization of health information and e-Health solutions for better health service delivery in selected pilot sites. While the phrasing “for better health service delivery” could be interpreted as the main objective, it was inferred from the RF and components that this project was focused specifically (and solely) on e-health as the name suggests and not on improving broader health service delivery or outcomes. Consultation with the team that designed the operation also confirmed that the extra phrase was there only to indicate the ultimate long-term outcome. Therefore, the assessment of efficacy 3 Report No. 132141-MN. Page 17 of 52 The World Bank E-Health Project (P131290) evaluates the Project’s progress towards improving integration and utilization of health information and e-Health solutions; better health service delivery is taken to be the long-term outcome expected upon achievement of the Project’s in the Theory of Change (see Table 1 and Annex 6). Further, as integration and utilization in terms of e-Health are very closely related, it is difficult to evaluate these domains separately. Integration is the bringing together of different inputs and only part of the process of making information usable by health providers, managers, and patients. Indeed, all the original PDO indicators measure usability. Therefore, it does not add much value to evaluate these domains separately and they have been assessed together. Box 1. Achievement of indicators PDO Indicators (as revised) • Not met: Percentage of statistical reports out of total report produced by pilot facilities that are transmitted electronically through the Health Information Exchange Platform (HIEP) to the National Center for Health Development (NCHD) (Baseline: 0 percent; Target: 75 percent; Actual: 0) • Not met: Percentage of electronic referrals out of total referrals at pilot facilities (Baseline: 0 percent; Target 50 percent; Actual: 0) • Met: Number of project area soums and family health centers transmitting monthly e-Health statistical reports (Baseline: 200; Target: 450; Actual: 453) • Met: Improvements in IT and health information management capacity among health professionals (Baseline: 0; Target: 1,000; Actual: 1,180). IRIs (as revised) Component 1: Build the E-Health Integration System • Partially met: Health Data Standards developed (Baseline: No; Target: Yes; Actual: 24 HDS developed and submitted for review by MOH E-health Committee) • Not met: Percentage of pilot facilities which design, develop and implement successful piloting of HIEP (Baseline: 0; Target: 80; Actual: 0) • Met: Percentage of digital images that are transmitted at central PACS (Baseline: 0; Target: 80; Actual: 100) • Partially met: Number of primary health care facilities having working computers (Baseline: 110; Target: 500; Actual: 192*) • Met: Storage capacities for health data information improved (Baseline: Limited data storage capacity; Target: Health data storage capacities enhanced; Actual: Yes (20 Tb)). *While data to verify the actual value of this indicator were not available at the time of the last Implementation Status and Results Report, the MOH provided evidence during consultation on the draft Implementation Completion and Results Report that 82 primary health care facilities were provided with working computers in 2019 as part of the Project. Component 2: Institutional Strengthening and Capacity Building • Met: Training Center has been designed, implemented and ready to use (Baseline: No; Target: Yes; Actual: Yes) • Met: Soums connected to high speed internet (Baseline: 0; Target: 300; Actual: 443) Page 18 of 52 The World Bank E-Health Project (P131290) • Partially met: E-signatures used as a valid endorsement for health insurance claims (Baseline: No; Target: Yes; Actual: Legislation passed and effective from May 1, but regulation not yet issued) • Met: Beneficiary feedback incorporated into implementation (Baseline: No; Target: Yes; Actual: Yes). 45. Although the Project had some important achievements, these were significantly less than expected either at appraisal or at the time of the second restructuring. The Project was in problem status for five out of its eight years of implementation, stymied by frequent transition, capacity constraints, and lack of familiarity with WB procedures (Section III B). Box 1 outlines which indicators were met. 46. Achievement of the Project’s objectives was highly dependent upon the successful implementation of the contract for the design and implementation of the HIEP platform and, to a lesser extent, to that of the PACS. The successful design and implementation of HIEP were prerequisites for improving integration and utilization of health information and e-Health solutions for better health service delivery. Because of this, the numerous issues with the procurement, duplication concerns, and eventual implementation of the HIEP put at risk the achievement of the Project’s objectives (see section III B). While the Project achieved less-than-expected results, it was able to implement some of the initial measures to build an enabling environment for establishing an e-Health system, with the PACS having achieved good results in pilot facilities. 47. The Project did not achieve integration or utilization of health information and e-Health solutions as envisaged. This is mostly because implementation of the contract for the HIEP came too late, was unsuccessful, and overlapped with a system to extract health information to support health financing developed by the Ministry of Finance (MOF). The independent assessment of the HIEP contract’s implementation progress found that of 11 expected objectives and 11 expected outcomes listed in the contract’s Inception Report, only 3 and 2 of these, respectively, could be considered partially completed and achieved conditional upon submission of required documentation. The objectives partially achieved are to (a) ensure integration of PACS – limited to radiology reports and not in the correct format ; (b) implement the full set of EHR registries and analytical functions necessary to furnish high-quality, readily available, accurate, and actionable information on public health and health system performance metrics—master indexes for patients, practitioners, and health care providers created, but this is not the full set; and (c) implement the Clinician Portal/EMR systems for Mongolia’s primary health care facilities— the Clinician Portal/EMR is implemented for primary health care facilities, but the scope is limited to a screening program and an assessment of usage shows that it is not used for anything other than screening. The following two expected outcomes were partially achieved: (a) the standardized digital images are available for consultation/decision not only within the clinical premises through the hospital LAN but also through the referral service, but their use is not mandated and (b) the Clinician Portal/EMR system mentioned above. The assessment found that the overall level of contract compliance was only 29 percent, with compliance for standard software at 52 percent, for custom software at 31 percent, with documentation at 13 percent, and objectives and outcomes at 0 percent. As a result, there was no movement toward targets of the first two PDO Indicators (see above) and IRI 2 defined, since progress toward all of them depended upon the successful completion of the HIEP. Page 19 of 52 The World Bank E-Health Project (P131290) 48. Despite the unsuccessful implementation of the HIEP, the Project was able to produce some tangible achievements that can provide the foundation for the design and implementation of a strong e-Health system. The PACS is reported to have achieved good results in five central, pilot hospitals. However, the independent assessment found that there are still some crucial functionalities outstanding, including a licensed clinical viewer, integration of existing local PACS with the central PACS, and data exchange between existing EMRs and PACS. In beneficiary workshops, clinicians and IT specialists have questioned why they have been asked to migrate to a new system that, because of the lack of integration, has created additional work and slowed clinical processes. Still, 100 percent of digital images generated in the pilot facilities were transmitted to the central PACS, exceeding the target for IRI 3. Notwithstanding the missing functionalities, the PACS pilot was estimated as cost-saving over 15 years (see Section II C). The enabling environment has been strengthened with the development of 24 HDS and protocols and legal frameworks, which are now awaiting approval by the e-Health Committee.4 Health facilities have been provided with digital equipment, including fixed X-rays, portable X-rays, and portable ultrasound equipment. Eighty-two primary health care facilities in Uvs, Khuvsgui, and Ulaanbaatar districts were each provided with 4 computers and other important ICT equipment 5 and 443 soums were provided with access to high-speed internet with counterpart financing (exceeding the target of IRI 6). This has allowed 453 soums and family health centers to transmit monthly e-Health reports (exceeding the target of PDO Indicator 3). 49. Several project-supported activities have helped build capacity for developing an integrated e- Health system and providing for its successful utilization by clinicians, IT specialists, and health units. These include building, equipping, and furnishing an e-Health Learning Center; providing training on IT literacy and e-Health basics to 1,180 health professionals (exceeding the PDO Indicator iv target); establishing certified training for information network professionals; and preparing a computer users’ textbook. Legislation to allow that e-signatures be used as a valid endorsement for health insurance claims was passed and became effective from May 1, 2022, but regulations have not yet been issued (IRI (vii)). Finally, the storage capacity for the health data was improved through an order issued by a Parliamentary Committee, which required that the HIEP be connected with the e-Mongolia system and that designated the NDC for storage and safeguarding of the health information database. 50. While these may appear to be a fragmented set of outputs, they are undoubtedly a step forward toward the development of an integrated e-Health system in a challenging context. An ICT system being developed in the MOF, and which led to a halt in implementation in 2020/21 due to concerns of duplication (see section III B), has closely followed the HIEP design and is exchanging information related to financing across several sectors, including health insurance claims. While the decision to continue with development of the HIEP under the Project apparently did not account for the development of the MOF’s 5 The MOH provided 82 primary health care facilities with 4 computers, 1 network switch, 1 identification reader, 1 finger print reader, and 4 voltage stabilizers each. Page 20 of 52 The World Bank E-Health Project (P131290) system, and the development of the HIEP essentially started a parallel system, the Project’s outputs should help integrate and utilize e-Health systems under the MOF’s system. Justification of Overall Efficacy Rating 51. The Project did not achieve either integration or utilization. Nevertheless, Overall Efficacy is rated Modest since it did produce some tangible achievements, and support some preliminary activities that will be of value once plans to develop an integrated health system are resumed. C. EFFICIENCY 52. An economic analysis can be undertaken for the PACS pilot. A cost-effectiveness assessment was not undertaken for appraisal, due to the difficulty in assigning a monetary value to expected improvements in health outcomes and reliably estimating the impact of the Project’s investments in infrastructure and training on health outcomes. These difficulties remain valid, and the value of a full cost- benefit analysis is further reduced by the lack of tangible benefits from the Project activities. However, the PACS pilot generated nonincremental benefits, as pilot facilities could exchange images for specific patients rather than patients having to repeat the same imaging when attending a new hospital. This means that there should be less duplication of images across the pilot facilities. These benefits can be valued at the resource costs saved. 53. The PACS pilot was estimated to be cost-saving over 15 years, with a net present value of US$6.5 million. This is likely to be a considerable underestimate of the true savings from the PACS system for the following reasons: (a) only selected imaging modalities were included and (b) the GOM intends to extend the PACS contract to all aimag hospitals in the near future and likely all health sector facilities in the next 15 years. However, there are still missing functionalities that need to be integrated into the system. 54. However, the Project’s implementation was not efficient (see annex 4). Although at completion it had financed some separate outputs, these were very few especially when viewed in the context of its eight-year implementation period. The successive implementation issues that continued even after streamlining its design and implementation arrangements (see section III B) constantly put the achievement of its objectives at risk. With a cumulative closing date extension of 36 months, only about 45 percent of credit funds were disbursed, and a refund of over US$1 million has been requested due to disbursements for ineligible expenditures (see section IV B). The undisbursed US$8.5 million was cancelled at closing in September 2022, and reallocated to other IDA countries as by that time Mongolia had graduated to IBRD. If the Project had cancelled funds or closed earlier while Mongolia was still an IDA country, it may have been possible for this concessional capital to be reallocated to other World Bank- financed projects in Mongolia, rather than lose the financing to other countries. The historical service and commitment fee charges over 2017 to 2023 total US$336,028, of which US$283,312 is from 2020 to 2023, that is, due to the additional three years of implementation. The MOF started making principal repayments in 2020, that is, before the Project had tangible benefits. Assessment of Efficiency and Rating Page 21 of 52 The World Bank E-Health Project (P131290) The positive economic benefits of one of the Project’s outputs is overshadowed by the inefficiencies of the eight-year implementation period. The Project’s Efficiency is rated Negligible. D. JUSTIFICATION OF OVERALL OUTCOME RATING 55. The Project’s overall outcome is rated Unsatisfactory. Its relevance continues to be High, both to the World Bank’s CPF, and also to the country’s health sector needs and current reforms. It became all the more relevant following the COVID-19 pandemic. However, there were significant shortcomings in the achievement of its objectives and major shortcomings in its efficiency. E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 56. The Project did not have a gender focus, per se, but its investment in information systems was to provide tracking of health care usage disaggregated by gender. Further, women were expected to be the main beneficiaries of investments to improve IT skills and capacity as they are disproportionately represented among health care professionals in lead government health institutions such as MOH, NCHD, and so on. Institutional Strengthening 57. The PDO and the activities designed to help achieve it were intrinsically defined for strengthening the institutional capacity for collecting, storing, monitoring, and utilizing data to bring greater effectiveness and efficiency into the health sector. In this sense, the development and design of systems, provision of equipment required to implement them, and the training provided to staff to utilize them all aimed at strengthening institutional capacity. For example, over 1000 health professionals were trained in ICT literacy, supported by preparation of a computer user’s textbook in Mongolian. Other health professionals were trained in specific systems, for example radiologists were trained in using PACS. The Project also established Cisco Academy-certified training for ICT professionals working in the health sector. The Project also provided 328 computers, fingerprint and ID card readers to primary health care facilities in Uvs, Khuvsgul and Ulaanbaatar districts, worth $2.15 million. Mobilizing Private Sector Financing N/A Poverty Reduction and Shared Prosperity 58. The Project’s impact on poverty reduction was limited to the lack of achievements durings implementation. Its beneficiaries were to include all patients visiting facilities at specific pilot sites. Page 22 of 52 The World Bank E-Health Project (P131290) Other Unintended Outcomes and Impacts 59. The Project procured diagnostic equipment required for the onset of the COVID-19 pandemic (see section III B). The project delivered $2.2 million worth of digital equipment, including fixed X-rays, portable X-rays, and portable ultrasound, to health facilities as part of the emergency COVID-19 response. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 60. Several factors during preparation affected the Project’s implementation and outcome. These included (a) other external financing, (b) design complexity, (c) coordination with the NEA framework, (c) institutional arrangements, and (d) identification of risks. Project preparation, including work on some of the foundational activities under Component 1, was financed in part by a US$125,000 World Bank grant for e-Health development preparation. 61. Other external financing meant that the GOM identified e-Health strengthening as the priority for World Bank support. The GOM has a well-established partnership with the Asian Development Bank, who at that time was supporting the MOH to prepare the latest in a sequence of health sector projects. These covered many of the priority sectoral needs, including health financing reforms to service delivery, human resources for health, and pharmaceuticals. While e-Health strengthening would not be an urgent reform area when viewed from a health system perspective at that time, this was identified by the GOM as an outstanding cross-cutting area that required reform and, as such, the priority for first-time World Bank financing. 62. The Project’s design was complex, and the effect of this complexity on implementation was underestimated during preparation. First, the original design and implementation of HIEP involved a series of parallel (foundational) activities (standardization, EA implementation, HIEP and relevant e- services development, IT infrastructure development, PACS, and so on) under Components 1 (e-Health Foundational Activities) and 2 (Clinical Data Collection, Access and Sharing) before the pilot testing under Subcomponent 2.4 (IT infrastructure, maintenance and support and pilot tests) could be implemented. Yet, as the PDO indicators were all defined to measure utilization of e-Health systems by pilots, the Project’s impact required convergence of advances on all of these foundations to show progress. Moreover, there was large initial fragmentation with five components and eight sub-components, despite a recommendation to consolidate the number of sub-components during the Decision Review. While this complexity was acknowledged during preparation, the Decision Review assessed that the project struck a middle ground between technical complexity and efforts to leap-frog technology on the one hand, and the need to keep things simple given the low-capacity environment. 63. The development of an EA for health depended upon the definition of the NEA framework which would be adapted to the specifics of the health sector. The NEA was being developed under a separate institutional framework with support from the World Health Organization. At the time of the Project’s preparation, the World Bank was working with the Government on the preparation of a SMART Page 23 of 52 The World Bank E-Health Project (P131290) Government Project to support development of the NEA framework, which it intended to finance. The Project Appraisal Document (PAD) mentioned specifically that “the e-Health Project’s success will draw on the proposed SMART Government Project’s planned work on a NEA,” and the Project’s Steering Committee was to be responsible for ensuring synergies between the two projects and with the e-Health project financed by China that was providing computers and servers to all secondary and tertiary hospitals and developing an entirely new EMR for all public hospitals. 64. Paralleling its complexity, the Project’s institutional arrangements were complex and lacked clarity. The Vice-Minister of Health was the Project Director, and the MOH’s Department of Monitoring and Evaluation and Internal Audit (DMEIA) was the Project’s Implementing Agency. An e-Health PIU in the DMEIA was responsible for implementation, including project management, FM, M&E, reporting, and preparation of technical specifications and terms of reference for procurement by the General Procurement Agency (GPA) (see section IV B). Also, there were agencies responsible for design and implementation of the Government’s ICT policy; the NDC that provided facilities for secure storage of, and access to, government data; and the NCHD that was responsible for implementation of health sector policies on health statistics, information, e-Health, and information technology. A Project Steering Committee, chaired by the Minister of Health, was to be established for oversight and to provide strategic advice and guidance to the Project and the MOH. To help ensure the long-term sustainability of investments, the Project was to finance the establishment of an NHIC that would oversee all aspects of e- Health. 65. The Project’s implementation risk was assessed as being Substantial during preparation, but likely underestimated several risks. The following risks were identified as Substantial: (a) stakeholder risks, including possible reduced government commitment, duplicative efforts by donors, and lack of capacity among beneficiaries; (b) implementing agency risks, including weak technical capacity, fiduciary weaknesses, inadequate oversight, and lack of familiarity with World Bank projects; (c) project risks, including compliance with standards, interface layer, and technical complexity of project and unproven internet environment. Mitigating measures were identified and reportedly addressed in project design, but risks did not consider the broader political landscape that resulted in numerous staffing changes at all levels and underestimated the difficulties of supporting a new client—this was the World Bank’s first operation in support of Mongolia’s health sector—and implementing a complex project involving notoriously challenging procurement of complex IT systems. The need for coordination of the Project’s implementation with other e-Government initiatives that were being implemented at the same time was not identified. B. KEY FACTORS DURING IMPLEMENTATION 66. The Project was in problem status since March 2017 through completion, with the Implementation Status and Results Report (ISR) Development Objective and Implementation Progress ratings varying between Moderately Unsatisfactory and Unsatisfactory. In addition to the design factors described above, there were several factors that made implementation challenging: (a) delays in start of implementation, (b) continuous leadership and staffing changes and vacancies and related capacity constraints, (c) streamlining of institutional arrangements, (d) limited implementation and disbursements by the midterm review (MTR), (e) issues with M&E and procurement, (f) issues with the HIEP contract, Page 24 of 52 The World Bank E-Health Project (P131290) and (g) COVID-19. These are described below; challenges with M&E are described in section IV A and those with procurement in section IV B. 67. The start of implementation suffered delays. The SDR 18.4 million credit for the Project was approved by the World Bank’s Board on June 6, 2014, but only became effective more than one year later on August 17, 2015. All new loans and credits in Mongolia require Parliamentary approval following review by the Cabinet. The Project’s Parliamentary approval was delayed in view of a government restructuring in late 2014. With this, Parliament ratified the project only on June 3, 2015. The Cabinet restructuring also resulted in changes in the MOH at the ministerial and management levels. Following this, there were delays in recruitment of staff of the PIU, issues with eventual selection of personnel, and the need to re- advertise the positions and reevaluate candidates as a result. 68. Subsequently also, the Project’s implementation continued to be plagued by the lack of continuity in management and capacity constraints, together with the lack of familiarity with World Bank procedures. Frequent changes in senior leadership at the MOH associated with Presidential elections in June/July 2017 and June 2021 and Parliamentary elections in June 2016 and June 2020, led to considerable instability in Project management. In some cases, changes in government were accompanied by changes in the PIU, and at times resulted in extended vacancies and the need to recruit new staff to the unit. In one of these difficult changes, the long-standing project coordinator’s contract was terminated in November 2021, at a time when the implementation was finally accelerating. On several occasions, staff of the e-Health Unit in the MOH transferred out, leaving critical vacancies. This lack of stability and continuity was compounded by capacity constraints at the MOH, especially for implementing a technically complex and institutionally fragmented project. Staff of the MOH faced a steep learning curve to implement this first World Bank-financed health project, requiring close support by World Bank staff in routine matters such as issuance of no-objections to terms of reference and contract awards. And, to complicate matters, the priorities of and commitment by specialists in the PIU wavered with each change in staff. 69. The MTR in May 2018 focused on attempting to address the above issues and the impact that they were having on the eventual achievement of the Project’s objectives. By the time of the MTR, the Project’s progress was limited. The only activities that had been completed were (a) the combined feasibility study on primary health care IT equipment needs and digital equipment for the PACS (the study reported major gaps and limitations on both); (b) a study tour to Estonia by MOH officials to increase awareness of successful e-Health systems; and (c) hiring of consultants for the PIU, drafting selected terms of reference and bidding documents. Procurement of the HIEP and PACS required definition of the NEA framework that was financed under the World Bank-financed SMART Project (see section III A) that was suffering delays. The Project’s ISR had rated progress toward achievement of PDO and implementation progress as either Moderately Unsatisfactory or Unsatisfactory for extended periods. Project management, procurement, and M&E were all rated less than Satisfactory. The PDO and IRIs had not been updated in view of the absence of data (see section IV A), and disbursements totaled only USS$1.6 million. The MTR conducted a thorough review of issues affecting implementation and this resulted in the second project restructuring that aimed to right-size and streamline the Project. Page 25 of 52 The World Bank E-Health Project (P131290) 70. The Project’s support for the establishment of the NHIC as the center that would oversee all aspects of e-Health—a proposal that would add to institutional complexity and the coordination required but provide ‘a home’ to e-Health—was dropped, and institutional arrangements were first clarified and later adjusted. The establishment of the NHIC as a separate institution became redundant when the Government established other institutional arrangements that were expected to ensure the long-term sustainability of e-Health investments. These involved the establishment of an e-Health Division within the MOH (that was later dissolved—see below), with dedicated full-time staff, with the mandate for defining standards, e-Health policy, and overall policy governance, and the Center for Health Development was already responsible for the technical aspects of data management. Specifically, the e- Health Division of MOH became responsible for defining and enforcing e-Health policy and overall policy governance; coordinating e-Health initiatives and health sector projects; determining and enforcing nationwide health system infrastructure (hardware and software); and enhancing the e-Health legal environment by improving health policies regulations, standards and legal acts. The Health Information Units of the Center for Health Development was responsible for continuous capacity building of existing and new health professionals and specialists, carrying out health information and health technology training, maintaining and improving health applications and hardware, operating the HIEP helpdesk, operating the health data dashboard to supply health information to policy and decision makers, and maintaining and improving the HIEP. The National Data Center would securely and safely store and maintain national health data, both during implementation and thereafter. The PIU continued to be responsible for the Project’s implementation following the dissolution of the e-Health Division within the MOH in August 2021. The e-Health Division had been created in 2018 and was responsible for governance of digital health. Since then, the MOH had planned to expand the division and establish a National Health Information Center under the ministry. But, with the establishment of a new Ministry of Digital Development and Communication this plan did not move forward. 71. The Project leveraged its financing and human resource capacity to rapidly meet the urgent needs for ensuring health system preparedness resulting from the COVID-19 pandemic. It responded to pandemic preparedness activities through a reallocation of funds to allow the procurement of digital diagnostic equipment on a fast-track basis: 12 mobile and 15 stationary X-ray and 41 mobile ultrasound machines. These were distributed to aimag and central hospitals and helped ensure that the full benefit of the PACS system was achieved. The PIU also played an important coordinating role for the MOH in the preparation of the World Bank-financed Mongolia COVID-19 Emergency Response and Health System Preparedness Project (P173799, approved on April 2, 2020). Both projects share institutional arrangements and are closely aligned in implementation and mutually supportive. However, the increased workload due to concurrent implementation of both projects, coupled with the dissolution of the e-Health Division, the resignation of the Project Coordinator and the resignation of the PIU’s IT specialist, limited the unit’s capacity at an important moment in the Project’s implementation. 72. The Project faced severe delays in awarding the two large contracts for HIEP and PACS, but the PACS contract was eventually implemented with only minor issues while implementation of the HIEP contract faced almost continuous issues. Soon after the HIEP contract was awarded, the MOF sent a communication to the World Bank in September 2020 that the contract could duplicate some new electronic health information solutions being developed (outside the scope of the Project). In its communication the MOF informed the World Bank that since the contract did not meet the set timeline Page 26 of 52 The World Bank E-Health Project (P131290) for launching certain elements of the e-Health platforms, it intended to renegotiate the terms of reference for the HIEP contract, outputs, and service contracts to address these concerns after discussing this with the MOH. However, based on its review of documents, the World Bank opined initially that there would be no duplication (but there was a need for coordination) since the MOF’s system related to a health insurance system and not a health information exchange platform. The World Bank also advised against proceeding with new activities under the Project until the issues with the contract were resolved and suggested that the MOH consider three options: (a) reengage on the HIEP and PACS contracts and redefine their scope, timelines, and costs; (b) follow the first option and include other key relevant health system actions that would contribute toward achievement of the PDO and needs of the health system; and (c) cancel remaining funds and reallocate them to other parts of the World Bank’s Mongolia portfolio. This decision suffered delays following changes in government in January 2021. The HIEP’s contract implementation resumed only in June 2021, after receiving communication from the MOF confirming the Government’s decision to continue with the Project following an assessment by a parliamentary working group. In addition, in June 2021, the Parliamentary Standing Committee on Innovation and E-Policy discussed all health sector projects. The committee issued an order in June 2021 that required the HIEP be connected with the e-Mongolia system and designating the NDC for storage and safeguarding of the integrated health information database. The Prime Minister established a working group to monitor the implementation of the order. However, progress by the HIEP contractor was not commensurate with the contract’s requirement and timeline. 73. With increasing concerns that the time remaining to completion would be insufficient to develop a fully functioning HIEP and that the contractor did not have sufficient technical capacity and familiarity with industry standards, the PIU attempted to procure a technical audit of the HIEP at the World Bank’s suggestion. Issuance of the HIEP contract was not without issues, but finally a consortium between international and national contractors was selected to implement the HIEP. However, the international partner pushed most of the responsibility to the local partner. With this, progress in implementation was not commensurate with the contractor’s progress reports, and there was increasing concern that the MOH was not receiving value for money. The MOH did not succeed in obtaining an expression of interest from a qualified firm to carry out an audit and hired an independent international consultant to carry out a technical assessment instead. The World Bank advised the Project Director and Coordinator (a manager of the HIEP contract was not in place at the time although this was a contractual obligation) that pending the consultant’s findings, no further payments should be made to the contractor since the system did not meet the description of services in the HIEP contract. Despite this, the MOH made payments (an advance and a first payment) to the contractor totaling US$1,718,360. As expected, the assessment completed in September 2022 found that none of the objectives or outcomes specified in the contract had been achieved, and no payments should have been made to the contractor (see section IV B). Page 27 of 52 The World Bank E-Health Project (P131290) IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 74. The PAD mentioned that the PDO’s progress would be monitored through four types of M&E: (a) regular/routine monitoring, (b) MTR, (c) completion review, and (d) impact assessments for the pilots under Component 2. The Project’s RF was streamlined and contained three PDO outcomes, seven IRIs, and several output indicators. To the extent possible, the Project’s M&E activities would be integrated into existing data collection mechanisms the MOH, NCHD, and health facilities under the existing Health Management Information System. The pilots’ impact assessments were built into their design and included baseline and similar surveys to be carried out at regular intervals during their implementation. M&E Implementation 75. Throughout implementation, the PIU submitted regular update reports to monitor and evaluate progress. However, during early implementation, it became clear that it was and would continue to be difficult to monitor progress according to indicators in the RF. This was partly because the PDO indicators were defined as end-of-project outcomes, and the activities relating to most of the IRIs were off to a slow start, making any attempt to measure progress difficult. In addition, the feasibility study of primary health care IT equipment needs and digital equipment of PACS revealed major gaps and limitations in IT capacity at the primary level that required attention, and the second project restructuring adjusted the project description, among others, to address these. Further, while it is desirable to define and count upon PDO indicators that measure actual outcomes, the delays in the Project’s implementation, coupled with the fact that only few of its indicators could be used to measure actual implementation progress limited the use of the RF. Also, as mentioned in section III B, all the PDO indicators had been defined to measure outcomes at pilot facilities only, and the MOH indicated its desire to measure the Project’s impact over the entirety of the health sector (for example, by building the capacity of primary health care with hardware and software solutions) and not just that of the pilot areas. The MTR resulted in a detailed proposal to adjust the Project, its scope, components, implementation arrangements, and RF to address these issues that was reflected in the second restructuring (see section I B). The borrower has not yet produced a completion report and impact assessments of the pilots were not carried out. M&E Utilization 76. In view of the above issues, it was not possible to use the RF to track the Project’s progress in its early stages. In fact, before the RF was restructured in September 2018, only three IRIs (all measuring outputs) had been reported in the Project’s ISRs: (a) health standards developed, validated, and promulgated; (b) planning for training center developed; and (c) health personnel receiving training. Following the MTR and second project restructuring, the revised indicators were used to monitor the Project’s achievements. Page 28 of 52 The World Bank E-Health Project (P131290) Justification of Overall Rating of Quality of M&E 77. The overall quality of M&E is rated Modest, in view of issues with Project design and the shortfalls in implementation described above. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE Environmental and Social 78. Environmental. The Project was classified as Category C and an environmental assessment was not carried out during preparation. As only minor works were to be financed (associated with the installation of new equipment in existing offices and buildings), no adverse environmental impacts were expected. 79. Social. The Project triggered the Indigenous Peoples Policy (OP/BP 4.10) due to the presence of indigenous peoples (IPs) in parts of Mongolia who would be project beneficiaries. The relevant elements of OP/BP 4.10 were integrated directly into Project design since it was expected that benefits would be the same for all groups of people including IPs in view of the type of Project activities and their presence in four pilot sites (of which three were to be in Ulaanbaatar and one in Khovsgul Aimag). The elements incorporated in project design included (a) conducting consultations with IPs on project activities during implementation; (b) ensuring the provision of culturally appropriate e-Health benefits by using the IP languages in the provision of services; (c) establishing the estimated number of IPs in the selected project areas and the proportion of IPs likely to have information in the system; and (d) affording opportunities for IPs who may not have had access to public health facilities by providing information about e-Health and addressing any related concerns. 80. Throughout implementation, compliance with the IP safeguard policy was considered satisfactory and no issues were reported. As of June 2022, the Project’s grievance redress mechanism had received five complaints related to HIEP and PACS contract payments. As of July 2022, the PIU had organized meetings with the respective contractors and with end users (IT specialist, director, and doctors of several hospitals) to obtain their feedback, but the complaints had not been fully resolved. Fiduciary 81. Financial Management (FM). The Project’s FM assessment considered its FM risk rating as Substantial after mitigating measures defined in a plan of action, mostly due to the MOH’s lack of knowledge and experience in implementing World Bank-financed projects. The FM plan of action included measures to recruit and assign qualified FM staff to the PIU, carry out FM/disbursement training for the FM staff, prepare an FM Manual with detailed procedures including sound internal control arrangements, a chart of accounts, their description and use, and selection, testing and adoption of an accounting and financial reporting system. 82. During implementation, FM performance was mostly rated as Moderately Satisfactory although some shortcomings in compliance with the Bank’s financial management requirements were noted early on. Areas for improvement included enhancement of internal control procedures, financial Page 29 of 52 The World Bank E-Health Project (P131290) reporting practices and consistency in the PIU staffing. Slow disbursement progress was a common theme throughout the project implementation cycle. In an effort to better and fully manage all project transactions, MOF has requested the Bank to officially transfer all project Designated Accounts to the Treasury Single Account from commercial banks. The Bank has performed its due diligence on the request and granted its approval in January 2021. 83. The World Bank carried out periodic FM reviews that found the Project maintained acceptable FM arrangements and practices and complied with the World Bank’s financial reporting requirements . FM actions were agreed during each such review and were closely followed up by the PIU. The government’s statutory audit of the Project’s FY21 financial statements contained a qualified opinion due to material misstatements found in the financial statements. The PIU was subsequently able to address those misstatements and revise the financial statements accordingly, and was advised by the Bank to prepare the financial statements submitted to the government well in advance to allow for time to consider and address the auditors findings before the specified statutory deadline. All 6 audit reports submitted to the Bank were received in a timely manner and were of unqualified opinion and all 25 submission of the Quarterly Interim Financial Reports were done in a timely manner. 84. The most serious FM issue related to findings of an independent ICT audit that reviewed the HIEP contract. The auditor found that (a) the contractor had not achieved any of the objectives or outcomes specified in the HIEP contract, (b) no payment should have been made against the invoices submitted and deliverables according to the contract conditions, and (c) components within the scope of the contract had not been implemented fully. In view of this, the World Bank was not able to consider any of the expenditures under the contract as required for the Project and declared the payments made against the contract in the amount of US$1,718,360 as ineligible expenditures. The World Bank, in a letter dated February 7, 2023, informed the Government of this decision and requested that the amounts disbursed in respect of these ineligible expenditures be refunded by March 31, 2023. 85. Procurement. Procurement processes were managed by the Government’s GPA that acts as procurement agency for all central government procurement. However, the procurement capacity and risk assessment assigned a High risk rating to procurement in view of (a) the GPA’s large workload coupled with low staff capacity, including unfamiliarity with World Bank procedures; (b) the intensive ICT nature of the Project’s procurement; (c) the MOH’s lack of experience with implementing World Bank-financed projects; (d) inadequate planning and scheduling to ensure that activities would be initiated and be concluded as planned; (e) lack of a procurement oversight mechanism; and (f) possible elite capture and political interference. Risk mitigation measures included (a) developing a Procurement Manual setting out responsibilities and processes, (b) carrying out procurement training for GPA and MOH staff, (c) involving technical staff and users in the preparation of specifications or hiring competent consultants to draft technical specifications and terms of reference, (d) ensuring realism in procurement planning and compliance with dates, and (e) providing just-in-time advice and implementation support. 86. Throughout Project implementation, delays in contracting procurement staff initially and subsequent staff rotation, coupled with the time-consuming processes of the evaluation committee established at the MOH and issues with the lack of suitable bidders (for example, for the EA, the independent ICT audit), led to long delays in contracting consultants and procuring goods and services. Page 30 of 52 The World Bank E-Health Project (P131290) The bid evaluation process was regulated by the public procurement law of Mongolia, whereby for review of all contracts, regardless of value, an evaluation committee was to be established by order of the State Secretary. The PIU had to follow many steps to acquire the signatures of relevant officials to get the State Secretary’s order. These administrative processes often added three months to any contracting process. Since the Procurement Law only required the establishment of bid evaluation committees above certain contract value thresholds, the World Bank suggested the Project’s Steering Committee request a simplification of these procedures by establishing smaller evaluation teams (the evaluation team for the HIEP comprised 11 members) at the PIU with involvement of more experienced technical experts from the relevant beneficiary departments familiar with the subject of the contracts. Nevertheless, issues with the evaluation committee persisted, and delays continued, especially following the resignation of key persons in the committee in early 2019, when the proposals for key contracts—for the HIEP and the PACS—were being procured. 87. To allow the possibility of using more modern methods of procurement and specific procedures that would be more suited to the development of the HIEP (and offer greater flexibility to the MOH), the second project restructuring adjusted the Project’s procurement requirements to adopt the Procurement Framework rather than the World Bank’s Procurement Guidelines. This was expected to provide greater flexibility for procurement of a comprehensive HIEP through a combination of a number of the initially planned activities into larger, more results-oriented procurement packages. One example of the flexibility that this introduced was following single-stage procurement using initial selection (as opposed to a full two-stage process) and using evaluation criteria at the initial selection stage, limiting the number of bidders invited to prepare full proposals. In following the World Bank’s Procurement Regulations, the PIU submitted a Project Procurement Strategy for Development, including a contract management plan. 88. The World Bank’s Integrity Department opened a case concerning allegations of sanctionable practices, including collusion, fraud and corruption related to contracts financed under the Project. The investigation is ongoing. C. BANK PERFORMANCE Quality at Entry 89. The World Bank supported the Government’s request for assistance with a project to integrate and expand existing health information systems and that could help it address emerging health sector priorities. The World Bank attempted to discuss support to other sectoral needs, however, given the government’s stated priorities and synergistic national reforms, it was difficult to shift focus towards these areas during preparation. This was to be the first World Bank project in Mongolia’s health sector which implied a steep learning curve for the MOH to understand and appreciate the World Bank’s policies and procedures. Yet, against this background, the Project was overly ambitious, with complex institutional arrangements (most of which had no experience working with the World Bank). Its design even proposed to establish a new institution that would have added further complexity. A simpler project, with fewer activities and institutions involved would have been preferable for developing information systems—a notoriously complex endeavor—and for an implementing agency with limited experience. Further, the Page 31 of 52 The World Bank E-Health Project (P131290) Project’s design was such that its successful implementation and the full achievement of its objectives relied upon the implementation of a single, large contract (HIEP) that suffered repeated delays and was not implemented. The procurement of that contract, in turn, depended upon, first, the completion of the EA under the Smart Government Project and then its adaptation to the health sector. The Project’s RF was flawed as appraised, as it did not prove suitable to measuring progress or, more importantly, to raise red flags when implementation faltered. There were several flags during preparation that should have suggested caution especially with respect to the HIEP, including large-scale financing for health information systems in general. There was also an expectation that international partners with successful experience in e-Health would partner with the Government (which did not happen) and that the experience of other, more technologically advanced countries could be introduced in Mongolia. Also, the World Bank’s support for the SMART Government Project that was being processed in parallel offered assurances of the Government’s commitment to e-Government, including health. However, while the Project had been designed to adapt to the already existing and planned systems in the health sector, it is not clear that the World Bank was apprised of the details of those systems and plans. In retrospect, the World Bank helped prepare a project without the benefit of a full picture of information systems in the health sector. The preparation of the Project offered an opportunity to engage in a dialogue in Mongolia’s health sector, but that support could have been more contained, for example, to the PACS system (which was requested by the Government shortly before negotiations) only. Finally, the Project’s risk assessment did not fully address these issues and severely underestimated the risks its implementation would face (see section III A). The risks would not have precluded the eventual approval of the financing but should have advised a simpler project design. These risks were acknowledged during the World Bank’s review process, however the evaluation was that while the project was high-risk, it was also potentially high- reward and transformational. The mitigation measures recommended – such as to include a change management specialist in the PIU – did not align with the context. Quality of Supervision 90. The World Bank’s implementation support team faced a difficult task, with a major restructuring only occurring after four years Following early delays in effectiveness, the Project’s implementation and prospects rapidly deteriorated. The team identified issues with project design that challenged its implementation and worked closely with the Government to address them. Unfortunately, the team’s efforts were not sufficient to improve implementation and providing support was an uphill battle in view of the continuous changes in management and staffing at all levels and fluctuating commitment to the Project, and by repeated requests by constantly rotating staff to continue with the Project, expressing their intentions to improve implementation. The team also dealt with the need to coordinate Project activities with the SMART Project, those of other donors, and with the Government’s e-Government initiative; and the repeated difficulties in the PIU’s attracting and working with contractors and, starting in 2020, with carrying out only virtual support and supervision. Fiduciary compliance was reviewed and supervised regularly. The team’s documentation was timely and detailed and expressed candid ratings of implementation performance and prospects for achievement of the PDO (and other ratings, including those related to risks). It was difficult to focus on development impact when implementation was almost hostage to the award and implementation of a single contract, but the team was constantly searching for and recommending which activities and outputs should be prioritized to make the best use of the Project’s financing to advance on the foundations needed for eventual Page 32 of 52 The World Bank E-Health Project (P131290) development and implementation of an e-Health system. A major restructuring could have taken place earlier, and the second restructuring should have adjusted the PDO at the least to eliminate reference to the pilot sites. The team should also have attempted to further clarify the MOF’s proposal to develop its own health information system, to identify whether or not it would duplicate efforts under the Project. More importantly, the World Bank possibly demonstrated undue optimism that implementation would improve with each change in project staffing, always giving the PIU and the MOH additional time, while a full retrospective review of implementation would have been warranted to decide upon the continuation of support. Had it done so, it may have concluded that the use of remedies—suspension and eventual cancellation--was justified. Justification of Overall Rating of Bank Performance 91. Overall, World Bank Performance is rated Moderately Unsatisfactory. There were significant shortcomings in the World Bank’s preparation and appraisal of the Project, especially the identification of risks for what was a first involvement in Mongolia’s health sector. The supervision team inherited an over- dimensioned project and made a timely and decisive attempt to address design issues, including by streamlining the Project; addressing issues with the RF; and, to the extent possible, clarifying and simplifying institutional arrangements. The Project’s achievement of the outputs described in section II B are due in no small measure to the efforts of the World Bank’s supervision. D. RISK TO DEVELOPMENT OUTCOME 92. Risk to development outcome for a project that did not fully achieve its objectives can be assessed from two angles. The first is the risk that the outputs (and outcomes) that were achieved will not be sustained and the second the risk that what was achieved will not feed into further development, for example, that the PACS financed and the staff trained will not be utilized in any future systems that were to be developed. From both these perspectives, there are indications that the Government continues to assign priority to its e-Government strategy which bodes well both for maintaining investments supported by the Project and continuing to pursue an integrated e-Health platform along the lines that had been envisaged. While there may be a risk related to financing of further development, the Government has routinely prioritized allocation of counterpart funding for key activities under the Project, and, in any event, should have no difficulties in securing funding should it desire to go ahead with development of the platform. Although the SMART Project suffered from many of the issues that affected the Project’s implementation (for example, continuous changes in staffing, lack of capacity, and so on), its Implementation Completion and Results Report found that it had achieved its objectives, albeit with less than expected efficiency. The World Bank has recently approved an SDR 29.5 million credit for a follow- on SMART Government II Project (P176631) in support of Mongolia’s e-Government strategy, but it does not contain support specific to the health sector. V. LESSONS AND RECOMMENDATIONS 93. Be very realistic about client capacity where the World Bank is financing a first project in a client’s sector, especially when the operation is a complex one. The Project was the first in Mongolia’s health sector, and there may have been a certain optimism about seeing this as a ‘foot in the door’. The World Bank had never worked with the implementing agencies and so had no experience upon which to Page 33 of 52 The World Bank E-Health Project (P131290) evaluate implementation capacity and issues. More importantly, the implementing agencies had no experience in working with the World Bank and, as a result, had no knowledge of the World Bank’s policies, procedures, and other requirements (which are different to those of the Asian Development Bank with which the MOH had extensively cooperated). The provision of technical assistance is often mentioned as a mitigating factor to address weak implementing capacity, but acquiring technical assistance requires not only capacity but knowledge of World Bank requirements. This was compounded by the Project’s design involving systems development, a notoriously complex activity that often challenges even clients with much more experience and capacity, as well as a highly fragmented set of activities with five components and eight subcomponents which would concurrently develop the roadmap (EA) to implement the system. This complexity made it challenging to establish an appropriate PDO indicator framework with intermediate indicators that could capture meaningful incremental progress. Add to this the numerous staffing changes in the MOH and PIU and it is clear that weak capacity, in part the result of these changes, was a factor that deserved greater consideration during preparation. Further, a detailed assessment of capacity should have led to either identifying credible and feasible mitigating efforts or to the scaling back of a complex project. 94. Digital health projects may benefit from an enhanced review process to adequately assess technical risk. Digital health is a growing area of need in modern health systems. Yet it is also a highly technical area with multiple risks, and numerous projects have failed to achieve their objectives from low- to high-income countries. This Project’s risk assessment severely underestimated the risks it would come to face (see section III A). Further, the mitigating factors come across as almost pro-forma ones that are the standard response to risks under any project, for example, providing technical assistance to support implementation capacity. While all risk assessments should be taken seriously, this is especially important for first-time projects, such as this Project, where there is an inherent desire to engage with a new client, especially when the needs are so compelling. Concurrently, the World Bank’s review process did not adequately evaluate the technical risk of an overly designed project in a low-capacity context. The risk appetite seems to have been heightened due to the prospect of a transformational set of activities, however the multitude of failed digital health projects around the world must by now temper the realism of that outcome. It may be that the reviewers were not sufficiently specialized in e-Health to assess the complexity of the design. Given that there will be continued need for investment into digital health, an enhanced technical review may be beneficial for digital health projects, similar to that undertaken for environmental or social risks. The technical review need not result in a go-no go with respect to financing but, rather, should lead to a reconsideration of a project and its scope and complexity, especially when it is not possible to mitigate risks. This should be done as soon as possible during a project’s design to avoid vested interests in a certain design. 95. It should not be assumed that similar projects and investments in other countries can be transferred to other countries, and Mongolia’s health sector may have benefited from a series of projects with the first establishing a more straightforward system. The Project’s design aimed to follow the model of an e-Health project in Estonia. Staff of the MOH visited Estonia to learn of that country’s system, and the Project aimed to contract Estonian staff to support the Project’s implementation. But the capacity and level of development of information technology in Estonia and Mongolia are very different. Estonia is one of the most advanced countries with respect to sophistication of its information systems, a result of its continued prioritization of the use of technology. At the time of preparation, Mongolia was Page 34 of 52 The World Bank E-Health Project (P131290) just recently prioritizing e-Government, including of the health sector. Perhaps instead of learning about Estonia’s system, it would have been better for those involved to help the MOH prioritize and divide design and implementation into smaller parts for phased implementation. For example, a PACS is far less technically complex than a HIEP yet establishes the principle of health information exchange and integration. A PACS (albeit with limitations) was achieved under the Project for pilot sites. An alternative design for this first engagement could have been to establish and quality assure a nationwide PACS, and only then to consider a HIEP. 96. Again, for a first-time project, full knowledge of donor involvement and previous, current, and future plans is essential, both at preparation and during implementation, and a structure for coordination should be established where needed. The Project was already designed in an attempt to unify previously designed, financed, and implemented HISs (see section I A). This should already have raised a flag as to the need for coordination. However, during implementation another source of financing, related to the Project and its rollout, became available. Apparently little if any information about this financing was provided to the World Bank’s implementation support team despite repeated attempts to request details that were needed to coordinate hardware and software requirements. Project preparation needs to have full details of complementary sources of financing in the sector—existing and planned, both to avoid overlaps and to ensure coordination which is inherently required to ensure compatibility of information systems. 97. Projects should not be designed to depend upon implementation of its critical activities under a different project, even if they are both World Bank-financed—greater complementarity is needed. The design of the HIEP required the definition of the system’s EA that was being financed under the SMART Project. While the design of the EA was on the Project’s critical path, it was not fundamental for the SMART Project. Initial delays in developing the EA under the SMART Project automatically led to delays in the design of the HIEP under the Project. Perhaps if its activities are dependent upon other sectoral initiatives, then a higher-level implementing agency would be more effective, for example, a smaller e- Health project could have been integrated as a component of SMART with the same PIU and Project Director. 98. Client commitment, relevance of the development objective, and the feasibility of achieving results are factors that need to be evaluated closely and repeatedly. This is especially the case when, as under the Project, repeated engagements offered expectations and optimism that all issues would be addressed satisfactorily to allow the achievement of important results. It is difficult to stop supporting a project given investments in time and effort, but the sunk costs should not be the only reason to continue supporting it. When client commitment fluctuates, as it did under the Project, and other factors may compromise the Project’s achievement of objectives, the World Bank needs to reconsider closely whether continued engagement makes sense. . Page 35 of 52 The World Bank E-Health Project (P131290) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Project Development Objectivehttps://operationsdashboard.worldbank.org/project/secure/sap/forms/isr? Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion (1) Percentage of statistical Percentage 0.00 75.00 0.00 reports out of total reports produced by pilot facilities 31-Dec-2013 31-Dec-2019 31-May-2022 that are transmitted electronically through HIEP to the National Centre for Health Development Comments (achievements against targets): HIEP was not operational at completion, meaning indicator could not be achieved. Page 36 of 52 The World Bank E-Health Project (P131290) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion (2) Percentage of electronic Percentage 0.00 50.00 0.00 referrals out of total referrals at pilot facilities 31-Dec-2013 31-Dec-2019 31-May-2022 Comments (achievements against targets): HIEP was not operational at completion, meaning indicator could not be achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion (3) Number of project area Number 200.00 450.00 453.00 soum and family health center facilities transmitting 13-Jun-2018 30-Sep-2020 31-May-2022 monthly e-health reports Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion (4) Improvements in IT Number 0.00 1,000.00 1,180.00 Page 37 of 52 The World Bank E-Health Project (P131290) literacy and health 13-Jun-2018 30-Sep-2020 31-May-2022 information management capacity among health professionals Comments (achievements against targets): A.2 Intermediate Results Indicators Component: Component 1: Build the E-Health Integration System Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Component 1: Health Data Text No Yes 24 health data Standards are developed, standards have been validated and promulgated developed and by MOH submitted for review by MoH E-Health Committee. 31-Dec-2013 31-Dec-2019 31-May-2022 Comments (achievements against targets): Page 38 of 52 The World Bank E-Health Project (P131290) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Component 1: Percentage of Percentage 0.00 80.00 0.00 pilot facilities which design, develop, and implement 31-Dec-2013 31-Dec-2019 31-May-2022 successful piloting of eHealth Portal, Viewer and HIEP. Comments (achievements against targets): HIEP was not operational at completion, meaning indicator could not be achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Component 1: Percentage of Percentage 0.00 80.00 100.00 digital images generated in pilot facilities which are 31-Dec-2013 31-Dec-2019 31-May-2022 transmitted to the central PACS. Comments (achievements against targets): Page 39 of 52 The World Bank E-Health Project (P131290) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Component 1: Number of Number 110.00 500.00 192.00 primary healthcare facilities (soum, FHC) having at least 1 13-Jun-2018 30-Sep-2020 15-Aug-2023 upgraded and working computer Comments (achievements against targets): While data to verify the actual value of this indicator were not available at the time of the last Implementation Status and Results Report, the MOH provided evidence during consultation on the draft Implementation Completion and Results Report that 82 primary health care facilities were provided with working computers in 2019 as part of the Project. The primary health care software part of HIEP contract was not finalized in time for further procurement of computers to achieve the target for this indicator. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Component 1:Health data Text Limited Data storage Yes Yes (20 Tb) information storage capacity capacities improved 13-Jun-2018 30-Sep-2020 31-May-2022 Comments (achievements against targets): Page 40 of 52 The World Bank E-Health Project (P131290) Component: Component 2: Institutional Strengthening and Capacity Building Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Component 2: Training Text No Yes Yes Center has been designed, implemented and is ready for 31-Dec-2013 31-Dec-2019 31-May-2022 use. Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Component 2: Number of Number 0.00 70.00 443.00 soum and family health centers connected to high 13-Jun-2018 30-Sep-2020 29-Jun-2022 speed internet highway Comments (achievements against targets): Indicator Name Unit of Measure Baseline Original Target Formally Revised Actual Achieved at Page 41 of 52 The World Bank E-Health Project (P131290) Target Completion Component 2: E signatures Text No Yes Legislation passed and used as a valid endorsement effective from May 1, for health insurance claims but regulation not yet issued 13-Jun-2018 30-Sep-2020 31-May-2022 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Component 2: Beneficiary Text No Yes Yes consultation outcomes and feedback integrated in 13-Jun-2018 30-Sep-2020 31-May-2022 project implementation Comments (achievements against targets): Page 42 of 52 The World Bank E-Health Project (P131290) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1 To Improve Integration and Utilization of Health Information and E-Health Solutions for Better Health Service Delivery in Selected Pilot sites. 1. Percentage of statistical reports out of total report produced by pilot facilities that are transmitted electronically through the Health Information Exchange Platform (HIEP) to the National Center for Health Development (NCHD) Outcome Indicators 2. Percentage of electronic referrals out of total referrals at pilot facilities 3. Number of project area soums and family health centers transmitting monthly e-Health statistical reports 4. Improvements in IT and health information management capacity among health professionals 1. Health Data Standards developed 2. Percentage of pilot facilities which design, develop and implement successful piloting of HIEP 3. Percentage of digital images that are transmitted at central PACS 4. Number of primary health care facilities having working computers Intermediate Results Indicators 5. Storage capacities for health data information improved 6. Training Center has been designed, implemented and ready to use 7. Soums connected to high speed internet 8. E-signatures used as a valid endorsement for health insurance claims 9. Beneficiary feedback incorporated into implementation 1. Monthly e-Health statistical reports 2. Improved health information management capacity among health professionals Key Outputs by Component 3. 24 health data standards (linked to the achievement of the 4. Digital images that are transmitted at central PACS Objective/Outcome 1) 5. Improved storage capacities for health data information 6. Training Center 7. Soums connected to high speed internet 8. Beneficiary feedback Page 43 of 52 The World Bank E-Health Project (P131290) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Aparnaa Somanathan Task Team Leader(s) Gerelgua Tserendagva Procurement Specialist(s) Songling Yao Social Specialist Peter Leonard Social Specialist Yiren Feng Social Specialist Supervision/ICR Kate Mandeville, Anna L Wielogorska, Pagma Genden Task Team Leader(s) Zheng Liu Procurement Specialist(s) Erdene Ayush Financial Management Specialist Akosua O. Dakwa Team Member Songling Yao Social Specialist Yan Zhang Procurement Team Yiren Feng Environmental Specialist Gantuya Paniga Team Member Otgonjargal Norovjav Procurement Team Wut Yi Win Procurement Team Badamkhand Bold Team Member Dean Georgakopoulos Procurement Team B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation Page 44 of 52 The World Bank E-Health Project (P131290) FY12 4.912 63,916.48 FY13 31.480 193,639.28 FY14 34.449 183,366.87 FY15 0 778.21 Total 70.84 441,700.84 Supervision/ICR FY13 0 3,990.72 FY15 18.900 43,585.10 FY16 27.675 77,877.96 FY17 34.750 119,613.32 FY18 37.479 193,855.83 FY19 40.317 226,361.04 FY20 33.737 154,056.67 FY21 21.753 114,893.64 FY22 21.596 114,955.48 FY23 22.819 104,789.94 Total 259.03 1,153,979.70 Page 45 of 52 The World Bank E-Health Project (P131290) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Components (US$, millions) Closing (US$, millions) Approval (%) 15.98 (original Build the e-Health Integration System 6.90 43.2 Component 1 and 2) Institutional Strengthening and 2.30 (original 0.31 13.5 Capacity Building Component 3 and 4) Project Management 1.22 0.76 62.3 Total 19.50 7.97 40.9 Page 46 of 52 The World Bank E-Health Project (P131290) ANNEX 4. EFFICIENCY ANALYSIS 1. In any consideration of efficiency, the opportunity cost must be considered—here the opportunity cost of capital by the GOM. Overall, the Government only utilized US$8.88 million out of a US$19.5 million IDA credit. The undisbursed US$8.5 million was cancelled at closing in September 2022, meaning that it was reallocated to other IDA countries as by that time Mongolia had graduated to IBRD. If the Project had cancelled funds or closed earlier, it may have been possible for the GOM to reallocate this concessional capital to other World Bank-financed projects. The opportunity cost of retaining this funding for this Project was the loss of potential impact in other projects. The GOM also committed US$4.25 million in government financing, which could have been used for other investments. 2. The administrative costs to the GOM must also be considered. The historical service and commitment fee charges over 2017 to 2023 total US$336,028, of which US$283,312 is from 2020 to 2023, that is, due to the additional three years of implementation. The MOF started making principal repayments in 2020, that is, before the Project had tangible benefits. By 2039, the principal repayments will total US$16,902,179 and the net charges US$3,138,114. 3. While the task team leader changed thrice over preparation and implementation, this is not excessive for a Project that lasted 10 years, from the start of preparation to closing. More concerning was the change in MOH and PIU staff, particularly in the last stage of implementation when familiarity with the Project and experienced staff were particularly needed. 4. When the actual component costs are compared to the estimated component costs at appraisal (annex 3), project management costs were only 62.3 percent of those anticipated despite three additional years of implementation. The variance in the actual costs for the other two components are due to underachievement of activities. 5. A cost-effectiveness assessment was not undertaken for appraisal, due to the difficulty in assigning a monetary value to expected improvements in health outcomes and reliably estimating the impact of the Project’s investments in infrastructure and training on health outcomes. These difficulties remain valid, and the value of a full cost-benefit analysis is further reduced by the lack of tangible benefits from the Project activities. 6. However, there is one Project activity that lends itself to an economic analysis, which is the PACS pilot. This generated nonincremental benefits, as pilot facilities could exchange images rather than patients having to undergo the same imaging when attending different hospitals. This means that a smaller proportion of images should be duplicated across the pilot facilities. Under the new Health Insurance Law implemented in 2018, patients who attend different hospitals should only have imaging in the first hospital visited. These benefits can be valued at the resource costs saved. 7. A net present value has been calculated for PACS, with the following assumptions: • A government perspective was taken. Page 47 of 52 The World Bank E-Health Project (P131290) • Costs were estimated as follows: o Investment costs. The total amount paid to the PACS contractors was US$1,640,826 in 2022. o Maintenance costs. One of the functional guarantees of the contractor was that the total cost of ownership of PACS for the entire country was not more than US$2.2 million over five years. This was broken down as US$440,000 over 2023 to 2027. It was then assumed that maintenance costs would increase by an average 5% per year. • Benefits were estimated as follows: o Unit cost. Diagnostic-related group tariffs were obtained from the General Agency for Health Insurance (GAHI) in Mongolia, who manages the social health insurance fund. These are the amounts reimbursed from GAHI to hospitals for performing these scans and represent the estimated total cost, including goods and labor. Separate tariffs were only available for magnetic resonance imaging (MRI) and computerized tomography (CT), as ultrasound and plain X-rays were included in an umbrella tariff for outpatients (table 4.1). Tariffs were estimated for ultrasound and X-ray. Only the cost to GAHI and not co-payments by patients were included in the analysis. Table 4.1. Costs for MRI and CT Imaging Type Cost (US$) Data Source MRI 69 GAHI CT 24 GAHI Ultrasound 10 Estimate X-ray 5 Estimate o Unit. Data from the contractors shows that 320,596 diagnostic images were produced across the five pilot hospitals over 18 months of system implementation (August 18, 2020, to February 21, 2022). This included 232,648 X-rays, 52,039 CT scans, 14,654 MRI scans, and 10, 579 ultrasound scans over 18 months. This was converted into an annual production for 2021 of 34,866 CT scans and 9,818 MRI scans. As there were no data available for the pilot facilities from before the implementation of PACS, an assumption was made that duplication in other facilities had been avoided in 30 percent of these scans, that is, these represented cost savings. Benefits started in 2021, with 11,506 duplicative CT scans avoided and 3,240 MRI scans. From 2023, it was also assumed that there is a 1.5 percent annual increase in these diagnostic images from patient demand, based on an analysis of utilization trends outlined in the PAD. • An operating period of 15 years (2020 to 2035) was assumed, as with rapid evolution in health information systems, it is likely that the technology may become obsolete or requires significant upgrade after that period. • A discount rate of 6 percent for social sector projects in Mongolia. Page 48 of 52 The World Bank E-Health Project (P131290) 8. The PACS pilot was cost-saving over 15 years, with a net present value of US$6.5 million. This is likely to be a considerable underestimate of the true savings from the PACS system for the following reasons: (a) only selected imaging modalities were not included and (b) the GOM intends to extend the PACS contract to all aimag hospitals in the near future and likely all health sector facilities in the next 15 years. 9. A sensitivity analysis was undertaken to account for uncertainty in the true value of the proportion of scans for which duplication was averted. If it was assumed 50 percent of all scans averted duplication in another facility, then the net present value increased to US$11.7 million. However, if it was assumed that only 10 percent of scans averted duplication, then the pilot was not cost-saving. Page 49 of 52 The World Bank E-Health Project (P131290) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS The Borrower, through MOH, provided comments that were addressed in the final version of the report. Key comments included: (i) MOH’s clarification of certain Project activities as implemented; (ii) several clarifications around implementation dates and measurement of various indicators; and (iii) incorporation of a suggestion regarding the tracking of meaningful incremental outcome targets in technically complex projects (rather than solely measuring end outcomes) in the Lessons and Recommendations section. The MOH also subsequently provided evidence that, on review, enabled updating of the actual value for one intermediate results indicator (Number of PHC facilities having at least one upgraded and working computer). Page 50 of 52 The World Bank E-Health Project (P131290) ANNEX 6. ORIGINAL THEORY OF CHANGE Activities Outputs Outcomes Long-Term Outcome Component 1: • EA for health, based on the • Foundations in place for e-Health NEA framework, adapted to deployment of e-Health Foundational the specifics of the health solutions and Activities sector prerequisites needed for • A suite of HDS, which would greater integration of form the ‘common language’ health information needed for diverse computer between health facilities, systems to interoperate the health insurance agency, public health agencies, private sector, and so on Component 2: • Requirements for e-Health • Standardized data Clinical Data portal and viewer developed: exchange permitted Collection, (a) the computer between health care Access and architecture; (b) a web-portal providers and access for Sharing application and links; and (c) users to the HIEP security infrastructure to initially, and links Improved assure privacy and between the HIS and integration confidentiality other individual systems and Improved • HIEP and the that could come online utilization health interconnections between in the future of health service existing HIS, new HIS, other • HIS providers able to information delivery health information systems, exchange data and, as and e- an enterprise service bus for the e-Health system Health secure internet-based data became fully functional, solutions exchange, and gateway for between health use authentication and insurance and health handling of queries designed services utilization and constructed reporting and • A central archiving and epidemiology communication environment • The sharing of imaging for digital medical images acquired in diagnostic developed and selected departments with health facilities to use digitally care professionals and acquired imagery (mini PACS) patients. The HIEP upgraded and local networks infrastructure would be upgraded to share digital used for image exchange imagery and distribution as the • Servers required for the HIEP PACS exchange would are housed and supported at functionally and the NDC, and the pilot testing physically be part of HIEP Page 51 of 52 The World Bank E-Health Project (P131290) Activities Outputs Outcomes Long-Term Outcome of the clinical and imaging data exchange mechanisms tested under four pilots Component 3: • NHIC established • Institutionalization of e- National • Standards enforced Health in Mongolia, thus Health comprising the ongoing work ensuring the long-term Information of maintaining, sustainability of e-heath Center (NHIC) disseminating, and enforcing investments the HDS access and • Complex data analysis specialized e-Health capabilities for policy application support in the development and health areas of clinical practice, statistics clinical care, and other processes relating to the health venues • Hardware, database management software, and financing analysis tools acquired for Health Data Statistics Component 4: • Training provided to improve • Strengthened human Institutional the computer literacy of capacity for the successful Strengthening health service providers implementation and and Capacity • IT technical training provided institutionalization of e- Building to improve the IT skills and Health in Mongolia competence of NHIC staff Component 5: • PIU established • Strengthened government Project • Domain expert consultation institutional capacity for Management provided project management • Annual M&E implemented Page 52 of 52