DIGITAL HEALTH INTERVENTIONS: AN EVIDENCE GAP MAP REPORT © International Bank for Reconstruction and Development / The World Bank 1818 H Street NW, Washington DC 20433 Internet: www.worldbank.org; Telephone: 202 473 1000 This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the Executive Directors of The World Bank or other partner institutions or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions This work is available under the Creative Commons Attribution 4.0 International (CC BY 4.0) unported license https://creativecommons.org/licenses/by/4.0/. Under the Creative Commons Attribution license, you are free to copy, distribute and adapt this work, including for commercial purposes, under the following conditions: Attribution – Please cite the work as follows: Digital Health Interventions: An Evidence Gap Map Report. Washington DC: World Bank. License: Creative Commons Attribution CC BY 4.0 Translations – If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by The World Bank and should not be considered an official World Bank translation. The World Bank shall not be liable for any content or error in its translation. All queries on rights and licenses should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington DC, 20433, USA; fax: 202-522-2625; email: pubrights@worldbank.org. DIGITAL HEALTH INTERVENTIONS: AN EVIDENCE GAP MAP REPORT The World Bank: Marelize Görgens, Nejma Cheikh and Thomas Wilkinson The International Initiative for Impact Evaluation (3ie): Collins Zamawe, Birte Snilstveit, Miriam Berretta and Richard Appell This page is for collation purposes only. CONTENTS SUMMARY ............................................................................................................... v Background ....................................................................................................... v Scope and methods ........................................................................................... v Main findings and gaps ..................................................................................... vi Conclusions and implications........................................................................... viii INTRODUCTION ............................................................................................... 1 1.1 Background and rationalle ........................................................................ 1 1.2 Objectives and questions.......................................................................... 2 SCOPE AND INCLUSION/EXCLUSION CRITERIA ......................................... 3 2.1 Theory of change...................................................................................... 3 2.2 Population ................................................................................................ 4 2.3 Interventions ............................................................................................. 4 2.4 Outcomes ................................................................................................. 6 2.5 Study designs ........................................................................................... 7 2.6 Additional criteria ...................................................................................... 9 METHODS ...................................................................................................... 11 3.1 Search strategy and searching ............................................................... 11 3.2 Reference management and screening procedures ............................... 11 3.3 Data extraction and critical appraisal ...................................................... 12 3.4 Analysis and reporting ............................................................................ 12 3.5 Limitations of the evidence gap map....................................................... 13 3.6 Procedures for representation of excluded studies ................................. 14 FINDINGS ....................................................................................................... 15 4.1 Search results ........................................................................................ 15 4.2 The geographical and economic context of the studies........................... 17 4.3 Intervention coverage ............................................................................. 18 4.4 Outcome coverage ................................................................................. 19 4.5 Frequency of artificial intelligence use .................................................... 23 4.6 Health domains ...................................................................................... 24 4.7 Evaluation methods ................................................................................ 25 4.8 Confidence rating for systematic reviews ................................................ 26 4.9 Equity and gender focus ......................................................................... 27 4.10 Characteristics of the excluded studies ................................................... 27 i DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT GAP ANALYSIS.............................................................................................. 33 5.1 Absolute evidence gaps.......................................................................... 33 CONCLUSIONS AND IMPLICATIONS FOR FUTURE RESEARCH ............... 39 REFERENCES ................................................................................................ 41 APPENDICES ........................................................................................................ 43 Appendix 1: Classification of Digital Health Interventions ................................. 43 Appendix 2: Minimum Reporting Requirements for studies to be included ....... 53 Appendix 3: Search strategy ............................................................................ 56 Appendix 4: The complete list of literature sources .......................................... 90 Appendix 5: Data extraction template .............................................................. 92 Appendix 6: Critical appraisal tool .................................................................... 99 TABLES 2.1 Intervention categories and names ........................................................... 5 2.2 Broad outcome categories ........................................................................ 6 A1.1 Classification of Digital Health Interventions ........................................... 43 FIGURES 2.1 Theory of change developed by the authors based on the WHO Classification of Digital Health Interventions ............................................. 4 4.1 PRISMA diagram .................................................................................... 15 4.2 Trends in the publication of DHIs IEs (n=632) and SRs (n=97) since 2000 and 2014, respectively .......................................................... 16 4.3 Location and number of impact evaluation studies globally (n=632) ....... 17 4.4 Frequency of IEs by Income levels of the countries (n=632) ................... 18 4.5 Interventions reported by IEs (n=632) and SRs (n=97) ........................... 19 4.6 Frequency of outcome reporting in IEs (n=632) and SRs (n=97) ............ 20 4.7 The distribution of the economic outcomes (n=178)................................ 22 4.8 Distribution of health outcomes along the theory of change by intervention types (n=729) ................................................................. 23 4.9 Common health domains covered in IEs (n=632) and SRs (n=97) ......... 25 4.10 Impact evaluation methods (n=632) ........................................................ 25 4.11 Economic evaluation methods conducted by the studies (n=187) ........... 26 4.12 Consideration of equity or gender (n=729) .............................................. 27 4.13 Digital health intervention reported by excluded studies ......................... 28 4.14 Reasons for exclusion related to evaluation design ................................ 29 ii 4.15 Distribution of the outcomes reported by excluded studies ..................... 31 5.1 Screenshot of DHI evidence gap map available online ........................... 34 iii ACRONYMS AI artificial intelligence ANOVA analysis of variance CBA cost benefit analysis CCA cost consequence analysis CDSS computerized decision support system CEA cost effectiveness analysis CMA cost minimization analysis CUA cost utility analysis DHI digital health intervention EGM Evidence Gap Map EID early infant diagnosis GHO Global Health Observatory HITSystem HIV Infant Tracking System ICER incremental cost effectiveness ratio IE impact evaluation ITS interrupted time series LMICs low and middle-income countries NCD non-communicable disease QALY quality-adjusted life-year RCT randomized controlled trial SDG Sustainable Development Goal SR systematic review SROI social return on investment StAR SMS-Text Adherence support WHO World Health Organization. iv SUMMARY BACKGROUND Digital Health Interventions (DHIs) and Artificial Intelligence (AI) have an increasingly important role in the healthcare landscape as advances in technology produce digital health products. Such tools enable countries to create health systems that deliver personalised, pre-emptive, predictive and participative healthcare. Currently, there is no conformity regarding how to evaluate the impact of DHIs and AI on healthcare outcomes of interest. To facilitate decision making with regards to development, investment and implementation of DHIs and AI, there is a need to develop a standardised methodological approach to the evaluation of their effects and costs. Such standardisation will help provide comparative evidence to inform future investments in this sector. The World Bank and 3ie have been working on an Evidence Gap Map (EGM) to systematically map rigorous evidence on the effects of a broad range of digital health interventions, including AI. The EGM was commissioned to gain a better understanding of both the characteristics of existing impact evaluations (IEs) and systematic reviews (SRs) and where there are gaps in the available evidence. The main goal of this EGM was to facilitate decision-making regarding investment in research assessing the effects and economic impact of DHIs. A more specific objective was to support the development of a standardised methodological approach for the economic evaluation of DHIs which can be used by stakeholders to address high priority evidence gaps in different contexts. SCOPE AND METHODS The intervention-outcome framework for this EGM was informed by the World Health Organisation’s (WHO) classification of DHIs (WHO, 2018). According to the WHO classification, DHIs are organised into the following four broad categories based on ‘targeted primary user’: clients (i.e., members of the public and caregivers who are potential and current users of health services), healthcare providers (i.e., involved in the delivery of services to clients), health system and resource managers (i.e., administrators of healthcare systems and healthcare providers), and data services (i.e., cross-cutting activities to support the overall functioning of data collection and digital management of health services). Within each of these categories, there are many intervention-type sub- categories. The framework for this EGM included all the four broad intervention categories and the 28 intervention types as classified by WHO. Interventions without a clear digital component were outside of the scope of this EGM and hence were not included. We did not exclude studies based on outcomes, but rather recorded outcome measures as well as their definitions and categorised these into broad outcome categories. We created 10 outcome categories (health status (natural units), process outcome (therapeutic), behaviour change, healthcare utilisation, client/provider satisfaction, process outcome (non-therapeutic), quality of care, health status (aggregated/summary units), knowledge and beliefs, economic outcomes and an additional category for any other outcome measures that did not fit in these categories. v DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT We searched a range of different sources of academic and grey literature, including 11 academic databases, 13 repositories of IEs and SRs, and 13 specialist organizational databases and websites of bilateral and multilateral agencies. We also conducted backwards citation tracking of SRs that met our inclusion criteria, as well as any relevant non-SRs and guidelines identified to identify additional studies. All the records identified through the searches were assessed against a set of detailed and rigorous inclusion criteria for this EGM to identify eligible IEs and SRs. We extracted the data using 3ie standardised templates and SR methods were critically appraised. MAIN FINDINGS AND GAPS We retrieved 63,014 records, and excluded 17,142 duplicates. Most of records (n=40,374) were excluded during the title and abstract screening. After the full-text screening, we included 632 IEs (436 completed and 196 ongoing) and 97 SRs (89 completed and 8 ongoing). The EGM reveals two major types of evidence gaps in the literature: ‘absolute’ gaps, where no or few primary studies have been conducted, and ‘synthesis’ gaps, where no SR exists despite a cluster of IEs or the quality of the SRs is sub-standard or the existing SRs are dated. The major gaps are presented below. • The evidence base is skewed towards high-income and Western countries with more limited evidence from low and middle-income countries (LMICs). In terms of location, there is limited evidence from the WHO regions of Africa, South-East Asia and Eastern Mediterranean as most of the studies (66%) were conducted in the two regions of America and Europe. Similarly, very few studies were conducted in LMICs and a huge gap in the volume of studies conducted in rich and poor countries has been noted. For instance, over three-quarters of the studies were conducted in high- income countries with the USA alone contributing one-third of the included IE studies. • There is limited evidence of DHIs for health system or resource managers and data services. Across the four intervention categories, almost all studies evaluated interventions for clients and healthcare providers. We found significant evidence gaps in the other two categories, which were together covered by five studies only. • Despite the potential role of AI in strengthening health systems and improving the quality of health care, we found a shallow evidence base in the published literature for AI. The number of studies that evaluated AI-powered interventions is extremely low. In total, only 13% (n=83) of the studies covered interventions that incorporated AI Possible reasons might be slow adoption of AI in the health sector, limited use of peer-reviewed publication as a mechanism of measuring impact, or long and rigorous vetting processes that may be required in the medical field. This could also be due to the inclusion criteria of the EGM, which was focused on IEs that use counterfactual analysis. This may not be a common approach in AI. • Notwithstanding the large volume of IEs covering clients and providers’ DHIs, the evidence base within these intervention categories is unevenly distributed. For example, there is limited literature on DHIs that empower clients to take lead on health issues affecting them through, for example, citizen reporting or client-client communication. The focus for most studies is largely on top-down or externally led interventions such as targeted communication interventions. For the healthcare providers, there is a heavy focus on DHIs concerned with service delivery e.g., vi INTRODUCTION telemedicine or decision support. Interventions that support the providers in planning or coordination of health services (e.g., referral or activity scheduling) have received relatively low attention. • • Not many studies covered critical global health domains such as maternal health, nutrition, HIV/AIDS and family planning/reproductive health. Not more than 45 studies covered each of these domains compared to 343 studies that covered NCDs alone. Most IEs were concerned with the NCDs and there is also a substantial focus on mental health, infectious diseases and child health. The plausible explanation for this evidence gap is that most of the infrequently covered domains predominantly concern LMICs, where very few DHI studies have been conducted. • The impact of DHIs on summary or impact health outcomes such as mortality, quality of life and healthcare has been under-explored. The focus of most studies was on short-term outputs and directly measurable health outcomes e.g., number of clinic visits. In fact, there is a huge gap in the number of studies that have reported the intermediate and summary outcomes. However, this gap is not surprising given that long-term health outcomes are relatively difficult and expensive to measure. For economic outcomes, there are evidence gaps in both intermediate and summary outcomes. • The cost data of the interventions are infrequently reported. For example, about 80% (n=551) of the studies did not report any cost data and out of those that reported more than half (n=93, 60%) simply provided cost data without performing any further analysis such cost-effectiveness analysis. Availability of relevant and usable costing data is as important as effectiveness data as decisions on policy, programming and scale-up depend on both. • There is neither up to date nor high-quality SRs of personal health tracking interventions even though many IE studies have addressed this topic. The majority of the studies covered process outcome (therapeutic), economic outcomes, healthcare utilisation, health status (natural units), client/provider satisfaction and behaviour change outcomes. We identified a few SRs (n=4), but none of them are rated as high confidence due to methodological limitations. • There are opportunities for possible syntheses relating to the effects of client health records and healthcare decision support system interventions. A substantial number of studies have explored the effects of client health records on healthcare utilisation, quality of care and health status (natural units), but no high-quality SR exists. Another area for potential synthesis is the link between practitioner decision support systems and each of the following outcomes: healthcare utilisation, process outcome (non- therapeutic), process outcome (therapeutic) and quality of care. • The effects of telemedicine on process outcome (therapeutic), healthcare utilisation, health status (aggregated/summary units), health status (natural units), knowledge and beliefs and process outcome (non-therapeutic) present additional synthesis gaps for future reviews. Telemedicine is one of the most common interventions with a large body of evidence across all the outcomes of interest in this EGM. Most of the studies have measured outcomes of health status (natural units), process outcome (therapeutic) and healthcare utilisation. We identified a cluster of completed and vii DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT ongoing reviews on telemedicine, but only two of them (under the outcome categories of client/provider satisfaction, economic outcomes and behaviour change) are rated as high confidence. • Synthesis gaps also exist among targeted digital health communication interventions as most of the included SR are of substandard quality. This is the most covered type of DHI in the literature as multiple studies have explored the link between targeted digital health communication and healthcare utilisation (n=56), process outcome (non- therapeutic) (n=47), health status (natural units) (n=134) and client/provider satisfaction (n=67). Although some SRs have been conducted to assess the effects of the intervention on these outcomes, none of them is rated as high confidence or updated. CONCLUSIONS AND IMPLICATIONS Overall, the global DHIs evidence base is rich. The EGM clearly show areas with sufficient, limited, non-existent, high/low-quality and updated/dated evidence for DHIs. Exploration of the findings from this EGM and the quality of the existing evidence presented could facilitate or inform decisions. For instance, policymakers can explore evidence of the effects of DHIs to inform strategy development, whereas funders and researchers could examine the priority areas for future research and funding in this field. Based on the findings, we believe that research or SR in the areas summarised below could improve the global DHIs evidence base. • There is a need to explore the availability and possible barriers to assessing the health system management and data service interventions. This is also a potential area for future IEs to focus. ‒ A very small proportion of the studies are from LMIC. We assume this is not only about a lack of data but also reflects a broader issue of access to digital innovations in health in LMIC as well as publication bias. Since LMIC shoulders the greatest burden of global health issues and experiences relatively high number of health system challenges (e.g., poor access to health services and information), the region stands to benefit more from digital health solutions. Therefore, more investment in digital health research, equipment/resources and training is recommended. • It is evident from the findings that most DHIs were used to address NCDs. Evidence of the effects in other priority global health issues such as HIV, maternal health and nutrition is limited. These health domains should be considered in future studies to improve the availability of evidence in this field. • Future studies on DHIs should consider measuring long-term health and economic outcomes, which are relatively neglected in the current literature. These outcomes include those relating to the quality of care, quality of life, survival rate and summary economic outcomes. • The role of AI in many sectors is on the rise. However, based on our data, it seems not many digital interventions incorporate AI components. This is one of the areas with massive evidence gaps that could be addressed by future studies viii INTRODUCTION • There is a limited economic evaluation evidence base for DHIs, which could limit decisions on policy, programming and scale-up. One potential mechanism to improve conduct of economic evaluation is development and promotion of standardised, high- quality economic evaluation methodological guidance tailored towards the DHIs. • Main synthesis gaps to be considered include areas where there is a large body of IEs but no SR has been performed e.g., effects of provider decision support systems on healthcare utilisation or process outcome (non-therapeutic). Areas with dated or low confidence reviews should also be prioritised e.g., impact of telemedicine on health status (natural units) or process outcome (therapeutic). These syntheses can provide rigorous evidence of the effects of various DHIs, which could improve the evidence base. ix This page is for collation purposes INTRODUCTION BACKGROUND AND RATIONALLE Digital health or the use of digital technologies for health (e.g. eHealth, mHealth, big data and artificial intelligence) is a new and fast-growing field of study in public health (World Health Organisation (WHO), 2019). It is considered integral to improving accessibility, quality and affordability of healthcare, which is at the centre of both primary health care and universal health coverage (WHO, 2018). Its importance in strengthening the health system is well reflected in the 2018 World Health Assembly resolution, which urges ministries of health to prioritise development, implementation and evaluation of digital health interventions (DHIs)(WHO, 2018). While the response has been overwhelming, there are concerns that the interventions are being rolled out without a careful assessment of the evidence base on effects and costs. The use of digital technologies for health is vital in meeting new challenges such as the increase of non-communicable diseases, shortage of the health workforce, ageing population, unplanned emergencies, and infectious disease outbreaks. Digital Health is also widely recognised as a potential tool to advance the Sustainable Development Goals (SDGs) and to support health systems across the world in increasing access to healthcare, health promotion and disease prevention among others. DHIs have targeted all levels of healthcare delivery, including the patient, facility, community and system levels. They aim to address a range of outcomes, including demand, access, behaviour, quality and integration. Increased importance has been placed on assessing the effects and costs of DHIs providing evidence to inform future investment, such as which interventions give the best value for money from limited available resources. This evidence is critical to ensure that DHIs do not improperly divert resources from more cost effective non-digital interventions. While some research is available, the evidence base is still relatively nascent and there is no conformity regarding how to evaluate the impact of DHIs on healthcare outcomes of interest. To facilitate governments’ decision making with regards to DHIs and artificial intelligence (AI), there is a need to develop standardised DHI economic evaluation methodology. Such standardisation will help provide comparative evidence to inform future investments in this sector. The current state of evidence, however, presents an opportunity to develop a strategic and coordinated research agenda that addresses important evidence gaps. To do so, it is important to establish what is already known and identify common economic and non-economic outcome measures across intervention types. This Evidence Gap Map (EGM) provides a better understanding of both the characteristics of existing impact evaluations and systematic reviews and where there are gaps in the evidence. 1 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT OBJECTIVES AND QUESTIONS The broad goal of this EGM was to facilitate decision-making regarding investment in research assessing the effects and costs of DHIs and AI globally. This was achieved through systematic literature searches, reviewing the available evidence, identifying both substantive and methodological characteristics of the literature and describing the evidence gaps. By identifying the ‘evidence baseline’ and best practice for future research, it aims to ensure that new studies are as rigorous and useful as possible. A more specific objective was to support the development of an economic evaluation framework which can be used by stakeholders conducting economic evaluations to address high priority evidence gaps in different contexts. The use of AI in health care can be applied in both digital and non-digital (analogue) interventions; this EGM focused on DHIs with further evidence searches specifically targeted towards AI. In particular, this EGM addressed the following questions: What are the characteristics of the current landscape of evaluations addressing the effects of DHIs (e.g., geographical distribution, interventions covered, outcomes measured, attention to equity, study design, measurement of costs and resource use)? What are the outcomes that are measured in evaluations assessing the impacts of different interventions and at different levels of DHIs? What are the methodological approaches and metrics used to evaluate the effects of DHIs? At what level of the health system are most of the evidence concentrated (e.g., patient, facility, system)? The report proceeds as follows. In the next section, we present the methodology of the EGM, including the scope, methods, interventions, outcomes and other inclusion criteria. Section 3 covers the findings of the EGM, comprising the descriptions of the IEs and SRs evidence base as well as the search results. In section 4, we present the conclusion and implications of the findings. 2 SCOPE AND INCLUSION/EXCLUSION CRITERIA The scope of this EGM was informed by the theory of change (TOC) developed by the authors based on the WHO’s recommendations on digital interventions for health system strengthening (WHO, 2018), which includes classification of DHIs. The TOC in turn informed the the inclusion/exclusion criteria of this EGM, including the target population, interventions and outcomes of interest. THEORY OF CHANGE Digital interventions for health system strengthening are technological solutions to support the achievement of health sector objectives. The WHO notes that digital health should be considered in the context of the health system challenges to demonstrate how technology is addressing health needs such as lack of service utilisation and access to care. For example, DHIs may be used to facilitate targeted communications to individuals to generate demand and broaden contact coverage. Health workers may also be targeted to give them access to clinical protocols through, for example, decision-support mechanisms or telemedicine consultations. Figure 2.1 presents the TOC for this EGM conceptualising our understanding of the link between digital technologies for health and health system outcomes. This TOC is based on the fundamental principle of the WHO’s classification of DHIs, which indicates that the interventions are developed or implemented to tackle or meet specific health system needs or challenges. The specific outcomes of the interventions vary widely and largely depend on the challenges targeted by the interventions. Assuming the interventions are effective and address the underlying problem, improvement in the targeted issues is expected if the interventions are properly implemented. Meeting these health needs is an important step towards improved clients’ health outcomes, satisfaction and behaviours as well as health system processes. For example, an intervention involving sending short message reminders to address poor treatment adherence may improve adherence and further improve health outcomes such as mortality or quality of life. 3 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Figure 2.1 Theory of change developed by the authors based on the WHO Classification of Digital Health Interventions1 Source: 3IE, World Bank. Note: WHO = World Health Organization. Some of the “outcomes categories” span outputs, intermediate outcomes and impact (e.g., economic outcomes), while others are more limited to the intermediate outcomes (e.g., knowledge and beliefs). POPULATION The target population groups for this EGM were: Clients receiving an intervention via a digital platform Healthcare providers who prescribe DHIs and monitor outcomes Healthcare systems collecting and reporting information through DHIs Healthcare workers utilizing DHIs to facilitate their work and interact with patients or other healthcare entities INTERVENTIONS We used the conceptual framework developed above and the WHO classification of DHIs to define the intervention-outcome framework of the EGM. According to the WHO classification, DHIs are organised into four broad categories based on ‘targeted primary user’, as below: Clients: Includes members of the public and caregivers who are potential and current users of health services. Healthcare providers: Health workers involved in the delivery of services to clients. Health system and resource managers: Administrators of healthcare systems and healthcare providers 1 WHO Classification of Digital Health Interventions v1.0 4 SCOPE AND INCLUSION/EXCLUSION CRITERIA Data services: Cross-cutting activities to support the overall functioning of data collection and digital management of health services. Within these categories, there are several specific intervention strategies as presented in Table 2.1 and further described in Appendix 1. The framework for this EGM included the four broad intervention categories which incorporated 28 specific types of interventions as classified by WHO. Interventions without a clear digital component were outside of the scope of this EGM and hence were not considered. Table 2.1 Intervention categories and names Intervention broad category Intervention name/type Client Targeted digital health communication Untargeted digital health communication Client-to-client communication Personal health tracking Citizen-based reporting On-demand information services to clients Client financial transactions Healthcare providers Client identification and registration Client health records Healthcare provider decision support Telemedicine Healthcare provider communication Referral coordination Scheduling and activity planning Training Prescription and medication management Laboratory and diagnostics imaging Health systems management Human resource management Supply chain management Public health event notification Civil registration and vital statistics Health financing Equipment and asset management Facility management Data services Data collection, management, and use Data coding Location mapping Data exchange and interoperability Source: 3IE, World Bank. DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT OUTCOMES An extensive range of outcomes were included in this EGM. Part of the objective of this EGM was to identify what outcomes are commonly measured, and establish meaningful categories of outcome measures. The rationale was that this would ultimately enable the broader project to propose standardise outcome measures for future research, in consultation with stakeholders. We, therefore, did not exclude studies based on outcomes, but rather recorded outcome measures as well as their definitions (according to the authors) and categorised these into broad outcome categories based on the similarities. This process was partly informed by the TOC for the project and hence most of the specific outcomes that have been put together to form outcome categories are well aligned to the TOC. Through this approach, we created 10 broad outcome categories and an additional category for any other outcome measures that did not fit the definition of these categories. Table 2.2 provides a list of all broad outcome categories included in this EGM along with their working definitions and examples. Table 2.2 Broad outcome categories Outcome category Definition Example Economic Cost of treatment, cost consequence analysis Comparative costs, incremental outcomes (CCA), cost minimization analysis (CMA), cost effectiveness ratio, return cost effectiveness analysis (CEA), cost utility on investment, net benefits analysis (CUA), cost benefit analysis (CBA), social return on investment (SROI) Behaviour Change in reported health seeking or lifestyle Smoking rates, condom use, change risky sexual practice Client or Reported comfort or acceptability of an Ease of use of technology provider intervention or treatment approach satisfaction Process Uptake rates for the intervention Number of people successfully outcome (non- enrolled in a programme, therapeutic) downloads of an app, login or interaction with digital platform Process Improvements in patient compliance in Number of people achieving outcome adherence to monitoring or treatment that are treatment adherence such as (therapeutic) the result of a digital intervention. taking antiretrovirals, following best practice guidance or blood pressure monitoring Healthcare Rates of healthcare consumption Attendance at GP clinics or utilisation hospital visits, elective surgeries Quality of care Improvements in standard measures of Quality of care survey treatment quality improvement, common care practices followed, risk factors identified and dealt with by healthcare professional. Table 2.2 continued on next page 6 SCOPE AND INCLUSION/EXCLUSION CRITERIA Table 2.2 Broad outcome categories (continued) Outcome category Definition Example Health status Measures of rates of disease or impact in Change in HIV infection rates; (natural units) natural (non synthesised) units hazard ratio, risk difference, odds ratio Health status Measures of rates of disease or impact in Improvements in survival rates, (aggregated or aggregated, synthesised, or generalised units lives saved, HRQoL, QALYs, summary DALYs units) Knowledge Both provider and client's change in Health care workers knowledge and beliefs awareness, knowledge, self-efficacy, of the management of neonatal perceptions, intentions, attitudes and beliefs resuscitation before and after eLearning Other Any other outcome not addressed above Source: 3IE, World Bank. Note: HIV = human immunodeficiency virus; HRQoL = health-related quality of life; QALYs = quality-adjusted life years; DALYs = disability-adjusted life years. STUDY DESIGNS This EGM included both primary studies and SRs. We only included studies that used an experimental or quasi-experimental design and/or analysis method, which seek to robustly measure the net change in outcomes that are attributed to an intervention or policy as compared to some appropriate counterfactual (i.e., impact evaluations) and SRs of such studies. Further details are provided below and in appendix 2. Impact evaluations For IE studies, we included the study designs listed below and Appendix 2 describes the minimum reporting requirements for each study design that was used to review if studies meet a quality threshold for inclusion as IEs. Randomised controlled trial (RCT) (individual or cluster) Regression discontinuity design Controlled before-and-after studies using appropriate methods to control for selection bias and confounding, such as: a. Propensity score matching or other matching methods b. Instrumental variable estimation or other methods using an instrumental variable such as the Heckman Two-step approach c. Difference-in-differences d. Fixed- or random-effects model with an interaction term between time and intervention for baseline and follow-up observations Natural experiments Other quasi-experimental studies including studies using synthetic controls Cross-sectional or panel studies with an intervention and comparison group using methods to control for selection bias and confounding as described above (see bullet DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT point 3). Purely observational studies (interviews, surveys, correlational studies, design workshops, etc.) were not eligible Interrupted time series (ITS) Mixed-methods studies that combined qualitative research with any of the above- mentioned study designs Finally, an inclusion criteria was that studies were required to be effectiveness studies. These types of studies stand in contrast to efficacy studies which test an intervention under ideal and controlled conditions to maximise the likelihood of observing an effect if one exists. The justification for this approach is that our interest was to identify evidence on the effects of an intervention implemented under circumstances that approach ‘real- world’ practice. Although there exists broad agreement on the type of study design characteristics of effectiveness (pragmatic) studies and efficacy (explanatory) studies, there is currently no validated definition of ‘effectiveness studies’ (Treweek et al., 2009; Gerthlener et al., 2006; Singal et al., 2014). We, therefore, developed five criteria to help us distinguish more clearly between efficacy and effectiveness studies, drawing on two existing tools (Gartlehner et al., 2006; Thorpe et al., 2009). Studies were considered efficacy and excluded if they fulfilled at least one of the criteria outlined below: Research Objective: Is the study primarily designed to determine to what extent a specific technique, technology, treatment, procedure or service works under ideal conditions rather than attempt to answer a question relevant to the roll-out of a large programme? Population: Are the participants highly selected and therefore unrepresentative of the general population (Are strict inclusion and exclusion criteria used to enrol a homogenous population which may limit the generalizability of the results? e.g., 'college students that have a disease of interest’ or ‘participants that are more likely to adhere to the treatment’) Providers: Is the intervention primarily delivered by the research study team rather than trained laypersons (parents/ teachers/ community members/ NGOs) who don’t have extensive expertise? Delivery of intervention: Is the intervention delivered with a high degree of assurance of delivery of the treatment? (Is the delivery tightly monitored/ supervised by the researcher following specific protocols; Is adherence to the treatment monitored closely with frequent follow-ups?) ‒ Delivery of intervention: Are concurrent interventions restricted to the study population for a witnessed effect to be attributed to the intervention? Systematic reviews SRs were considered for inclusion if they clearly described the search, data collection and synthesis methods according to the 3ie database of systematic reviews protocols e.g., reported the criteria used for deciding which studies to include in the review, such as types of studies, participants/population, intervention(s), setting and outcome(s) (Snilstveit et al. 2014). We excluded non-systematic reviews, systematic reviews of the efficacy studies, qualitative reviews and descriptive reviews. Moreover, we only included reviews with an explicit focus on DHIs and a reasonably comprehensive search strategy e.g. no restrictions 8 SCOPE AND INCLUSION/EXCLUSION CRITERIA on language and publication status and searched for records from sufficient number of databases (at least two). ADDITIONAL CRITERIA Below are the general inclusion/exclusion criteria applied to both IEs and SRs: There were no restrictions on the population for the geographic region or income status of countries included in the map. Given the relatively recent state of the evidence base in this field and the fact that most programs/initiatives have started to flourish in the 2000s, we conducted the searches for IEs from 2000 onwards and SRs from 2014 onwards. We did not exclude studies based on the publication status This page is for collation purposes only METHODS We followed the standards and methods for EGMs developed by 3ie (Snilstveit et al., 2016). An EGM aims to establish what we know, and do not know, about the effects of interventions in a thematic area through systematic searching, screening and data extraction of all relevant completed or ongoing IEs and SRs (Snilstveit et al., 2016). SEARCH STRATEGY AND SEARCHING Due to the broad scope of this EGM, we implemented a sensitive search strategy primarily constructed by a combination of intervention and study design terms. We engaged an experienced information specialist to develop the search strategy. This was reviewed by internal 3ie staff as well as a representative of the World Bank and revised following the comments that were provided. The final strategy was translated according to the requirements and functionalities of different databases that were searched. The search strategies for each database are provided in Appendix 3. We searched a range of different sources of academic and grey literature, including 11 academic databases (a combination of general social science and health-focused databases), 13 repositories of impact evaluations and systematic reviews, and 13 specialist organizational databases and websites of bilateral and multilateral agencies. The complete list of sources is provided in Appendix 4. We also conducted backwards citation tracking of systematic reviews that met our inclusion criteria, as well as any relevant non- systematic reviews and guidelines identified to identify additional studies. REFERENCE MANAGEMENT AND SCREENING PROCEDURES The search results were imported into EPPI-Reviewer 4, a web-based software program for managing references and data. We used this platform to manage references, identify and remove duplicate studies as well as screen records for inclusion at title, abstract and full-text levels and conduct quality assurance. We trained all screeners in the implementation of the screening protocol. As part of the training, the team members (including expert reviewers) independently screened the same set of 100 abstracts and 30 full-text papers. We then compared the results, discussed any discrepancy in coding decisions and clarified the inclusion criteria when needed. At the title and abstract stage, we planned to combine ‘safety first’ single screening (with any study where the first screener is uncertain about inclusion assessed by a second (i.e., more senior) reviewer) (Shemilt et al., 2016) with EPPI-reviewer’s machine learning functionality to speed up the screening process. We started by screening a randomly selected set of around 1000 studies to provide a training set for the machine-learning algorithm and used the prioritisation functionality to prioritise studies for screening according to their likelihood of inclusion. However, due to the broad scope of this review, the prioritisation function was not helpful. We explored the potential of using other machine 11 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT learning strategies for reducing screening workload and speeding up the study selection process within EPPI-Reviewer, including by building a study classifier based on existing screening data, but none of them could achieve this without compromising the quality of data. As such, we screened all the records using the safety-first screening approach at both abstract and full-text stages. DATA EXTRACTION AND CRITICAL APPRAISAL We extracted meta-data from both IEs and SRs using standardised forms adapted from the general 3ie data extractions tools for EGMs. We captured bibliographic details, geographic information and substantive data, as well as standardised methods information and data on how studies addressed equity. Besides, we extracted data on interventions (type and description), outcomes (type and description), population and costs. We have included the data extraction template in Appendix 5. We piloted the tools and provided the training to all coders, which included extracting data from the same set of five studies to ensure consistency in coding and resolve any issues or ambiguities. A single researcher conducted the data extraction for each study; however, a sample (5%) was double coded to check for consistency. SRs were critically appraised following the 3ie systematic review database protocol, which draws on Lewin et al. (2009). The tool assesses the review on how the search, screening, data extraction, and synthesis was conducted and covers all the most common areas where biases are introduced. Overall, each SR was rated as low, medium, or high confidence. One reviewer conducted the initial critical appraisal, and a systematic review methods expert conducted a final review of all appraisals. The complete tool can be found in Appendix 6. It is important to note that we did not critically appraised primary studies, as this was beyond the scope of the project. ANALYSIS AND REPORTING We performed descriptive analyses such as frequencies and counts to describe the size and characteristics of the existing evidence base and identify major evidence gaps. The findings are presented in tables and figures as appropriate. We have provided data on interventions, outcomes, types of studies (and designs), the geographical location,and how studies addressed cross-cutting issues like gender and equity. We further conducted a systematic analysis of evidence gaps, focusing on (a) absolute gaps (no or few primary studies); (b) synthesis gaps (clusters of IEsbut no high-quality systematic reviews; and (c) other gaps (any gaps or in-balance in the evidence base, including for example type of studies, or geographical locations). The included studies were mapped onto the framework of interventions and outcomes and presented on an interactive platform which provides a graphical display of the evidence in a grid-like framework. This provides a visual display of the volume of evidence for intervention-outcome combination, the type of evidence (IE, SR, completed or ongoing), and a confidence rating for SRs. The final map is published on an online interactive platform that provides additional filters so that the users can further explore the available evidence, for example by country or population. 12 METHODS LIMITATIONS OF THE EVIDENCE GAP MAP This EGM has provided insight into the availability of rigorous evidence for DHIs, which can be used to inform policy-related decision making. However, like with any other empirical work, the EGMs have limitations, which must be considered both in the interpretation and application of the findings. By their nature, EGMs simply map out and categorise the evidence according to a pre- defined intervention-outcome framework. It neither synthesizes the evidence to provide effects sizes nor assess the effectiveness of an intervention, program or policy. In other words, it tells us about what evidence is there and not what the evidence says. As such, compared to SRs, the EGMs have relatively limited policy implications such as signposting high-quality evidence to policymakers and identifying gaps to be filled through evidence synthesis or IEs. Similarly, this EGMs does not say much about the overall quality of the DHIs evidence base as critical appraisal was restricted to SRs. Although some elements of quality control were embedded within the IE inclusion criteria (e.g., rigorous study designs), lack of critical appraisal on the internal and external validity of the studies means that not every IE included in this EGM is an important source of evidence due to possible biases Besides, EGMs often do not have search strategies that are as comprehensive as that of an SR. As such, there is a risk that this EGM missed some studies (i.e., subject-specific studies) and hence the findings may not reflect the richness of the DHIs evidence base. We minimised this risk by engaging an experienced information specialist during the development of the search strategy and searching for literature in several databases (including grey literature sources). Therefore, this EGM still provides a very good overview of the global DHIs evidence base. The findings should also be interpreted with caution due to restrictions on the date of publication. We included IEs and SRs published from 2000 to 2019 and 2014 to 2019, respectively. Thus, this EGM does not include all available evidence in this field as we excluded some of the records on the date of publication. Bearing in mind that digital health is a fast-growing field, these restrictions were necessary to ensure that the EGM captures contemporary digital interventions and health outcomes which would be relevant to the current issues. Since all health outcomes were included, we grouped the outcomes into broad categories, which were then used in the EGM framework. Although this classification was vital to ensure there was a manageable number of outcomes in the EGM, the broad outcomes limit our understanding of the evidence base. For instance, behaviour change outcome captures the change in the rate of smoking, condom use and breastfeeding, among others. Hence, it may not be possible to specify the distribution of the evidence across these specific outcomes. Nevertheless, we conducted a separate analysis to assess the distribution of the evidence across common health domains such as HIV/AIDS, mental health and NCDs. Finally, the records were screened for inclusion by individual reviewers at both abstract and full-text levels. We did not employ double screening in which each record is independently screened by at least two reviewers. Studies have shown that single screening is relatively less robust as it misses a substantial number of studies (Mahtani et DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT al., 2020; Waffenschmidt et al., 2019). Nevertheless, the safety-first approach that was used kept this at a minimum by allowing first screeners to refer to expert reviewers all papers that they were not sure about. Moreover, a random proportion (5%) of the records were checked by expert reviewers as part of the quality assurance and only a few issues were identified and addressed. PROCEDURES FOR REPRESENTATION OF EXCLUDED STUDIES We collected limited metadata from a sample of the excluded IEs to provide an indication of the characteristics of this body of evidence, including key issues that led to their exclusion. This section describes the inclusion criteria and the procedure that was used to select these studies. We considered for inclusion all IEs that were excluded during full-text screening for reasons of study design/methods. Hence, IEs excluded from the main analysis for any of the following reasons were not eligible: duplicate, does not have empirical data, full-text not accessible, date of publication, intervention not relevant and other reasons. Studies that have no empirical data are less likely to have relevant metadata needed for this analysis. Similarly, the focus of this EGM is on DHIs and hence studies about other types of interventions (e.g., non-DHI) were outside the scope of this work. We reviewed 5% of the eligible excluded studies. This gave us a realistic number of studies to assess given that around 5500 studies were excluded during full-text screening. Based on the inclusion criteria above, a total of 1,828 excluded studies were eligible and hence the sample size for this analysis was 91 studies. We used a simple random sampling approach to select these studies. We generated a unique ID for each study in Stata software and then randomly selected 91 studies from the pool of all eligible IE studies. The following metadata were collected from the studies: type of intervention, study/evaluation design, analytical methods and outcomes. We created a matrix capturing the metadata and provided a descriptive summary of the data. 14 FINDINGS SEARCH RESULTS Screening outcomes As shown in the PRISMA diagram (Figure 4.1), we retrieved a total of 63,014 records (both IEs and SRs) through searches in academic databases (n=62,684) and grey literature searches (n=330). After the removal of duplicates (n=17,142), we retained 45,872 records for the title and abstract screening. We excluded most of the records (n=40,374) during the title and abstract screening as they did not meet the inclusion criteria e.g., no digital intervention, qualitative study or ineligible quantitative study design. Since the date of publication cut off point was different for IEs (i.e., 2000) and SRs (i.e., 2014), we excluded most of the SRs at this stage because the general date limiter in our search strategy was set to 2000. Figure 4.1 PRISMA diagram Source: 3IE, World Bank. We retained 5,498 records for full-text screening after the title and abstract review. Of these, we could not retrieve full-text papers for 344 (6.3%) records mostly because the 15 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT papers were not available e.g., only conference abstract is published. Of the 5,154 full-text papers that we screened, 4,425 were excluded for various reasons as shown in the PRISMA diagram. The most common reasons for exclusion at this stage were irrelevant intervention (i.e., not digital), studies not meeting minimum reporting requirements and others (e.g., non-systematic reviews, qualitative reviews/studies and surveys). After the full-text screening, we included 729 completed and ongoing studies in this EGM (632 IEs and 97 SRs). We further critically appraised the completed SRs (not protocols) that were included and rated our confidence in the findings (i.e., high, medium or low) based on the methods and reporting procedures followed by the reviews. Publication of IEs and SRs over time Figure 4.2 depicts the number of IEs and SRs of DHIs published each year from 2000 to 2019 and 2014 to 2019, respectively. In general, the evidence base for IEs has substantially expanded over the last decade. For instance, the number of studies has increased steadily since 2011, reaching a peak of more than 70 (ongoing and completed) studies per year between 2014 and 2018. The fluctuation of ongoing and completed IEs is depicted by the blue and grey lines in the graph, respectively. The blue bars indicate the total number of IEs for each year, with 2017 being the year with the highest number of publications. Overall, of the 632 included IEs, 436 were completed and 196 were ongoing studies. Figure 4.2 Trends in the publication of DHIs IEs (n=632) and SRs (n=97) since 2000 and 2014, respectively 100 92 90 80 80 73 75 75 70 60 50 46 40 37 35 30 23 22 19 17 20 10 12 11 9 7 8 10 4 6 6 1 1 3 0 Total IE Total SR Completed IE Completed SR Ongoing IE Ongoing SR Source: 3IE, World Bank. Of the 97 SRs included in this EGM, 89 were completed while eight were protocols (i.e., ongoing). As can be seen, the total number of SRs published (orange bars) fluctuated from year to year between 2014 and 2017. The yellow and green lines of the graph show the volume of completed and ongoing SRs, respectively and together represent the total 16 FINDINGS number of SRs for each year. We have excluded from the graph IEs (n=41) and SRs (n=16) published in 2019 to avoid distorting the trend as the last database search for this EGM was conducted in April 2019. Thus, we missed all the studies and protocols published later that year. THE GEOGRAPHICAL AND ECONOMIC CONTEXT OF THE STUDIES We have included IEs conducted in 69 countries globally. However, most of the countries (64%) contributed less than 5 studies. The distribution of the IEs across countries is relatively uneven with the Unites States alone accounting for 32% (n=203) of the studies. The geographical location of the studies is captured by the global map in Figure 4.3 The map reveals an uneven and polarised distribution of IEs across regions and countries. Across the WHO Regions, the Region of Americans (n=233) and the European Region (n=188) have the highest number of IEs. These are followed by the Regions of Western Pacific (n=92), Africa (n=70), South-East Asia (n=47) and the Eastern Mediterranean (n=10). The number of studies conducted in high-income countries (n=490) are much higher than those conducted in middle-income (n=113) or low-income (n=29) countries. Most of the IEs (77%) were performed in high-income countries while the LMICs together contributed 23% of the studies (Figure 4.4). This likely reflects the gap in access to technology and innovations between rich and poor countries. Except for China (n = 31), only high-income countries have more than 20 IE studies on DHIs. Kenya and India are the only LMICs with over 10 IE studies. Figure 4.3 Location and number of impact evaluation studies globally (n=632) Source: 3IE, World Bank. DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Figure 4.4 Frequency of IEs by Income levels of the countries (n=632) Income level of the countries Upper middle income 47 Lower middle income 66 Low income 29 High income 490 0 100 200 300 400 500 600 Number of impact evaluations Source: INTERVENTION COVERAGE Figure 4.5 presents the distribution of IEs (blue) and SRs (orange) across the interventions included in this EGM. The intervention framework comprises four DHIs categories divided further into 28 specific types of DHIs. The IEs in this EGM covered 19 of the 28 interventions, which means 9 are yet to be rigorously evaluated. Nearly all interventions covered by the IEs are for clients (n=403) or healthcare providers (n=224). There are only three and two IEs covering health systems management and data science interventions, respectively. Of the 28 specific types of DHIs, the SRs covered 11 of them. We further observed that SRs only covered clients (n=59) and healthcare providers’ (n=38) interventions. We did not find any SR that covered DHIs related to health systems management or data services. By type of intervention, over half of the IEs covered targeted digital health communication (n=324). This is followed by telemedicine, which was covered by 92 IE studies. Fifty-seven IE studies evaluated interventions that support decision making by healthcare providers. The DHIs that facilitate personal health tracking were evaluated by 49 studies. The interventions on the training of providers and management of client health records were evaluated by 28 and 20 IEs, respectively. The other interventions covered by at least 5 IEs are, in order of frequency: untargeted digital health communication (n=12), on-demand information services to clients (n=9), healthcare provider communication (n=9), prescription and medication management (n=9), and client-to-client communication (n=8). Similarly, more than half of the SRs (n=47) evaluated targeted digital health communication (e.g., SMS reminder). The other relatively common interventions covered by the SRs include telemedicine (n=14), training for healthcare providers (n=9) and healthcare provider decision support (n=8). The other included SRs covered personal health tracking (n=6), untargeted digital health communication (n=5), prescription and medication management (n=3), laboratory and diagnostics imaging (n=1), and client-to- client communication (n=1). 18 FINDINGS Figure 4.5 Interventions reported by IEs (n=632) and SRs (n=97) Data services Data exchange and interoperability Location mapping Data coding Data collection, management, and use 2 Facility management Equipment and asset management Health systems management Health financing Civil registration and vital statistics Public health event notification 1 Supply chain management 2 Human resource management Laboratory and diagnostics imaging 1 1 Prescription and medication management 3 9 9 Healthcare providers Training 28 Scheduling and activity planning 5 Referral coordination 1 Healthcare provider communication 2 9 Telemedicine 14 92 Healthcare provider decision support 8 57 Client health records 1 20 Client identification and registration 2 Client financial transactions On-demand information services to clients 9 Citizen-based reporting 1 Client Personal health tracking 6 49 Client-to-client communication 1 8 Untargeted digital health communication 5 12 Targeted digital health communication 47 324 0 50 100 150 200 250 300 350 SRs IEs Source: 3IE, World Bank. OUTCOME COVERAGE The studies included in this EGM have reported both economic and health-related results or outcomes of various types of DHIs, which covers the whole causal pathway from outputs to a sequence of outcomes to impacts. Figure 4.6 presents all the outcomes reported by the included IEs and SRs. DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Figure 4.6 Frequency of outcome reporting in IEs (n=632) and SRs (n=97) SRs units) and beliefs Others IEs summary Knowledge SRs IEs (aggregated/ SRs Health status IEs SRs (natural Health status units) IEs SRs outcome Healthcare Quality of care IEs change satisfaction therapeutic) (therapeutic) utilisation SRs IEs Process SRs IEs outcome Process SRs (non IEs Client or SRs provider IEs Behaviour SRs IEs Economic outcomes SRs IEs 0 50 100 150 200 250 300 Clients Healthcare providers Health system managers Data services Source: 3IE, World Bank. Economic outcomes The economic outcomes have been reported in 159 IEs and 19 SRs, representing 25% and 20% of the included studies, respectively (Figure 4.6). The majority of the IEs and SRs that reported these outcomes covered DHIs for clients (n=119 IEs and n=11 SRs) and healthcare providers (n=45 IEs and n=8 SRs). Only three IEs and no SR covering DHIs for 20 FINDINGS health system managers reported economic outcomes. Moreover, none of the studies on DHIs for data services included any cost data. The economic outcomes reported by the studies are further organised into simple, intermediate and summary/impact based on the DHI’s TOC causal pathway. The simple economic outcomes refer to reporting of cost data without any reference to health or well- being outcomes e.g., cost incurred/saved. However, some make reference to outputs such as cost per user or persons reached. Studies reporting simple economic outcomes are generally variations on forms of costing analyses. Intermediate and summary economic outcomes link intervention cost data to some health outcomes. The difference between the two is the level of aggregation and finality of the health outcome. Intermediate economic outcomes focus on immediate or natural outcomes such as cost per infection averted whereas the summary outcomes result from impact level analysis such as cost per life saved or cost per quality adjusted life-year gained. Highly aggregated outcomes, such as net benefits calculated under a cost benefit analysis framework would also be classified as a summary outcome, however no cost-benefit analyses were identified in the included studies. Figure 4.7 presents the distribution of the outcomes across the three categories by intervention types. As can be seen, most studies reported simple (n=153) and summary (n=39) economic outcomes. More than half (n=93, 60%) of the simple economic outcomes relate to studies that only reported the cost of programme implementation and the cost borne by programme participants such as the cost of making phone calls, sending texts or routine care. For instance, three studies documented the mean cost of treatment and control groups and also looked into the mean difference across the groups (Dinesen et al., 2012; Mailuhu et al., 2015 and Kao et al., 2016). The bootstrapping method has been applied in some studies to generate confidence intervals since cost data is generally not normally distributed. For instance, McCrossan et al. (2012) employed non-parametric bootstrapping to analyse the difference in healthcare costs between the study groups. The remaining studies (n=61, 39%) under this category utilised summary metrics that gauge cost per output e.g., a simple incremental cost effectiveness ratio (ICER). For instance, Mark et al. (2017) estimate cost savings per hospitalized beneficiary from potentially avoided readmissions attributable to event notifications. Most of the reported summary and intermediate economic outcomes compare costs with health benefits in natural units or aggregated units such as quality-adjusted life-years (QALYs). For instance, Band et al. (2016) examines the cost-effectiveness of adding the HOME BP intervention into primary care for the self-management of hypertension in comparison to usual care and estimated cost per unit reduction in systolic blood pressure (a “natural” unit) and a related study reported cost per smoking quitter (Harrington et al., 2016). Udsen et al (2014) delineate that their proposed study would compute incremental cost-effectiveness by using utility scores from EQ-5D health-related quality of life questionnaire from which QALYs would be calculated. Similarly, Mudiyanselage et al. (2019) calculated the incremental cost-effectiveness ratio (ICER) using differences in costs between treatment and control group and health-related quality of life derived using ANCOVA. DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Figure 4.7 The distribution of the economic outcomes (n=178) Summary economic 31 7 0 outcomes or impact Intermediate economic 9 5 0 outcomes Simple economic 38 21 1 outcomes 0 10 20 30 40 50 60 70 Clients Healthcare providers Health system managers Data services Source: 3IE, World Bank. The standard methodology of an EGM is to identify both completed studies and protocols as this identifies the existing and soon-to-be published outcomes. A significant proportion of economic evaluation outcomes, however, are detailed in protocols only, with a final study reporting the economic outcomes not identified in this search strategy. This may be due to a lag between protocol development and publishing of economic outcomes, or a systematic problem achieving final publication of on economic evalution despite signalling the intent to identify economic outcomes in the trial protocol. Health outcomes We found an unequal distribution of IEs across the health outcomes reported by the included studies. As presented in Figure 4.6, the most frequently reported health outcomes in IEs were health status (natural units) (n=284), behaviour change (n=242) and process outcome (therapeutic) (n=242). Most of the other outcomes were less frequently covered. For instance, relatively few IEs examined healthcare utilisation (n=137), client/provider satisfaction (n=113) and process outcome (non-therapeutic) (n=85). A limited number of studies reported quality of care (n=60), health status (aggregated/summary units) (n=77) and and knowledge and beliefs (n=49) as outcomes of DHIs. Across the health outomes, most of the IE outcomes are linked to client and healthcare provider interventions. For instance, 1101 and 387 IE outcomes were for the client and healthcare provider related interventions, respectively. Only 15 and 3 IE outcomes were for health system management and data service interventions. As with IEs, there is uneven distribution of SRs across the reported health outcome (Figure 4.6). The top 3 frequently measured outcome categories were health status (natural units) (n=40), behaviour change (n=35) and process outcome (therapeutic) (n=31). Client/provider satisfaction and health status (aggregated/summary units) were covered by 16 SRs each. The outcomes of quality of care, process outcome (non-therapeutic), healthcare utilisation and knowledge and beliefs were least measured health outcomes as they were each covered by less than 10 SRs. For almost every health outcome, the largest share of SRs covered outcomes related to the DHIs for clients (n=135). There is a lack of 22 FINDINGS outcome data among SRs on interventions concerning health systems management and data services. The health outcomes reported by the studies can also be organised along the DHI TOC casual pathway from outputs through impact. For this analysis, outputs comprised process outcome (non-therapeutic) as well as knowledge and beliefs. Intermediate outcomes include behaviour change, client/provider satisfaction, health status (natural units), process outcome (therapeutic) and healthcare utilisation. The final outcomes or impact consist of health status (aggregated/summary units) and quality of care. Based on this classification, most outcomes reported by the included studies are intermediate (n=2,306), followed by impact (n=342) and outputs (n=296). Figure 4.8 shows the distribution of the reported outcomes along the ‘output-outcome-impact’ TOC by intervention types. Figure 4.8 Distribution of health outcomes along the theory of change by intervention types (n=729) Impact Intermediate outcomes Output 0 500 1000 1500 Clients Healthcare providers Health system managers Data services Source: 3IE, World Bank. FREQUENCY OF ARTIFICIAL INTELLIGENCE USE AI is defined as the “use of computers for automated decision-making to perform tasks that normally require human intelligence”2. We found 84 IEs (13%) reporting interventions that utilised some form of AI. This implies that 87% of the studies did not use AI (n=576). Of the 84 IEs that covered AI, over half (n=47, 56%) covered AI interventions for clients, 36 (43%) for healthcare providers, one (1%) for health system managers and none for data services. A very small proportion of the included SRs (4%, n=4) covered interventions with AI elements. Hence, AI was not utilised in 96% (n=94) of the interventions in SRs. Three SRs reported AI interventions for healthcare providers and one for clients. The typical AI building block comprises of the following categories: data analytics (e.g., natural language processing), information processing (e.g., machine learning) and 2 Artificial Intelligence in Global Health: Defining a Collective Path Forward (https://www.usaid.gov/cii/ai-in-global-health) DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT automated actions (e.g., speech generation)2. Figure 4.8 shows the distribution of the AI interventions reported in included IEs across these categories. As shown, most AI- powered interventions in IEs were used in information processing (n=67). The use of AI in data analytics (n=9) and automated actions (n=7) is only present in a handful of studies. All the AI interventions reported in SRs were used in information processing. Across these building blocks, AI has been used to address various health system challenges. Finocchario-Kessler et al. (2015) evaluated the impact of HIV Infant Tracking System (HITSystem), a web-based tool that tracks the care of infants enrolled in early infant diagnosis (EID) so that those who drop out of care can be easily identified and targeted for outreach. Using the infant’s date of birth, the HITSystem configures automated electronic alerts for time-sensitive interventions to optimise care for HIV-exposed infants. Studies have also documented the use of AI that incorporated automated text-message alerts or advice for patients. For example, one of the studies looked into the use of SMS- Text Adherence support (StAR) that delivered automated text messages on blood pressure to participants to help them maintain and improve treatment adherence and blood pressure control (Bobrow, 2016). AI is also used in computerised decision support system (CDSS) or the use of prediction algorithms to aid clinicians with inpatient care (Sheibani, 2018; Van Der Gugten, 2015). Trinanes et al. (2015), for instance, underscore in their SR that the technology could serve to optimise care of depression in various scenarios by providing recommendations based on the best evidence available and facilitating decision-making in clinical practice. HEALTH DOMAINS We further explored the use of DHIs in management or prevention of common morbidities and other health-related issues. The analysis included 10 health domains (Figure 4.9) whose selection was informed by data from the WHO’s Global Health Observatory (GHO)3. As shown in the Figure, the IEs are disproportionately distributed across the health domains. The most featured health domain is non-communicable disease (NCD), which was covered by over half of the studies (n=343). Other health domains with high coverage in IEs include child health (n=66); infectious, parasitic and vector-borne disease (n=75) and mental health (n=76). Maternal health (n=44), nutrition (n=38), family planning and reproductive health (n=32), HIV/AIDS (n=26), and unintentional injury (n=24) were less frequently covered domains. Violence as a health domain lags, it was covered by only two studies. As with IEs, we found an unequal distribution of SRs across the health domains. Most of the SRs covered interventions for NCDs (n=36), mental health (n=11) and nutrition (n=10). The domains least covered by the SRs were infectious, parasitic and vector-borne disease (n=5); unintentional injury (n=5); family planning and reproductive health (n=4); child health (n=4); and maternal health (n=4). There were no SRs that focused on violence and HIV/AIDS domains. 3 Global Health Observatory (GHO) data: https://www.who.int/data/gho 24 FINDINGS Figure 4.9 Common health domains covered in IEs (n=632) and SRs (n=97) Systematic reviews Impact evaluations Violence 0 2 Unintentional injury 0 26 HIV/AIDS 4 32 Family planning and reproductive health 4 45 Nutrition 5 24 Maternal health 4 66 Child health 10 38 Infectious, parasitic and vector-borne disease 5 75 Mental health 11 76 Non-communicable disease 36 343 0 100 200 300 400 Number of studies Source: 3IE, World Bank. EVALUATION METHODS The graph in Figure 4.10 displays the evaluation methods that were employed by IEs included in this EGM. As shown, most studies (n=583, 91%) evaluated the interventions through RCTs. The rest of the studies used quasi-experimental designs, including statistical matching (n=24), difference-in-difference (n=13), interrupted time series (n=13), fixed effect estimation (n=4) and regression discontinuity (n=1). Figure 4.10 Impact evaluation methods (n=632) Impact evaluation study designs Regression discontinuity design 1 Fixed effects 4 Interrupted time series 13 Difference-in-difference 13 Statistical matching (including PSM) 24 Randomised controlled trial 583 0 100 200 300 400 500 600 700 Number of impact evaluations Source: 3IE, World Bank. DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT We further explored the specific methods used in economic evaluation studies. Figure 4.11 shows how many times each of the major economic evaluation methods were used among included studies. Of the 159 studies that reported cost data associated with the DHIs, 93 did not adopt any specific evaluation method as they simply reported the cost incurred or saved. Cost-effectiveness analysis was conducted in 63 studies, which makes it the most frequently used economic evaluation method. A substantial number of studies also reported using cost utility analysis (n=28). Cost minimisation and cost consequence analyses were reported in one and two studies, respectively. None of the included studies used cost benefit analysis. Figure 4.11 Economic evaluation methods conducted by the studies (n=187) Cost only/Cost analysis 85 8 Cost minimization analysis 1 Cost consequence analysis 2 Cost effectiveness analysis 55 8 Cost utility analysis 26 2 Cost benefit analysis 0 0 20 40 60 80 100 Impact evaluation Systematic review Source: 3IE, World Bank. CONFIDENCE RATING FOR SYSTEMATIC REVIEWS We give each review a confidence rating of low, medium or high based on the critical review of their methods, which included evaluating the extent of biases in the methods that the reviewers employed to identify, screen, extract, analyze and report data (see section 3.3). This rating should be considered as our guidance for how much confidence users may have in the findings of the SRs in this EGM. More than half of the SRs (n=51, 56%) were given a low confidence rating and a few were rated as high (n=15, 17%) or medium (n=24, 27%) confidence. This means that 83% (n=75) of the SRs had minor to major methodological issues that affected our confidence in their findings. The key issues that affected the confidence rating include a lack of risk of bias assessment and limitations in the comprehensiveness of the search strategy, or the procedures used to avoid bias in study selection and data extraction. 26 FINDINGS EQUITY AND GENDER FOCUS Figure 4.12 presents the different ways equity or gender has been considered in both IEs and SRs. The most prominent way equity or gender has been considered is through interventions targeting a specific disadvantaged or marginalised population e.g., rural residents (n=103). A few IE studies stratified the data by sex (n=42) while others consider the effects of the intervention on equity or the gender-sensitivity of the outcomes (n=6). Some IE studies performed subgroup analyses with one or more equity dimensions (n=29). Only a few studies have reflected equity or gender-sensitivity through a cogent theory of change (n=9) or analytical framework (n=3). The majority of the IEs (n=572) did not consider any gender or equity issue. Just like IEs, most SRs did not consider equity or gender in any way (n=81). A small number of SRs that considered equity or gender had interventions that targeted a specific marginalised or vulnerable population (n=12), integrated gender inequality considerations in their theory of change (n=3), included sub- group analysis of gendered inequality (n=1) and used gender or equity sensitive methodologies in their research process (n=1). Figure 4.12 Consideration of equity or gender (n=729) Sex-disaggregated data 42 Research process informed by gender and/ 6 or equity Measures effects on gender and/ or equity 6 outcome Gender and/or equity-sensitive analytical 3 frameworks Equity focus Approach to ethics informed by gender and/ 1 or equity considerations Sub-group or population analysis by gender 29 and/ or equity (trigger) 1 Gender and/ or equity sensitive methodologies – other 1 Theory of change 9 3 Intervention targeting a specific vulnerable 103 population (s) 12 Does not address gender or equity 572 81 0 200 400 600 800 Number of studies Impact evaluations Systematic reviews Source: 3IE, World Bank. CHARACTERISTICS OF THE EXCLUDED STUDIES In this analysis, we included 91 IEs that were excluded from the EGM during full-text screening due to methodological reasons. The analysis focused on intervention type, evaluation design, analytical approaches and outcomes of the excluded studies. We DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT provide a landscape of these IEs because although excluded from the EGM, they are relevant to the boarder workstream. Intervention types Overall, most of the studies (n=34, 37%) covered targeted digital health communication followed by telemedicine (n=18, 20%) and training of health providers (n=12, 13%). Most of the studies (97.8%) focus on clients and healthcare providers, while one study each for health system managements and data services. Notably, as it can be seen in Figure 4.13 below, the overall distribution of the studies across the 28 intervention types is very similar to the results of included IEs. Figure 4.13 Digital health intervention reported by excluded studies Data services Data exchange and interoperability Location mapping 1 Data coding Data collection, management, and use Facility management Equipment and asset management Health systems management Health financing Civil registration and vital statistics Public health event notification Supply chain management 1 Human resource management Laboratory and diagnostics imaging 4 Prescription and medication management 2 Healthcare providers Training 12 Scheduling and activity planning 1 Referral coordination 2 Healthcare provider communication 5 Telemedicine 18 Healthcare provider decision support 8 Client health records 7 Client identification and registration Client financial transactions On-demand information services to clients 3 Citizen-based reporting Client Personal health tracking 11 Client-to-client communication 3 Untargeted digital health communication 2 Targeted digital health communication 34 0 5 10 15 20 25 30 35 40 Source: 3IE, World Bank. 28 FINDINGS Evaluation design Regarding evaluation design, 68 studies are experimental, and 9 used quasi-experimental design, including difference-in-difference and statistical matching. Mixed methods design was used in 22 studies, most of which are using qualitative methods, such as interviews and focus group discussions to understand the target population’s perspective, as supplementary to the quantitative analysis. Many studies used the form of interview to conduct survey, which is not counted as mixed method if no qualitative data is collected or open-ended questions asked. Most IEs were excluded because they used RCT design but failed to apply robust methodological approach. In the screening stage, each study was checked in sequence on whether: it has a control group, it is an efficacy study, it has selection bias, and it meets all required reporting standards for the study design it has used (Appendix 2). There are four primary reasons why studies were excluded at this stage (Figure 4.14). Firstly, absence of an appropriate control group. Instead, before and after analysis with the same population appeared for many cases without other methods to control potential confounders. Secondly, efficacy trials are widely used to measure the impact. From this sample, 16 studies were excluded for being efficacy trials. The study populations are restricted to a specific setting or to homogeneous groups without any plausible explanations. Thirdly, limited studies applied balance tests and control for confounders. With the intervention and control group, not so many studies have conducted tests on key observables after randomization. Lastly, in many studies, the sample sizes are too small to give a valid measure or even to conduct statistical analysis. The adopted EGM standard has a minimum total sample size of 40 participants for RCT and 80 for quasi-RCT. However, there are 10 studies with less than 40 participants. Figure 4.14 Reasons for exclusion related to evaluation design 35 30 29 26 25 20 20 16 15 10 5 0 No comparison Efficacy study Selection Minimum reporting group bias/Confounding requirements not met Source: 3IE, World Bank. DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Analytical methods For analytical methods, some patterns were observed during the review process, which can be categorized as: • Descriptive analysis, represented as frequencies and percentages/counting numbers of specific behaviors, mean and standard deviation, range • Balance analysis is conducted at baseline to compare basic characteristics among treatment and control groups if applicable • Difference analysis (and efficacy analysis) are methods and models explaining how the study measures the difference among groups or effectiveness of the intervention • Association/Correlation analysis is used to test the relationship among variables • Sensitivity analysis, feasibility analysis, factor analysis (predictor, barrier, influential factors) and subgroup analysis if mentioned explicitly in the study Most studies applied descriptive analysis (n=81), and some of them used simple difference to measure the intervention impact. We found that 18 studies used balance tests in different rigor. For instance, T-test, chi-square test and Fisher's exact tests are commonly used to check if there is statistically significant difference of control and intervention characteristics. These methods are also commonly used to identify difference in pre-post intervention or between intervention and control groups. Other ways to measure effects are analysis of variance (ANOVA) and different regression models with or without adjustment. Some studies conducted supplementary association or correlation analysis to identify if the variables are interrelated before analysing the effects. And in some cases, sensitivity and specificity analysis is applied, which is common to measure the effectiveness of diagnosis methods. The common lacks in most of studies excluded as “Minimum reporting requirements not met” are simple analytical approaches, such as simple difference among two groups without any statistical measurement, missing descriptions of the randomisation process and key analytical results. Outcomes As can be seen in Figure 4.15 below, the most commonly reported outcomes in the excluded studies are health status (natural units), process outcome (therapeutic), and client/provider satisfaction. Economic outcomes and Health status (aggregated/ summary units) are least reported outcomes among the excluded studies. No study was excluded based on outcomes. 30 FINDINGS Figure 4.15 Distribution of the outcomes reported by excluded studies Other 9 Health status (aggregated/summary units) 5 Quality of care 26 Process outcome (non-therapeutic) 26 Client/provider Satisfaction 29 Healthcare utilisation 11 Behaviour change 27 Process outcome (therapeutic) 31 Economic outcomes 6 Health status (natural units) 32 0 5 10 15 20 25 30 35 Source: 3IE, World Bank. This page is for collation purposes GAP ANALYSIS There is a steady increase in number of IEs and SRs evaluating DHIs. The focus of the studies varies in many ways, including type of DHI, setting or location, targeted population, health domains and outcomes of interest. As such, although the DHI evidence base has significantly expanded over the last few decades, this EGM has identified several evidence gaps in the literature. Specifically, the map reveals two major types of gaps: ‘absolute’ gaps, where no or few primary studies have been conducted, and ‘synthesis’ gaps, where no SR exists despite a cluster of IEs or the quality of the reviews is substandard or the existing SRs are dated. The evidence gaps are shown in Figure 5.1, which presents IEs and SRs at the intersection(s) of interventions and outcomes they considered. For instance, the bubbles in the left upper corner represent the volume of IEs and SRs that measured economic outcomes in targeted digital health communication interventions. An interactive online map that graphically presents the findings of this EGM can be accessed here. The grey and coloured bubbles denote IEs and SRs, respectively. The size of the bubbles corresponds to the relative size of the evidence base (i.e., the number of IEs or SRs), whereas the colours of the SRs signify the confidence ratings (i.e., green = high, orange = medium and red = low). ABSOLUTE EVIDENCE GAPS The evidence base is skewed towards high-income and Western countries with more limited evidence from LMICs. In terms of location, there is limited evidence from the WHO regions of Africa, South-East Asia and Eastern Mediterranean as most of the studies (66%) were conducted in the two regions of America and Europe. Similarly, very few studies were conducted in LMICs and a huge gap in the volume of studies conducted in rich and poor countries has been noted. For instance, over three-quarters of the studies were conducted in high-income countries with the USA alone contributing one-third of the included IE studies. The proportion of studies performed in the USA is greater than that of studies conducted in LMICs combined. Limited access to or slow diffusion of technology and lack of research capacity in LMIC due to resource constraints are possible contributing factors (Olawuyi, 2018). However, this could also be due to publication bias. This observation is worrisome especially considering that LMICs shoulder the greatest burden of global health issues and would, therefore, benefit from innovations that come with DHIs. 33 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Figure 5.1 Screenshot of DHI evidence gap map available online Source: 3IE, World Bank. 34 GAP ANALYSIS There is limited evidence of DHIs for health system or resource managers and data services. Across the four intervention categories, almost all studies evaluated interventions for clients and healthcare providers. We found significant evidence gaps in the other two categories, which were together covered by five studies only. Of the seven and four specific types of DHIs for health system managers and data services, only two and one have been evaluated, respectively. These two DHI categories have received very little attention in the literature probably because their direct link to patient outcomes is not as clear as that of DHI for clients and healthcare providers. There is also an increased focus on patients/clients-centred care model worldwide and digital health technologies are being used to facilitate or support its implementation (Papavasiliou et al., 2020). This suggests there are more incentive to invest in DHIs that address direct patient/client issues. Similar gaps have also been identified in outcomes. Most of the studies measured outcomes concerning service users. There is a lack of studies that have reported health systems related outcomes such as workforce development, supply chain management and governance. Notwithstanding the large volume of IEs covering clients and providers’ DHIs, the evidence base within these intervention categories is unevenly distributed. For example, there is limited literature on DHIs that empower clients to take lead on health issues affecting them through, for example, citizen reporting or client-client communication. The focus of most studies is largely on top-down or externally led interventions such as targeted communication interventions. Although interventions that sufficiently engage users are more effective, they are, however, less preferred from the supply point of view because they are relatively time consuming and expensive (Chegini et al., 2019). For the healthcare providers, there is a heavy focus on DHIs concerned with service delivery e.g., telemedicine or decision support systems. Interventions that support the providers in planning or coordination of health services (e.g., referral or activity scheduling) have received relatively low attention. As already noted, this is likely because the latter has weaker connections to health outcomes. Despite the potential role of AI in strengthening health systems and improving the quality of healthcare, we found a shallow evidence base for AI. The number of studies that evaluated AI-powered interventions is extremely low. In total, only 13% (n=83) of the studies covered interventions that incorporated AI. Since some scientific innovations cannot be subjected to evaluation approaches adopted in this EGM (e.g., counterfactual analysis), the dearth of evidence could be due to methodological disparities. Other possible reasons might be slow adoption of AI in the health sector or long and rigorous vetting processes often required in the medical field. Also, publication bias cannot be ruled out. Not many studies have covered critical global health domains such as maternal health, nutrition, HIV/AIDS and family planning/reproductive health. Not more than 45 studies covered each of these domains compared to 343 that covered NCDs alone. Most of IEs were concerned with the NCDs and there is also a substantial focus on mental health, infectious diseases and child health. The plausible explanation for this evidence gap is that most of the in-frequently covered domains predominantly concern LMICs, where very few DHI studies have been conducted. For instance, HIV/AIDS and maternal DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT mortality disproportionately affect LMICs4. Thus, the coverage of these domains is likely to improve as more IEs are being implemented in LMICs. Nevertheless, this remains an important evidence gap in DHI literature. The impact of DHIs on long-term health outcomes such as mortality, quality of life and healthcare has been under-explored. The focus of most studies was on short-term and directly measurable health outcomes e.g., number of clinic visits. For instance, the top three frequently measured outcomes in IEs (i.e., health status (natural units), process outcomes (therapeutic) and behaviour change) are all intermediate health outcomes. In fact, there is a huge gap in the number of studies that have reported the intermediate and long-term outcomes such as health status (aggregated/summary units). However, this gap is not surprising given that long-term health outcomes are relatively difficult and expensive to measure. For economic outcomes, there are evidence gaps in both intermediate and summary outcomes as most of the studies have simply reported cost incurred/saved. Data and analysis of costs of interventions are infrequently reported. For example, about 80% (n=551) of the studies did not report any cost data and out of those that reported nearly half (n=79, 45%) simply provided cost data without performing any further investigation such cost-effectiveness analysis. Availability of relevant and usable costing data is as important as evidence on intervention effects. For instance, an intervention may be found to be effective, but if it is too expensive to scale-up or utilise resources inefficiently it may have limited feasibility as a policy option. Hence, the paucity of cost data limits the recommendations that can be made using existing data. Synthesis evidence gaps We have also mapped a huge body of SR evidence published between 2014 and 2019. The reviews cover various interventions and outcomes across health domains, regions and beneficiaries, among others. The EGM reveals many areas of interest with a relatively large number of relevant IEs but no SRs or the quality of the reviews is not satisfactory, or the SRs are dated. This section describes the specific DHI areas with potential for evidence synthesis. There is neither up to date nor high-quality SRs of personal health tracking interventions even though many IE studies have addressed this topic. Most studies covered the following outcomes: health status (natural units), healthcare utilisation, health status (aggregated/summary units), client/provider satisfaction, economic outcomes and behaviour change. We identified a few SRs (n=4), but none of them is rated as high confidence due to methodological limitations. There is one ongoing review of personal health tracking covering four outcomes of this EGM, which may address some of the synthesis gaps. Nonetheless, the ongoing review is not considering process outcome (therapeutic), economic outcomes and healthcare utilisation, which have a cluster of 12, 18 and 18 IEs, respectively. There are opportunities for possible syntheses relating to the effects of client health records and healthcare decision support system interventions. A substantial number of studies have explored the effects of client health records on healthcare utilisation, quality 4 Global Health Observatory (GHO) data: https://www.who.int/data/gho 36 GAP ANALYSIS of care and health status (natural units), but no high-quality SR exists. We identified one SR covering economic outcomes, health status (natural units) and process outcomes (non-therapeutic), but owing to poor methodologies, this SR has been rated as low confidence. Another area for potential synthesis is the link between practitioner decision support systems and healthcare utilisation as well as process outcome (non-therapeutic). These intersections have at least nine IEs, but no SR has been performed. On the same, there is a cluster of 24 and 11 IE studies falling under process outcome (therapeutic) and quality of care, respectively but no high confidence SR is available. The effects of telemedicine on process outcome (therapeutic), healthcare utilisation, health status (aggregated/summary units), health status (natural units), knowledge and beliefs and process outcome (non-therapeutic) present additional synthesis gaps for future reviews. Telemedicine is one of the most common interventions with a large body of evidence across all the outcomes of interest in this EGM. Most of the studies have measured outcomes of health status (natural units), process outcome (therapeutic) and healthcare utilisation. We identified a cluster of completed and ongoing reviews on telemedicine, but only two of them (under the outcome categories of client/provider satisfaction, economic outcomes and behaviour change) are rated as high confidence. For instance, 62 IE studies covered health status (natural units), but no high confidence review is available. There are also eight and 10 IEs under process outcome (non-therapeutic) and knowledge and beliefs respectively, which are yet to be synthesised. The dearth of evidence or high confidence SRs indicates gaps to be filled by rigorous evidence synthesis that can provide reliable evidence to inform policy and practice. Synthesis gaps also exist among targeted digital health communication interventions as most of the SR have important or major limitations. This is the most covered type of DHI in the literature as multiple studies have explored the link between targeted digital health communication and healthcare utilisation (n=56), process outcome (non-therapeutic) (n=47), health status (natural units) (n=134) and client/provider satisfaction (n=67). Although some SRs have been conducted to assess the effects of the intervention on these outcomes, none of them is updated or rated as high confidence. Therefore, these are areas where high-quality syntheses are needed to improve the evidence base. This page is for collation purposes CONCLUSIONS AND IMPLICATIONS FOR FUTURE RESEARCH We have systematically mapped rigorous evidence from impact evaluations and systematic reviews assessing the effects of DHIs across a range of different outcomes. The evidence base is relatively large and increasing and we identified over 600 IEs and almost 100 SR, published over the last two decades. However, the distributions of the studies and reviews are highly disproportionate across a range of parameters. In terms of location, most of the evidence is from high-income countries as less than 25% of the data came from LMICs. Most IEs and reviews covered interventions for clients/patients and healthcare providers. DHIs for strengthening health system management and data services have had very little coverage in the literature. Similarly, only a tiny share of the included studies covered interventions that incorporate AI. The focus of the majority of studies has been on NCDs. Other health domains such as HIV/AIDS and maternal health that could equally benefit from DHIs are rarely covered. The EGM clearly show areas with sufficient, limited, non-existent, high/low-quality and updated/dated evidence for DHIs. Exploration of the findings from this EGM and the quality of the existing evidence presented could facilitate or inform decisions. For instance, policymakers can explore evidence of the effects of DHIs to inform strategy development, whereas funders and researchers can examine the priority areas for future research and funding in this field. Assuming 5% sample is representative of the 5500 studies excluded during full-text screening, then this means there are a substantial amount of studies in circulation that might not be reliable for decision making. This highlights the need for (i) improved methods and (ii) consistent quality appraisal of evidence before it is used in decision making. Also, since the distribution of studies in sub-sample across intervention types is similar to the main results, there is no indication for intervention-related factors that are leading to poor methodological approach amongst published studies. However, is it likely that there are intervention type factors that limit publication in the first place given very low numbers of impact evaluaiton in particular intervention types. Based on these and other observations in this EGM, we believe that research or reviews in the areas summarised below could improve the global DHIs evidence base. • Lack of evidence on DHIs for health system management, data services and other areas could be because the concerned interventions have not been widely adopted for use or that they have been adopted but the evaluation has not been prioritised by researchers or the evaluation methods used are not rigorous. As such, there is a need to explore the availability and possible barriers to assessing the interventions. This is also a potential area for future IEs to focus. • A small proportion of the studies are from LMICs. We assume this is not only about a lack of data but also reflects a broader issue of access to digital innovations in health in LMICs as well as publication bias. Since LMICs shoulder the greatest burden of 39 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT global health issues and experiences relatively high number of health system challenges (e.g., poor access to health services and information), the region stands to benefit more from digital health solutions. Therefore, more investment in digital health research, equipment/resources and training is recommended. • It is evident from the findings that most DHIs were used to address NCDs. Evidence of the effects in other priority global health issues such as HIV/AIDS, maternal health and nutrition is limited. These health domains should be considered in future studies to improve the availability of evidence in this field. • Future studies on DHIs should consider measuring long-term health and economic outcomes, which are relatively neglected in the current literature. These outcomes include those relating to the quality of care, quality of life, survival rate and summary economic outcomes. • The role of AI in many sectors is on the rise. However, our data show that not many DHIs incorporate AI components. This is one of the areas with massive evidence gaps that could be addressed by future studies. • There is a limited economic evaluation evidence base for DHIs, which could limit decisions on policy, programming and scale-up. One potential mechanism to improve conduct of economic evaluation is development and promotion of standardised, high- quality economic evaluation methodological guidance tailored towards the DHIs. • Main synthesis gaps to be considered include areas where there is a large body of IEs but no SR has been performed e.g., effects of provider decision support systems on healthcare utilisation or process outcome (non-therapeutic). Areas with dated or low confidence reviews should also be prioritised e.g., impact of telemedicine on health status (natural units) or process outcome (therapeutic). These syntheses can provide rigorous evidence of the effects of various DHIs, which could improve the evidence base. 40 REFERENCES Chegini, Z., et al., Exploring the barriers to patient engagement in the delivery of safe care in Iranian hospitals: A qualitative study. Nursing open, 2019. 7(1): p. 457-465. Gartlehner, G., Hansen, R.A., Nissman, D., Lohr, K.N. and Carey, T.S., 2006. Criteria for distinguishing effectiveness from efficacy trials in systematic reviews. Higgins, J.P. and Green, S. eds., 2011. Cochrane handbook for systematic reviews of interventions (Vol. 4). John Wiley & Sons. https://publications.iadb.org/publications/english/document/Pathways-toward-Zero-Carbon-Electricity- Required-for-Climate-Stabilization.pdf Hugh Waddington, Edoardo Masset & Emmanuel Jimenez (2018) What have we learned after ten years of systematic reviews in international development? Journal of Development Effectiveness, 10:1, 1-16, DOI: 10.1080/19439342.2018.1441166 Joober, R., Schmitz, N., Annable, L. and Boksa, P., 2012. Publication bias: what are the challenges and can they be overcome? Journal of psychiatry & neuroscience: JPN, 37(3), p.149. Mahtani, K.R., C. Heneghan, and J. Aronson, Single screening or double screening for study selection in systematic reviews? BMJ Evidence-Based Medicine, 2020. 25(4): p. 149-150. Olawuyi, D.S., From technology transfer to technology absorption: addressing climate technology gaps in Africa. Journal of Energy & Natural Resources Law, 2018. 36(1): p. 61-84. Papavasiliou, S., C. Reaiche, and S. Papavasiliou, Digital health and patient-centred care: A digital systems view. Systems Research and Behavioral Science. Shemilt, I., et al., Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Systematic Reviews, 2016. 5(1): p. 140. Singal, A.G., P.D.R. Higgins, and A.K. Waljee, A primer on effectiveness and efficacy trials. Clinical and translational gastroenterology, 2014. 5(1): p. e45-e45. Snilstveit, B. 2012. “Systematic Reviews: From ‘Bare Bones’ to Policy Relevance.” Journal of Development Effectiveness 4 (3): 388–408. doi:10.1080/19439342.2012.709875. Snilstveit, B., M. Vojtkova, A. Bhavsar, and M. Gaarder, 2013. Evidence Gap Maps: A Tool for Promoting Evidence-Informed Policy and Prioritizing Future Research. Policy Research Working Paper WPS6725. Washington, D.C.: World Bank. Snilsveit, B., Vojtkova, M., Bhavsar, A., Stevenson, J., and Gaarder, M. (2016). Evidence and gap maps: A tool for promoting evidence informed policy and strategic research agendas. Journal of clinical epidemiology, 79, pp. 120 – 129. Treweek, S. and M. Zwarenstein, Making trials matter: pragmatic and explanatory trials and the problem of applicability. Trials, 2009. 10(1): p. 37. Waffenschmidt, S., et al., Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Medical Research Methodology, 2019. 19(1): p. 132. World Health Organisation, WHO guideline: recommendations on digital interventions for health system strengthening. 2018, World Health Organisation: Geneva. This page is for collation purposes APPENDICES APPENDIX 1 CLASSIFICATION OF DIGITAL HEALTH INTERVENTIONS Table A1.1 Classification of Digital Health Interventions Category Intervention Definition Synonyms and other descriptions 1. Clients 1.1 Targeted client Interventions that transmit – Public health event notification; disease notification to specific or pre-identified communication health event or information populations alerts and reminders to a – Notification of health events to specific populations based on demographic specified population group characteristics or demographic – Health promotion messaging – Health education, behavior change communication, health promotion communication, client-centered messaging; - Health communication based on a known client’s health status or clinical history 43 – Alerts for preventive services and wellness - Notifications and reminders for appointments, medication adherence, or follow-up services – Communication for retention in care, continuity of care – Laboratory results management, tests results management – Mass messaging campaign or communication to an undefined target group – Health messaging to undefined target group regardless of demographic characteristics or health status – Public health event notification; disease notification – Mass messaging campaign 1.2 Untargeted client Transmit untargeted health – Mass messaging campaign or communication to an undefined target group communication information or health event – Health messaging to undefined target group regardless of demographic alert to an undefined characteristics or health status population – Public health event notification; disease notification – Mass messaging campaign Table A1.1 continued on next page DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Table A1.1 Classification of Digital Health Interventions (continued) Category Intervention Definition Synonyms and other descriptions 1. Clients 1.3 Client-to-client Interventions that facilitate – Peer learning, peer group, peer-to-peer groups, peer network, peer support communication the formation and implementation of peer groups 1.4 Personal Health Interventions that allow – Self-access to client health record; patient’s own access to their health record. Tracking clients to access medical – Ability for clients to track their health history and clinical record records, monitor one’s – Personal health monitoring, self-tracking self-care, self-monitoring health, and/or document – Sensors and wearables for personal health monitoring health data – Client’s health data is collected based on a machine a client uses on their own – Personal health monitoring, self-tracking self-care, self-monitoring, journaling; capture patient originated data, client documentation of health status and activities 1.5 Citizen-based Interventions that enable – Public reporting on health system issues, such as the availability and quality of reporting: clients to report health services received, interaction with health worker, satisfaction with services 44 system feedback and – Accountability monitoring, accountability reporting public health events – Surveillance notification, disease notification, client reporting – Crowdsourced feedback, patient/client feedback, quality of care feedback – Surveillance notification, disease notification, client reporting 1.6 Citizen-based Interventions that enable – Public reporting on health system issues, such as the availability and quality of reporting: clients to report health services received, interaction with health worker, satisfaction with services system feedback and – Accountability monitoring, accountability reporting public health events – Surveillance notification, disease notification, client reporting – Crowdsourced feedback, patient/client feedback, quality of care feedback – Surveillance notification, disease notification, client reporting Table A1.1 continued on next page Table A1.1 Classification of Digital Health Interventions (continued) Category Intervention Definition Synonyms and other descriptions 1. Client 1.7 On-demand Interventions that facilitate – Client searches or looks up information on a health topic information clients’ searches for – Decision support for client services to clients information on health topic 1.8 Client financial Interventions that – Mobile money payments directly made by client (this can include payments for transactions transmit/manage out of any health services such as emergency transportation, health fees, etc.) pocket payments, – Health voucher issuance (e.g., for mosquito nets, for transport, etc.) and vouchers, and/or redemption services incentives (e.g., cash – Cash transfers to clients conditional on health-related behaviours transfers to clients conditional on health- related behaviours) 1.9 Client Interventions that verify – Biometrics, client registry identification and client unique identity, – Register client/patient for health services registration potentially via biometrics; 45 interventions that enrol/register clients for health services/clinical care plan Table A1.1 continued on next page APPENDICES DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Table A1.1 Classification of Digital Health Interventions (continued) Category Intervention Definition Synonyms and other descriptions 2. Healthcare 2.1 Client health Interventions that track – Domain-specific registers; eRegistries; eRegister, digital register, digital service providers records clients’ health status and record, immunization registry services received, manage – Digitized registers for longitudinal health program, including tracking of migrant clinical records, and collect populations’ benefits and health status routine health indicator – Case management logs within specific target populations, including migrant data populations – Clinical record of an individual with information that spans across multiple clinical domains. – Electronic medical record, personal health record – Electronic medical record and personal health records with that are not based on structured data, and instead including notes, images, documents – Data collection for Health Management Information System (HMIS) – Client health data collection 2.2 Healthcare Interventions that provide – Clinical decision support, job aid--linked to clients’ digital health record 46 provider decision alerts, checklists according – Provision of alerts for abnormal findings/lab values, “IF THEN statements,” support to protocol, and/or screen – Process algorithms to support service delivery according to care plans, clients by risk or other guidelines, and protocols health status – Job aid and assessment tools to support service delivery—may or may not be linked to a digital health record – Decision trees to support service delivery according to care plans, guidelines, and protocols – Tools for screening, risk assessment, triage and client prioritization – Job aid to support service delivery according to care plans, guidelines, and protocols Table A1.1 continued on next page Table A1.1 Classification of Digital Health Interventions (continued) Category Intervention Definition Synonyms and other descriptions 2. Healthcare 2.3 Telemedicine – Remote consultation, tele consultation, client-based telemedicine, hotlines, call Interventions that connect providers client and provider centers, helpline remotely for health – Real-time telemedicine, interactive telemedicine, synchronous telemedicine – Client calls a health worker or hotline to receive clinical guidance on health issue consultation, monitor client health, transmit data to – Telemonitoring, virtual monitoring. provider, and/or enable – Provider able to monitor client’s health through an implanted sensor/diagnostic case management equipment. consultation between – Store and forward providers – Asynchronous telemedicine – Inter-provider communication, closed user group, health, health worker to health worker communication. – Consulting other health care providers, particularly specialists, for patient case management; seeking second opinion for patient case management 2.4 Healthcare Interventions that allow for – Supportive supervision, coaching/mentoring, audit and feedback - 47 Provider communication between Communication to healthcare provider based on their performance Communication healthcare providers, – Alerts and reminders to healthcare provider - Motivational communication transmit alerts to healthcare provider healthcare providers, – Transmission of workflow updates to healthcare provider provide feedback to – Public health related updates to health workers healthcare providers, and – Emergency alerts to healthcare providers form peer groups for – Mass messaging to healthcare providers healthcare providers 2.5 Referral Interventions that – Peer support, peer learning, closed user groups Coordination coordinate emergency – Communication mechanisms for healthcare providers to discuss among logistics and manage themselves referrals – Ambulance systems, emergency response management – Care coordination – Clinical task linking APPENDICES – Referral management – Intersectoral referral management Table A1.1 continued on next page DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Table A1.1 Classification of Digital Health Interventions (continued) Category Intervention Definition Synonyms and other descriptions 2. Healthcare 2.6 Scheduling and Interventions that help – Automated scheduling of client’s health appointments providers Activity Planning schedule client – Work planning appointments and – Prioritization of daily activities/tasks healthcare providers’ – Task management activities 2.7 Training Interventions that provide – mLearning, eLearning, virtual learning training content, reference – Educational videos, multimedia learning and access to clinical guidance material, and assessments – Training reinforcement and refreshers (quizzes, interactive – Quizzes and interactive exercises to assess knowledge and competence exercises, etc.) 2.8 Prescription and Interventions that track – Tracking medication orders Medication prescription orders and – Tools to place prescription orders or track the status of prescriptions and refills Management refills, medication – Monitoring adherence to medications and drugs adherence/consumption – Monitoring/observing whether patients have taken their prescribed medications 48 and reporting of adverse – Reporting contraindications, drug interactions, adverse effects drug events 2.9 Laboratory and Interventions that transmit – Laboratory results management diagnostics diagnostic results to – Tests results communication between healthcare providers imaging healthcare providers, – Laboratory test requisition and management management diagnostic orders, and – Point of care diagnostics track biological specimens – Diagnostic accessories added to digital devices – Tracking of blood donations Table A1.1 continued on next page Table A1.1 Classification of Digital Health Interventions (continued) Category Intervention Definition Synonyms and other descriptions 2. Healthcare 2.10 Human Interventions that track – Health worker registry; provider registry providers Resource health workforce – Documentation of healthcare providers’ demographics, identification, health Management identification information, facility assignment, and other identifier information monitor performance of – Remote monitoring of healthcare providers healthcare provider, – Workforce management manage healthcare work – Audit and feedback Supervision, supportive supervision force – Clinical task tracking registration/certification, – Management of health worker registration and record training – Certification or licensure with regulatory authority such as a professional council information on healthcare – Track or manage preservice and/or in-service training received by a health providers worker 3. Health 3.1 Supply Chain Interventions that manage – Stock monitoring of health commodities Systems Management inventory, stock, status of – Logistics management Management health commodities, – Stock management 49 register drugs and health – Commodity security commodities, and report – Stockout prevention and monitoring counterfeit drugs – Alerts and notifications of stock levels – Restocking coordination – Sensors to monitor temperature and stability of vaccines – Drug regulation and registration – Logistics management – Procurement management – Counterfeit drug notification – Monitoring drug authenticity and quality – Pharmacovigilance 3.2 Public Health Interventions that notify – Public health surveillance Event public health events (e.g., – Surveillance from laboratory systems – Disease surveillance APPENDICES Notification disease surveillance) Table A1.1 continued on next page DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Table A1.1 Classification of Digital Health Interventions (continued) Category Intervention Definition Synonyms and other descriptions 3. Health 3.3 Civil Interventions that notify – Birth event alert Systems Registration and and register birth/death – Birth registration (can include registration for health system purposes, as well as Management Vital Statistics event and certify registration to civil registrar) birth/death event – Civil Registration and Vital Statistics (CRVS) – Issuance of birth certificate – Death surveillance – Death event alert – Death surveillance; mortality surveillance – Issuance of death certificate 3.4 Health Interventions that register – Eligibility verification for insurance Financing and verify client insurance, – Determination of insurance coverage track insurance billing and – Recording and verifying that a client is a member of a scheme or entitled to claims submission, benefits track/manage insurance – Social protection 50 reimbursement, – Social protection, administrative transaction processing; claims management transmit/manage payment – Claims and encounter reports for reimbursement to health providers, and – Insurance financial transactions manage budget – Health worker routine payments - Payroll management – Financial incentives for health worker motivation – Conditional payments, performance-based financing for health workers, results based financing – Financial management – Resource planning 3.5 Equipment and Interventions that monitor – Listing of available equipment and physical assets, e.g. hospital beds Asset status/maintenance of – Tracking maintenance of equipment Management health equipment and – Physical asset management track regulation/licensing – Regulation of physical assets of medical equipment Table A1.1 continued on next page Table A1.1 Classification of Digital Health Interventions (continued) Category Intervention Definition Synonyms and other descriptions 3. Health 3.6 Facility Interventions that list – Register health facilities Systems Management health facility information – List unique IDs and locations of health facilities Management and assess services – Health facility registry provided by health facilities – Assess performance and capacity of services provided at health facilities – Regulate and monitor services provided at health facilities – Supervision of health facilities 3.7 Data Collection, Interventions that collect – Electronic data collection, digital data collection Management, data from mobile based – Mobile based surveys, using applications such as OpenDataKit (ODK), Enketo, and Use surveys, store/aggregate FormHub, etc data, synthesize/visualize – Data warehouse, repository data, and automate – Reporting dashboards, report generation analysis of data – Presentations of data - Business intelligence – Predictive analytics – Machine learning 51 – Artificial intelligence Data Data Coding Interventions that parse – Dirty data management Servic unstructured data into – Automated data cleaning es structured data, merge, – Maintenance and versioning of health informatics terminology standards de-duplicate and curate – Terminology services coded datasets or – Semantic interoperability terminologies, and classify – Recording cause of death disease codes and cause – ICD coding, clinical coding for reporting and insurance of mortality – Mapping local terminology, codes, and formats Table A1.1 continued on next page APPENDICES DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Table A1.1 Classification of Digital Health Interventions (continued) Category Intervention Definition Synonyms and other descriptions 4. Data Location Interventions that map – Global Positioning System (GPS) mapping Services Mapping location of health facilities, – Geospatial visualization health events, coverage of – GPS mapping clients, healthcare – Demarcation of catchment areas providers, and health – Mapping coverage areas worker routes – Geospatial visualization - GPS mapping – Mapping of health worker route to track the services provided Data Exchange Interventions that enable – Data mediation and data exchange across – Interoperability and accessibility Interoperability systems – Information exchange – Interoperability layer – Data orchestration Source: 3IE, World Bank. 52 APPENDIX 2 MINIMUM REPORTING REQUIREMENTS FOR STUDIES TO BE INCLUDED We used the checklist below to operationalise the study design inclusion criteria. RANDOMISED CONTROLLED TRIALS Definition An impact evaluation design in which random assignment has been used to allocate the intervention amongst members of the eligible population Reporting requirements for establishing counterfactual • Clear description of random assignment process • Balance between all groups at baseline reported either in a table or in the narrative • Balance calculated using statistical tests for significance Requirements for reporting results • Post-intervention differences between groups should be calculated using method of data analysis such as single difference/OLS • Results can be reported as odds ratios or confidence intervals • Statistical tests for significance are required • Report sample size and it must be at least 40 (both groups combined) REGRESSION DISCONTINUITY DESIGN Definition An impact evaluation design in which the treatment and comparison groups are identifying as being those just on either side of a threshold value of a variable. Reporting requirements for establishing counterfactual • The threshold is clearly defined • Established continuity at threshold • Distribution of covariates and outcome measures around threshold is compared to ensure “balance” Requirements for reporting results • Post-intervention differences between groups should be calculated using method of data analysis such as single difference/OLS • Results can be reported as odds ratios or confidence intervals • Statistical tests for significance are required STATISTICAL MATCHING (PSM AND OTHERS) Definition An impact evaluation design in which the comparison group is contstructed using statistical matching techniques such as propensity scores. A propensity score is the probability of participating in the intervention, as given by a probit regression on observed characteristics Reporting requirements for establishing counterfactual • Covariates used to estimate propensity score are clearly listed 53 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT • The authors test the quality of the matching procedures using one of the following tests: ‒ Covariate balanced comparison test before and after the matching ‒ Histogram of propensity score before and after the matching ‒ Pseudo R2 before and after the matching ‒ Sensitivity analysis to address the issue of hidden bias related to unobservable variables Requirements for reporting results • Post-intervention differences between groups should be calculated using method of data analysis such as single difference/OLS • Results can be reported as odds ratios or confidence intervals • Statistical tests for significance are required DIFFERENCE-IN-DIFFERENCE & FIXED EFFECTS ESTIMATION DID calculates the change in the outcome observed in the treatment group compared to the change observed in the comparison group. Fixed effects when using panel data control for time-invariant characteristics by exploring the relationship between the dependent and explanatory variables within an entity (individual, household, etc.) Requirement for testing assumptions To test parallel trends assumptions, the paper must meet at least one (1) of the criteria below. See Gertler et al.'s handbook (pages 137‒138) for a deeper explanation of the tests below [1] Use at least two serial observations on the treatment and comparison groups before the start of the program. This means that the evaluation would require three serial observations: two pre-intervention observations to assess the preprogram trends, and at least one post-intervention observation to assess impact with the difference-in- differences method. [2] Perform a "placebo test" by conducting an additional difference-in-differences estimation using a “fake” treatment group: that is, a group that you know was not affected by the program. [3] Perform the placebo test not only with a fake treatment group, but also with a fake outcome. [4] Perform the difference-in-differences estimation using different comparison groups. Requirements for reporting results • Post-intervention differences between groups should be calculated using a regression with a time X treatment interaction • Statistical tests for significance are required INSTRUMENTAL VARIABLE ESTIMATION The IV method is used to estimate causal relationships when controlled experiments are not feasible or when a treatment is not successfully delivered to every unit in a randomized experiment. A valid instrument induces changes in the explanatory variable but has no independent effect on the dependent variable. The explanatory variable only affects the dependent variable through the instrument. 54 APPENDICES Requirement for testing assumptions • Test of underlying assumptions ‒ Theoretical discussion on why the instrument is correlated with the explanatory variable and not with the outcome variable or error term. ‒ Instrument must meet the relevance condition: authors should test for significant correlation between instrument and explanatory variable Requirements for reporting results • Statistical tests for significance are required 55 APPENDIX 3 SEARCH STRATEGY We searched for studies in academic databases using a search strategy focused on our study design and inclusion criteria. The search strategies for all the databases searched are included below. 1. OVID MEDLINE(R) AND EPUB AHEAD OF PRINT, IN-PROCESS & OTHER NON- INDEXED CITATIONS AND DAILY <1946 TO APRIL 19, 2019> SEARCHED 22ND APRIL 2019 ("quasi experiment*" or quasi-experiment* or "random* control* trial*" or "random* trial*" or RCT or (random* adj3 allocat*) or matching or "propensity score" or PSM or "regression discontinuity" or "discontinuous design" or RDD or "synthetic control" or "difference in difference*" or difference-in-difference* or "diff in diff" or cohort or "propensity weighted" or propensity-weighted or "interrupted time series" or (pre adj5 post) or ((pretest or "pre test") and (posttest or "post test")) or "rapid evidence assessment" or "systematic literature review" or "systematic review" or "meta- analy*" or metaanaly* or "meta analy*" or "instrumental variable*" or heckman or ((evaluation or impact or quantitative or "comparison group*" or counterfactual or "counter factual" or counter-factual or experiment*) adj3 (design or study or analysis)) or QED).ti,ab,kw. (1319328) clinical trial/ or clinical trial, phase i/ or clinical trial, phase ii/ or clinical trial, phase iii/ or clinical trial, phase iv/ or controlled clinical trial/ or randomized controlled trial/ or pragmatic clinical trial/ (823920) controlled clinical trials as topic/ or non-randomized controlled trials as topic/ or randomized controlled trials as topic/ or pragmatic clinical trials as topic/ or retrospective studies/ or controlled before-after studies/ or interrupted time series analysis/ or random allocation/ or cohort studies/ or follow-up studies/ or longitudinal studies/ or prospective studies/ or retrospective studies/ or propensity score/ (2052307) meta-analysis/ or "systematic review"/ (162006) or/1-4 (3451391) Annotation: Study Design Terms Cell Phones/ (7747) Smartphone/ (2828) MP3-Player/ (178) Computers, Handheld/ (3339) ((cell* or mobile*) adj1 (phone* or telephone* or technolog* or device*)).ti,ab,kw. (16410) (handheld or hand-held).ti,ab,kw. (11353) (smartphone* or smart-phone* or cellphone* or mobiles).ti,ab,kw. (9358) ((personal adj1 digital) or (PDA adj3 (device* or assistant*)) or MP3 player* or MP4 player*).ti,ab,kw. (1319) (samsung or nokia).ti,ab,kw. (1026) (windows adj3 (mobile* or phone*)).ti,ab,kw. (50) 56 APPENDICES android.ti,ab,kw. (2072) (ipad* or i-pad* or ipod* or i-pod* or iphone* or i-phone*).ti,ab,kw. (2470) (tablet* adj3 (device* or computer*)).ti,ab,kw. (1360) Telemedicine/ (19200) Webcasts as topic/ (303) Text Messaging/ (2219) Telenursing/ (199) (mhealth or m-health or "mobile health" or ehealth or e-health or "electronic health" or "digital health" or uhealth or u-health).ti,ab,kw. (22465) (telemedicine or tele-medicine or telehealth or tele-health or telecare or tele-care or telenursing or tele-nursing or telepsychiatry or tele-psychiatry or telemonitor* or tele-monitor* or teleconsult* or tele-consult* or telecounsel* or tele-counsel* or telecoach* or tele-coach*).ti,ab,kw. (16107) (webcast* or web-cast*).ti,ab,kw. (241) (((text* or short or voice or multimedia or multi-media or electronic or instant) adj1 messag*) or instant messenger).ti,ab,kw. (4531) (texting or texted or texter* or ((sms or mms) adj (service* or messag*)) or interactive voice response* or IVR or voice call* or callback* or voice over internet or VOIP).ti,ab,kw. (3040) (Facebook or Twitter or Whatsapp* or Skyp* or YouTube or "You Tube" or Google Hangout*).ti,ab,kw. (6274) Mobile Applications/ (3968) "mobile app*".ti,ab,kw. (3357) Reminder Systems/ (3200) (remind* adj3 (text* or system* or messag*)).ti,ab,kw. (1617) Medical informatics/ or Medical informatics applications/ (13259) Nursing informatics/ or Public health informatics/ (2586) ((medical or clinical or health or healthcare or nurs*) adj3 informatics).ti,ab,kw. (5223) Computer-Assisted Instruction/ or interactive tutorial/ (11722) ((interactive or computer-assisted) adj1 (tutor* or technolog* or learn* or instruct* or software or communication)).ti,ab,kw. (2379) mass media/ or radio/ or television/ or social media/ (30086) ((media or radio or television or tv or online or public) adj3 campaign*).ti,ab,kw. (4948) Geographic Information Systems/ or (GIS or "geographic information system*" or "global positioning").ti,ab,kw. (14096) or/6-40 (169608) Annotation: Digital Health Terms patients/ or inpatients/ or no-show patients/ or outpatients/ (51363) (patient* or client or clients or outpatient* or out-patient* or inpatient* or in- patient*).ti,ab,kw. (6277851) 57 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT or/42-43 (6286225) Annotation: Patients/Clients health services accessibility/ or health equity/ (69167) ((access* or "use" or "using" or "uses") adj3 "health service*").ti,ab,kw. (11522) health records, personal/ or patient generated health data/ or patient portals/ or electronic health records/ or health information exchange/ (17588) ((health or medical or personal) adj2 (record* or data or monitor* or self-monitor* or track* or self-track* or registr*)).ti,ab,kw. (153398) communication/ or "cell phone use"/ or access to information/ or health communication/ or information dissemination/ or information seeking behavior/ or consumer health information/ (104143) ((communicat* or messag* or online or access* or seek*) adj3 (health or information)).ti,ab,kw. (66251) peer group/ or peer influence/ (19253) ((peer or peer-to-peer) adj2 (group* or influenc* or pressure or network* or support)).ti,ab,kw. (9725) Social Responsibility/ (18747) ((monitor* or satisfact* or accountab* or feedback) adj3 (consumer* or public or patient* or client or clients) adj3 (service or services)).ti,ab,kw. (1200) ((cash adj3 (transfer* or voucher* or grant* or aid)) or health voucher*).ti,ab,kw. (666) "fees and charges"/ or fees, dental/ or fees, medical/ or fees, pharmaceutical/ or prescription fees/ or hospital charges/ (21523) ((fee or fees or charge or charges or payment*) adj2 (service* or hospital* or prescription* or medication or drug* or treatment*)).ti,ab,kw. (12458) or/45-57 (451202) Annotation: Patient/Client Interventions exp administrative personnel/ or exp health personnel/ (490222) (((health or medical or healthcare or frontline or front-line) adj (personnel or worker* or auxiliar* or staff or professional* or assistant* or provider* or administrator*)) or doctor* or physician* or GP or general practitioner? or family doctor or nurse* or midwi* or clinical officer* or pharmacist* or dentist* or vaccinator* or supervisor* or ((birth or childbirth or labor or labour) adj (attendant? or assistant?))).ti,ab,kw. (962899) (((lay or voluntary or volunteer? or untrained or unlicensed or nonprofessional? or non professional? or support or link or outreach or out reach) adj5 (worker? or visitor? or attendant? or aide or aides or support$ or person$ or helper? or carer? or caregiver? or care giver? or consultant? or assistant? or staff)) or promotores or promotora or promotoras or embajadoras or comodrones or abuela or "lay advocate*" or "lay health" or "lay advisors" or "lay educators" or "lay counselor*" or "lay counsellor*" or "lady health worker*" or "lay facilitator*" or "natural helpers" or linkworker? or monitrice* or consejeras).ti,ab,kw. (910624) (paraprofessional? or paramedic or paramedics or paramedical worker? or paramedical personnel or allied health personnel or allied health worker? or support worker? or home health aide?).ti,ab,kw. (8205) 58 APPENDICES ((community or village? or peer or indigenous or treatment) adj3 (health worker? or health care worker? or healthcare worker? or health advisor* or volunteer* or educator* or facilitator* or distributor* or extension worker* or supporter* or counselor* or counsellor*)).ti,ab,kw. (10336) (doula? or douladural? or barefoot doctor?).ti,ab,kw. (458) or/59-64 (2042029) Annotation: Healthcare Personnel hospital records/ or medical records/ or health records, personal/ or medical records systems, computerized/ or electronic health records/ or health information exchange/ or biometric identification/ or patient identification systems/ (104723) ((record* or registration* or registry or registries or e-registr* or eregistr* or ((medical or health) adj2 data)) adj2 (patient* or hospital or hospitals or medical or client or clients or in-patient* or inpatient* or out-patient* or outpatient*)).ti,ab,kw. (163687) data collection/ (87740) ((decision* adj3 (make or makes or making or made or support* or algorithm* or aid or aids or app or apps or application* or technique*)) or expert system* or job-aid* or "job aid*").ti,ab,kw. (169699) (((guideline* or protocol*) adj4 (adher* or comply or complian* or observ*)) or checklist*).ti,ab,kw. (54081) checklist/ or clinical protocols/ or guideline/ or practice guideline/ or diagnosis, computer-assisted/ or decision support systems, management/ or therapy, computer-assisted/ or "drug therapy, computer-assisted"/ or electronic prescribing/ or expert systems/ (96098) Risk Assessment/ec, mt, og, st, td [Economics, Methods, Organization & Administration, Standards, Trends] (30416) Triage/ec, mt, og, st, td [Economics, Methods, Organization & Administration, Standards, Trends] (4931) ((risk* adj2 (assess* or estimat* or calculat*)) or triage).ti,ab,kw. (136817) therapy, computer-assisted/ or "Drug Therapy, Computer-Assisted"/ or electronic prescribing/ or Clinical Pharmacy Information Systems/ (9878) ((therap* or prescrib* or prescript* or diagnos* or pharmacy or pharmacies or pharmacist*) adj2 (computer* or digital or electronic)).ti,ab,kw. (8536) Patient Care Planning/ (37611) ((patient* or client or clients or outpatient* or out-patient* or inpatient* or in-patient*) adj2 (plan* or goal*)).ti,ab,kw. (16277) remote consultation/ or call centers/ or hotlines/ or case management/ (16844) education, continuing/ or mentoring/ or work performance/ (10303) (mentor* or supervis* or coaching or motivat*).ti,ab,kw. (193676) ((remote* adj2 consult*) or "call center*" or "call centre*" or hotline* or (case* adj2 manag*)).ti,ab,kw. (21436) ((emergenc* or mass) adj2 (alert* or messag* or reminder* or updat*)).ti,ab,kw. (422) "delivery of health care"/ or "delivery of health care, integrated"/ or patient care team/ or point-of-care systems/ (165881) 59 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT (((health or management or medical or clinical) adj3 (system or systems or information or referral or (task* adj2 link*))) or ("health care" adj2 delivery)).ti,ab,kw. (230394) emergency medical services/ or emergency medical service communication systems/ (41185) ((emergenc* or ambulance*) adj3 (system* or manag*)).ti,ab,kw. (12984) "appointments and schedules"/ or "personnel staffing and scheduling information systems"/ or "task performance and analysis"/ or work simplification/ or time management/ (41378) ((appointment* or schedul* or task* or work) adj3 (plan* or manag* or priorit*)).ti,ab,kw. (12492) educational measurement/ or inservice training/ or staff development/ or computer- assisted instruction/ (72236) ((in-service or inservice or staff) adj2 (train* or learning or quiz* or "interactive exercise*")).ti,ab,kw. (9297) prescription drug monitoring programs/ or medication therapy management/ or prescriptions/ or drug prescriptions/ or medication adherence/ or adverse drug reaction reporting systems/ or pharmacovigilance/ (54033) (((prescription* or medication* or drug*) adj2 (monitor* or manag* or adher* or compliance or ((report* or notif*) adj2 system*))) or pharmacovigilance).ti,ab,kw. (43978) Clinical Laboratory Information Systems/ or "diagnostic techniques and procedures"/ or clinical laboratory techniques/ or diagnostic imaging/ (63757) ((laborat* or diagnostic) adj2 (system* or technique* or imaging or requisition* or communicat* or report*)).ti,ab,kw. (48541) or/66-95 (1642295) Annotation: Healthcare Providers’ Interventions (workforce/ or health workforce/) and (employee performance appraisal/ or personnel management/ or personnel administration, hospital/) (2671) ((worker* or workforce or "human resource*" or employee* or personnel) adj2 (manag* or appraisal or registr* or certific* or licens* or ((task* or work) adj2 (track* or monitor*)) or assign* or supervis*)).ti,ab,kw. (5584) exp Vaccines/ec, st, sd or exp "Equipment and Supplies"/ec, sn, sd, td, ut or (Pharmaceutical Preparations/ec, st, sd, ut not Veterinary Drugs/) or hospital distribution systems/ or materials management, hospital/ or inventories, hospital/ or medication systems, hospital/ or product line management/ or drug storage/mt (77851) ((commodit* or consumable* or stock or stocks or supply or supplies) adj3 (inventor* or level* or notif* or track* or count* or report* or chain or out or outs or manag* or order* or logistic* or system or systems or shortage* or manag* or monitor* or maintain* or maintenance or audit or auditing)).ti,ab,kw. (11760) ((health or medical or medicines or vaccine* or drug or drugs or laborator* or diagnos*) adj3 (product* or supply or supplies or consumable* or commodit* or stock or stocks or stockout* or "stock out*" or shortage*)).ti,ab,kw. (38349) public health surveillance/ or sentinel surveillance/ (8464) 60 APPENDICES ((surveillance or monitoring or notification) adj2 (outbreak* or epidemic* or "public health event*")).ti,ab,kw. (1286) registries/ or hospital records/ or electronic health records/ or vital statistics/ (103391) data collection/ or records as topic/ or birth certificates/ or death certificates/ or medical records/ or medical record linkage/ or medical records systems, computerized/ (182492) information management/ or health information management/ or health information exchange/ or "information storage and retrieval"/ (23852) (birth adj3 (registr* or notif* or report* or record* or log* or certif* or collection or survey* or surveillance)).ti,ab,kw. (10589) (((death* or mortality or vital) adj3 (registr* or notif* or report* or record* or log* or certif* or collection or survey* or surveillance)) or verbal autops*).ti,ab,kw. (45887) insurance claim reporting/ or insurance, health/ or insurance, health, reimbursement/ or financial management/ (62817) ((insurance or financ* or budget*) adj2 (coverage or incentiv* or conditional or performance-based or results-based or manag*)).ti,ab,kw. (15324) exp Health Facilities/ and ("facilities and services utilization"/ or health facility administration/ or exp management information systems/) (15311) (("health facilit*" or hospital or hospitals) adj3 (registr* or regulat* or manag* or performance or monitor* or (service* adj2 (capacit* or provid* or provision)))).ti,ab,kw. (21439) or/97-112 (573503) Annotation: Health Systems Management data collection/ or artificial intelligence/ or machine learning/ or data warehousing/ or database management systems/ or electronic data processing/ (133257) ((data adj2 (collect* or aggregat* or manag* or synthes* or analys*) adj2 automated) or (data adj2 (electronic or mobile-based or dashboard* or analytics)) or OpenDataKit or Enketo or Formhub).ti,ab,kw. (7093) Clinical Coding/ (1858) mortality/ or "cause of death"/ or child mortality/ or fetal mortality/ or hospital mortality/ or infant mortality/ or maternal mortality/ or mortality, premature/ or disease notification/ (150517) (coding or "cause of death" or ICD or ((disease* or dataset*) adj2 (code* or coding or coded)) or "dirty data" or (automat* adj2 "data cleaning")).ti,ab,kw. (262946) (((disease* or dataset* or "cause of death" or "international classification of diseases" or ICD) adj2 (code* or coding or coded)) or "dirty data" or (automat* adj2 "data cleaning")).ti,ab,kw. (7087) geographical information systems/ or geographic mapping/ (8180) ("geographic information systems" or "geographic* mapping" or GIS or (geospatial adj2 visual*) or gps or "global positioning system").ti,ab,kw. (31527) health information exchange/ or systems integration/ (10006) ((data or information) adj2 (mediat* or interoperability or accessib* or exchange or integrat* or orchestrat*)).ti,ab,kw. (28920) 61 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT or/114-123 (603126) Annotation: Data Services 5 and 41 and 44 and 58 (4699) limit 125 to yr="2000 -Current" (4637) Annotation: Patients/Clients +Patient/Client Interventions - 2000- 5 and 41 and 65 and 96 (5948) limit 127 to yr="2000 -Current" (5670) Annotation: Healthcare providers+Healthcare provider interventions, 2000- 5 and 41 and 113 (3831) limit 129 to yr="2000 -Current" (3694) Annotation: Health Systems Management, 2000- 5 and 41 and 124 (3639) limit 131 to yr="2000 -Current" (3523) Annotation: Data Services. 2000- 126 or 128 or 130 or 132 (11865) Annotation: All results added together 2. EBSCO DISCOVERY SEARCH – SEARCHED 23RD APRIL 2019 – RESULTS LIMITED TO REPEC, ECONLIT & AFRICA-WIDE DATABASES: PATIENTS /CLIENTS: 177 HITS; HEALTHCARE PROVIDERS: 197; HEALTH SYSTEMS MANAGEMENT: 111; DATA SERVICES: 617 S52 S1 AND S5 AND S51 Limiters - Date of Publication: 20000101-20191231 25,767 S51 S47 OR S48 OR S49 OR S50 Limiters - Date of Publication: 20000101- 20191231 2,045,614 S50 TI ( ((data or information) N2 (mediat* or interoperability or accessib* or exchange or integrat* or orchestrat*)) ) OR AB ( ((data or information) N2 (mediat* or interoperability or accessib* or exchange or integrat* or orchestrat*)) ) OR SU ( ((data or information) N2 (mediat* or interoperability or accessib* or exchange or integrat* or orchestrat*)) ) Limiters - Date of Publication: 20000101-20191231 448,959 S49 TI ( ("geographic information systems" or "geographic* mapping" or GIS or (geospatial N2 visual*) or gps or "global positioning system") ) OR AB ( ("geographic information systems" or "geographic* mapping" or GIS or (geospatial N2 visual*) or gps or "global positioning system") ) OR SU ( ("geographic information systems" or "geographic* mapping" or GIS or (geospatial N2 visual*) or gps or "global positioning system") ) Limiters - Date of Publication: 20000101- 20191231 653,096 S48 TI ( (coding or "cause of death" or ICD or ((disease* or dataset*) N2 (code* or coding or coded)) or "dirty data" or (automat* N2 "data cleaning")) ) OR AB ( (coding or "cause of death" or ICD or ((disease* or dataset*) N2 (code* or coding 62 APPENDICES or coded)) or "dirty data" or (automat* N2 "data cleaning")) ) OR SU ( (coding or "cause of death" or ICD or ((disease* or dataset*) N2 (code* or coding or coded)) or "dirty data" or (automat* N2 "data cleaning")) ) Limiters - Date of Publication: 20000101-20191231 1,148,148 S47 TI ( ((data N2 (collect* or aggregat* or manag* or synthes* or analys*) N2 automated) or (data N2 (electronic or mobile-based or dashboard* or analytics)) or OpenDataKit or Enketo or Formhub) ) OR AB ( ((data N2 (collect* or aggregat* or manag* or synthes* or analys*) N2 automated) or (data N2 (electronic or mobile- based or dashboard* or analytics)) or OpenDataKit or Enketo or Formhub) ) OR SU ( ((data N2 (collect* or aggregat* or manag* or synthes* or analys*) N2 automated) or (data N2 (electronic or mobile-based or dashboard* or analytics)) or OpenDataKit or Enketo or Formhub) ) Limiters - Date of Publication: 20000101- 20191231 138,916 S46 S1 AND S5 AND S45 Limiters - Date of Publication: 20000101-20191231 3,976 S45 S37 OR S38 OR S39 OR S40 OR S41 OR S42 OR S43 OR S44 Limiters - Date of Publication: 20000101-20191231 1,771,412 S44 TI ( (("health facilit*" or hospital or hospitals) N3 (registr* or regulat* or manag* or performance or monitor* or (service* N2 (capacit* or provid* or provision)))) ) OR AB ( (("health facilit*" or hospital or hospitals) N3 (registr* or regulat* or manag* or performance or monitor* or (service* N2 (capacit* or provid* or provision)))) ) OR SU ( (("health facilit*" or hospital or hospitals) N3 (registr* or regulat* or manag* or performance or monitor* or (service* N2 (capacit* or provid* or provision)))) ) Limiters - Date of Publication: 20000101-20191231 91,348 S43 TI ( ((insurance or financ* or budget*) N2 (coverage or incentiv* or conditional or performance-based or results-based or manag*)) ) OR AB ( ((insurance or financ* or budget*) N2 (coverage or incentiv* or conditional or performance-based or results-based or manag*)) ) OR SU ( ((insurance or financ* or budget*) N2 (coverage or incentiv* or conditional or performance-based or results-based or manag*)) ) Limiters - Date of Publication: 20000101-20191231 307,705 S42 TI ( (((death* or mortality or vital) N3 (registr* or notif* or report* or record* or log* or certif* or collection or survey* or surveillance)) or "verbal autops*") ) OR AB ( (((death* or mortality or vital) N3 (registr* or notif* or report* or record* or log* or certif* or collection or survey* or surveillance)) or "verbal autops*") ) OR SU ( (((death* or mortality or vital) N3 (registr* or notif* or report* or record* or log* or certif* or collection or survey* or surveillance)) or "verbal autops*") ) Limiters - Date of Publication: 20000101-20191231 157,182 S41 TI ( (birth N3 (registr* or notif* or report* or record* or log* or certif* or collection or survey* or surveillance)) ) OR AB ( (birth N3 (registr* or notif* or report* or record* 63 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT or log* or certif* or collection or survey* or surveillance)) ) OR SU ( (birth N3 (registr* or notif* or report* or record* or log* or certif* or collection or survey* or surveillance)) ) Limiters - Date of Publication: 20000101-20191231 40,033 S40 TI ( ((surveillance or monitoring or notification) N2 (outbreak* or epidemic* or "public health event*")) ) OR AB ( ((surveillance or monitoring or notification) N2 (outbreak* or epidemic* or "public health event*")) ) OR SU ( ((surveillance or monitoring or notification) N2 (outbreak* or epidemic* or "public health event*")) ) Limiters - Date of Publication: 20000101-20191231 6,468 `S39 TI ( ((health or medical or medicines or vaccine* or drug or drugs or laborator* or diagnos*) N3 (product* or supply or supplies or consumable* or commodit* or stock or stocks or stockout* or "stock out*" or shortage*)) ) OR AB ( ((health or medical or medicines or vaccine* or drug or drugs or laborator* or diagnos*) N3 (product* or supply or supplies or consumable* or commodit* or stock or stocks or stockout* or "stock out*" or shortage*)) ) OR SU ( ((health or medical or medicines or vaccine* or drug or drugs or laborator* or diagnos*) N3 (product* or supply or supplies or consumable* or commodit* or stock or stocks or stockout* or "stock out*" or shortage*)) ) Limiters - Date of Publication: 20000101-20191231 274,021 S38 TI ( ((commodit* or consumable* or stock or stocks or supply or supplies or equipment) N3 (inventor* or level* or notif* or track* or count* or report* or chain or out or outs or manag* or order* or logistic* or system or systems or shortage* or manag* or monitor* or maintain* or maintenance or audit or auditing) or (asset* N2 manag*)) ) OR AB ( ((commodit* or consumable* or stock or stocks or supply or supplies or equipment) N3 (inventor* or level* or notif* or track* or count* or report* or chain or out or outs or manag* or order* or logistic* or system or systems or shortage* or manag* or monitor* or maintain* or maintenance or audit or auditing) or (asset* N2 manag*)) ) OR SU ( ((commodit* or consumable* or stock or stocks or supply or supplies or equipment) N3 (inventor* or level* or notif* or track* or count* or report* or chain or out or outs or manag* or order* or logistic* or system or systems or shortage* or manag* or monitor* or maintain* or maintenance or audit or auditing) or (asset* N2 manag*)) ) Limiters - Date of Publication: 20000101-20191231 852,442 S37 TI ( ((worker* or workforce or "human resource*" or employee* or personnel) N2 (manag* or appraisal or registr* or certific* or licens* or ((task* or work) N2 (track* or monitor*)) or assign* or supervis*)) ) OR AB ( ((worker* or workforce or "human resource*" or employee* or personnel) N2 (manag* or appraisal or registr* or certific* or licens* or ((task* or work) N2 (track* or monitor*)) or assign* or supervis*)) ) OR SU ( ((worker* or workforce or "human resource*" or employee* or personnel) N2 (manag* or appraisal or registr* or certific* or licens* or ((task* or work) N2 (track* or monitor*)) or assign* or supervis*)) ) Limiters - Date of Publication: 20000101-20191231 352,718 64 APPENDICES S36 S1 AND S5 AND S19 AND S35 Limiters - Date of Publication: 20000101- 20191231 8,768 S35 S20 OR S21 OR S22 OR S23 OR S24 OR S25 OR S26 OR S27 OR S28 OR S29 OR S30 OR S31 OR S32 OR S33 OR S34 Limiters - Date of Publication: 20000101-20191231 4,218,597 S34 TI ( ((laborat* or diagnostic) N2 (system* or technique* or imaging or requisition* or communicat* or report*)) ) OR AB ( ((laborat* or diagnostic) N2 (system* or technique* or imaging or requisition* or communicat* or report*)) ) OR AB ( ((laborat* or diagnostic) N2 (system* or technique* or imaging or requisition* or communicat* or report*)) ) OR SU ( ((laborat* or diagnostic) N2 (system* or technique* or imaging or requisition* or communicat* or report*)) ) Limiters - Date of Publication: 20000101-20191231 370,431 S33 TI ( (((prescription* or medication* or drug*) N2 (monitor* or manag* or adher* or compliance or ((report* or notif*) N2 system*))) or pharmacovigilance) ) OR AB ( (((prescription* or medication* or drug*) N2 (monitor* or manag* or adher* or compliance or ((report* or notif*) N2 system*))) or pharmacovigilance) ) OR SU ( (((prescription* or medication* or drug*) N2 (monitor* or manag* or adher* or compliance or ((report* or notif*) N2 system*))) or pharmacovigilance) ) Limiters - Date of Publication: 20000101-20191231 159,861 S32 TI ( ((in-service or inservice or staff) N2 (train* or learning or quiz* or "interactive exercise*")) ) OR AB ( ((in-service or inservice or staff) N2 (train* or learning or quiz* or "interactive exercise*")) ) OR SU ( ((in-service or inservice or staff) N2 (train* or learning or quiz* or "interactive exercise*")) ) Limiters - Date of Publication: 20000101-20191231 159,693 S31 TI ( ((appointment* or schedul* or task* or work) N3 (plan* or manag* or priorit*)) ) OR AB ( ((appointment* or schedul* or task* or work) N3 (plan* or manag* or priorit*)) ) OR SU ( ((appointment* or schedul* or task* or work) N3 (plan* or manag* or priorit*)) ) Limiters - Date of Publication: 20000101-20191231 322,667 S30 TI ( ((emergenc* or ambulance*) N3 (system* or manag*)) ) OR AB ( ((emergenc* or ambulance*) N3 (system* or manag*)) ) OR SU ( ((emergenc* or ambulance*) N3 (system* or manag*)) ) Limiters - Date of Publication: 20000101-20191231 140,147 65 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT S29 TI ( (((health or management or medical or clinical) N3 (system or systems or information or referral or (task* N2 link*))) or ("health care" N2 delivery)) ) OR AB ( (((health or management or medical or clinical) N3 (system or systems or information or referral or (task* N2 link*))) or ("health care" N2 delivery)) ) OR SU ( (((health or management or medical or clinical) N3 (system or systems or information or referral or (task* N2 link*))) or ("health care" N2 delivery)) ) Limiters - Date of Publication: 20000101-20191231 1,752,397 S28 TI ( ((emergenc* or mass) N2 (alert* or messag* or reminder* or updat*)) ) OR AB ( ((emergenc* or mass) N2 (alert* or messag* or reminder* or updat*)) ) OR SU ( ((emergenc* or mass) N2 (alert* or messag* or reminder* or updat*)) ) Limiters - Date of Publication: 20000101-20191231 7,528 S27 TI ( ((remote* N2 consult*) or "call center*" or "call centre*" or hotline* or (case* N2 manag*)) ) OR AB ( ((remote* N2 consult*) or "call center*" or "call centre*" or hotline* or (case* N2 manag*)) ) OR SU ( ((remote* N2 consult*) or "call center*" or "call centre*" or hotline* or (case* N2 manag*)) ) Limiters - Date of Publication: 20000101-20191231 167,048 S26 TI ( (mentor* or supervis* or coaching or motivat*) ) OR AB ( (mentor* or supervis* or coaching or motivat*) ) OR SU ( (mentor* or supervis* or coaching or motivat*) ) Limiters - Date of Publication: 20000101-20191231 2,837,924 S25 TI ( ((patient* or client or clients or outpatient* or out-patient* or inpatient* or in- patient*) N2 (plan* or goal*)) ) OR AB ( ((patient* or client or clients or outpatient* or out-patient* or inpatient* or in-patient*) N2 (plan* or goal*)) ) OR SU ( ((patient* or client or clients or outpatient* or out-patient* or inpatient* or in-patient*) N2 (plan* or goal*)) ) Limiters - Date of Publication: 20000101-20191231 80,218 S24 TI ( ((therap* or prescrib* or prescript* or diagnos* or pharmacy or pharmacies or pharmacist*) N2 (computer* or digital or electronic)) ) OR AB ( ((therap* or prescrib* or prescript* or diagnos* or pharmacy or pharmacies or pharmacist*) N2 (computer* or digital or electronic)) ) OR SU ( ((therap* or prescrib* or prescript* or diagnos* or pharmacy or pharmacies or pharmacist*) N2 (computer* or digital or electronic)) ) Limiters - Date of Publication: 20000101-20191231 41,456 S23 TI ( ((risk* N2 (assess* or estimat* or calculat*)) or triage) ) OR AB ( ((risk* N2 (assess* or estimat* or calculat*)) or triage) ) OR SU ( ((risk* N2 (assess* or estimat* or calculat*)) or triage) ) Limiters - Date of Publication: 20000101- 20191231 782,514 S22 TI ( (((guideline* or protocol*) N4 (adher* or comply or complian* or observ*)) or checklist*) ) OR AB ( (((guideline* or protocol*) N4 (adher* or comply or complian* or observ*)) or checklist*) ) OR SU ( (((guideline* or protocol*) N4 (adher* or 66 APPENDICES comply or complian* or observ*)) or checklist*) ) Limiters - Date of Publication: 20000101-20191231 284,044 S21 TI ( ((decision* N3 (make or makes or making or made or support* or algorithm* or aid or aids or app or apps or application* or technique*)) or "expert system*" or job- aid* or "job aid*") ) OR AB ( ((decision* N3 (make or makes or making or made or support* or algorithm* or aid or aids or app or apps or application* or technique*)) or "expert system*" or job-aid* or "job aid*") ) OR SU ( ((decision* N3 (make or makes or making or made or support* or algorithm* or aid or aids or app or apps or application* or technique*)) or "expert system*" or job-aid* or "job aid*") ) Limiters - Date of Publication: 20000101-20191231 1,881,361 S20 TI ( ((record* or registration* or registry or registries or e-registr* or eregistr* or ((medical or health) N2 data)) N2 (patient* or hospital or hospitals or medical or client or clients or in-patient* or inpatient* or out-patient* or outpatient*)) ) OR AB ( ((record* or registration* or registry or registries or e-registr* or eregistr* or ((medical or health) N2 data)) N2 (patient* or hospital or hospitals or medical or client or clients or in-patient* or inpatient* or out-patient* or outpatient*)) ) OR SU ( ((record* or registration* or registry or registries or e-registr* or eregistr* or ((medical or health) N2 data)) N2 (patient* or hospital or hospitals or medical or client or clients or in-patient* or inpatient* or out-patient* or outpatient*)) ) Limiters - Date of Publication: 20000101-20191231 481,444 S19 S14 OR S15 OR S16 OR S17 OR S18 Limiters - Date of Publication: 20000101- 20191231 4,430,329 S18 TI ( (doula? or douladural? or "barefoot doctor?") ) OR AB ( (doula? or douladural? or "barefoot doctor?") ) OR SU ( (doula? or douladural? or "barefoot doctor?") ) Limiters - Date of Publication: 20000101-20191231 16,505 S17 TI ( ((community or village? or peer or indigenous or treatment) N3 ("health worker?" or "health care worker?" or "healthcare worker?" or "health advisor*" or volunteer* or educator* or facilitator* or distributor* or "extension worker*" or supporter* or counselor* or counsellor*)) ) OR AB ( ((community or village? or peer or indigenous or treatment) N3 ("health worker?" or "health care worker?" or "healthcare worker?" or "health advisor*" or volunteer* or educator* or facilitator* or distributor* or "extension worker*" or supporter* or counselor* or counsellor*)) ) OR SU ( ((community or village? or peer or indigenous or treatment) N3 ("health worker?" or "health care worker?" or "healthcare worker?" or "health advisor*" or volunteer* or educator* or facilitator* or distributor* or "extension worker*" or supporter* or counselor* or counsellor*)) ) Limiters - Date of Publication: 20000101-20191231 44,231 S16 TI ( (paraprofessional? or paramedic or paramedics or "paramedical worker?" or "paramedical personnel" or "allied health personnel" or "allied health worker?" or "support worker?" or "home health aide?") ) OR AB ( (paraprofessional? or 67 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT paramedic or paramedics or "paramedical worker?" or "paramedical personnel" or "allied health personnel" or "allied health worker?" or "support worker?" or "home health aide?") ) OR SU ( (paraprofessional? or paramedic or paramedics or "paramedical worker?" or "paramedical personnel" or "allied health personnel" or "allied health worker?" or "support worker?" or "home health aide?") ) Limiters - Date of Publication: 20000101-20191231 29,371 S15 TI ( (((lay or voluntary or volunteer? or untrained or unlicensed or nonprofessional? or "non professional?" or support or link or outreach or "out reach") N5 (worker? or visitor? or attendant? or aide or aides or support$ or person$ or helper? or carer? or caregiver? or care giver? or consultant? or assistant? or staff)) or promotores or promotora or promotoras or embajadoras or comodrones or abuela or "lay advocate*" or "lay health" or "lay advisors" or "lay educators" or "lay counselor*" or "lay counsellor*" or "lady health worker*" or "lay facilitator*" or "natural helpers" or linkworker? or monitrice* or consejeras) ) OR AB ( (((lay or voluntary or volunteer? or untrained or unlicensed or nonprofessional? or "non professional?" or support or link or outreach or "out reach") N5 (worker? or visitor? or attendant? or aide or aides or support$ or person$ or helper? or carer? or caregiver? or care giver? or consultant? or assistant? or staff)) or promotores or promotora or promotoras or embajadoras or comodrones or abuela or "lay advocate*" or "lay health" or "lay advisors" or "lay educators" or "lay counselor*" or "lay counsellor*" or "lady health worker*" or "lay facilitator*" or "natural helpers" or linkworker? or monitrice* or consejeras) ) OR SU ( (((lay or voluntary or volunteer? or untrained or unlicensed or nonprofessional? or "non professional?" or support or link or outreach or "out reach") N5 (worker? or visitor? or attendant? or aide or aides or support$ or person$ or helper? or carer? or caregiver? or care giver? or consultant? or assistant? or staff)) or promotores or promotora or promotoras or embajadoras or comodrones or abuela or "lay advocate*" or "lay health" or "lay advisors" or "lay educators" or "lay counselor*" or "lay counsellor*" or "lady health worker*" or "lay facilitator*" or "natural helpers" or linkworker? or monitrice* or consejeras) ) Limiters - Date of Publication: 20000101-20191231 270,932 S14 TI ( (((health or medical or healthcare or frontline or front-line) N1 (personnel or worker* or auxiliar* or staff or professional* or assistant* or provider* or administrator*)) or doctor* or physician* or GP or "general practitioner?" or "family doctor" or nurse* or midwi* or "clinical officer*" or pharmacist* or dentist* or vaccinator* or supervisor* or ((birth or childbirth or labor or labour) N1 (attendant? or assistant?))) ) OR AB ( (((health or medical or healthcare or frontline or front- line) N1 (personnel or worker* or auxiliar* or staff or professional* or assistant* or provider* or administrator*)) or doctor* or physician* or GP or "general practitioner?" or "family doctor" or nurse* or midwi* or "clinical officer*" or pharmacist* or dentist* or vaccinator* or supervisor* or ((birth or childbirth or labor or labour) N1 (attendant? or assistant?))) ) OR SU ( (((health or medical or healthcare or frontline or front-line) N1 (personnel or worker* or auxiliar* or staff or professional* or assistant* or provider* or administrator*)) or doctor* or physician* or GP or "general practitioner?" or "family doctor" or nurse* or midwi* or "clinical officer*" or pharmacist* or dentist* or vaccinator* or supervisor* or ((birth or 68 APPENDICES childbirth or labor or labour) N1 (attendant? or assistant?))) ) Limiters - Date of Publication: 20000101-20191231 4,150,912 S13 TI ( ((fee or fees or charge or charges or payment*) N2 (service* or hospital* or prescription* or medication or drug* or treatment*)) ) OR AB ( ((fee or fees or charge or charges or payment*) N2 (service* or hospital* or prescription* or medication or drug* or treatment*)) ) OR SU ( ((fee or fees or charge or charges or payment*) N2 (service* or hospital* or prescription* or medication or drug* or treatment*)) ) Limiters - Date of Publication: 20000101-20191231 79,442 S12 TI ( ((cash N3 (transfer* or voucher* or grant* or aid)) or "health voucher*") ) OR AB ( ((cash N3 (transfer* or voucher* or grant* or aid)) or "health voucher*") ) OR SU ( ((cash N3 (transfer* or voucher* or grant* or aid)) or "health voucher*") ) Limiters - Date of Publication: 20000101-20191231 19,391 S11 TI ( ((monitor* or satisfact* or accountab* or feedback) N3 (consumer* or public or patient* or client or clients) N3 (service or services)) ) OR AB ( ((monitor* or satisfact* or accountab* or feedback) N3 (consumer* or public or patient* or client or clients) N3 (service or services)) ) OR SU ( ((monitor* or satisfact* or accountab* or feedback) N3 (consumer* or public or patient* or client or clients) N3 (service or services)) ) Limiters - Date of Publication: 20000101-20191231 13,975 S10 TI ( ((peer or peer-to-peer) N2 (group* or influenc* or pressure or network* or support)) ) OR AB ( ((peer or peer-to-peer) N2 (group* or influenc* or pressure or network* or support)) ) OR SU ( ((peer or peer-to-peer) N2 (group* or influenc* or pressure or network* or support)) ) Limiters - Date of Publication: 20000101- 20191231 112,999 S9 TI ( ((communicat* or messag* or online or access* or seek*) N3 (health or information)) ) OR AB ( ((communicat* or messag* or online or access* or seek*) N3 (health or information)) ) OR SU ( ((communicat* or messag* or online or access* or seek*) N3 (health or information)) ) Limiters - Date of Publication: 20000101-20191231 999,392 S8 TI ( ((health or medical or personal) N2 (record* or data or monitor* or self-monitor* or track* or self-track* or registr*)) ) OR AB ( ((health or medical or personal) N2 (record* or data or monitor* or self-monitor* or track* or self-track* or registr*)) ) OR SU ( ((health or medical or personal) N2 (record* or data or monitor* or self- monitor* or track* or self-track* or registr*)) ) Limiters - Date of Publication: 20000101-20191231 627,468 69 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT S7 TI ( ((access* or "use" or "using" or "uses") N3 "health service*") ) OR AB ( ((access* or "use" or "using" or "uses") N3 "health service*") ) OR SU ( ((access* or "use" or "using" or "uses") N3 "health service*") ) Limiters - Date of Publication: 20000101-20191231 108,484 S6 TI ( (patient* or client or clients or outpatient* or out-patient* or inpatient* or in- patient*) ) OR AB ( (patient* or client or clients or outpatient* or out-patient* or inpatient* or in-patient*) ) OR SU ( (patient* or client or clients or outpatient* or out- patient* or inpatient* or in-patient*) ) Limiters - Date of Publication: 20000101- 20191231 13,221,480 S5 S2 OR S3 OR S4 Limiters - Date of Publication: 20000101-20191231 1,643,699 S4 TI ( (mhealth or m-health or "mobile health" or ehealth or e-health or "electronic health" or "digital health" or uhealth or u-health or telemedicine or tele-medicine or telehealth or tele-health or telecare or tele-care or telenursing or tele-nursing or telepsychiatry or tele-psychiatry or telemonitor* or tele-monitor* or teleconsult* or tele-consult* or telecounsel* or tele-counsel* or telecoach* or tele-coach* or webcast* or web-cast* or ((text* or short or voice or multimedia or multi-media or electronic or instant) N1 messag*) or "instant messenger" or texting or texted or texter* or ((sms or mms) N1 (service* or messag*)) or "interactive voice response*" or IVR or "voice call*" or callback* or "voice over internet" or VOIP or Facebook or Twitter or Whatsapp* or Skyp* or YouTube or "You Tube" or "Google Hangout*" or "mobile app*" or (remind* N3 (text* or system* or messag*)) or ((medical or clinical or health or healthcare or nurs*) N3 informatics) or ((interactive or computer- assisted) N1 (tutor* or technolog* or learn* or instruct* or software or communication)) or ((media or radio or television or tv or online or public) N3 campaign*) or GIS or "geographic information system*" or "global positioning") ) OR AB ( (mhealth or m-health or "mobile health" or ehealth or e-health or "electronic health" or "digital health" or uhealth or u-health or telemedicine or tele- medicine or telehealth or tele-health or telecare or tele-care or telenursing or tele- nursing or telepsychiatry or tele-psychiatry or telemonitor* or tele-monitor* or teleconsult* or tele-consult* or telecounsel* or tele-counsel* or telecoach* or tele- coach* or webcast* or web-cast* or ((text* or short or voice or multimedia or multi- media or electronic or instant) N1 messag*) or "instant messenger" or texting or texted or texter* or ((sms or mms) N1 (service* or messag*)) or "interactive voice response*" or IVR or "voice call*" or callback* or "voice over internet" or VOIP or Facebook or Twitter or Whatsapp* or Skyp* or YouTube or "You Tube" or "Google Hangout*" or "mobile app*" or (remind* N3 (text* or system* or messag*)) or ((medical or clinical or health or healthcare or nurs*) N3 informatics) or ((interactive or computer-assisted) N1 (tutor* or technolog* or learn* or instruct* or software or communication)) or ((media or radio or television or tv or online or public) N3 campaign*) or GIS or "geographic information system*" or "global positioning") ) OR SU ( (mhealth or m-health or "mobile health" or ehealth or e-health or "electronic health" or "digital health" or uhealth or u-health or telemedicine or tele- medicine or telehealth or tele-health or telecare or tele-care or telenursing or tele- nursing or telepsychiatry or tele-psychiatry or telemonitor* or tele-monitor* or teleconsult* or tele-consult* or telecounsel* or tele-counsel* or telecoach* or tele- 70 APPENDICES coach* or webcast* or web-cast* or ((text* or short or voice or multimedia or multi- media or electronic or instant) N1 messag*) or "instant messenger" or texting or texted or texter* or ((sms or mms) N1 (service* or messag*)) or "interactive voice response*" or IVR or "voice call*" or callback* or "voice over internet" or VOIP or Facebook or Twitter or Whatsapp* or Skyp* or YouTube or "You Tube" or "Google Hangout*" or "mobile app*" or (remind* N3 (text* or system* or messag*)) or ((medical or clinical or health or healthcare or nurs*) N3 informatics) or ((interactive or computer-assisted) N1 (tutor* or technolog* or learn* or instruct* or software or communication)) or ((media or radio or television or tv or online or public) N3 campaign*) or GIS or "geographic information system*" or "global positioning") ) Limiters - Date of Publication: 20000101-20191231 690,777 S3 TI ( (handheld or hand-held or smartphone* or smart-phone* or cellphone* or mobiles or ((personal N1 digital) or (PDA N3 (device* or assistant*)) or MP3 player* or MP4 player* or samsung or nokia or (windows N3 (mobile* or phone*)) or android or ipad* or i-pad* or ipod* or i-pod* or iphone* or i-phone* or (tablet* N3 (device* or computer*)) ) OR AB ( (handheld or hand-held or smartphone* or smart-phone* or cellphone* or mobiles or ((personal N1 digital) or (PDA N3 (device* or assistant*)) or MP3 player* or MP4 player* or samsung or nokia or (windows N3 (mobile* or phone*)) or android or ipad* or i-pad* or ipod* or i-pod* or iphone* or i-phone* or (tablet* N3 (device* or computer*)) ) OR SU ( (handheld or hand-held or smartphone* or smart-phone* or cellphone* or mobiles or ((personal N1 digital) or (PDA N3 (device* or assistant*)) or MP3 player* or MP4 player* or samsung or nokia or (windows N3 (mobile* or phone*)) or android or ipad* or i- pad* or ipod* or i-pod* or iphone* or i-phone* or (tablet* N3 (device* or computer*)) ) Limiters - Date of Publication: 20000101-20191231 2,312,335 S2 TI ( ((cell* or mobile*) N1 (phone* or telephone* or technolog* or device*)) ) OR AB ( ((cell* or mobile*) N1 (phone* or telephone* or technolog* or device*)) ) OR SU ( ((cell* or mobile*) N1 (phone* or telephone* or technolog* or device*)) ) Limiters - Date of Publication: 20000101-20191231 440,786 S1 TI ( ("quasi experiment*" or quasi-experiment* or "random* control* trial*" or "random* trial*" or RCT or (random* N3 allocat*) or matching or "propensity score" or PSM or "regression discontinuity" or "discontinuous design" or RDD or "synthetic control" or "difference in difference*" or difference-in-difference* or "diff in diff" or cohort or "propensity weighted" or propensity-weighted or "interrupted time series" or (pre N5 post) or ((pretest or "pre test") and (posttest or "post test")) or "rapid evidence assessment" or "systematic literature review" or "systematic review" or "meta-analy*" or metaanaly* or "meta analy*" or "instrumental variable*" or heckman or ((evaluation or impact or quantitative or "comparison group*" or counterfactual or "counter factual" or counter-factual or experiment*) N3 (design or study or analysis)) or QED) ) OR AB ( ("quasi experiment*" or quasi-experiment* or "random* control* trial*" or "random* trial*" or RCT or (random* N3 allocat*) or matching or "propensity score" or PSM or "regression discontinuity" or "discontinuous design" or RDD or "synthetic control" or "difference in difference*" or difference-in-difference* or "diff in diff" or cohort or "propensity weighted" or propensity-weighted or "interrupted time series" or (pre N5 post) or ((pretest or "pre 71 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT test") and (posttest or "post test")) or "rapid evidence assessment" or "systematic literature review" or "systematic review" or "meta-analy*" or metaanaly* or "meta analy*" or "instrumental variable*" or heckman or ((evaluation or impact or quantitative or "comparison group*" or counterfactual or "counter factual" or counter-factual or experiment*) N3 (design or study or analysis)) or QED) ) OR SU ( ("quasi experiment*" or quasi-experiment* or "random* control* trial*" or "random* trial*" or RCT or (random* N3 allocat*) or matching or "propensity score" or PSM or "regression discontinuity" or "discontinuous design" or RDD or "synthetic control" or "difference in difference*" or difference-in-difference* or "diff in diff" or cohort or "propensity weighted" or propensity-weighted or "interrupted time series" or (pre N5 post) or ((pretest or "pre test") and (posttest or "post test")) or "rapid evidence assessment" or "systematic literature review" or "systematic review" or "meta- analy*" or metaanaly* or "meta analy*" or "instrumental variable*" or heckman or ((evaluation or impact or quantitative or "comparison group*" or counterfactual or "counter factual" or counter-factual or experiment*) N3 (design or study or analysis)) or QED) ) Limiters - Date of Publication: 20000101-20191231 6,054,311 3. SOCIAL SCIENCES CITATION INDEX (WEB OF SCIENCE) – SEARCHED 23RD APRIL 2019 # 54 2,306 #53 AND #5 AND #1 Indexes = SSCI Timespan=2000-2019 # 53 100,309 #52 OR #51 OR #50 OR #49 # 52 16,469 TS = ( ((data or information) NEAR/2 (mediat* or interoperability or accessib* or exchange or integrat* or orchestrat*)) ) # 51 18,859 TS = ( ("geographic information systems" or "geographic* mapping" or GIS or (geospatial NEAR/2 visual*) or gps or "global positioning system") ) # 50 63,122 TS = ( (coding or "cause of death" or ICD or ((disease* or dataset*) NEAR/2 (code* or coding or coded)) or "dirty data" or (automat* NEAR/2 "data cleaning")) ) # 49 4,199 TS = ( ((data NEAR/2 (collect* or aggregat* or manag* or synthes* or analys*) NEAR/2 automated) or (data NEAR/2 (electronic or mobile-based or dashboard* or analytics)) or OpenDataKit or Enketo or Formhub) ) # 48 420 #47 AND #5 AND #1 # 47 84,353 #46 OR #45 OR #44 OR #43 OR #42 OR #41 OR #40 OR #39 72 APPENDICES # 46 5,498 TS = ( (("health facilit*" or hospital or hospitals) NEAR/3 (registr* or regulat* or manag* or performance or monitor* or (service* NEAR/2 (capacit* or provid* or provision)))) ) # 45 14,070 TS = ( ((insurance or financ* or budget*) NEAR/2 (coverage or incentiv* or conditional or performance-based or results-based or manag*)) ) # 44 7,659 TS = ( (((death* or mortality or vital) NEAR/3 (registr* or notif* or report* or record* or log* or certif* or collection or survey* or surveillance)) or "verbal autops*") ) # 43 3,383 TS = ( (birth NEAR/3 (registr* or notif* or report* or record* or log* or certif* or collection or survey* or surveillance)) ) # 42 191 TS = ( ((surveillance or monitoring or notification) NEAR/2 (outbreak* or epidemic* or "public health event*")) ) # 41 8,362 TS = ( ((health or medical or medicines or vaccine* or drug or drugs or laborator* or diagnos*) NEAR/3 (product* or supply or supplies or consumable* or commodit* or stock or stocks or stockout* or "stock out*" or shortage*)) ) # 40 33,101 TS = ( ((commodit* or consumable* or stock or stocks or supply or supplies or equipment) NEAR/3 (inventor* or level* or notif* or track* or count* or report* or chain or out or outs or manag* or order* or logistic* or system or systems or shortage* or manag* or monitor* or maintain* or maintenance or audit or auditing) or (asset* NEAR/2 manag*)) ) # 39 15,081 TS = ( ((worker* or workforce or "human resource*" or employee* or personnel) NEAR/2 (manag* or appraisal or registr* or certific* or licens* or ((task* or work) NEAR/2 (track* or monitor*)) or assign* or supervis*)) ) # 38 3,141 #37 AND #21 AND #5 AND #1 # 37 485,569 #36 OR #35 OR #34 OR #33 OR #32 OR #31 OR #30 OR #29 OR #28 OR #27 OR #26 OR #25 OR #24 OR #23 OR #22 # 36 3,874 TS = ((laborat* or diagnostic) NEAR/2 (system* or technique* or imaging or requisition* or communicat* or report*)) # 35 16,257 TS = ( (((prescription* or medication* or drug*) NEAR/2 (monitor* or manag* or adher* or compliance or ((report* or notif*) NEAR/2 system*))) or pharmacovigilance) ) 73 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT # 34 5,483 TS = ( ((in-service or inservice or staff) NEAR/2 (train* or learning or quiz* or "interactive exercise*")) ) # 33 13,866 TS = ( ((appointment* or schedul* or task* or work) NEAR/3 (plan* or manag* or priorit*)) ) # 32 4,064 TS = ( ((emergenc* or ambulance*) NEAR/3 (system* or manag*)) ) # 31 86,774 TS = ( (((health or management or medical or clinical) NEAR/3 (system or systems or information or referral or (task* NEAR/2 link*))) or ("health care" NEAR/2 delivery)) ) # 30 241 TS = ( ((emergenc* or mass) NEAR/2 (alert* or messag* or reminder* or updat*)) ) # 29 10,423 TS = ( ((remote* NEAR/2 consult*) or "call center*" or "call centre*" or hotline* or (case* NEAR/2 manag*)) ) # 28 164,926 TS = ( (mentor* or supervis* or coaching or motivat*) ) # 27 4,354 TS = ( ((patient* or client or clients or outpatient* or out-patient* or inpatient* or in- patient*) NEAR/2 (plan* or goal*)) ) # 26 1,644 TS = ( ((therap* or prescrib* or prescript* or diagnos* or pharmacy or pharmacies or pharmacist*) NEAR/2 (computer* or digital or electronic)) ) # 25 34,535 TS = ( ((risk* NEAR/2 (assess* or estimat* or calculat*)) or triage) ) # 24 23,964 TS = ( (((guideline* or protocol*) NEAR/4 (adher* or comply or complian* or observ*)) or checklist*) ) # 23 150,857 TS = ( ((decision* NEAR/3 (make or makes or making or made or support* or algorithm* or aid or aids or app or apps or application* or technique*)) or "expert system*" or job-aid* or "job aid*") ) # 22 23,707 TS = ( ((record* or registration* or registry or registries or e-registr* or eregistr* or ((medical or health) NEAR/2 data)) NEAR/2 (patient* or hospital or hospitals or medical or client or clients or in-patient* or inpatient* or out-patient* or outpatient*)) ) 74 APPENDICES # 21 666,202 #19 OR #18 OR #17 OR #16 # 20 121 TS = ( (doula? or douladural? or "barefoot doctor?") ) # 19 6,111 TS = ( ((community or village? or peer or indigenous or treatment) NEAR/3 ("health worker?" or "health care worker?" or "healthcare worker?" or "health advisor*" or volunteer* or educator* or facilitator* or distributor* or "extension worker*" or supporter* or counselor* or counsellor*)) ) # 18 1,860 TS = ( (paraprofessional? or paramedic or paramedics or "paramedical worker?" or "paramedical personnel" or "allied health personnel" or "allied health worker?" or "support worker?" or "home health aide?") ) # 17 475,379 TS = ( (((lay or voluntary or volunteer? or untrained or unlicensed or nonprofessional? or "non professional?" or support or link or outreach or "out reach") NEAR/5 (worker? or visitor? or attendant? or aide or aides or support$ or person$ or helper? or carer? or caregiver? or "care giver?" or consultant? or assistant? or staff)) or promotores or promotora or promotoras or embajadoras or comodrones or abuela or "lay advocate*" or "lay health" or "lay advisors" or "lay educators" or "lay counselor*" or "lay counsellor*" or "lady health worker*" or "lay facilitator*" or "natural helpers" or linkworker? or monitrice* or consejeras) ) # 16 245,444 TS = ( (((health or medical or healthcare or frontline or front-line) NEAR/1 (personnel or worker* or auxiliar* or staff or professional* or assistant* or provider* or administrator*)) or doctor* or physician* or GP or "general practitioner?" or "family doctor" or nurse* or midwi* or "clinical officer*" or pharmacist* or dentist* or vaccinator* or supervisor* or ((birth or childbirth or labor or labour) NEAR/1 (attendant? or assistant?))) ) # 15 2,048 #14 AND #6 AND #5 AND #1 # 14 120,568 #13 OR #12 OR #11 OR #10 OR #9 OR #8 OR #7 # 13 5,610 TS = ( ((fee or fees or charge or charges or payment*) NEAR/2 (service* or hospital* or prescription* or medication or drug* or treatment*)) ) # 12 1,607 TS = ( ((cash NEAR/3 (transfer* or voucher* or grant* or aid)) or "health voucher*") ) # 11 989 TS = ( ((monitor* or satisfact* or accountab* or feedback) NEAR/3 (consumer* or public or patient* or client or clients) NEAR/3 (service or services)) ) 75 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT # 10 13,952 TS = ( ((peer or peer-to-peer) NEAR/2 (group* or influenc* or pressure or network* or support)) ) #9 59,590 TS = ( ((communicat* or messag* or online or access* or seek*) NEAR/3 (health or information)) ) #8 39,035 TS = ( ((health or medical or personal) NEAR/2 (record* or data or monitor* or self- monitor* or track* or self-track* or registr*)) ) #7 8,708 TS = ( ((access* or "use" or "using" or "uses") NEAR/3 "health service*") ) #6 501,662 TS = ( (patient* or client or clients or outpatient* or out-patient* or inpatient* or in- patient*) ) #5 84,001 #4 OR #3 OR #2 #4 57,198 TS = ( (mhealth or m-health or "mobile health" or ehealth or e-health or "electronic health" or "digital health" or uhealth or u-health or telemedicine or tele-medicine or telehealth or tele-health or telecare or tele-care or telenursing or tele-nursing or telepsychiatry or tele-psychiatry or telemonitor* or tele-monitor* or teleconsult* or tele-consult* or telecounsel* or tele-counsel* or telecoach* or tele-coach* or webcast* or web-cast* or ((text* or short or voice or multimedia or multi-media or electronic or instant) NEAR/1 messag*) or "instant messenger" or texting or texted or texter* or ((sms or mms) NEAR/1 (service* or messag*)) or "interactive voice response*" or IVR or "voice call*" or callback* or "voice over internet" or VOIP or Facebook or Twitter or Whatsapp* or Skyp* or YouTube or "You Tube" or "Google Hangout*" or "mobile app*" or (remind* NEAR/3 (text* or system* or messag*)) or ((medical or clinical or health or healthcare or nurs*) NEAR/3 informatics) or ((interactive or computer-assisted) NEAR/1 (tutor* or technolog* or learn* or instruct* or software or communication)) or ((media or radio or television or tv or online or public) NEAR/3 campaign*) or GIS or "geographic information system*" or "global positioning") ) #3 32,626 TS = (handheld or hand-held or smartphone* or smart-phone* or cellphone* or mobiles or (personal NEAR/1 digital) or (PDA NEAR/3 (device* or assistant*)) or MP3 player* or MP4 player* or samsung or nokia or (windows NEAR/3 (mobile* or phone*)) or android or ipad* or i-pad* or ipod* or i-pod* or iphone* or i-phone* or (tablet* NEAR/3 (device* or computer*))) #2 12,715 TS = ( ((cell* or mobile*) NEAR/1 (phone* or telephone* or technolog* or device*)) ) 76 APPENDICES #1 466,070 TS = ( ("quasi experiment*" or quasi-experiment* or "random* control* trial*" or "random* trial*" or RCT or (random* NEAR/3 allocat*) or matching or "propensity score" or PSM or "regression discontinuity" or "discontinuous design" or RDD or "synthetic control" or "difference in difference*" or difference-in-difference* or "diff in diff" or cohort or "propensity weighted" or propensity-weighted or "interrupted time series" or (pre NEAR/5 post) or ((pretest or "pre test") and (posttest or "post test")) or "rapid evidence assessment" or "systematic literature review" or "systematic review" or "meta-analy*" or metaanaly* or "meta analy*" or "instrumental variable*" or heckman or ((evaluation or impact or quantitative or "comparison group*" or counterfactual or "counter factual" or counter-factual or experiment*) NEAR/3 (design or study or analysis)) or QED) ) 4. CINAHL (EBSCO) – SEARCHED 24TH APRIL 2019 S86 S23 OR S60 OR S75 OR S85 Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records Database - CINAHL Plus with Full Text 3,623 S85 S1 AND S5 AND S84 Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records Search modes - Boolean/Phrase Interface - EBSCOhost Research Databases Search Screen - Advanced Search Database - CINAHL Plus with Full Text 889 S84 S76 OR S77 OR S78 OR S79 OR S80 OR S81 OR S82 OR S83 51,195 S83 (MH "Electronic Data Interchange") Limiters - Published Date: 20000101- 20191231; Exclude MEDLINE records 1,827 S82 TI ( ((data or information) N2 (mediat* or interoperability or accessib* or exchange or integrat* or orchestrat*)) ) OR AB ( ((data or information) N2 (mediat* or interoperability or accessib* or exchange or integrat* or orchestrat*)) ) OR SU ( ((data or information) N2 (mediat* or interoperability or accessib* or exchange or integrat* or orchestrat*)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 3,249 S81 (MH "Global Positioning System") OR (MH "Geographic Information Systems") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 1,291 S80 TI ( ("geographic information systems" or "geographic* mapping" or GIS or (geospatial N2 visual*) or gps or "global positioning system") ) OR AB ( ("geographic information systems" or "geographic* mapping" or GIS or (geospatial N2 visual*) or gps or "global positioning system") ) OR SU ( ("geographic information systems" or "geographic* mapping" or GIS or (geospatial N2 visual*) or gps or "global positioning system") ) Limiters - Published Date: 20000101- 20191231; Exclude MEDLINE records 4,584 77 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT S79 (MH "Coding, Computer-Assisted") OR (MH "Coding") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 16,272 S78 TI ( (coding or "cause of death" or ICD or ((disease* or dataset*) N2 (code* or coding or coded)) or "dirty data" or (automat* N2 "data cleaning")) ) OR AB ( (coding or "cause of death" or ICD or ((disease* or dataset*) N2 (code* or coding or coded)) or "dirty data" or (automat* N2 "data cleaning")) ) OR SU ( (coding or "cause of death" or ICD or ((disease* or dataset*) N2 (code* or coding or coded)) or "dirty data" or (automat* N2 "data cleaning")) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 26,577 S77 (MH "Data Collection") OR (MH "Data Collection, Computer Assisted") OR (MH "Data Collection Methods") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 13,500 S76 TI ( ((data N2 (collect* or aggregat* or manag* or synthes* or analys*) N2 automated) or (data N2 (electronic or mobile-based or dashboard* or analytics)) or OpenDataKit or Enketo or Formhub) ) OR AB ( ((data N2 (collect* or aggregat* or manag* or synthes* or analys*) N2 automated) or (data N2 (electronic or mobile- based or dashboard* or analytics)) or OpenDataKit or Enketo or Formhub) ) OR SU ( ((data N2 (collect* or aggregat* or manag* or synthes* or analys*) N2 automated) or (data N2 (electronic or mobile-based or dashboard* or analytics)) or OpenDataKit or Enketo or Formhub) ) Limiters - Published Date: 20000101- 20191231; Exclude MEDLINE records 4,834 S75 S1 AND S5 AND S74 1,129 S74 S61 OR S62 OR S63 OR S64 OR S65 OR S66 OR S67 OR S68 OR S69 OR S70 OR S71 OR S72 OR S73 Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 301,141 S73 TI ( (("health facilit*" or hospital or hospitals) N3 (registr* or regulat* or manag* or performance or monitor* or (service* N2 (capacit* or provid* or provision)))) ) OR AB ( (("health facilit*" or hospital or hospitals) N3 (registr* or regulat* or manag* or performance or monitor* or (service* N2 (capacit* or provid* or provision)))) ) OR SU ( (("health facilit*" or hospital or hospitals) N3 (registr* or regulat* or manag* or performance or monitor* or (service* N2 (capacit* or provid* or provision)))) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 3,536 S72 (MH "Insurance Coverage") OR (MH "Managed Care Information Systems+") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 5,457 S71 TI ( ((insurance or financ* or budget*) N2 (coverage or incentiv* or conditional or performance-based or results-based or manag*)) ) OR AB ( ((insurance or financ* or budget*) N2 (coverage or incentiv* or conditional or performance-based or 78 APPENDICES results-based or manag*)) ) OR SU ( ((insurance or financ* or budget*) N2 (coverage or incentiv* or conditional or performance-based or results-based or manag*)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 12,789 S70 (MH "Birth Certificates") OR (MH "Death Certificates") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 678 S69 TI ( (((death* or mortality or vital) N3 (registr* or notif* or report* or record* or log* or certif* or collection or survey* or surveillance)) or "verbal autops*") ) OR AB ( (((death* or mortality or vital) N3 (registr* or notif* or report* or record* or log* or certif* or collection or survey* or surveillance)) or "verbal autops*") ) OR SU ( (((death* or mortality or vital) N3 (registr* or notif* or report* or record* or log* or certif* or collection or survey* or surveillance)) or "verbal autops*") ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 3,335 S68 TI ( (birth N3 (registr* or notif* or report* or record* or log* or certif* or collection or survey* or surveillance)) ) OR AB ( (birth N3 (registr* or notif* or report* or record* or log* or certif* or collection or survey* or surveillance)) ) OR SU ( (birth N3 (registr* or notif* or report* or record* or log* or certif* or collection or survey* or surveillance)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 1,214 S67 (MH "Disease Surveillance") OR (MH "Biosurveillance") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 5,528 S66 TI ( ((surveillance or monitoring or notification) N2 (outbreak* or epidemic* or "public health event*")) ) OR AB ( ((surveillance or monitoring or notification) N2 (outbreak* or epidemic* or "public health event*")) ) OR SU ( ((surveillance or monitoring or notification) N2 (outbreak* or epidemic* or "public health event*")) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 125 S65 TI ( ((health or medical or medicines or vaccine* or drug or drugs or laborator* or diagnos*) N3 (product* or supply or supplies or consumable* or commodit* or stock or stocks or stockout* or "stock out*" or shortage*)) ) OR AB ( ((health or medical or medicines or vaccine* or drug or drugs or laborator* or diagnos*) N3 (product* or supply or supplies or consumable* or commodit* or stock or stocks or stockout* or "stock out*" or shortage*)) ) OR SU ( ((health or medical or medicines or vaccine* or drug or drugs or laborator* or diagnos*) N3 (product* or supply or supplies or consumable* or commodit* or stock or stocks or stockout* or "stock out*" or shortage*)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 7,822 79 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT S64 (MH "Equipment and Supplies+") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 133,409 S63 TI ( ((commodit* or consumable* or stock or stocks or supply or supplies or equipment) N3 (inventor* or level* or notif* or track* or count* or report* or chain or out or outs or manag* or order* or logistic* or system or systems or shortage* or manag* or monitor* or maintain* or maintenance or audit or auditing) or (asset* N2 manag*)) ) OR AB ( ((commodit* or consumable* or stock or stocks or supply or supplies or equipment) N3 (inventor* or level* or notif* or track* or count* or report* or chain or out or outs or manag* or order* or logistic* or system or systems or shortage* or manag* or monitor* or maintain* or maintenance or audit or auditing) or (asset* N2 manag*)) ) OR SU ( ((commodit* or consumable* or stock or stocks or supply or supplies or equipment) N3 (inventor* or level* or notif* or track* or count* or report* or chain or out or outs or manag* or order* or logistic* or system or systems or shortage* or manag* or monitor* or maintain* or maintenance or audit or auditing) or (asset* N2 manag*)) ) Limiters - Published Date: 20000101- 20191231; Exclude MEDLINE records 6,109 S62 (MH "Personnel Management+") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 136,916 S61 TI ( ((worker* or workforce or "human resource*" or employee* or personnel) N2 (manag* or appraisal or registr* or certific* or licens* or ((task* or work) N2 (track* or monitor*)) or assign* or supervis*)) ) OR AB ( ((worker* or workforce or "human resource*" or employee* or personnel) N2 (manag* or appraisal or registr* or certific* or licens* or ((task* or work) N2 (track* or monitor*)) or assign* or supervis*)) ) OR SU ( ((worker* or workforce or "human resource*" or employee* or personnel) N2 (manag* or appraisal or registr* or certific* or licens* or ((task* or work) N2 (track* or monitor*)) or assign* or supervis*)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 7,519 S60 S1 AND S5 AND S30 AND S59 1,304 S59 S31 OR S32 OR S33 OR S34 OR S35 OR S36 OR S37 OR S38 OR S39 OR S40 OR S41 OR S42 OR S43 OR S44 OR S45 OR S46 OR S47 OR S48 OR S49 OR S50 OR S51 OR S52 OR S53 OR S54 OR S55 OR S56 OR S57 OR S58 Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 322,518 S58 (MH "Clinical Laboratory Information Systems") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 411 S57 TI ( ((laborat* or diagnostic) N2 (system* or technique* or imaging or requisition* or communicat* or report*)) ) OR AB ( ((laborat* or diagnostic) N2 (system* or technique* or imaging or requisition* or communicat* or report*)) ) OR AB ( ((laborat* or diagnostic) N2 (system* or technique* or imaging or requisition* or 80 APPENDICES communicat* or report*)) ) OR SU ( ((laborat* or diagnostic) N2 (system* or technique* or imaging or requisition* or communicat* or report*)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 11,385 S56 (MH "Prescription Drug Monitoring Programs") OR (MH "Pharmacovigilance") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 467 S55 TI ( (((prescription* or medication* or drug*) N2 (monitor* or manag* or adher* or compliance or ((report* or notif*) N2 system*))) or pharmacovigilance) ) OR AB ( (((prescription* or medication* or drug*) N2 (monitor* or manag* or adher* or compliance or ((report* or notif*) N2 system*))) or pharmacovigilance) ) OR SU ( (((prescription* or medication* or drug*) N2 (monitor* or manag* or adher* or compliance or ((report* or notif*) N2 system*))) or pharmacovigilance) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 13,981 S54 (MH "Staff Development") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 12,727 S53 TI ( ((in-service or inservice or staff) N2 (train* or learning or quiz* or "interactive exercise*")) ) OR AB ( ((in-service or inservice or staff) N2 (train* or learning or quiz* or "interactive exercise*")) ) OR SU ( ((in-service or inservice or staff) N2 (train* or learning or quiz* or "interactive exercise*")) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 5,274 S52 (MH "Appointment and Scheduling Information Systems") OR (MH "Personnel Staffing and Scheduling Information Systems") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 270 S51 TI ( ((appointment* or schedul* or task* or work) N3 (plan* or manag* or priorit*)) ) OR AB ( ((appointment* or schedul* or task* or work) N3 (plan* or manag* or priorit*)) ) OR SU ( ((appointment* or schedul* or task* or work) N3 (plan* or manag* or priorit*)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 3,267 S50 (MH "Emergency Service Information Systems") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 222 S49 TI ( ((emergenc* or ambulance*) N3 (system* or manag*)) ) OR AB ( ((emergenc* or ambulance*) N3 (system* or manag*)) ) OR SU ( ((emergenc* or ambulance*) N3 (system* or manag*)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 3,346 81 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT S48 (MH "Health Information Systems+") OR (MH "Hospital Information Systems") OR (MH "Health Care Delivery, Integrated") Limiters - Published Date: 20000101- 20191231; Exclude MEDLINE records 26,671 S47 TI ( (((health or management or medical or clinical) N3 (system or systems or information or referral or (task* N2 link*))) or ("health care" N2 delivery)) ) OR AB ( (((health or management or medical or clinical) N3 (system or systems or information or referral or (task* N2 link*))) or ("health care" N2 delivery)) ) OR SU ( (((health or management or medical or clinical) N3 (system or systems or information or referral or (task* N2 link*))) or ("health care" N2 delivery)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 78,741 S46 TI ( ((emergenc* or mass) N2 (alert* or messag* or reminder* or updat*)) ) OR AB ( ((emergenc* or mass) N2 (alert* or messag* or reminder* or updat*)) ) OR SU ( ((emergenc* or mass) N2 (alert* or messag* or reminder* or updat*)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 150 S45 (MH "Remote Consultation") OR (MH "Case Management") OR (MH "Telephone Information Services") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 8,917 S44 TI ( ((remote* N2 consult*) or "call center*" or "call centre*" or hotline* or (case* N2 manag*)) ) OR AB ( ((remote* N2 consult*) or "call center*" or "call centre*" or hotline* or (case* N2 manag*)) ) OR SU ( ((remote* N2 consult*) or "call center*" or "call centre*" or hotline* or (case* N2 manag*)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 11,936 S43 (MH "Mentorship") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 7,692 S42 TI ( (mentor* or supervis* or coaching or motivat*) ) OR AB ( (mentor* or supervis* or coaching or motivat*) ) OR SU ( (mentor* or supervis* or coaching or motivat*) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 49,773 S41 (MH "Nursing Care Plans+") OR (MH "Patient Care Plans+") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 3,399 S40 TI ( ((patient* or client or clients or outpatient* or out-patient* or inpatient* or in- patient*) N2 (plan* or goal*)) ) OR AB ( ((patient* or client or clients or outpatient* or out-patient* or inpatient* or in-patient*) N2 (plan* or goal*)) ) OR SU ( ((patient* or client or clients or outpatient* or out-patient* or inpatient* or in-patient*) N2 (plan* or goal*)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 6,859 82 APPENDICES S39 TI ( ((therap* or prescrib* or prescript* or diagnos* or pharmacy or pharmacies or pharmacist*) N2 (computer* or digital or electronic)) ) OR AB ( ((therap* or prescrib* or prescript* or diagnos* or pharmacy or pharmacies or pharmacist*) N2 (computer* or digital or electronic)) ) OR SU ( ((therap* or prescrib* or prescript* or diagnos* or pharmacy or pharmacies or pharmacist*) N2 (computer* or digital or electronic)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 2,665 S38 (MH "Risk Assessment") OR (MH "Triage") Limiters - Published Date: 20000101- 20191231; Exclude MEDLINE records 21,487 S37 TI ( ((risk* N2 (assess* or estimat* or calculat*)) or triage) ) OR AB ( ((risk* N2 (assess* or estimat* or calculat*)) or triage) ) OR SU ( ((risk* N2 (assess* or estimat* or calculat*)) or triage) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 28,961 S36 (MH "Practice Guidelines") OR (MH "Guideline Adherence") OR (MH "Checklists") OR (MH "Protocols+") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 53,094 S35 TI ( (((guideline* or protocol*) N4 (adher* or comply or complian* or observ*)) or checklist*) ) OR AB ( (((guideline* or protocol*) N4 (adher* or comply or complian* or observ*)) or checklist*) ) OR SU ( (((guideline* or protocol*) N4 (adher* or comply or complian* or observ*)) or checklist*) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 19,735 S34 (MH "Decision Support Systems, Clinical") OR (MH "Decision Support Systems, Management") OR (MH "Decision Support Techniques+") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 4,457 S33 TI ( ((decision* N3 (make or makes or making or made or support* or algorithm* or aid or aids or app or apps or application* or technique*)) or "expert system*" or job- aid* or "job aid*") ) OR AB ( ((decision* N3 (make or makes or making or made or support* or algorithm* or aid or aids or app or apps or application* or technique*)) or "expert system*" or job-aid* or "job aid*") ) OR SU ( ((decision* N3 (make or makes or making or made or support* or algorithm* or aid or aids or app or apps or application* or technique*)) or "expert system*" or job-aid* or "job aid*") ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 54,010 S32 (MH "Patient Record Systems+") OR (MH "Electronic Health Records+") 27,811 S31 TI ( ((record* or registration* or registry or registries or e-registr* or eregistr* or ((medical or health) N2 data)) N2 (patient* or hospital or hospitals or medical or client or clients or in-patient* or inpatient* or out-patient* or outpatient*)) ) OR AB ( ((record* or registration* or registry or registries or e-registr* or eregistr* or 83 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT ((medical or health) N2 data)) N2 (patient* or hospital or hospitals or medical or client or clients or in-patient* or inpatient* or out-patient* or outpatient*)) ) OR SU ( ((record* or registration* or registry or registries or e-registr* or eregistr* or ((medical or health) N2 data)) N2 (patient* or hospital or hospitals or medical or client or clients or in-patient* or inpatient* or out-patient* or outpatient*)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 23,430 S30 S24 OR S25 OR S26 OR S27 OR S28 OR S29 514,447 S29 TI ( (doula? or douladural? or "barefoot doctor?") ) OR AB ( (doula? or douladural? or "barefoot doctor?") ) OR SU ( (doula? or douladural? or "barefoot doctor?") ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 246 S28 TI ( ((community or village? or peer or indigenous or treatment) N3 ("health worker?" or "health care worker?" or "healthcare worker?" or "health advisor*" or volunteer* or educator* or facilitator* or distributor* or "extension worker*" or supporter* or counselor* or counsellor*)) ) OR AB ( ((community or village? or peer or indigenous or treatment) N3 ("health worker?" or "health care worker?" or "healthcare worker?" or "health advisor*" or volunteer* or educator* or facilitator* or distributor* or "extension worker*" or supporter* or counselor* or counsellor*)) ) OR SU ( ((community or village? or peer or indigenous or treatment) N3 ("health worker?" or "health care worker?" or "healthcare worker?" or "health advisor*" or volunteer* or educator* or facilitator* or distributor* or "extension worker*" or supporter* or counselor* or counsellor*)) ) Limiters - Published Date: 20000101- 20191231; Exclude MEDLINE records 1,465 S27 TI ( (paraprofessional? or paramedic or paramedics or "paramedical worker?" or "paramedical personnel" or "allied health personnel" or "allied health worker?" or "support worker?" or "home health aide?") ) OR AB ( (paraprofessional? or paramedic or paramedics or "paramedical worker?" or "paramedical personnel" or "allied health personnel" or "allied health worker?" or "support worker?" or "home health aide?") ) OR SU ( (paraprofessional? or paramedic or paramedics or "paramedical worker?" or "paramedical personnel" or "allied health personnel" or "allied health worker?" or "support worker?" or "home health aide?") ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 4,020 S26 TI ( (((lay or voluntary or volunteer? or untrained or unlicensed or nonprofessional? or "non professional?" or support or link or outreach or "out reach") N5 (worker? or visitor? or attendant? or aide or aides or support$ or person$ or helper? or carer? or caregiver? or care giver? or consultant? or assistant? or staff)) or promotores or promotora or promotoras or embajadoras or comodrones or abuela or "lay advocate*" or "lay health" or "lay advisors" or "lay educators" or "lay counselor*" or "lay counsellor*" or "lady health worker*" or "lay facilitator*" or "natural helpers" or linkworker? or monitrice* or consejeras) ) OR AB ( (((lay or voluntary or volunteer? or untrained or unlicensed or nonprofessional? or "non professional?" or support or link or outreach or "out reach") N5 (worker? or visitor? or attendant? or aide or aides or support$ or person$ or helper? or carer? or caregiver? or care giver? or 84 APPENDICES consultant? or assistant? or staff)) or promotores or promotora or promotoras or embajadoras or comodrones or abuela or "lay advocate*" or "lay health" or "lay advisors" or "lay educators" or "lay counselor*" or "lay counsellor*" or "lady health worker*" or "lay facilitator*" or "natural helpers" or linkworker? or monitrice* or consejeras) ) OR SU ( (((lay or voluntary or volunteer? or untrained or unlicensed or nonprofessional? or "non professional?" or support or link or outreach or "out reach") N5 (worker? or visitor? or attendant? or aide or aides or support$ or person$ or helper? or carer? or caregiver? or care giver? or consultant? or assistant? or staff)) or promotores or promotora or promotoras or embajadoras or comodrones or abuela or "lay advocate*" or "lay health" or "lay advisors" or "lay educators" or "lay counselor*" or "lay counsellor*" or "lady health worker*" or "lay facilitator*" or "natural helpers" or linkworker? or monitrice* or consejeras) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 13,040 S25 (MH "Health Personnel+") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 281,049 S24 TI ( (((health or medical or healthcare or frontline or front-line) N1 (personnel or worker* or auxiliar* or staff or professional* or assistant* or provider* or administrator*)) or doctor* or physician* or GP or "general practitioner?" or "family doctor" or nurse* or midwi* or "clinical officer*" or pharmacist* or dentist* or vaccinator* or supervisor* or ((birth or childbirth or labor or labour) N1 (attendant? or assistant?))) ) OR AB ( (((health or medical or healthcare or frontline or front- line) N1 (personnel or worker* or auxiliar* or staff or professional* or assistant* or provider* or administrator*)) or doctor* or physician* or GP or "general practitioner?" or "family doctor" or nurse* or midwi* or "clinical officer*" or pharmacist* or dentist* or vaccinator* or supervisor* or ((birth or childbirth or labor or labour) N1 (attendant? or assistant?))) ) OR SU ( (((health or medical or healthcare or frontline or front-line) N1 (personnel or worker* or auxiliar* or staff or professional* or assistant* or provider* or administrator*)) or doctor* or physician* or GP or "general practitioner?" or "family doctor" or nurse* or midwi* or "clinical officer*" or pharmacist* or dentist* or vaccinator* or supervisor* or ((birth or childbirth or labor or labour) N1 (attendant? or assistant?))) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 415,575 S23 S1 AND S5 AND S8 AND S22 1,926 S22 S9 OR S10 OR S11 OR S12 OR S13 OR S14 OR S15 OR S16 OR S17 OR S18 OR S19 OR S20 OR S21 Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 133,020 S21 (MH "Fees and Charges+") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 5,985 S20 TI ( ((fee or fees or charge or charges or payment*) N2 (service* or hospital* or prescription* or medication or drug* or treatment*)) ) OR AB ( ((fee or fees or 85 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT charge or charges or payment*) N2 (service* or hospital* or prescription* or medication or drug* or treatment*)) ) OR SU ( ((fee or fees or charge or charges or payment*) N2 (service* or hospital* or prescription* or medication or drug* or treatment*)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 2,244 S19 TI ( ((cash N3 (transfer* or voucher* or grant* or aid)) or "health voucher*") ) OR AB ( ((cash N3 (transfer* or voucher* or grant* or aid)) or "health voucher*") ) OR SU ( ((cash N3 (transfer* or voucher* or grant* or aid)) or "health voucher*") ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 179 S18 (MH "Patient Satisfaction+") OR (MH "Consumer Satisfaction+") OR (MH "Feedback") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 27,890 S17 TI ( ((monitor* or satisfact* or accountab* or feedback) N3 (consumer* or public or patient* or client or clients) N3 (service or services)) ) OR AB ( ((monitor* or satisfact* or accountab* or feedback) N3 (consumer* or public or patient* or client or clients) N3 (service or services)) ) OR SU ( ((monitor* or satisfact* or accountab* or feedback) N3 (consumer* or public or patient* or client or clients) N3 (service or services)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 382 S16 (MH "Peer Pressure") OR (MH "Peer Counseling") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 936 S15 TI ( ((peer or peer-to-peer) N2 (group* or influenc* or pressure or network* or support)) ) OR AB ( ((peer or peer-to-peer) N2 (group* or influenc* or pressure or network* or support)) ) OR SU ( ((peer or peer-to-peer) N2 (group* or influenc* or pressure or network* or support)) ) Limiters - Published Date: 20000101- 20191231; Exclude MEDLINE records 7,231 S14 (MH "Information Seeking Behavior") Limiters - Published Date: 20000101- 20191231; Exclude MEDLINE records 2,119 S13 TI ( ((communicat* or messag* or online or access* or seek*) N3 (health or information)) ) OR AB ( ((communicat* or messag* or online or access* or seek*) N3 (health or information)) ) OR SU ( ((communicat* or messag* or online or access* or seek*) N3 (health or information)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 63,273 S12 (MH "Patient Access to Records") 775 86 APPENDICES S11 TI ( ((health or medical or personal) N2 (record* or data or monitor* or self-monitor* or track* or self-track* or registr*)) ) OR AB ( ((health or medical or personal) N2 (record* or data or monitor* or self-monitor* or track* or self-track* or registr*)) ) OR SU ( ((health or medical or personal) N2 (record* or data or monitor* or self- monitor* or track* or self-track* or registr*)) ) Limiters - Published Date: 20000101- 20191231; Exclude MEDLINE records 32,333 S10 (MH "Health Services Accessibility+") Limiters - Published Date: 20000101- 20191231; Exclude MEDLINE records 34,319 S9 TI ( ((access* or "use" or "using" or "uses") N3 "health service*") ) OR AB ( ((access* or "use" or "using" or "uses") N3 "health service*") ) OR SU ( ((access* or "use" or "using" or "uses") N3 "health service*") ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 35,136 S8 S6 OR S7 Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 587,324 S7 (MH "Patients+") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 110,241 S6 TI ( (patient* or client or clients or outpatient* or out-patient* or inpatient* or in- patient*) ) OR AB ( (patient* or client or clients or outpatient* or out-patient* or inpatient* or in-patient*) ) OR SU ( (patient* or client or clients or outpatient* or out- patient* or inpatient* or in-patient*) ) Limiters - Published Date: 20000101- 20191231; Exclude MEDLINE records 585,398 S5 S2 OR S3 OR S4 56,523 S4 TI ( (mhealth or m-health or "mobile health" or ehealth or e-health or "electronic health" or "digital health" or uhealth or u-health or telemedicine or tele-medicine or telehealth or tele-health or telecare or tele-care or telenursing or tele-nursing or telepsychiatry or tele-psychiatry or telemonitor* or tele-monitor* or teleconsult* or tele-consult* or telecounsel* or tele-counsel* or telecoach* or tele-coach* or webcast* or web-cast* or ((text* or short or voice or multimedia or multi-media or electronic or instant) N1 messag*) or "instant messenger" or texting or texted or texter* or ((sms or mms) N1 (service* or messag*)) or "interactive voice response*" or IVR or "voice call*" or callback* or "voice over internet" or VOIP or Facebook or Twitter or Whatsapp* or Skyp* or YouTube or "You Tube" or "Google Hangout*" or "mobile app*" or (remind* N3 (text* or system* or messag*)) or ((medical or clinical or health or healthcare or nurs*) N3 informatics) or ((interactive or computer- assisted) N1 (tutor* or technolog* or learn* or instruct* or software or communication)) or ((media or radio or television or tv or online or public) N3 campaign*) or GIS or "geographic information system*" or "global positioning") ) OR AB ( (mhealth or m-health or "mobile health" or ehealth or e-health or 87 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT "electronic health" or "digital health" or uhealth or u-health or telemedicine or tele- medicine or telehealth or tele-health or telecare or tele-care or telenursing or tele- nursing or telepsychiatry or tele-psychiatry or telemonitor* or tele-monitor* or teleconsult* or tele-consult* or telecounsel* or tele-counsel* or telecoach* or tele- coach* or webcast* or web-cast* or ((text* or short or voice or multimedia or multi- media or electronic or instant) N1 messag*) or "instant messenger" or texting or texted or texter* or ((sms or mms) N1 (service* or messag*)) or "interactive voice response*" or IVR or "voice call*" or callback* or "voice over internet" or VOIP or Facebook or Twitter or Whatsapp* or Skyp* or YouTube or "You Tube" or "Google Hangout*" or "mobile app*" or (remind* N3 (text* or system* or messag*)) or ((medical or clinical or health or healthcare or nurs*) N3 informatics) or ((interactive or computer-assisted) N1 (tutor* or technolog* or learn* or instruct* or software or communication)) or ((media or radio or television or tv or online or public) N3 campaign*) or GIS or "geographic information system*" or "global positioning") ) OR SU ( (mhealth or m-health or "mobile health" or ehealth or e-health or "electronic health" or "digital health" or uhealth or u-health or telemedicine or tele- medicine or telehealth or tele-health or telecare or tele-care or telenursing or tele- nursing or telepsychiatry or tele-psychiatry or telemonitor* or tele-monitor* or teleconsult* or tele-consult* or telecounsel* or tele-counsel* or telecoach* or tele- coach* or webcast* or web-cast* or ((text* or short or voice or multimedia or multi- media or electronic or instant) N1 messag*) or "instant messenger" or texting or texted or texter* or ((sms or mms) N1 (service* or messag*)) or "interactive voice response*" or IVR or "voice call*" or callback* or "voice over internet" or VOIP or Facebook or Twitter or Whatsapp* or Skyp* or YouTube or "You Tube" or "Google Hangout*" or "mobile app*" or (remind* N3 (text* or system* or messag*)) or ((medical or clinical or health or healthcare or nurs*) N3 informatics) or ((interactive or computer-assisted) N1 (tutor* or technolog* or learn* or instruct* or software or communication)) or ((media or radio or television or tv or online or public) N3 campaign*) or GIS or "geographic information system*" or "global positioning") ) or (MH "Telehealth+") OR (MH "Telemedicine+") Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 46,254 S3 TI ( (handheld or hand-held or smartphone* or smart-phone* or cellphone* or mobiles or ((personal N1 digital) or (PDA N3 (device* or assistant*)) or MP3 player* or MP4 player* or samsung or nokia or (windows N3 (mobile* or phone*)) or android or ipad* or i-pad* or ipod* or i-pod* or iphone* or i-phone* or (tablet* N3 (device* or computer*)) ) OR AB ( (handheld or hand-held or smartphone* or smart-phone* or cellphone* or mobiles or ((personal N1 digital) or (PDA N3 (device* or assistant*)) or MP3 player* or MP4 player* or samsung or nokia or (windows N3 (mobile* or phone*)) or android or ipad* or i-pad* or ipod* or i-pod* or iphone* or i-phone* or (tablet* N3 (device* or computer*)) ) OR SU ( (handheld or hand-held or smartphone* or smart-phone* or cellphone* or mobiles or ((personal N1 digital) or (PDA N3 (device* or assistant*)) or MP3 player* or MP4 player* or samsung or nokia or (windows N3 (mobile* or phone*)) or android or ipad* or i- pad* or ipod* or i-pod* or iphone* or i-phone* or (tablet* N3 (device* or computer*)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 16,611 88 APPENDICES S2 TI ( ((cell* or mobile*) N1 (phone* or telephone* or technolog* or device*)) ) OR AB ( ((cell* or mobile*) N1 (phone* or telephone* or technolog* or device*)) ) OR SU ( ((cell* or mobile*) N1 (phone* or telephone* or technolog* or device*)) ) Limiters - Published Date: 20000101-20191231; Exclude MEDLINE records 3,771 S1 TI ( ("quasi experiment*" or quasi-experiment* or "random* control* trial*" or "random* trial*" or RCT or (random* N3 allocat*) or matching or "propensity score" or PSM or "regression discontinuity" or "discontinuous design" or RDD or "synthetic control" or "difference in difference*" or difference-in-difference* or "diff in diff" or cohort or "propensity weighted" or propensity-weighted or "interrupted time series" or (pre N5 post) or ((pretest or "pre test") and (posttest or "post test")) or "rapid evidence assessment" or "systematic literature review" or "systematic review" or "meta-analy*" or metaanaly* or "meta analy*" or "instrumental variable*" or heckman or ((evaluation or impact or quantitative or "comparison group*" or counterfactual or "counter factual" or counter-factual or experiment*) N3 (design or study or analysis)) or QED) ) OR AB ( ("quasi experiment*" or quasi-experiment* or "random* control* trial*" or "random* trial*" or RCT or (random* N3 allocat*) or matching or "propensity score" or PSM or "regression discontinuity" or "discontinuous design" or RDD or "synthetic control" or "difference in difference*" or difference-in-difference* or "diff in diff" or cohort or "propensity weighted" or propensity-weighted or "interrupted time series" or (pre N5 post) or ((pretest or "pre test") and (posttest or "post test")) or "rapid evidence assessment" or "systematic literature review" or "systematic review" or "meta-analy*" or metaanaly* or "meta analy*" or "instrumental variable*" or heckman or ((evaluation or impact or quantitative or "comparison group*" or counterfactual or "counter factual" or counter-factual or experiment*) N3 (design or study or analysis)) or QED) ) OR SU ( ("quasi experiment*" or quasi-experiment* or "random* control* trial*" or "random* trial*" or RCT or (random* N3 allocat*) or matching or "propensity score" or PSM or "regression discontinuity" or "discontinuous design" or RDD or "synthetic control" or "difference in difference*" or difference-in-difference* or "diff in diff" or cohort or "propensity weighted" or propensity-weighted or "interrupted time series" or (pre N5 post) or ((pretest or "pre test") and (posttest or "post test")) or "rapid evidence assessment" or "systematic literature review" or "systematic review" or "meta- analy*" or metaanaly* or "meta analy*" or "instrumental variable*" or heckman or ((evaluation or impact or quantitative or "comparison group*" or counterfactual or "counter factual" or counter-factual or experiment*) N3 (design or study or analysis)) or QED) ) OR (MH "Double-Blind Studies") OR (MH "Intervention Trials") OR (MH "Randomized Controlled Trials") OR (MH "Clinical Trials") OR (MH "Controlled Before-After Studies") OR (MH "Interrupted Time Series Analysis") OR (MH "Pretest-Posttest Design+") OR (MH "Quasi-Experimental Studies") OR (MH "Nonequivalent Control Group") OR (MH "Retrospective Design") OR (MH "Panel Studies+") OR (MH "Prospective Studies+") Limiters - Published Date: 20000101- 20191231; Exclude MEDLINE records 246,740 89 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT APPENDIX 4 THE COMPLETE LIST OF LITERATURE SOURCES Academic databases: • Econlit (Ovid): http://www.ovid.com/site/catalog/databases/52.jsp • Ebsco Discovery (limited to Repec, World Bank E-Library, Econlit, Africa-Wide): https://www.ebscohost.com/discovery • Scopus: https://www.scopus.com/ • Social Sciences Citation Index (SSCI) (via Web of Science): https://library.maastrichtuniversity.nl/collections/databases/ssci/ • Medline • Embase • CAB Global Health • Popline • Epistemonikos • PsycInfo • CINAHL Repositories of evaluations and systematic reviews: • 3ie Repository of Impact Evaluations Error! Reference source not found. • 3ie RIDIE (Registry for International Development Impact Evaluations): http://ridie.3ieimpact.org/ • USAID Evaluation Clearing House: https://dec.usaid.gov/dec/content/evaluations.aspx • Innovations for Poverty Action (IPA) www.poverty-action.org/project-evaluations • The Abdul Latif Jameel Poverty Action Lab (JPAL) :www.povertyactionlab.org • AEA RCT Registry : https://www.socialscienceregistry.org/ • Campbell Collaboration, www.campbellcollaboration.org • Cochrane Collaboration • African Development Bank (AfDB): https://www.afdb.org/en/documents/publications/ • BREAD: http://ibread.org/bread/papers • Center for Effective Global Action (CEGA): http://cega.berkeley.edu/evidence/ • DFID Research for Development (R4D): http://r4d.dfid.gov.uk/ • CENTRAL (Cochrane) • Epistemonikos Specialist organizational databases • World Health Organization Database • Health Technology Assessment Database • GSMA mWomen • Mobile Active • mHealthevidence.org • mHealth Alliance: https://mhealthknowledge.org 90 APPENDICES • Knowledge for Health • International Center for Research on Women • Women for Women International • Mobiles for Education Alliance • mHealth Info, and • Health Unbound. • Charity Science Health 91 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT APPENDIX 5: DATA EXTRACTION TEMPLATE We extracted data from all included studies, using the standardised coding templates provided below. Table A5.1 Data extraction protocol Variable Name Variable Description Coder Name Coder's name Study id Unique id ascribed to each record Title of Use only the English version of the publication's main title. If paper is not written publication in English and has the title translated, use the translated version of the title. If the publication does not provide an English version, include the title in its original language. Please enter title in sentence case. Ensure there are no line breaks Author name Enter the first authors name in the format "Last name/s", "First name" "Second initial/s" (if any). Publication type Select from list: – Journal article – Working paper (these include discussion papers and technical reports/papers, if they are part of a series) – Report – 3ie Series Report – Book or book chapter Doi Code the study's DOI. If no information is found, code as "no DOI". Example: 10.1007/s11127-017-0452-x Abstract Copy and paste study's abstract. If there's no abstract code as: "no abstract" Ensure there are no line breaks. Journal name Use full journal name. Do not abbreviate name. Do not include "The" at the beginning Example: Journal of Development Effectiveness If publication is a working paper, write the series name. If publication is a report, write the publishing institution Example: World Bank Policy Research Working Paper Journal volume Use Arabic numberals (do not use Roman numerals) For working papers, include series number Journal issue Add journal issue if any Pages For example: 321‒340 If no page numbers given in reference (i.e., working papers that are only online), indicate “not applicable” Table A5.1 continued on next page 92 APPENDICES Table A5.1 Data extraction protocol (continued) Variable Name Variable Description Year of Select the year when the print version of the study was published. The format is publication YYYY. If only publication online use this. If study does not have year information select 9999 URL If study is a journal article enter URL of the landing page from the journal publisher's website if study is a published working paper or published report, enter URL of the document’s landing page from the publishing website If study is a published working paper or published report and there is not a landing page, provide url of the full-text PDF Open access If the study's (full-text) content is available, code as "Yes". If study has paywalls code as "No" Please save the pdf in the Dropbox folder called “Full Text PDFs” using the following format Firstauthorsurname_year_record id Equity focus How does this study consider gender and/or* equity? Choose as many factors as you find from the below list: – Sex-disaggregates data – Does not address gender or equity – Gender and/or equity-sensitive analytical frameworks – Theory of change – Sub-group or population analysis by gender and/ or equity (trigger) – Gender and/ or equity sensitive methodologies – other – Intervention targeting a specific vulnerable population (s) – Measures effects on gender and/ or equity outcome – Research process informed by gender and/ or equity – Study refers to ethics approval – Approach to ethics informed by gender and/ or equity considerations Equity Which dimensions(s) of gender and/or equity does the intervention target? dimension Please select one or more answer from the below list as applicable: – Place of residence (rural, urban, peri-urban, informal dwellings) – Ethnicity – Culture (includes language) – Sex (includes the use of the term gender meaning the biological sex of a person) – Religion – Education – Socioeconomic status (income or poverty status) – Land size – Land ownership Table A5.1 continued on next page 93 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Table A5.1 Data extraction protocol (continued) Variable Name Variable Description Equity – Head of household (female or male) dimension – Social capital – Age – Disability (medical, physical, neurological, mental disorders) – Sexual orientation – Sexual identity – Gendered social norms – Refugees – Conflict-Affected – Other (vulnerable group not typified by any of the above) – Power dynamics or relations between the studied population(s) or subpopulation and a power holder(s) Equity Open answer – provide a description of how the study considers gender and description equity, and for which population to corroborate answers above (include page numbers where relevant) Keywords Enter all author provided keywords, one per row. If the author does not provide any or if there are important keywords you think are missing, please add them (maximum 6 in total) Country name Select the countries in which the study was conducted (drop down menu) (can select multiple) WHO region Automatically indicates WHO /region in which the study was conducted when country name selected. Where multiple country study, option to select predominant region – African region – Americas – South-East Asia – European – Eastern Mediterranean – Western Pacific Country income Automatically indicates income level when country name selected. Where level multiple country study, option to select predominant income level Table A5.1 continued on next page 94 APPENDICES Table A5.1 Data extraction protocol (continued) Variable Name Variable Description Evaluation Select one of two options defined as: design 1 Experimental: a. RCT defined as prospective randomised assignment, where randomisation is implemented by researchers (or by decision makers in the context of an evaluation study) 2 Quasi-experimental: a. Quasi-random assignment: i) regression discontinuity design (sharp designs) or ii) natural experiment in which exposure to treatment is random b. Non-random assignment: i) Studies that control for unobservables (DID, FE, IV, Fuzzy RDD, ITS) or ii) studies that control for observables only (e.g., statistical matching, synth control, regression adjustment) Evaluation If Experimental then select: method – Randomised controlled trials If Quasi-experimental then select: – Regression discontinuity design – Difference-in-difference – Fixed effects – Instrumental variables – Statistical matching (includes PSM) – Synthetic controls – Interrupted time series Mixed methods Select YES if study includes quantitative and qualitative analyses, otherwise select NO. Additional Select additional method if any. If none, use N/A methods1 Additional Select additional method if any. If none, use N/A methods2 Project/program Code the name of the project/program being evaluated (if any) name Program Select one of the following: implementation – Government agency agency category – International Aid Agency – International Financial Institution – Non-Profit Organization – For-Profit Firm – Academic Institution – Charitable Foundation or Private Foundation – Not specified Table A5.1 continued on next page 95 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Table A5.1 Data extraction protocol (continued) Variable Name Variable Description Program Input the name of the agenc(ies) implementing the program implementation agency name Program What category of funding agency funded the program? funding agency Note: only code if reported in the study, no need to do additional research to find. category Select one of the following: – Government agency – International Aid Agency – International Financial Institution – Non-Profit Organization – For-Profit Firm – Academic Institution – Charitable Foundation or Private Foundation – Not specified Program Input the name of the agenc(ies) funding the research (note: this is not the same funding agency as organizations that fund the research of the evaluation) name Research What category of funding agency funded the research? funding agency Note: only code if reported in the study, no need to do additional research to find. category Select one of the following: – Government agency – International Aid Agency – International Financial Institution – Non-Profit Organization – For-Profit Firm – Academic Institution – Charitable Foundation or Private Foundation – Not specified Research Input the name of the agenc(ies) funding the research (note: this is not the same funding agency as organizations that fund the program) name Intervention Select all that apply: category First – Client tier – Healthcare providers – Health systems management – Data services Table A5.1 continued on next page 96 APPENDICES Table A5.1 Data extraction protocol (continued) Variable Name Variable Description Intervention Look to WHO DHI classifications, drop down boxes depending on what first tier category category chosen – e.g., Clients chosen at first tier should enable choice of: Second tier – Targeted Client Communication – Untargeted client communication – Client to client communication – Personal health tracking – Citizen based reporting – On-demand information services to clients – Client financial transactions Intervention Look to WHO DHI classifications, drop down boxes depending on what 2nd tier category third category chosen e.g., if 1.2 chosen above then enable choice between: tier – 1.2.1 Transmit untargeted health information to an undefined population – 1.2.2 Transmit untargeted health event alerts to undefined group Intervention Select all that apply: name – Targeted Digital health communication – Untargeted Digital health communication – Client-to-client communication – Personal Health Tracking – Citizen-based reporting: – On-demand information services to clients – Client financial transactions – Client identification and registration – Client health records – Healthcare provider decision support – Telemedicine – Healthcare Provider Communication – Referral Coordination – Scheduling and Activity Planning – Training – Prescription and Medication Management – Laboratory and diagnostics imaging management – Human Resource Management – Supply Chain Management – Public Health Event Notification – Civil Registration and Vital Statistics – Health Financing – Equipment and Asset Management – Facility Management – Data Collection, Management, and Use – Data Coding – Location Mapping – Data Exchange and Interoperability Table A5.1 continued on next page 97 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Table A5.1 Data extraction protocol (continued) Variable Name Variable Description Intervention Input description of the intervention description Broad Outcome First-order outcomes: category – Indication/therapeutic - specific outcomes (e.g., number of people achieving treatment adherence) – Quality of care outcomes (e.g., quality of care survey improvements) – Client Satisfaction outcomes (e.g., reported ease of use of technology) – Process outcomes (e.g., number of people successfully enrolled in a program) – Utilisation outcomes (e.g., number of usages of app functionality) Behaviour change – Generalised health outcomes: – Health status – Lives saved – Quality-adjusted life-years (QALYs) Disability-adjusted life years (DALYs) Outcome label Input outcome label (as described in the study) Outcome Input description/ definition of the outcome(s) description Economic data Does the study assess costs and resource use in any way? Economic Does the study report any measure of costs relative to effects eg cost- evaluation effectiveness analysis benefit cost analysis Economic data How does the study measure cost and resource use? description Study What are the population groups for which outcome data is collected? Select all Population that apply: – Healthcare professionals – Beneficiaries – children – Beneficiaries – adult – Beneficiaries – newborn – Beneficiaries – adolescents – Beneficiaries - other Health Domain Use lists available here: https://www.mindmeister.com/264463850/mhealth- evidence-taxonomy Target User Use lists available here: https://www.mindmeister.com/264463850/mhealth- evidence-taxonomy Health System Use lists available here: https://www.mindmeister.com/264463850/mhealth- Application evidence-taxonomy Source: 3IE, World Bank. 98 APPENDICES APPENDIX 6 CRITICAL APPRAISAL TOOL All included systematic reviews will be critically appraised using the standardised checklist below, and given a rating of high, medium or low confidence in the review findings. Checklist for making judgements about how much confidence to place in a systematic review of effects (adapted version of SURE checklist)i Note for reviewers: always include page numbers to supporting information in the comments section for each question to guide peer reviewers and final reviewers. Copy and paste relevant sections into the comments if this is useful. Assessed by: Date: Section A: Methods used to identify, include and critically appraise studies A.1 Were the criteria used for deciding which  Yes studies to include in the review reported?  Partially Did the authors specify:  No  Types of studies  Participants/ settings/ population Coding guide - check the answers above  Intervention(s) YES: All four should be yes  Outcome(s) NO: All four should be no PARTIALLY: Any other Comments (note important limitations or uncertainty) Section A continued on next page 99 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Section A: Methods used to identify, include and critically appraise studies (continued) A.2 Was the search for evidence reasonably  Yes comprehensive?  Partially Were the following done:  No  Language bias avoided (no restriction of  Can’t tell inclusion based on language). If authors search in English but do not restrict inclusion based on language, we mark this Coding guide - check the answers above: as yes. If they are not explicit about language, we assume yes but flag in the YES: All five should be yes comments. PARTIALLY: Relevant databases and reference  No restriction of inclusion based on lists are both reported publication status NO: Any other  Relevant databases searched (Minimum criteria: All reviews should search at least one source of grey literature such as Google. For academic searches, health SRs should use Medline/ Pubmed + Cochrane Library; for social sciences IDEAS + at least one database of general social science literature and one subject specific database)  Reference lists in included articles checked  Authors/experts contacted Comments (note important limitations or uncertainty) Section A continued on next page 100 APPENDICES Section A: Methods used to identify, include and critically appraise studies (continued) A.3 Does the review cover an appropriate time  Yes period?  Can't tell (only use if no information about time Is the search period comprehensive enough period for search) that relevant literature is unlikely to be  No omitted?  Unsure Coding guide: YES: Generally this means searching the literature at least back to 1990 NO: Generally if the search does not go back to 1990 CAN’T TELL: No information about time period for search Note: With reference to the above – there may be important reasons for adopting different dates for the search, e.g., depending on the intervention. If you think there are limitations with the timeframe adopted for the search which have not been noted and justified by the authors, you should code this item as a NO and specify your reason for doing so in the comment box below. Older reviews should not be downgraded, but the fact that the search was conducted some time ago should be noted in the quality assessment. Always report the time period for the search in the comment box. Comments (note search period, any justification provided for the search period, or uncertainty) Section A continued on next page 101 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Section A: Methods used to identify, include and critically appraise studies (continued) A.4 Was bias in the selection of articles  Yes avoided?  Partially Did the authors specify:  No  Independent screening of full text by at Coding guide: least 2 reviewers YES: All three should be yes, although reviews  List of included studies provided published in journals are unlikely to have a list of  List of excluded studies provided excluded studies (due to limits on word count) and the review should not be penalised for this. PARTIALLY: Independent screening and list of included studies provided are both reported NO: All other. If list of included studies provided, but the authors do not report whether or not the screening has been done by 2 reviewers review is downgraded to NO. Comments (note important limitations or uncertainty): A.5 Did the authors use appropriate criteria to  Yes assess the quality and risk of bias in  Partially analysing the studies that are included?ii  No  The criteria used for assessing the quality/ risk of bias were reported Coding guide:  A table or summary of the assessment of YES: All three should be yes each included study for each criterion was PARTIALLY: The first and third criteria should be reported reported. If the authors report the criteria for  Sensible criteria were used that focus on assessing risk of bias and report a summary of this the quality/ risk of bias (and not other assessment for each criterion, but the criteria may qualities of the studies, such as precision be only partially sensible (e.g., do not address all or applicability/external validity). “Sensible” possible risks of bias, but do address some), we is defined as a recognised quality appraisal downgrade to PARTIALLY. tool/ checklist, or similar tool which NO: Any other assesses bias in included studies. Please see footnotes for details of the main types of bias such a tool should assess. Comments (note important limitations or uncertainty) Section A continued on next page 102 APPENDICES Section A: Methods used to identify, include and critically appraise studies (continued) A.6 Overall – how much confidence do you  Low confidence (limitations are important enough have in the methods used to identify, that the results of the review are not reliable) include and critically appraise studies?  Medium confidence (limitations are important Summary assessment score A relates to the 5 enough that it would be worthwhile to search for questions above. another systematic review and to interpret the results of this review cautiously, if a better review High confidence applicable when the answers cannot be found) to the questions in section A are all assessed as ‘yes’  High confidence (only minor limitations) Low confidence applicable when any of the following are assessed as ‘NO’ above: not reporting explicit selection criteria (A1), not conducting reasonably comprehensive search (A2), not avoiding bias in selection of articles (A4), not assessing the risk of bias in included studies (A5) Medium confidence applicable for any other – i.e., section (A3) is assessed as ‘NO’ or can’t tell and remaining sections are assessed as ‘partially’ or ‘can’t tell’ Comments (note important limitations). 103 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Section B: Methods used to analyse the findings B.1 Were the characteristics and results of the  Yes included studies reliably reported?  No Was there:  Partially  Independent data extraction by at least 2  Not applicable (e.g., no included studies) reviewers  A table or summary of the characteristics of the participants, interventions and outcomes Coding guide: for the included studies YES: All three should be yes  A table or summary of the results of all the PARTIALLY: Criteria one and three are yes, but included studies some information is lacking on second criteria. No: None of these are reported. If the review does not report whether data was independently extracted by 2 reviewers (possibly a reporting error), we downgrade to NO. NOT APPLICABLE: if no studies/no data Comments (note important limitations or uncertainty) Section B continued on next page 104 APPENDICES Section B: Methods used to analyse the findings (continued) B.2 Are the methods used by the review authors  Yes to analyse the findings of the included studies  Partially clear, including methods for calculating effect sizes if applicable?  No  Not applicable (e.g., no studies or no data) Coding guide: YES: Synthesis methods used clearly reported. If it is clear that the authors use narrative synthesis, they don't need to say this explicitly. PARTIALLY: Some reporting on methods but lack of clarity NO: Nothing reported on methods NOT APPLICABLE: if no studies/no data Note: Question is not asking you to make a judgement about the appropriateness / application of synthesis methods; rather, whether authors report their synthesis methods / make it clear which method they use. Comments (note important limitations or uncertainty) B.3 Did the review describe the extent of  Yes heterogeneity?  Partially  Did the review ensure that included studies  No were similar enough that it made sense to combine them, sensibly divide the included  Not applicable (e.g., no studies or no data) studies into homogeneous groups, or sensibly Coding guide: conclude that it did not make sense to YES: First should be yes, and second category combine or group the included studies? should be yes if applicable  Did the review describe the extent to which PARTIALLY: The first category is yes there were important differences in the results of the included studies? NO: Any other  If a meta-analysis was done, was the I2, chi NOT APPLICABLE: if no studies/no data square test for heterogeneity or other Note: This question is interested in whether the appropriate statistic reported? authors DESCRIBED heterogeneity in results. Question B.6 assesses whether the authors explored reasons for observed heterogeneity in results between studies. Comments (note important limitations or uncertainty) Section B continued on next page 105 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Section B: Methods used to analyse the findings (continued) B.4 Were the findings of the relevant studies  Yes combined (or not combined) appropriately  Partially relative to the primary question the review addresses and the available data?  No How was the data analysis done?  Not applicable (e.g., no studies or no data)  Descriptive only  Can’t tell  Vote counting based on direction of effect  Vote counting based on statistical Coding guide: significance YES: If appropriate table, graph, text summary or  Description of range of effect sizes meta-analysis AND appropriate weights AND unit of  Meta-analysis analysis errors addressed (if appropriate). If  Meta-regression narrative synthesis only used, YES if authors report  Other: specify and discuss magnitude of effects for all included studies (i.e., the authors DO NOT use vote-counting  Not applicable (e.g., no studies or no data) based on direction, statistical significance or How were the studies weighted in the selectively report results) analysis? PARTIALLY: If appropriate table, graph, text  Equal weights (this is what is done when summary or meta-analysis AND appropriate vote counting is used) weights AND unit of analysis errors not addressed  By quality or study design (this is rarely (and should have been). done) NO: If vote counting based on direction of effect or  Inverse variance (this is what is typically statistical significance used OR authors selectively done in a meta-analysis) describe results OR inappropriate reporting of table,  Number of participants (sample size) graph or meta-analyses.  Other: specify NOT APPLICABLE: if no studies/no data  Not clear CAN’T TELL: if unsure (note reasons in comments  Not applicable (e.g., no studies or no data) below) Did the review address unit of analysis errors?  Yes - took clustering into account in the analysis (e.g., used intra-cluster correlation coefficient)  No, but acknowledged problem of unit of analysis errors  No mention of issue  Not applicable - no clustered trials or studies included Comments (note important limitations or uncertainty) Section B continued on next page 106 APPENDICES Section B: Methods used to analyse the findings (continued) B. 5 Does the review report evidence  Yes appropriately?  No  The review makes clear which evidence is  Partially subject to low risk of bias in assessing causality (attribution of outcomes to  Not applicable intervention), and which is likely to be biased, and does so appropriately Coding guide:  Where studies of differing risk of bias are included, results are reported and YES: Both criteria should be fulfilled (where analysed separately by risk of bias status applicable) NO: Criteria not fulfilled PARTIALLY: Only one criteria fulfilled, or when there is limited reporting of quality appraisal (the latter applies only when inclusion criteria for study design are appropriate) NOT APPLICABLE: No included studies Note on reporting evidence and risk of bias: For reviews of effects of ‘large n’ interventions, experimental and quasi-experimental designs should be included (if available). For reviews of effects of ‘small n’ interventions, designs appropriate to attribute changes to the intervention should be included (e.g., pre-post with assessment of confounders) Please specify included study designs and any other comments (note important limitations or uncertainty): Section B continued on next page 107 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT Section B: Methods used to analyse the findings (continued) B.6 Did the review examine the extent to which  Yes specific factors might explain differences  Partially in the results of the included studies?  No  Were factors that the review authors considered as likely explanatory factors  Not applicable clearly described? Coding guide:  Was a sensible method used to explore YES: Explanatory factors clearly described and the extent to which key factors explained appropriate methods used to explore heterogeneity heterogeneity? PARTIALLY: Explanatory factors described but for  Descriptive/textual meta-analyses, sub-group analysis or meta-  Graphical regression not reported (when they should have been)  Meta-analysis by sub-groups NO: No description or analysis of likely explanatory  Meta-regression factors  Other NOT APPLICABLE: e.g., too few studies, no important differences in the results of the included studies, or the included studies were so dissimilar that it would not make sense to explore heterogeneity of the results Comments (note important limitations or uncertainty) B.7 Overall - how much confidence do you  Low confidence (limitations are important enough have in the methods used to analyse the that the results of the review are not reliable) findings relative to the primary question  Medium confidence (limitations are important addressed in the review? enough that it would be worthwhile to search for Summary assessment score B relates to the 5 another systematic review and to interpret the questions in this section, regarding the results of this review cautiously, if a better review analysis. cannot be found) High confidence applicable when all the  High confidence (only minor limitations) answers to the questions in section B are assessed as ‘yes’. Low confidence applicable when any of the following are assessed as ‘NO’ above: critical characteristics of the included studies not reported (B1), not describing the extent of heterogeneity (B3), combining results inappropriately (B4), reporting evidence inappropriately (B5). Medium confidence applicable for any other: i.e., the “Partial” option is used for any of the 6 preceding questions or questions and/or B.2 and/ or B.6 are assessed as ‘no’. Use comments to specify if relevant, to flag uncertainty or need for discussion 108 APPENDICES Section C: Overall assessment of the reliability of the review C.1 Are there any other aspects of the review  Additional methodological concerns – only one not mentioned before which lead you to person reviewing question the results?  Robustness  Interpretation  Conflicts of interest (of the review authors or for included studies)  Other  No other quality issues identified Note: if one or more of the additional methodological concerns above are noted, the confidence level can be downgraded at the agreement of the reviewers. C.2 Are there any mitigating factors which  Limitations acknowledged should be taken into account in  No strong policy conclusions drawn (including in determining the reviews reliability? abstract/ summary)  Any other factors Note: if the authors acknowledge limitations of the review process and as a result do not draw strong policy conclusions, the confidence level can be upgraded at the agreement of the reviewers. Use comments to specify if relevant, to flag uncertainty or need for discussion C.3 Based on the above assessments of the methods how would you rate the reliability of the review?  Low confidence in conclusions about effects: The systematic review has the following major limitations...  Medium confidence in conclusions about effects: The systematic review has the following limitations...  High confidence in conclusions about effects : If applicable: The review has the following minor limitations... Coding guide: High confidence in conclusions about effects: high confidence noted overall for sections A and B, unless moderated by answer to C1. Medium confidence in conclusions about effects: medium confidence noted overall for sections A or B, unless moderated by answer to C1 or C2. Low confidence in conclusions about effects: low confidence noted overall for sections A or B, unless moderated by answer to C1 or C2. Limitations should be summarised above, based on what was noted in Sections A, B and C. 109 DIGITAL HEALTH INTERVENTIONS AN EVIDENCE GAP MAP REPORT NOTES i Adapted from Supporting the Use of Research Evidence (SURE) Collaboration. SURE checklist for making judgements about how much confidence to place in a systematic review. In: SURE guides for preparing and using policy briefs. www.evipnet.org/sure ii Risk of bias is the extent to which bias may be responsible for the findings of a study. Bias is a systematic error or deviation from the truth in results or inferences. In studies of the effects of social, economic and health care interventions, the main types of bias arise from systematic differences in the groups that are compared (selection bias), the intervention that is provided, or exposure to other factors apart from the intervention of interest (performance bias/contamination), withdrawals or exclusions of people entered into a study (attrition bias) or how outcomes are assessed (detection bias) and reported (reporting bias). Reviews of social science studies may be particularly affected by reporting bias, where a biased subset of all the relevant data and analyses is presented. Assessments of the risk of bias are sometimes also referred to as assessments of the validity or quality of a study. Validity is the extent to which a result (of a measurement or study) is likely to be true. Quality is a vague notion of the strength or validity of a study, often indicating the extent of control over bias. 110