EVALUATION REPORT 2018 Big Data, Artificial Intelligence, and Decision Science in Health and Nutrition Skills Building Program MARCH 2019 Decision and Delivery Science, Health, Nutrition and Population Global Practice 1 © International Bank for Reconstruction and Development / The World Bank 1818 H Street NW, Washington DC 20433 Internet: www.worldbank.org; Telephone: 202 473 1000 This work is a product of the staff of The World Bank. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the Executive Directors of The World Bank or other partner institutions or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions This work is available under the Creative Commons Attribution 3.0 Unported licence (CC BY 3.0 IGO) https://creativecommons.org/licenses/by/3.0/igo/. Under the Creative Commons Attribution license, you are free to copy, distribute and adapt this work, including for commercial purposes, under the following conditions: Attribution – Please cite the work as follows: The World Bank. 2019. “Evaluation Report: 2018 Big Data, Artificial Intelligence, and Decision Science in Health and Nutrition Skills Building Program”. Washington DC: World Bank. License: Creative Commons Attribution CC BY 3.0 IGO. Translations – If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by The World Bank and should not be considered an official World Bank translation. The World Bank shall not be liable for any content or error in its translation. All queries on rights and licenses should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington DC, 20433, USA; fax: 202-522-2625; email: pubrights@worldbank.org. 2 Contents Overview........................................................................................................................ 4 Introduction ............................................................................................................... 4 Skills Building Course Objectives ................................................................................ 5 Training Format ......................................................................................................... 5 Evaluation of the Skills Building Program ....................................................................... 7 Methodology ............................................................................................................. 7 Achievement of Objectives ........................................................................................ 8 Objective 1 and 2: .................................................................................................. 8 Objectives 3 to 6: ................................................................................................. 11 Track-specific results ................................................................................................ 14 Big Data Analytics track ........................................................................................ 15 Optima HIV track .................................................................................................. 16 Health Services Prioritization Tool for Universal Health Coverage track............... 17 Implementation Cascades track ........................................................................... 18 Optima Nutrition .................................................................................................. 20 Optima TB track ................................................................................................... 21 Logistics ................................................................................................................... 23 Sustainability and follow-up ..................................................................................... 23 Key takeaways ............................................................................................................. 24 APPENDIX .................................................................................................................... 26 SurveyMonkey training evaluation survey form: English.......................................... 26 3 Overview Introduction The World Bank’s Decision & Delivery Science and Nutrition teams and the Bill & Melinda Gates Foundation are working in partnership to improve the use of data for decision making and implementation, and to apply big data and cognitive analytical approaches to address complex problems in health and development. Over recent years, both groups have invested in the development and application of analytical and mathematical decision science models that help policy makers make evidence-based decisions that maximize allocative efficiency. The World Bank and the Gates Foundation are also exploring how innovative technologies and machine learning techniques can support decision and delivery choices in health. To build skills and demand for the use of big data, artificial intelligence and analytical and mathematical decision science models amongst countries and local partners, a series of 5-day skills building courses were conducted in 2018 in the following 3 locations: 10 – 14 September 2018: Bucharest, Romania 24 – 28 September 2018: Bangkok, Thailand 1 – 5 October 2018: Pretoria, South Africa Over 400 participants from 49 countries from across the world attended one of these three trainings, with representation from government health ministries (over 50% of total participants), academics and researchers, World Bank country teams and staff, NGOs and civil society organizations working with public health issues, implementing partners, and other international organizations such as UNAIDS and the WFP. To encompass the rich and diverse experiences of all the participants and to cater to their specific needs, simultaneous translations were provided in Spanish and Russian (in Bucharest) and in French (in Pretoria), with additional support from French-speaking instructors in Pretoria. 4 Skills Building Course Objectives The 5-day skills building course aimed to: ✓ Create awareness of the need for analytics to improve decision and delivery choices in health and development ✓ Introduce big data and cognitive analytics for transforming health and development systems ✓ Provide training on analytical optimization tools that can answer pertinent policy and implementation questions for specific policy sectors ✓ Discuss data availability and syntheses for use in analytical frameworks ✓ Discuss the interpretation and use of analytical results, including case studies where analytics has been useful in improving outcomes. ✓ Provide practical experience in using decision science tools Training Format The 5-day training was broken down into two parts. During plenary days (Day 1 and 2), participants were introduced to big data, cognitive analytics, data science, decision and delivery optimization, and their current and future applications in the field of public health. Between 10-12 presentations were made over the course of these two days by leading experts in this field, and aimed to provide participants with a base level of knowledge and build their interest in the tools and techniques they were going to learn. During the track-specific days (Days 3-5), participants selected to be trained on one of the following 6 tools: ✓ Track 1: Health Services Prioritization Tool for the improvement of Universal Health Coverage ✓ Track 2: Optima HIV ✓ Track 3: Optima TB ✓ Track 4: Optima Nutrition ✓ Track 5: Implementation cascade analysis and optimization ✓ Track 6: Big data analytics in health 5 By location, the proportion of participants who took part in each track was as follows: Bucharest Bangkok Pretoria 7% 10% 16% 26% Big Data 29% 18% Cascades 8% 42% 30% HIV HSP 12% 5% Nutrition 12% 18% TB 8% 14% 16% 9% 20% 6 Evaluation of the Skills Building Program Methodology To assess the extent to which the skill-building program achieved its objectives, the program evaluation team looked at various sources of feedback received by the World Bank organizing team and assessed the extent to which the training objectives were fulfilled. Some of the sources of feedback are: ✓ SurveyMonkey evaluation forms sent out to all participants: A total of 146 people took the survey, though only 124 (85%) completed the full evaluation form. These forms were published in English (92 respondents), French (14 respondents), Spanish (6 respondents), and Russian (12 respondents). A sample of the English feedback form can be found in the Appendix. ✓ Feedback videos from all 3 venues: 7 participant feedback videos from Bucharest, 12 from Bangkok, and 13 from Pretoria. To ensure equal representation from all 6 training tracks, at least one participant per track was interviewed in all locations. ✓ End of the day surveys on CrowdCompass app: for all 3 locations, participants were asked for feedback every day through the conference app on how the training day went. ✓ Direct feedback from trainers, country teams, and participants: via email or in person to the World Bank organizing team. During analysis, the evaluation team further disaggregated the data to check if there was any specific positive or negative feedback specific to track, location, or by language of instruction. For certain tracks and languages, the sample size of respondents was not large enough for us to reach conclusions, which has been mentioned in the track-specific analysis. A lot of sources of feedback were also more subjective and open-ended, all of which have been incorporated into the analysis in this report. This evaluation is laid out in the following format: ✓ Assessment of to what extent the objectives were fulfilled ✓ Feedback on logistics ✓ Sustainability and follow-up 7 Achievement of Objectives Objective 1 and 2: ✓ To create awareness of the need for analytics to improve decision and delivery choices in health and development ✓ Introduce big data and cognitive analytics for transforming health and development systems The overall consensus from all 3 training venues was the fact that participants were extremely enthusiastic about getting to “I am being sincere when I say know more about how big data, decision science, and artificial the skills I acquired and the intelligence were revolutionizing the field of public health, and knowledge impacted on me are this enthusiasm started even before they arrived at the invaluable” – participant in training venue. From an analysis of feedback for the Plenary Pretoria days (Day 1 and 2), which were specifically to achieve the objectives of building awareness about and introducing these techniques, participants in all three venues consistently expressed how transformative it was for them to understand the role of Big Data and AI in decision making and resource optimization. Participants also emphasized on how important it was for them to learn about the practical applications of these technologies, citing that the sessions where real-world examples (case studies) were shared were pertinent in increasing their curiosity, and building their interest in learning more and applying these techniques in their own countries. 8 Analysis of Plenary Days (Day 1 and 2) On a scale of 1 (very poor/extremely dissatisfied/not useful) to 7 (very good/extremely satisfied/extremely useful): 0% 20% 40% 60% 80% 100% % favorable (>4 out of 7) How would you rate the Day 1 and 2 training 21% 19% 26% 24% 69% content? How would you rate the order in which the 16% 24% 28% 26% 78% content was presented? How would you rate the materials used during 23% 21% 27% 22% 70% Day 1 and 2? How would you rate the quality of the Day 1 and 22% 22% 29% 18% 69% 2 program? How would you rate the overall usefulness of it? 17% 15% 27% 32% 74% How would you rate the relevance of it to your 20% 12% 22% 38% 72% organization’s mission? My knowledge/skills increased as a result of the 18% 25% 23% 23% 71% Day 1 and 2 program The knowledge/skills gained through it are 69% 20% 21% 21% 27% directly applicable to my work 1 2 3 4 5 6 7 On an average, 72% participants consistently rated the overall training as well as its content, presentation, usefulness and relevance to their organization’s mission as good or excellent (a score of 5 or greater out of 7). 70% participants also said that their skills increased as a result of the plenary days and that these skills they learnt were directly applicable to their work. Overall, 70% participants stated that the course made them curious about and helped them feel more competent in understanding and engaging in issues relating to AI in health . Overall impact of course 0% 20% 40% 60% 80% 100% % favorable (>3 out of 5) Extent to which this course made you curious about and helped you feel more competent in understanding and 26% 34% 36% 70% engaging in issues relating to AI in health (1= not at all 3 = somewhat 5 = totally changed how I feel about AI in health) 1 2 3 4 5 When asked whether they would use what they learned in the program in their day to day work, 82% said an enthusiastic yes, stating applying this knowledge in M&E, analytics and predictive modeling for health data, strategic planning, health services prioritization, program design and 9 implementation, decision making, and more. One of the common thoughts heard in all sources of feedback was the participants’ wish to apply this knowledge to their local settings and existing work in data analytics in health, which a few participants stated was already happening in their countries. In terms of the specific program, group work and interactive exercises were most popular amongst participants, and case studies from various places where these techniques were already in use built more interest. Other specific parts of the plenary days that participants liked were the taxonomy of machine learning in health, the cost-effectiveness exercise, conversations on the current applications and future of AI in health, and the to-do list for governments in making this possible. Even so, many participants wished there were “The workshop was successful in creating more case studies for them to learn from, and awareness of the need for countries to apply also expressed interest in having more hands-on AI and big data approach and analytics to experience. These were no longer concerns after improve decision and delivery choices in the evaluation of Days 3-5 was done, since the health and development” – participant in tool-specific training tracks provided them with Bangkok this much needed interactivity and potential for real-world application. Additionally, some expressed interest in learning more about examples from developing rather than developed nations, especially on applications of decision science and AI techniques in primary health care, which would have helped build stronger connections in how best to incorporate these technologies in their own contexts. Other challenges that participants faced during Days 1 and 2 was the fact that some groups during group work were too large (for instance: in Pretoria) and participants felt that maintaining country teams as groups would have been a better way to promote thinking about applications based on their country context. Some also noted the need for more technical facilitators for the group work sessions, and many maintained that the total number of presentations during the plenary was too large, which made their learning more rushed and didn’t give them time to process. Another specific point of feedback was to not use self-narration in presentations (which was the case for one session in the plenary), specifically since it took away the element of discussion. Overall, it can be said that Objectives 1 and 2 were achieved with 70-80% satisfaction levels. 10 Objectives 3 to 6: ✓ Provide training on analytical optimization tools that can answer pertinent policy and implementation questions for specific policy sectors ✓ Discuss data availability and syntheses for use in analytical frameworks ✓ Discuss the interpretation and use of analytical results, including case studies where analytics has been useful in improving outcomes ✓ Provide practical experience in using decision science tools After the introduction to analytical tools and data techniques during Plenary days 1 and 2, participants chose one of 6 different training tracks based on their interest and applicability in their current work. All 6 of these tracks were taught in all 3 locations from days 3 to 5 of the skills building program, except for Optima Nutrition in Bucharest where it was condensed into a one day program. Overall, participants were very happy to have hands-on experience over the course of 3 days “You don’t need a background in computer working on optimization tools (Optima HIV, TB, science or statistics; you can just be a person and Nutrition), utilizing other tools to with a basic understanding of concepts and understand health services prioritization and the teacher encourages you to understand analyze implementation cascades, and also learn how to apply this in the field” – participant in about techniques used in big data analytics. 81% Bangkok said that the training track fulfilled their learning needs, with 86% agreeing that the stated objectives (objectives 3-6) were met by the program. Satisfaction with the group work and practicals increased in the tracks to 82%, while satisfaction with the content, presentation, material, and quality stayed between 80-85%. 85% participants also found these tools to be very useful in their work and relevant to their organizations’ missions, and 84% said that their knowledge and skills increased as a result of the training. Participants’ satisfaction with attention paid to theoretical and practical work was a bit lower at 69% and 64% respectively, and 60% felt that the time management of their training tracks (time taken for instruction, time for their participation, pace of training track) was excellent (90% rated it average or better than average). 11 Overall response for Track-specific training days (Days 3-5) 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Q17 Q18 Q19 Q20 Q21 Q22 Q23 Q24 Q25 Q26 Q27 Q28 Q29 Q30 Q31 Q32 Q33 Q34 Q35 Q36 Q37 % Favorable Median Poly. (% Favorable) Poly. (Median) In both the evaluation surveys and individual feedback, all the participants were very happy with the trainers; trainers’ knowledge of the subject and answering questions, quality of delivery, and effectiveness in encouraging participation all showed satisfaction scores of 85-89%. Perhaps an impactful metric that tells us the extent to which the training objectives were achieved would be that 69% of participants said that due to the trainers’ efforts, they definitely felt ready to apply the content that they learned. Some of the feedback around how the training tracks could be “These tools are helping us make improved were both specific to the tools being used, as well intelligent decisions, and in as the format of the training. Many felt the need for more generating better results in hands-on work and expressed their wish to work with public health” – participant in datasets that are applicable to their country contexts, and Bangkok. that the links to real world data were made from the first day of the track. Due to the diversity in backgrounds of participants, this skills building program was designed to aid those without much of a background in data analytics to also learn how to use these techniques, which had mostly positive, but some negative implications. Many participants lauded the fact that the program took them from the basics and proceeded to increase their skills step-by-step, while some participants who were already trained in these techniques found the pace to be slow, making a suggestion to perhaps design training tracks should be based more on existing skill level. Since some training tools, such as the HSP tool, Implementation Cascades analysis tool, and Optima Nutrition, are still being finalized, some participants expressed that they would have preferred to work on a fully functional tool, and to have a full demonstration of what these tools can accomplish 12 before they start working with them. In many of the participant interviews, participants expressed how important it was for them for these tools to be able to be adaptable to their country contexts. Some also wished that user guides for these tools were available in other languages, and that the interface was a bit more user-friendly and intuitive (mostly for unfinalized tools). Another conversation during the trainings was around the lack of good, usable data in some countries, which could potentially impact the usability of these tools. One of the things that participants appreciated from the trainings was the chance to network with their peers, and to learn more about work being done in public health spheres across the world. The chance to be part of group work with other contemporaries was much appreciated by many. Almost all participants that the World Bank organizing team talked to and many who answered the feedback surveys stated that they wished the training was longer than 3 days, since they wanted to learn more about analyzing the data generated from the tool in more granularity, and get better at using these tools. Many expressed interest in follow-up trainings, and wanting to bring back these techniques to their countries and their existing work, which is discussed in detail in the “Sustainability” section of this report. Overall, it can be said that Objectives 3-6 were achieved with 70-80% satisfaction levels. 13 Track-specific results One of the most significant sources for this evaluation was the SurveyMonkey training evaluation form, which additionally gave insight on how well individual training tracks did in terms of trainer prowess, quality of material, and applicability of the training material to participants’ existing work and training needs. The following 21 questions were asked of respondents about their respective tracks, and were grouped in the analysis into two categories: ✓ Quality of training track material and applicability to participant needs (rated on a scale of 1=very poor/strongly disagree to 7=very good/strongly agree): Q17) To what extent did the Day 3-5 program fulfill your learning needs? Q18) To what extent did the Day 3-5 program achieve its announced objectives? How did you like the group work and practicals? Q19) How would you rate the Day 3-5 training content? Q20) How would you rate the order in which the content was presented? Q21) How would you rate the materials used during the Day 3-5 training? Q22) How would you rate the quality of the Day 3-5 training? Q23) How would you rate the overall usefulness of it? Q24) How would you rate the relevance of the Day 3-5 training to your organization’s mission? Q25) My knowledge and skills increased as a result of the Day 3-5 training Q26) The knowledge and skills gained through it are directly applicable to my work ✓ Design of curriculum and training, and trainer prowess (rated on a scale of 1=very poor to 5=very good): Q27) Attention to theoretical content Q28) Attention to practical content Q29) Time for instructors’ presentations Q30) Time for your participation Q31) Pace of the training Q32) Knowledge of the subject Q33) Quality of delivery Q34) Effectiveness in encouraging participation Q35) Effectiveness in answering participant questions Q36) Effectiveness in making you feel ready to apply the content 14 Big Data Analytics track Based on this analysis, the overall satisfaction with the Big Data Analytics track was 73%. Quality of training track material and applicability to participant needs (rated on a scale of 1=very poor/strongly disagree to 7=very good/strongly agree) (n=45): % favorable 0% 50% 100% (>4 out of 7) To what extent did the Day 3-5 program fulfill your learning needs? 27% 11% 31% 27% 69% To what extent did the Day 3-5 program achieve its announced objectives? 11% 22% 27% 29% 78% How did you like the group work and practicals? 13% 18% 27% 31% 76% How would you rate the Day 3-5 training content? 20% 18% 31% 27% 76% How would you rate the order in which the content was presented? 27% 20% 22% 27% 69% How would you rate the materials used during the Day 3-5 training? 13% 20% 24% 40% 84% How would you rate the quality of the Day 3-5 training? 16% 18% 24% 36% 78% How would you rate the overall usefulness of it? 18% 20% 16% 42% 78% How would you rate the relevance of the Day 3-5 training to your organization’s mission? 16% 29% 13% 36% 78% My knowledge and skills increased as a result of the Day 3-5 training 20% 29% 18% 31% 78% The knowledge and skills gained through it are directly applicable to my work 13% 36% 11% 31% 78% 1 2 3 4 5 6 7 Design of curriculum and training, and trainer prowess (rated on a scale of 1=very poor to 5=very good) (n=45): % favorable 0% 20% 40% 60% 80% 100% (>3 out of 5) Attention to theoretical content 24% 42% 29% 71% Attention to practical content 16% 33% 38% 71% Time for instructors’ presentations 38% 20% 29% 49% Time for your participation 47% 29% 22% 51% Pace of the training 31% 40% 16% 56% Knowledge of the subject 7% 36% 56% 91% Quality of delivery 16% 33% 49% 82% Effectiveness in encouraging participation 18% 31% 47% 78% Effectiveness in answering participant questions 20% 31% 49% 80% Effectiveness in making you feel ready to apply the content 27% 31% 33% 64% 1 2 3 4 5 Based on this analysis and other open-ended feedback received, some recommendations for this track are: 15 ✓ Some participants recommended that trainers should use public health examples to make the training more applicable to their work. ✓ In Bucharest there were some shortcomings with respect to practical content, but this improved significantly in the other locations. ✓ The balance of time between instructors’ presentations and participant engagement, and the pace of the training should be improved. Optima HIV track Based on this analysis, the overall satisfaction with the Optima HIV track was 90%. Quality of training track material and applicability to participant needs (rated on a scale of 1=very poor/strongly disagree to 7=very good/strongly agree) (n=11): % favorable 0% 50% 100% (>4 out of 7) To what extent did the Day 3-5 program fulfill your learning needs? 36% 36% 27% 100% To what extent did the Day 3-5 program achieve its announced objectives? 18% 64% 18% 100% How did you like the group work and practicals? 36% 36% 27% 100% How would you rate the Day 3-5 training content? 27% 36% 36% 100% How would you rate the order in which the content was presented? 36% 27% 36% 100% How would you rate the materials used during the Day 3-5 training? 45% 27% 27% 100% How would you rate the quality of the Day 3-5 training? 18% 55% 27% 100% How would you rate the overall usefulness of it? 27% 27% 45% 100% How would you rate the relevance of the Day 3-5 training to your organization’s mission? 18% 18% 55% 91% My knowledge and skills increased as a result of the Day 3-5 training 27% 9% 64% 100% The knowledge and skills gained through it are directly applicable to my work 36% 18% 36% 91% 1 2 3 4 5 6 7 16 Design of curriculum and training, and trainer prowess (rated on a scale of 1=very poor to 5=very good) (n=11): % favorable 0% 20% 40% 60% 80% 100% (>3 out of 5) Attention to theoretical content 9% 55% 36% 91% Attention to practical content 9% 55% 27% 82% Time for instructors’ presentations 27% 36% 36% 73% Time for your participation 18% 55% 27% 82% Pace of the training 36% 45% 18% 64% Knowledge of the subject 18% 18% 64% 82% Quality of delivery 18% 36% 45% 82% Effectiveness in encouraging participation 9% 64% 27% 91% Effectiveness in answering participant questions 9% 45% 45% 91% Effectiveness in making you feel ready to apply the content 18% 64% 18% 82% 1 2 3 4 5 Health Services Prioritization Tool for Universal Health Coverage track Based on this analysis, the overall satisfaction with the Health Services Prioritization track was 67%. Quality of training track material and applicability to participant needs (rated on a scale of 1=very poor/strongly disagree to 7=very good/strongly agree) (n=19): % favorable 0% 50% 100% (>4 out of 7) To what extent did the Day 3-5 program fulfill your learning needs? 21% 26% 11% 32% 68% To what extent did the Day 3-5 program achieve its announced objectives? 11% 26% 21% 32% 79% How did you like the group work and practicals? 32% 16% 5% 42% 63% How would you rate the Day 3-5 training content? 26% 16% 26% 26% 68% How would you rate the order in which the content was presented? 21% 42% 11% 26% 79% How would you rate the materials used during the Day 3-5 training? 26% 26% 11% 26% 63% How would you rate the quality of the Day 3-5 training? 26% 11% 21% 32% 63% How would you rate the overall usefulness of it? 26% 16% 21% 37% 74% How would you rate the relevance of the Day 3-5 training to your organization’s mission? 11% 21% 21% 42% 84% My knowledge and skills increased as a result of the Day 3-5 training 11% 32% 21% 26% 79% The knowledge and skills gained through it are directly applicable to my work 21% 16% 32% 32% 79% 1 2 3 4 5 6 7 17 Design of curriculum and training, and trainer prowess (rated on a scale of 1=very poor to 5=very good) (n=19): % favorable 0% 20% 40% 60% 80% 100% (>3 out of 5) Attention to theoretical content 47% 16% 32% 47% Attention to practical content 53% 16% 16% 32% Time for instructors’ presentations 53% 21% 26% 47% Time for your participation 37% 26% 21% 47% Pace of the training 37% 32% 21% 53% Knowledge of the subject 11% 26% 53% 79% Quality of delivery 21% 16% 58% 74% Effectiveness in encouraging participation 5% 37% 53% 89% Effectiveness in answering participant questions 16% 21% 53% 74% Effectiveness in making you feel ready to apply the content 32% 32% 37% 68% 1 2 3 4 5 Based on this analysis and other open-ended feedback received, some recommendations for this track are: ✓ A few participants in Pretoria in the Health Services Prioritization track stated that it would have helped them understand the training material better if they had a French-speaking trainer. ✓ Many participants wished they were trained on a fully functional HSP tool, since it was still being finalized. ✓ Participants’ satisfaction with its practical content stayed consistently at around 32% across all three venues. Similarly, the balance between time for instructors’ presentation and time for participant engagement could be improved (satisfaction levels of 47%). ✓ On a more granular note, many participants rated the training materials and quality average and better, but the proportion of those satisfied with it was only 63%. These showcase the need to rethink the design and materials of the training track. Implementation Cascades track Only two respondents from the Implementation Cascades track responded to the SurveyMonkey evaluation form, hence it is not possible to reach conclusions by merely using data from this source. 18 Quality of training track material and applicability to participant needs (rated on a scale of 1=very poor/strongly disagree to 7=very good/strongly agree) (n=2): 0% 20% 40% 60% 80% 100% To what extent did the Day 3-5 program fulfill your learning needs? To what extent did the Day 3-5 program achieve its announced objectives? How did you like the group work and practicals? How would you rate the Day 3-5 training content? How would you rate the order in which the content was presented? How would you rate the materials used during the Day 3-5 training? How would you rate the quality of the Day 3-5 training? How would you rate the overall usefulness of it? How would you rate the relevance of the Day 3-5 training to your organization’s mission? My knowledge and skills increased as a result of the Day 3-5 training The knowledge and skills gained through it are directly applicable to my work 1 2 3 4 5 6 7 Design of curriculum and training, and trainer prowess (rated on a scale of 1=very poor to 5=very good) (n=2): 0% 20% 40% 60% 80% 100% Attention to theoretical content Attention to practical content Time for instructors’ presentations Time for your participation Pace of the training Knowledge of the subject Quality of delivery Effectiveness in encouraging participation Effectiveness in answering participant questions Effectiveness in making you feel ready to apply the content 1 2 3 4 5 Based on other sources of open-ended feedback, a recommendation for this track is: ✓ Many participants wished they were trained on a fully functional Implementation Cascades analysis tool, since it was still being finalized. 19 Optima Nutrition Based on this analysis, the overall satisfaction with the Optima Nutrition track was 86%. Quality of training track material and applicability to participant needs (rated on a scale of 1=very poor/strongly disagree to 7=very good/strongly agree) (n=37): % favorable 0% 20% 40% 60% 80% 100% (>4 out of 7) To what extent did the Day 3-5 program fulfill your learning needs? 22% 27% 49% 97% To what extent did the Day 3-5 program achieve its announced objectives? 22% 19% 57% 97% How did you like the group work and practicals? 16% 24% 51% 92% How would you rate the Day 3-5 training content? 11% 35% 43% 89% How would you rate the order in which the content was presented? 16% 24% 46% 86% How would you rate the materials used during the Day 3-5 training? 19% 22% 46% 86% How would you rate the quality of the Day 3-5 training? 16% 38% 35% 89% How would you rate the overall usefulness of it? 19% 24% 51% 95% How would you rate the relevance of the Day 3-5 training to your organization’s mission? 16% 22% 57% 95% 92% My knowledge and skills increased as a result of the Day 3-5 training 22% 27% 43% 92% The knowledge and skills gained through it are directly applicable to my work 32% 24% 35% 1 2 3 4 5 6 7 Design of curriculum and training, and trainer prowess (rated on a scale of 1=very poor to 5=very good) (n=37): % favorable 0% 20% 40% 60% 80% 100% (>3 out of 5) Attention to theoretical content 24% 41% 27% 68% Attention to practical content 22% 24% 38% 62% Time for instructors’ presentations 24% 35% 32% 68% Time for your participation 27% 43% 24% 68% Pace of the training 30% 32% 35% 68% Knowledge of the subject 5% 30% 65% 95% Quality of delivery 46% 51% 97% Effectiveness in encouraging participation 32% 65% 97% Effectiveness in answering participant questions 5% 27% 68% 95% Effectiveness in making you feel ready to apply the content 22% 43% 32% 76% 1 2 3 4 5 Based on this analysis and other open-ended feedback received, some recommendations for this track are: 20 ✓ Specifically amongst French speakers (all in Pretoria venue), the attention paid to practical and theoretical content was a bit below average overall (2.9 out of 5), though this is based on a sample size of 11 ✓ Many participants wished they were trained on a fully functional Optima Nutrition tool, since it was still being finalized. ✓ There is room for improvements to the attention paid to practical content. Optima TB track Based on this analysis, the overall satisfaction with the Optima TB track was 83%. Quality of training track material and applicability to participant needs (rated on a scale of 1=very poor/strongly disagree to 7=very good/strongly agree) (n=10): % favorable 0% 20% 40% 60% 80% 100% (>4 out of 7) To what extent did the Day 3-5 program fulfill your learning needs? 10% 20% 50% 20% 90% To what extent did the Day 3-5 program achieve its announced objectives? 10% 20% 20% 50% 90% How did you like the group work and practicals? 20% 30% 50% 100% How would you rate the Day 3-5 training content? 10% 10% 30% 50% 90% How would you rate the order in which the content was presented? 10% 10% 20% 60% 90% How would you rate the materials used during the Day 3-5 training? 10% 10% 40% 40% 90% How would you rate the quality of the Day 3-5 training? 20% 10% 30% 40% 80% How would you rate the overall usefulness of it? 20% 40% 40% 100% How would you rate the relevance of the Day 3-5 training to your organization’s mission? 10% 30% 10% 50% 90% My knowledge and skills increased as a result of the Day 3-5 training 20% 10% 20% 50% 80% The knowledge and skills gained through it are directly applicable to my work 30% 20% 50% 70% 1 2 3 4 5 6 7 21 Design of curriculum and training, and trainer prowess (rated on a scale of 1=very poor to 5=very good) (n=10): % favorable 0% 20% 40% 60% 80% 100% (>3 out of 5) Attention to theoretical content 10% 40% 40% 80% Attention to practical content 20% 40% 40% 80% Time for instructors’ presentations 20% 50% 30% 80% Time for your participation 30% 10% 60% 70% Pace of the training 30% 10% 60% 70% Knowledge of the subject 10% 40% 50% 90% Quality of delivery 10% 30% 60% 90% Effectiveness in encouraging participation 20% 20% 50% 70% Effectiveness in answering participant questions 20% 30% 50% 80% Effectiveness in making you feel ready to apply the content 40% 20% 40% 60% 1 2 3 4 5 22 Logistics One of the common challenges noticed with the skills building program was around language. The training in Bucharest had participants who spoke Spanish and Russian and the training in Pretoria had a significant number of French-speaking participants. Both locations had simultaneous translations from English to these languages, but only Pretoria had some trainers who were able to instruct in French; there was consensus that having a trainer who could fluently speak their language helped immensely. At the same time, the organizing team cannot ascertain for sure whether having translators helped Spanish, Russian, and French speakers learn to the same extent that the course helped English speakers, since they received equal amounts of feedback that praised the translation facilities or said it was not enough. The team also received some feedback on issues with the quality of translations, both for content (PPTs, handouts) and speech, which were corrected on location to the extent possible. In a training program that utilized data techniques and technology, one of the biggest shortcomings was the lack of stable WiFi in the Bangkok and Pretoria trainings. Many participants expressed disappointment with the lack of stable, fast internet, and since most tools used online resources or were web-applications, participants also felt that their excitement to use these tools was reduced due to hampered or slow access to the same. Sustainability and follow-up There was immense interest in having longer and additional trainings like the 2018 skills building program, clearly showing the growing interest in utilizing big data and cognitive analytics techniques for transforming health and development systems globally, and how this training course helped build that interest. Participants from a few countries, like Nepal, eSwatini, and Kenya, said that they were either already using or were about to use the tools they were getting trained on in real-world scenarios in their countries. Many also expressed interest in additional, in-depth trainings in their countries, wishing to bring this knowledge to their governments, research institutions, and existing projects. One of the most common requests in the training evaluation survey was for support and monitoring from the World Bank to apply these tools/models “Now that I understand at their country level, and/or requesting for a contact person modelling and the model better, I or expert (whether from the group of trainers or the World think there is no excuse for policy Bank) to have continuous engagement with as they go back makers not to use evidence from and try to implement what they learnt in these courses. More modeling studies” – participant specifically, there has been an additional interest in in Pretoria conducting geospatial analyses using Optima HIV in Botswana, Kenya, South Africa and Malawi. 23 Key takeaways The 2018 Big Data, Artificial Intelligence, and Decision Science in Health and Nutrition Skills Building Program was successful in building awareness and curiosity about new and existing data science and optimization analytics technologies amongst policymakers, academics, and other people working in the field of public health. As with any initiative, it had its own share of hurdles and challenges, but also comes with a future where engagement with and demand for the knowledge and skills training shared in this program will only be on the rise. Some of the key takeaways from the overall skills building program are: ✓ The skills building program achieved its objectives with 70-80% satisfaction rates. ✓ The trainers and speakers selected for the program were highly skilled. ✓ Participants felt that the skills and knowledge that they gained from the program were very useful in their work and for their organizations. ✓ Participants especially liked all the hands-on work and group exercises, and wished the program had more interactive sessions. ✓ Examples from real-world scenarios, especially when used in hands-on exercises, were instrumental in building the interest and engagement of participants. ✓ Some of the tools that participants were trained on need to be finalized and/or made more adaptable and applicable to their country contexts to facilitate usability and engagement. ✓ Technological impairments, such as slow WiFi, are extremely detrimental to training programs like these. ✓ There were issues with translations and facilitation in languages other than English that can potentially be done away by having facilitators who speak these languages on board. ✓ There is immense interest in follow up trainings that are more in-depth and focused on countries, as well as continuous engagement with trainers and the World Bank as participants go back to their countries and start using these tools in their work. This potential for engagement should be explored at the earliest. Recommendations for improvements are suggested for the following tracks: Big Data Analytics track: ✓ Trainers should use public health examples to make the training more applicable to the participants’ fields of work. ✓ The balance of time between instructors’ presentations and participant engagement, and the pace of the training should be improved. 24 Health Services Prioritization track: ✓ Having a French-speaking trainer might improve the understanding of this training track for participants who speak French. ✓ It is important to finalize the HSP tool and ensure that it is fully functional. ✓ More attention needs to be paid to practical content. ✓ The balance between time for instructors’ presentation and time for participant e ngagement needs to be improved. ✓ There is an overall need to rethink the design and materials of this training track. Implementation Cascades track: ✓ It is important to finalize the Implementation Cascades tool and ensure that it is fully functional. Optima Nutrition track: ✓ More attention needs to be paid to practical content. ✓ It is important to finalize the Optima Nutrition tool and ensure that it is fully functional. 25 APPENDIX SurveyMonkey training evaluation survey form: English Evaluation of Skills Building Program Training location: Bucharest Bangkok Pretoria Training track: Big Data HIV TB Implementation cascades Nutrition HSP tool Where you are from: World Bank staff/consultant | Government representative | Academic |Other FEEDBACK ABOUT THE COURSE OVERALL 1. What was your main reason for coming to the skills building program? (select all that apply) • To be able to conduct current/planned analyses effectively myself • To understand the analyses, their results and implications better (analyses done by others) • To be able to request such analyses by knowing more about them (better informed demand) • To network and share information • For professional interest and growth • To learn more about the Optima model that has or will be applied in my country • To learn about big data and artificial intelligence in health • Other (please specify): ________________________________________________________ 2. Extent to which this course made you curious about and helped you feel more competent in understanding and engaging in issues relating to AI in health 1= not at all 3 = Somewhat 5 = Totally changed how I feel about AI in health 3. Will you use what you learned at the Skills Building Program? If Yes, how? 26 FEEDBACK ABOUT DAYS 1 & 2 OF THE PROGRAM (2 PLENARY DAYS) Very Ave- Very poor rage good 4. How would you rate the Day 1 and 2 training content? ① ② ③ ④ ⑤ ⑥ ⑦ 5. How would you rate the order in which the content was ① ② ③ ④ ⑤ ⑥ ⑦ presented? 6. How would you rate the materials used during Day 1 and ① ② ③ ④ ⑤ ⑥ ⑦ 2? Very Me- Very low dium high 7. How would you rate the quality of the Day 1 and 2 ① ② ③ ④ ⑤ ⑥ ⑦ program? 8. How would you rate the overall usefulness of it? ① ② ③ ④ ⑤ ⑥ ⑦ 9. How would you rate the relevance of it to your ① ② ③ ④ ⑤ ⑥ ⑦ organization’s mission? Strongly Nei- Strongly Disagree ther Agree 10. My knowledge/skills increased as a result of the Day 1 and 2 agr ① ② ③ ④ ⑤ ⑥ ⑦ program 11. The knowledge/skills gained through it are directly applicable ① ② ③ ④ ⑤ ⑥ ⑦ to my work. 12. What worked best in the Day 1 and 2 program? 13. What would you recommend to improve the Day 1 and 2 program in the future? FEEDBACK ABOUT DAYS 3 TO 5 OF THE PROGRAM (TRACK-SPECIFIC TRAINING) Not Very 14. To what extent did the Day 3-5 program fulfill your learning needs? at ① ② ③ ④ ⑤ ⑥muc ⑦ all h 15. To what extent did the Day 3-5 program achieve its announced ① ② ③ ④ ⑤ ⑥ ⑦ objectives? 16. How did you like the group work and practicals? ① ② ③ ④ ⑤ ⑥ ⑦ Very Average Very 17. How would you rate the Day 3-5 training content? poor ① ② ③ ④ ⑤ goo ⑥ ⑦ d 18. How would you rate the order in which the content was ① ② ③ ④ ⑤ ⑥ ⑦ presented? 27 19. How would you rate the materials used during the Day 3-5 ① ② ③ ④ ⑤ ⑥ ⑦ training? 20. How would you rate the quality of the Day 3-5 training? ① ② ③ ④ ⑤ ⑥ ⑦ 21. How would you rate the overall usefulness of it? ① ② ③ ④ ⑤ ⑥ ⑦ 22. How would you rate the relevance of the Day 3-5 training to ① ② ③ ④ ⑤ ⑥ ⑦ your organization’s mission? Stron Neither Stron 23. My knowledge and skills increased as a result of the Day 3-5 gly gly ① ② ③ ④ ⑤ ⑥ ⑦ training. Disa Agre 24. The knowledge and skills gained through it are directly gree e ① ② ③ ④ ⑤ ⑥ ⑦ applicable to my work. 25. To assess how balanced the Day 3-5 training was, please rate each aspect below with respect to quantity. Very Very good poor a. Attention to theoretical content ① ② ③ ④ ⑤ b. Attention to practical content ① ② ③ ④ ⑤ c. Time for instructors’ presentations ① ② ③ ④ ⑤ d. Time for your participation ① ② ③ ④ ⑤ e. Pace of the training (insufficient = too slow; excessive=too fast) ① ② ③ ④ ⑤ 26. How would you rate the facilitators for the training tracks on each of the following aspects? Very Very good poor a. Knowledge of the subject ① ② ③ ④ ⑤ b. Quality of delivery ① ② ③ ④ ⑤ c. Effectiveness in encouraging participation ① ② ③ ④ ⑤ d. Effectiveness in answering participant questions ① ② ③ ④ ⑤ e. Effectiveness in making you feel ready to apply the content ① ② ③ ④ ⑤ 27. What would you recommend to improve the track-specific training in the future? 28. Any recommendations to improve the tool (either Optima HIV, Optima TB, Optima Nutrition, Implementation Cascades or the Health Service Prioritisation Tool for improved UHC) that was used in your training track 28 29. Do you have any technical support needs that you would like to discuss with the World Bank? Please detail them here. Thanks for completing the questionnaire. 29