Policy Research Working Paper 10981
Behaviorally Informed Messages Increase
COVID-19 Vaccination Intentions
Insights from a Global Meta-Analysis
Daniel Alejandro Pinzon Hernandez
JungKyu Rhys Lim
Michelle Dugas
Ellen Moscoe
Mohamad Chatila
Corey Cameron
Renos Vakis
Zeina Afif
Victor Hugo Orozco Olvera
Development Economics A verified reproducibility package for this paper is
Development Impact Group available at http://reproducibility.worldbank.org,
November 2024 click here for direct access.
Policy Research Working Paper 10981
Abstract
During the COVID-19 pandemic, low- and middle-in- data summarizes the results of this research program testing
come countries struggled with lower vaccination rates the impact of behaviorally informed messaging on vaccine
compared to wealthier countries, posing challenges to intentions. Results from the meta-analysis show that among
reducing virus transmission, mitigating healthcare system unvaccinated survey respondents, behaviorally informed
pressures, and promoting economic recovery. Communica- messages significantly increased the odds of vaccination
tions campaigns offer low-cost opportunities to overcome intention by 1.28 times overall and up to 1.93 times in
such challenges by strengthening vaccine confidence and individual studies (safety messages in Papua New Guinea).
intentions to get vaccinated, but empirical testing is needed Significant pooled effects of specific framings ranged from
to identify which messages will be most effective in different increasing the odds of vaccination intention by 1.16 times
contexts. To support policy-making efforts to design effec- (variant framing) to 1.45 times (experts and religious lead-
tive communication rapidly during the pandemic, a global ers framing). This research underscores the importance of
research program of 28 online experiments was conducted communication tailored to address different drivers of vac-
by recruiting respondents (123,270 individuals) through cine hesitancy and offers insights for handling future health
social media between January 2021 and June 2022 across crises with behavioral communication strategies leveraging
23 mostly low- and middle-income countries and territories. rapid insights afforded by social media.
An individual participant data (IPD) meta-analysis of these
This paper is a product of the Development Impact Group, Development Economics. It is part of a larger effort by the
World Bank to provide open access to its research and make a contribution to development policy discussions around the
world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may
be contacted at dpinzonhernandez@worldbank.org, rhyslim@worldbank.org, and mdugas@worldbank.org. A verified
reproducibility package for this paper is available at http://reproducibility.worldbank.org, click here for direct access.
RESEA
CY
LI R
CH
PO
TRANSPARENT
ANALYSIS
S
W
R
R E
O
KI P
NG PA
The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development
issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the
names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those
of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and
its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.
Produced by the Research Support Team
Behaviorally Informed Messages Increase COVID-19
Vaccination Intentions: Insights from a Global Meta-Analysis
Daniel Alejandro Pinzon Hernandez 1,2 dpinzonhernandez@worldbank.org
JungKyu Rhys Lim1,2 rhyslim@worldbank.org
Michelle Dugas1,2 mdugas@worldbank.org
Ellen Moscoe1 emoscoe@worldbank.org
Mohamad Chatila1 mchatila@worldbank.org
Corey Cameron1 ccameron90@gmail.com
Renos Vakis1 rvakis@worldbank.org
Zeina Afif1 zafif@worldbankgroup.org
Victor Hugo Orozco Olvera1 vorozco@worldbank.org
JEL Codes: I10, I12, I18, D91
Keywords: infectious disease, coronavirus, immunization, message framing
1
World Bank Development Impact (DIME).
2
Corresponding authors: Daniel Pinzon (dpinzonhernandez@worldbank.org), Rhys Lim (rhyslim@worldbank.org),
Michelle Dugas (mdugas@worldbank.org).
Acknowledgments
The authors express their gratitude to all who contributed to the implementation of this project,
especially colleagues from regional and global Health, Nutrition, and Population (HNP) teams, Poverty
and Equity teams, External and Corporate Relations (ECR) teams, the Virtual Lab, Meta for their ad
credits and support, and country government partners. The authors would like to thank Aneesa
Arur (Program Leader, HAEDR), Reena Badiani-Magnusson (Senior Economist, Program Leader, EECDR),
Tom Bundervoet (Lead Economist, EAEPV), Agnes Couffinhal (Senior Economist, HHNGE), Laura Di
Giorgio (Senior Economist, HLCHN), Olena Doroshenko (Senior Health Economist, HECHN), Yohana
Dukhan (Senior Health Economist, HAWH3), Denizhan Duran (Senior Health Economist, HMNHN), Diya
Nitham Mousa Elfadel (Health Specialist, HMNHN), Rochelle Se Yun Eng (Senior Health Economist,
HEAH1), Takahiro Hasumi (Senior Health Specialist, HECHN), Christopher H. Herbst (Senior Health
Specialist, Program Leader, HMNDR), Christopher Alexander Hoy (Economist, EPVGE), Timothy
A. Johnston (Manager, IEGHC), Lombe Kasonde (Senior Health Specialist, HHNGE), Phillis Kim
(Consultant, HHNGE), Haena Kim (Consultant, EPVGE), Leonardo Ramiro Lucchetti (Senior Economist,
EAEPV), Fernando Xavier Montenegro Torres (Senior Health Economist, HAEH1), Son Nam Nguyen (Lead
Health Specialist, HMNHN), Nicolas Rosemberg (Senior Economist, Health, HAWH3), Sering Touray
(Senior Economist, EAWPV), Opope Oyaka Tshivuila Matala (Senior Health Specialist, HAWH2), Sherin
Varkey (Program Leader, HAEDR), David Wilson (Director, HHNDR), and Laura Zoratto (Senior Economist,
EGVPI). The team is grateful to Alexandru Cojocaru (Senior Economist, EECPV) and an anonymous peer
reviewer for their valuable comments as peer reviewers, and to Charles Arnold for his input into an
earlier version of this paper. This study was partially funded by the Advancing Health Online (AHO)
Initiative, a fiscally sponsored project of Global Impact and funded by Meta and MSD to advance public
understanding of how social media and behavioral sciences can be leveraged to improve the health of
communities around the world. The findings, opinions, interpretations, recommendations, and
conclusions expressed herein do not reflect the views of the World Bank, its board of executive
directors, or the governments they represent.
2
1. Introduction and Literature Review
COVID-19 upended daily life and exacerbated global inequalities, resulting in about 6.88 million
reported deaths globally by March 2023 (Johns Hopkins University, 2023) and 14.83 million excess
deaths (i.e., the number of deaths during a crisis that exceeded the expected number under normal
conditions, including those directly caused by the crisis and those resulting from indirect impacts like
healthcare disruptions) (Msemburi et al., 2023). The development and regulatory approval of COVID-19
vaccines changed the trajectory of the pandemic. However, limited access to and acceptance of
vaccination resulted in global vaccine coverage inequalities that continued to persist throughout the
pandemic. Although 72.3% of the world’s population had received at least one dose of the COVID-19
vaccine as of March 2023 when many countries stopped regularly reporting vaccination data, there
remained stark gaps in vaccination rates across regions (Holder, 2023). For example, 37% of the
population received at least one dose of vaccination in Africa, 58% in the Middle East, 70% in Europe,
81% in the U.S. and Canada, and 82% in Asia-Pacific and Latin America (Holder, 2023). Among LMICs,
surveys found that Europe and Central Asia showed the highest vaccine hesitancy (58%), followed by the
Middle East and North Africa (47%), East Asia and Pacific (26%), Sub-Saharan Africa (15%), and Latin
America and the Caribbean (8%) (Dayton Eberwein et al., 2023). In particular in LMICs, COVID-19
vaccination can reduce transmission and mortality, prevent healthcare system overload, support
economic recovery, and address global vaccine inequity (Ali et al., 2022; WHO, 2021).
The journey to vaccination is a complex process involving structural factors, individual decision-
making, and last-mile actions. The success of vaccination programs is determined by promoting
vaccination intentions and translating these intentions into vaccine uptake. Issues of vaccine confidence
(i.e., lack of trust in the vaccine or provider), complacency (i.e., perceived lack of need for a vaccine),
and low convenience (i.e., limited access) can contribute to vaccine hesitancy 3 (Larson et al., 2014) and
ultimately discourage vaccine uptake.
Communications campaigns offer one approach to strengthening vaccine confidence and
promote intentions to get vaccinated. Although public health messages generally have small to
moderate effects on attitudes and behaviors (e.g., Noar, 2006; Wakefield et al., 2010), these messages
can have societal impacts as they reach large populations repeatedly (Funder & Ozer, 2019; Noar, 2006).
A recent summary of research found that media campaigns, both through traditional (e.g., television,
radio, and printed materials) and new social media, can significantly impact health knowledge, attitudes,
and behaviors (Orozco-Olvera & Malhotra, 2023). These approaches might be particularly effective
when coupled with other behavioral and policy interventions such as vaccine certificates or automated
vaccination appointment systems (e.g., van Bavel et al., 2020; Ruggeri et al., 2024).
As a key antecedent of vaccine uptake, it is important to understand the type of communication
that fosters vaccine confidence and intentions to get vaccinated. Given the limited empirical evidence to
guide such communication strategies as the pandemic was ongoing (Lim et al., in press), we launched a
global research program of 28 online survey experiments leveraging social media across 23 mostly low-
3
Vaccine hesitancy refers to a “delay in acceptance or refusal of vaccines despite availability of vaccination
services” on a continuum between full acceptance and refusal (WHO, 2014).
3
and middle-income countries and territories to help inform policy makers’ approach to communicate for
promoting uptake of the COVID-19 vaccine.
While research has explored vaccine acceptance globally, including new research on COVID-19
vaccine hesitancy, evidence on crafting effective communications to promote COVID-19 vaccine
acceptance in low- and middle-income countries is still limited (Batteux et al., 2022; Xia & Nan, 2023).
The gap in experimental evidence providing insights into effective messaging in LMICs and non-Western,
Educated, Industrial, Rich, and Democratic (WEIRD) contexts remains notable. Additionally, existing
syntheses of effective communication for encouraging COVID-19 vaccination were largely systematic
reviews (e.g., Batteux et al., 2022; Xia & Nan, 2023) and evidence synthesis (e.g., Ruggeri et al., 2024;
Nan et al., 2022; van Bavel et al., 2020) that did not use meta-analytic methods, which could identify
overall effect sizes, moderators, and sources of heterogeneity (Card, 2012).
Our research complements this work by quantifying and comparing messages’ overall effect
sizes and moderators across countries using meta-analytic methods, which can help generate more
precise policy recommendations about effective messaging to promote vaccine confidence. COVID-19
vaccine hesitancy may differ from hesitancy toward long-established vaccines like measles, mumps, and
Rubella (MMR); seasonal influenza; and diphtheria, tetanus, and pertussis (DTaP); or newer ones like
human papillomavirus (HPV), due to its rapid introduction and promotion on a large scale (e.g.,
Dombrádi et al., 2021).
The purpose of this research is to summarize the work completed in support of policy makers to
evaluate the effectiveness of different messages to increase COVID-19 vaccination intentions by
summarizing the findings of 28 survey experiments with 123,720 adults recruited from social media in
23 countries and territories between January 2021 and June 2022 using two-stage individual patient
data (IPD) meta-analysis (MA) methods. Results from the meta-analysis show that, among unvaccinated
survey respondents, behaviorally informed messages significantly increased the odds of vaccination
intention by 1.28 times overall. Significant pooled effects of specific framings ranged from increasing
odds of vaccination intentions by 1.16 times (framing that highlighted the shifting nature of the
pandemic and need to protect against different variants of the virus) to 1.45 times (framing that
leveraged messenger effects and endorsement of the vaccine by experts and religious leaders). While
messages featuring experts with religious leaders or celebrities, and dynamic social norms had
consistent impacts, others, such as endemic, safety, experts, efficacy and pro-social, and variant
messages, showed moderate-to-high heterogeneity. This research underscores the importance of
communication tailored to address different drivers of vaccine hesitancy and offers insights for handling
future health crises with behavioral communication strategies leveraging rapid insights afforded by
social media.
While this research offers evidence to help close the gap in understanding how to effectively
promote vaccine intentions in primarily LMICS, the study has limitations that should be noted. In
particular, this research leverages non-representative Facebook sampling that under-represents older,
poorer, and less educated groups, and could reflect self-selection bias. In addition, this study examines
impact of messaging on self-reported intentions instead of actual vaccination behavior. Still, this work
offers new evidence to fill gaps in understanding message effects on vaccine confidence in non-WEIRD
and primarily low- and middle-income countries that could inform future research that aims to address
this study’s limitations and future public policy efforts.
4
Literature Review
As a part of pandemic responses, effective public health communication and messaging can help
address vaccine hesitancy through strengthening confidence in the vaccine’s effectiveness and safety,
and mitigating complacency about the potential risks of the target virus (Batteux et al., 2022; Nan et al.,
2022; Xia & Nan, 2023). Who (i.e., messengers, information sources) says what (i.e., messages) are key
factors influencing message effectiveness (Lasswell, 1948; McGuire, 1989). Using trustworthy
messengers can help improve attitudes and encourage health behaviors (Huang & Sundar, 2022; Lu et
al., 2021; Ruggeri et al., 2024). For example, during COVID-19, some studies found that vaccine uptake
and vaccination intentions were increased when messages were conveyed by healthcare professionals
(Bartoš et al., 2022), politicians (Pink et al., 2021), and messengers with religious orientation (Chu et al.,
2021) and races (Alsan & Eichmeyer, 2024) aligned and/or matched to the target audience. However,
other research has not found evidence for the messenger effect (Huang & Liu, 2022; Motta et al., 2021),
underscoring the importance of testing messages and messengers in different contexts.
Regarding ‘what to say’ (content of the messages), prior research has identified promising
message features applicable to encouraging COVID-19 vaccination including social norms, individual and
collective benefits, and vaccine safety (Batteux et al., 2022; Ruggeri et al., 2024; Nan et al., 2022; Xia &
Nan, 2023). In the COVID-19 context, some studies testing descriptive norms (i.e., what others do) in
messages highlighting wide vaccine uptake increased vaccination intentions (Ryoo & Kim, 2021; Sinclair
& Agerström, 2021), whereas other studies did not find effectiveness (Santos et al., 2021; Sasaki et al.,
2022). Prior research found that messages focusing on individual benefits did not significantly increase
COVID-19 vaccination intentions (Bokemper et al., 2021; Freeman et al., 2021; James et al., 2021), while
messages focusing on collective benefits could increase the intentions (Freeman et al., 2021; James et
al., 2021; Trueblood et al., 2022).
As some vaccine hesitancy reflected concerns over vaccine safety (Dayton Eberwein et al.,
2023), messages focusing on vaccine safety have also been found to increase vaccination intentions
(Argote Tironi et al., 2021; Ashworth et al., 2021; Davis et al., 2022; James et al., 2021; Palm et al., 2021).
Lastly, to overcome the message fatigue (i.e., overexposed, redundant, exhausted, tedious) that people
felt from prolonged, extensive COVID-19 vaccination communication (So et al., 2017) and consequent
lack of engagement in protective behaviors (Guan et al., 2023), it is important to develop and use
diverse messages (Okuhara et al., 2023). In our study, we developed messages focusing on evolving
contexts, such as recognizing that the virus was becoming endemic and the growing number of new
variants.
As noted, prior systematic reviews found that most experimental studies testing messages
encouraging COVID-19 vaccinations were conducted in WEIRD societies and upper-income countries
(Batteux et al., 2022; Xia & Nan, 2023), warranting research to test what messages can encourage
vaccination behaviors in non-WIERD contexts. Given limited evidence from experiments testing COVID-
19 vaccine messages conducted in LMICs to address vaccine hesitancy, we aim to close this gap in the
literature with a summary of insights from our global research program testing messages to encourage
COVID-19 vaccinations across 23 non-WEIRD and mostly lower- and middle-income countries with meta-
analyses.
5
2. Method
2.1. Study Sample
To identify whether different behaviorally informed vaccine messages can increase intent to
vaccinate, we conducted 28 randomized survey experiments using social media in 23 countries and
territories between January 2021 (when vaccines were not widely available) and June 2022 (when
vaccine supply was widely available). See Appendix A for a list of sampled countries and when studies
were implemented in each country. Multiple studies were conducted in Lebanon, Iraq, and Jordan
throughout the course of the pandemic. Each study followed similar protocols for recruitment,
experimental design, and survey measurement, which are described below. The results from individual
studies have been reported in other publications for Belize, Haiti, Jamaica (Margolies et al., 2023), Papua
New Guinea (Hoy et al., 2023), Saudi Arabia (Lim et al., 2024), and Zambia (Hoy et al., 2022). Also, a
separate survey with family doctors in Romania (Badiani-Magnusson et al., 2022) was conducted as part
of the same global research program, yet was not included in this IPD meta-analysis. Our experiments
were approved by the Health Media Lab Institutional Review Board (HML IRB), approval number #
1017TWBG21.
2.2. Participant Recruitment and Sampling
Participants were recruited through Facebook using quota sampling by displaying ads inviting
participants to interact with a chatbot in Facebook Messenger to complete the survey (using languages of
preference) for a chance to win a lottery reward.
The sample in each country was drawn from all Facebook accounts 18 years and older in the
country. We used quota sampling by exposing the recruitment ads to pre-defined demographic groups
based on age, gender, and region to mirror the country’s populations. To get a quota sample and optimize
recruitment, we updated the ad placement in real-time based on participating Facebook users’
characteristics to ensure proper representation of each demographic group in the final sample using
marketing applications such as the Virtual Lab (Rao et al., 2020) and RevealBot (RevealBot, 2024). For
example, if a subgroup was underrepresented and had higher attrition or non-response, then the budget
for ads targeting that demographic subgroup was automatically increased to better approximate the
required sample for that subgroup. 4
Across all studies, the final sample included 229,886 responses in 28 studies with an average of
7,800 responses per online study. Only participants who had not received the COVID-19 vaccine were
included in the experiment (n = 123,720), which is the focus of the results reported in this paper. No
4
For example, we recruited participants in Iraq targeting ads based on a combination of gender, age group, and
governorate. This combination created x strata for recruitment with each stratum reflecting a combination of
these three characteristics. For instance, one set of ads targeted women aged 40-49 who lived in the governorate
of Baghdad while another set of ads targeted men aged 40-49 who lived in the governorate of Baghdad with quota
targeting based on the latest available population data. During data collection for the study, if the proportion of
participants from this demographic group in the study sample was lower than its proportion in the broader
population, the recruitment ads were dynamically adjusted to spend more money to better target this population
and increase its proportion in the study sample.
6
additional data cleaning was necessary because the Facebook Messenger chatbot ensures that only one
survey is recorded per Facebook account and discards responses that deviate from the questions asked.
2.3. Study Procedures
Participants who clicked on the study advertisement were redirected to Facebook Messenger,
where the survey was conducted via chatbot. Participants selected their preferred language and gave
informed consent at the beginning of the survey before proceeding to the survey modules.
First, we asked participants’ COVID-19 vaccination status. Those who were unvaccinated were
included in the survey experiment that randomized participants to different messaging arms and asked
for their vaccination intentions, our main outcome. We then asked participants about their perceptions
of the main benefits of or concerns about COVID-19 vaccination, additional questions about attitudes
and beliefs related to COVID-19, use of different information channels (e.g., social media, radio, etc.),
and trusted information sources. Lastly, we asked participants to provide information about their
demographics. The study took approximately 10 minutes, on average. See Figure 1 for a summary of
participant inclusion and exclusion at each step of the studies.
Randomization. After answering a question about their COVID-19 vaccination status,
unvaccinated participants were randomized to study arms with different behaviorally-informed message
framings about the COVID-19 vaccine. Randomization was stratified such that participants had a greater
probability of being allocated to the control message compared to other arms, and all treatment
message arms had an equal probability of assignment. See Appendix B for more detail on the proportion
of samples randomized to each arm. 5
5
More participants were assigned to the control condition to provide a subsample for diagnostics to understand
attitudes and beliefs about a country’s social media population with potential impacts of framing. These
diagnostics were used to inform the government’s COVID-19 strategy along with results from the survey
experiments.
7
Figure 1. Study flow diagram.
Stimuli: Message Framings
Message framings were informed by prior research and leveraged principles like the messenger
effect, targeting beliefs about the safety and efficacy of the vaccine, and social norms. Reflecting unique
country contexts and the evolving pandemic circumstances, the content of messages varied across
countries and over time. For example, messages highlighting variants and COVID-19 becoming endemic
were tested only at later stages of the pandemic, whereas framings about expert endorsement and
safety were tested early and throughout much of the pandemic. In addition, the selection of messages
to be tested in each country was informed by expectations of what would be most effective in a given
8
context—for instance, messages relating to religious leader endorsement were tested in contexts where
religion was expected to have greater impacts such as religious countries in the Middle East and North
Africa. As such, direct comparisons of the magnitude of impacts across different framings should be
interpreted with caution—as differences in impacts may reflect systematic differences in country and
timeline of rollout. Two framings (financial incentives, mortal risk of COVID-19) were tested in fewer
than three studies and were thus excluded from the meta-analysis. See Appendix C for the
operationalization of message framings used in each study.
Control framing (k = 28, n = 62,046). The control framing was a pure control with no additional
text before the vaccine intention question.
Experts framing (k = 16, n = 14,152). This framing aimed to leverage a messenger effect by
emphasizing that experts consider the vaccines safe and effective (e.g., ‘The COVID-19 vaccines available
in {country} ({type of vaccines}) are considered safe and highly effective by national and international
experts’).
Experts + celebrities framing (k = 7, n = 9,492). This framing leveraged the messenger effect by
alluding to endorsements of experts as well as celebrities, who have also been demonstrated to have an
influential effect on health behavior (Alatas et al., 2024; Chu et al., 2021) (e.g., ‘The COVID-19 vaccines
available in {country} ({type of vaccines}) are considered safe and highly effective by national and
international experts, and celebrities and famous athletes get it themselves’).
Experts + religious leaders framing (k = 8, n = 10,082). This framing leveraged the messenger
effect by alluding to endorsement of religious leaders as well as experts, as religious leaders also play an
important role in shaping health behaviors in many contexts (Rakotoniana et al., 2014) (e.g., ‘If a COVID-
19 vaccine is considered safe and effective by national and international experts, and religious leaders in
your community get it themselves, would you plan to take the vaccine?’).
Safety framing (k = 12, n = 6,298). This framing contrasted the safety of the COVID-19 vaccines
to the dangers of COVID-19 infection, addressing concerns about the safety of vaccination (e.g., ‘COVID-
19 vaccines are safe - there have been no reported hospitalizations in {Country} due to vaccinations
compared to {number} deaths due to COVID-19’).
Efficacy and pro-social framing (k = 14, n = 8,841). This framing emphasized the efficacy of
COVID-19 vaccines for protecting the vaccinated person and their family and friends from serious health
complications. (e.g., ‘The latest studies from around the world confirm that the COVID-19 vaccines
protect you, your friends and family from COVID-19 by substantially reducing or eliminating
hospitalizations and deaths).
Dynamic social norms framing (k = 12, n = 6,258) This framing used messaging to convey that
descriptive norms were shifting toward a growing number of people getting vaccinated against COVID-
19 (e.g., ‘{Nationality} are getting vaccinated against COVID-19! More than {number} have done it so far,
with {number} just in the past 2 weeks alone’). Messages used data available from governments or
external sources to ensure the numbers provided were credible.
Endemic framing (k = 7, n = 3,215). This framing emphasized the vaccination needs for the
future, as the COVID-19 virus will continue to exist (e.g., ‘As we learn to live with COVID-19, it's likely we
9
will all be exposed to the virus eventually. The best way to prepare yourself is to get fully vaccinated
because it greatly reduces the risk of hospitalization and death’).
Variants framing (k = 6, n = 3,336). This framing focused on the vaccine’s efficacy against new
variants, to account for concerns at the time about whether the vaccines would be protective against
different strains of the virus (e.g., ‘New variants like Omicron and Delta can be worrying, but the best
evidence so far indicates that vaccines are still highly protective against serious illness and death from
COVID-19’).
2.4. Measures and Indices
Primary Outcome
Vaccine Intention. The outcome of interest is self-reported vaccination intention, which was
measured for respondents who reported that they were not yet vaccinated at the time of the study.
Vaccine intention was measured as a categorical outcome by asking a question “Are you willing to/do
you plan to get the COVID-19 vaccine?” Response options included “Yes,” “Unsure,” and “No”. For our
analysis, we recode these responses into a binary classification of “Yes” or “No/Unsure”.
Potential Individual-Level Moderators
Gender. Participants were asked to indicate whether their gender was male, female, or other. In
some country contexts, it was considered inappropriate to include the ‘other’ category, and participants
were simply asked if they were male or female.
Age. Age was measured categorically, asking participants to indicate whether they were 18-29,
30-39, 40-49, 50-59, or 60+.
Education. Participants were asked to indicate the highest level of education they had
completed – No Education, Primary Education, Secondary Education, or Tertiary Education.
Health worker. Participants were asked to indicate whether they were employed as a health
worker or not.
Potential Study-Level Moderators
Based on discussion with public health and behavioral science experts and a review of the
literature, we explore the following potential moderators: the month of study launch, the GDP per
capita of the study country, the vaccination rate in the country at the time of the study, and trust and
vaccine perception measures from the Wellcome Global Monitor Survey (2018). The Wellcome Global
Monitor survey is a survey of attitudes toward science and health based on a survey with over 140,000
people in more than 140 countries. Since these data from the Wellcome Global Monitor survey are
measured at the individual level, we aggregate these measures to the country level by taking their mean
value. This approach is consistent with practices in cross-cultural and health research (e.g., Gelfand et
al., 2011; Weinberg et al., 2021), including research estimating generalized trust at the country level
(e.g., Marozzi, 2014; Reeskens & Hooghe, 2008; Robbins, 2012). See Appendix D for more details on the
empirical rationale for country-level aggregation of the Wellcome Global Monitor survey indicators. The
Wellcome Global Monitor 2018 data were not available for Belize, Djibouti, Jamaica, Papua New Guinea,
and West Bank and Gaza.
10
Country-level vaccination rate (at least one dose). We include an estimate of the proportion of
the country’s population that received at least one dose of a COVID-19 vaccine. Country level data was
obtained at the time of data collection from Our World in Data (2024), which collected data from public
official sources. The effect of the intervention could be smaller in countries with higher vaccination rate
because the remaining unvaccinated sample may be particularly reluctant to receive the vaccine. It
might also reflect the level of deployment of the vaccine as well as the general acceptance of the
vaccine.
Intervention start date. We included the start month of the data collection. The dates were
centered so that the starting month of the first intervention (i.e., January 2021) was set to month 0, and
so on until month 16, corresponding to the start of the last intervention (May 2022). The intervention
start date might also reflect the available knowledge about the virus and availability of the vaccine at
the time of the intervention.
GDP per capita PPP. We also explore country wealth using gross domestic product (GDP)
converted to international dollars using purchasing power parity (PPP) rates from the World Bank
(2024). An international dollar possesses equivalent purchasing power relative to GDP as the U.S. dollar
does within the United States. The GDP per capita has been shown to correlate with a large number of
social and health outcomes across countries (Bloom & Canning, 2000), which may influence the
effectiveness of the behaviorally informed messages targeting vaccine intentions. When including this
variable in the study-level moderator analysis, we center the data by subtracting the median GDP per
capita PPP of the sample countries from the GDP per capita PPP of each country.
Trust in healthcare workers. We explore whether trust in healthcare worders (e.g., doctors and
nurses) before the onset of the pandemic moderates impacts of message framings. We use data from
the Wellcome Global Monitor (Wellcome Trust & Gallup, 2018), which assessed trust in healthcare
workers with an item rating “How much do you trust each of the following?” for “Doctors and nurses in
this country” on a 1-4 scale, with response options of “A lot”, “Some,” “Not much,” “Not at all”.
Respondents could also respond with “Don’t know,” and “Refused”—such responses were dropped
from our analysis. For ease of interpretation, we reverse-coded the responses, so higher values indicate
stronger trust such that 1 = Not at all and 4 = A lot. We aggregate trust to the country level by
calculating the mean response within each country. We then center country-level trust measures by
subtracting the median country-level trust measure from each country-level means.
Perceived vaccine safety. We included perceived vaccine safety before the COVID-19 pandemic.
We used data from the Wellcome Global Monitor (Wellcome Trust & Gallup, 2018). Perceived vaccine
safety was measured by asking “Do you agree, disagree, neither agree nor disagree with the following
statement? Vaccines are safe.” on a 1-5 scale, with response options of “Strongly agree,” “Somewhat
agree,” “Neither agree nor disagree,” “Somewhat disagree,” “Strongly disagree,” “Don’t know,” and
“Refused.” We reverse-coded the responses, so higher values indicate stronger agreement, such that 1 =
Strongly disagree and 5 = Strongly agree. We aggregate perceived vaccine safety to the country level by
calculating the mean response within each country. We then center country-level vaccine safety
perceptions by subtracting the median country-level trust measure from each country-level mean.
Perceived vaccine effectiveness. We included perceived vaccine effectiveness before the COVID-
19 pandemic using data from the Wellcome Global Monitor (Wellcome Trust & Gallup, 2018). Perceived
vaccine effectiveness was measured by asking “Do you agree, disagree, neither agree nor disagree with
11
the following statement? Vaccines are effectiveness.” on a 1-5 scale, with response options of “Strongly
agree,” “Somewhat agree,” “Neither agree nor disagree,” “Somewhat disagree,” “Strongly disagree,” As
with the measure of vaccine safety, we reverse-coded the responses, so higher values indicate stronger
agreement, and responses of “Don’t know” and “Refused” were dropped. We aggregate perceived
vaccine effectiveness to the country level by calculating the mean response within each country. We
then center country-level vaccine safety perceptions by subtracting the median country-level trust
measure from each country-level mean.
2.5. Meta-Analytic Approach
To summarize findings from our research program, we conduct a meta-analysis using a random-
effects, two-stage individual participant data (IPD) meta-analytic approach (Riley et al., 2021). The
random effects model allows for variation in study effects, appropriate for our context where
operationalizations of independent variables and outcome measurements differ slightly across studies.
Moreover, IPD enables us to fully leverage the individual level data collected, allowing us to explore
study-level moderators that could explain variation in messaging effects across studies as well as
heterogeneity in treatment effects that could be explained by individual-level characteristics. IPD meta-
analyses are commonly used in contexts such as clinical research, where individual participant data is
more readily available, offering advantages over the traditional meta-analytic approach that rely solely
on aggregate statistics (Burke et al., 2016; Debray et al., 2015; Tierney et al., 2015). As noted earlier, the
operationalization of message framings and the primary outcome were tailored to each study’s context.
All analyses were conducted with Stata (version 18).
To estimate the treatment effects for the framing experiment, we use logistic regression with
treatment arm as the exposure and a binary measure of vaccine intention as the outcome variable (1 =
intend to vaccinate). In our analysis, we control for a range of individual-level covariates, including
gender, age, education, and whether the participant was a health worker. For covariates and
moderators, we use categorical variables coded to have a category for non-response (i.e., missing) to
avoid dropping observations where possible. In the first stage of the IPD MA, we calculate the logit
model for each one of the studies as follows:
1
������������(������������ ) = (1)
1+������������ −(������������������������������������ + ������������������������������������ + )
For s =1,2,3…,28 studies
Where ������������(������������ ) is the probability of intent to vaccinate, ������������������������ is the behaviorally informed message assigned
and ������������������������ a set of individual level covariates.
̂ are pooled across studies (s) using the common-
In the second stage, the treatment effects estimated ������������
effect inverse-variance model as follows:
∑������������ ������������������������ ������������������������ 1
̂=
������������ and ̂) =
������������������������(������������
∑������������ ������������������������ ∑������������ ������������������������
1
Where ������������������������ =
������������������������(������������������������ )2
12
We consider pooled effects to be significant when the 95% confidence intervals do not include 0
in their range.
For heterogeneity, we use the I2 index, or the ratio of excess to total variability, to measure
inconsistency of effect sizes. To interpret the I2 index, 25 is generally considered low, 50 is moderate,
and 75 is high (Higgins, 2003).
Missing Data Analysis
We conduct Little's Missing Completely at Random (MCAR) test to determine whether missing
data in our datasets are MCAR –meaning, whether missingness is unrelated to any observed or
unobserved data. Little’s MCAR test showed that the missing data in gender, education, age and health
worker status are not MCAR in 19 of the studies (see Appendix E). While the studies conducted in the
Facebook Messenger chatbot required participants to answer each question before moving on, there
are other reasons for missing data. First, the three studies in Tunisia and the study in Ukraine were
conducted in SurveyMonkey and allowed participants to skip questions. Second, some studies added the
response options "other" or "prefer not to answer" to demographic questions that were coded as
missing due to small sample sizes. Third, in some cases, at the request of the in-country counterpart(s)
or to assess whether demographic questions increased attrition, these questions were asked after the
experiment, resulting in valid observations with no responses to demographic questions. Finally, some
questions were not included at the request of the country counterpart(s), particularly those related to
health worker status.
Following best practices non-random missingness, instead of analyzing only complete cases, we
use the model-based multiple imputation for our analysis (Allison, 2002; Enders, 2022; Schafer &
Graham, 2002) with ordered logistic regression, creating 18 imputations and randomly sorting the
dataset. Although multiple imputation assumes that data is Missing at Random (MAR), it can often
remain unbiased when data is Missing Not at Random (NMAR) (Schafer & Graham, 2002). This
imputation method estimates multiple values based on the distribution of the observed data to capture
uncertainty. Multiple imputation involves three steps: first, the missing data is filled in with estimated
values m times to create m complete datasets; second, each of these datasets is analyzed using a
statistical method; finally, the results and parameter estimates are pooled for inference. Following
recommendations that m should equal the percentage of incomplete cases (Bodner, 2008; White et al.,
2011), we used 18 iterations, with missing cases ranging from 1% to 17%. Multiple imputation provides
more accurate results by considering the variability from both sampling and imputation.
Individual-Level Moderator Analysis
To analyze individual-level moderators, we perform subgroup analyses for gender, age,
education level, and health worker occupation. Interactions between treatment and gender, treatment
and age, treatment and education level, and treatment and occupation are evaluated by logistic
regression analyses. For ease of interpretation, we explore interaction effects be recoding the age and
education categorical variables as binary. When interactions were statistically significant, further post
hoc analyses are performed with participants of male vs. female, 40 years old or older vs. younger than
40 years old, secondary education or below vs. tertiary education, and health workers vs. non-health
workers. More specifically, we estimate the pooled effects within each subgroup of interest to
13
decompose significant interactions, following the same procedure as for main effects but only including
observations of the relevant subgroup.
Study-Level Moderator Analysis
Study-level moderator analysis is conducted for framings where significant study-level variation
is observed according to the I2 index. Specifically, random-effects meta-regressions where study-level
moderators are regressed on the effect sizes of a given framing across different studies are performed,
with a meta-regression estimated for each framing. In this model, regression coefficients indicate the
relationship between a covariate and the impact of a given framing. For example, we can interpret a
regression coefficient of 0.03 for intervention start date as indicating that an increase in one month of
the start date of a study corresponds to an increase of 0.03 units in log odds ratio in the impact of the
experts framing on vaccination intentions.
Given the number of explored moderators (7) and the small number of studies testing certain
framings (i.e., endemic, variants), we first examined the bivariate correlation between moderators and
effect sizes and included only those with a significant relationship in our final meta-regression model for
a given framing. While other IPD MA studies also had small numbers of studies (i.e., 6-7) when
conducting moderator analysis (e.g., Jenum et al., 2019; Louise et al., 2021; Smelt et al., 2018), we note
that our results should be interpreted with caution as we have a modest number of 6-16 studies
included in our analyses to explain heterogeneity across studies. For efficacy and pro-social (k = 14),
endemic (k = 7) and variant (k = 6) framings, none of the tested moderators explained a significant
amount of variation in effect sizes.
3. Results
3.1. Characteristics of Participants and the Study Sample
Table 1 presents the demographic breakdown of the study sample, including the total sample
that completed the surveys—both vaccinated participants who did not participate in the survey
experiment and unvaccinated participants who were included in the survey experiments. Reflecting the
typical social media user in LMICs, the sample skewed toward being male (52.4%), highly educated
(43.6% tertiary education), and somewhat young (29.2% 18-29 years old).
14
Table 1. Summary of individual-level characteristics of full sample and survey experiment sample
included in the meta-analysis.
Unvaccinated
Vaccinated
Total Experiment
Variable Group Sample
(n=217,254) Sample
(n=93,554)
(n=123,720)
Male 52.35 51.64 53.30
Gender Female 47.15 47.80 46.29
N/A 0.50 0.56 0.41
18-29 29.18 33.34 23.68
30-39 21.78 22.90 20.29
40-49 17.68 16.04 19.84
Age
50-59 14.20 10.58 19.00
60+ 6.68 4.41 9.68
N/A 10.48 12.73 7.51
No education 1.43 1.40 1.48
Primary 8.02 8.00 8.05
Education Secondary 34.78 33.69 36.22
Tertiary 43.57 42.22 45.35
N/A 12.20 14.70 8.90
No 70.51 70.21 70.91
Health Worker Yes 12.27 12.83 11.52
N/A 17.22 16.96 17.57
Note. Distribution of respondents by key demographic characteristics and COVID-19 vaccination status. N/A includes
participants who chose this response option or did not complete the survey. Gender, age, and education were asked in all
surveys, whereas health worker status was missing in The Gambia.
Given the large size of the sample across all studies, all differences in the demographic
characteristics between the unvaccinated and vaccinated samples are significant. Qualitatively,
however, the sample characteristics are similar between unvaccinated and vaccinated in gender and
education, with slightly larger proportions of more educated participants in the vaccinated sample. The
biggest difference between the unvaccinated sample used for the intervention and the vaccinated
sample is the age distribution, potentially reflecting most countries’ policies of vaccinating the older
population first. Unexpectedly, however, the proportion of participants indicating that they were health
workers was slightly greater among the unvaccinated (12.8%) than vaccinated (11.5%) samples.
The studies were conducted over 18 months in mostly low- and middle-income countries and
territories, reflecting demand from policy makers for more evidence to guide their communication
strategies. Table 2 summarizes the timeline of implementation across different studies and the
corresponding rate of vaccination at time of data collection by region (see Appendix A for statistics for
each study). Early on, implementation of the studies began before any vaccines were distributed
including Lebanon Round 1 (R1), Iraq R1, West Bank and Gaza, Tunisia R1, Libya, Chad, Cameroon, and
Honduras. Towards the end of the research program, studies were implemented in some contexts
where about half the country population or more had been vaccinated (Saudi Arabia, Tunisia R3,
15
Lebanon R2, Belize). Given the wide range in timing of studies and vaccination rates, we take into
account these differences in our moderator analysis examining study-level heterogeneity.
Our samples recruited from social media reported significantly higher vaccination levels than the
national average at the time of the survey in all studies. This suggests that the unvaccinated population
in countries with higher coverage already had access to the vaccine (compared to countries with low
vaccine access) and, therefore, it is plausible that the unvaccinated in these cases were particularly
resistant to vaccination. In addition, this finding highlights some of the potential uniqueness of our study
samples—these samples recruited through social media are not representative of the general
population, and results should be interpreted with this in mind.
Table 2. Summary of study-level moderators of full sample and survey experiment sample included in the
meta-analysis.
Region No. of Start dates of Mean Mean Mean Mean Mean Mean
Studies Data Collection COVID-19 National GDP per capita, National National National
vaccination COVID-19 PPP Trust in Perceived Perceived
rate in vaccination (constant 2017 doctors vaccine vaccine
sample rate at international $) and nurses safety effectivenes
time of the (1-4) (1-5) s
experiment (1-5)
Middle East & North 37% 22% 16,655 3.0 4.5 4.5
14 Jan-21 to Apr-22
Africa (0%, 86%) (0%, 54%) (4,768; 47,315) (2.7, 3.2) (4.3, 4.6) (4.4, 4.7)
Sub-Saharan Africa and 39% 9% 2,367 2.9 4.3 4.5
5 Aug-21 to May-22
the Sahel (0%, 72%) (0%, 19%) (1,060; 3,757) (2.5, 3.4) (4.1, 4.7) (4.4, 4.7)
45% 21% 12,630 3.0 3.9 4.1
Europe, Asia & Pacific 5 Jun-21 to Dec-21
(16%, 64%) (1%, 39%) (3,948; 18,307) (2.9, 3.1) (3.3, 4.2) (3.8, 4.4)
Latin America & 41% 22% 6,997 3.0 4.3 4.6
4 Mar-21 to Apr-22
Caribbean (0%, 89%) (0%, 59%) (3,102; 10,244) (3.0, 3.1) (4.1, 4.6) (4.5, 4.6)
Note. Experiment start date is the month of Facebook ad launch for the first and last intervention in the region. COVID-19
vaccination rate and GDP per capita show the mean for the region with the minimum and maximum in parenthesis.
16
3.2. Main Effects on Vaccine Intentions
We first explored the overall effect of behaviorally-informed framings compared to the control
group. The pooled estimate of all tested messages’ impacts on vaccination intentions was a log odds
ratio [LOR] of 0.24 (95% CI: [0.22, 0.27], OR: 1.28), indicating that he use of any behaviorally-informed
messaging increased participants’ odds of having the intention to get vaccinated against COVID-19 by
1.28 times compared to the control group (see Appendix F for meta-analysis results transformed into
odds ratios).
Next, we estimated the effect of each framing on vaccination intentions—see Figures 2 and 3
for the study level effect by message framings compared to the control group. In descending order of
magnitude, the pooled estimates of the messages’ with significant 6 impacts on vaccination intention
was a 0.37 LOR (95% CI: [0.32, 0.42], OR: 1.45) for the message using experts and religious leaders, 0.35
LOR (95% CI: [0.27, 0.44], OR: 1.42) for the endemic framing, 0.28 LOR (95% CI: [0.23, 0.33], OR: 1.33) for
the message using experts and celebrities, 0.24 LOR (95% CI: [0.18, 0.29], OR: 1.27) for the efficacy and
pro-social framing, 0.23 LOR (95% CI: [0.19, 0.27], OR: 1.26) for the message using experts, 0.15 LOR
(95% CI: [0.06, 0.24], OR: 1.16) for the variants framing, and 0.14 LOR (95% CI: [0.07, 0.20], OR: 1.15) for
the dynamic social norms framing, meaning that these message framings were effective. Conversely, no
difference between the safety message group and the control group was observed from the pooled
estimates (LOR: 0.04, 95% CI: [-0.02, 0.11], OR: 1.05).
Messages using experts and religious leaders (I2 = 39.5%, p = 0.115), experts and celebrities (I2 =
0.0%, p = 0.686), and dynamic social norms (I2 = 0.0%, p = 0.489) showed non-significant levels of
heterogeneity, meaning that these messages showed a consistent level of impacts across studies and
countries.
Conversely, results indicated significant moderate-to-high heterogeneity across the studies in
the impact of the endemic framing (I2 = 75.7%, p < 0.001), the safety framing (I2 = 70.1%, p < 0.001), the
experts framing (I2 = 69.4%, p < 0.001), the efficacy and prosocial framing (I2 = 60.3%, p = 0.002), and the
variant framing (I2 = 60.7%, p = 0.026), indicating that that these messages’ impacts significantly vary by
study.
Specifically, the endemic messages significantly increased vaccination intentions in Iraq,
Lebanon, and Haiti, but not in Tunisia, The Gambia, Belize, and Jamaica. The safety messages were
effective only in Papua New Guinea and Serbia, but not in Kuwait, Jordan, Saudi Arabia, Djibouti, Tunisia,
Jordan, the Republic of Congo, The Gambia, Kosovo, and North Macedonia. The messages using experts
were effective in Lebanon, Iraq, West Bank and Gaza, Tunisia, Libya, Chad, Cameroon, Serbia, and
Honduras, but not in Kuwait, Jordan, Saudi Arabia, Djibouti, Papua New Guinea, Kosovo, and North
Macedonia. The efficacy and prosocial messages were effective in Jordan (R1 and R2), Zambia, Kosovo,
Servia, and Haiti, but not in Kuwait, Saudi Arabia, Djibouti, Lebanon, North Macedonia, Ukraine, Belize,
and Jamaica. Lastly, the variant messages significantly increased vaccination intentions in Iraq and Haiti,
but not in Tunisia, Lebanon, Belize, and Jamaica.
6
When reporting results, we use the term 'significance' to imply statistical significance.
17
Figure 2. Effect of experts and efficacy framings on intentions by study.
18
Figure 3. Effect of safety, dynamic social norms, endemic and variants framings on intentions by study.
3.3. Study-Level Moderation
We conduct a moderator analysis to understand factors explaining the differences in the
message impacts across studies for the endemic, safety, experts, efficacy and prosocial, and variant
framings where significant between-study heterogeneity was observed. Table 3 shows the random-
effects meta-regressions where study-level moderators are regressed on the effect sizes of a given
framing across different studies, with a meta-regression estimated for each framing. See Appendix G for
bivariate correlations for each study-level moderator and effect sizes for framings with significant
heterogeneity.
Country-Level Vaccination Rate
Country-level vaccination rate was not a significant moderator for any of the framings.
Intervention Start Month
The impacts of expert messages are larger in interventions conducted later than earlier (β =
0.031, p = 0.02), but only when not controlling for covariates from the Wellcome Global Monitor (2018).
Specifically, on average, for each month increase of the intervention start date, there is an increase of
0.031 units in the impact of expert messages on vaccination intentions.
19
The impacts of endemic framing are marginally larger in interventions conducted later than
earlier (β = 0.046, p = 0.057), but only when controlling for Wellcome indicators. Specifically, on average,
for each month increase of the intervention start date, there is a marginally corresponding increase of
0.046 units in the impact of endemic messages on vaccination intentions.
Country-Level Trust in Healthcare Workers
Trust in healthcare workers (e.g., doctors and nurses) was not a significant moderator for any of
the framings.
Country-Level Perceived General Vaccine Safety
General perceived vaccine safety (i.e., not COVID-specific) was not a significant moderator for
any of the framings.
Country-Level Perceived General Vaccine Effectiveness
The impacts of variants framing (β = 1.21, p = 0.096) are marginally larger in countries with
higher general perceived vaccine effectiveness. Specifically, on average, for each unit increase in level of
agreement with the statement “Vaccines are effective”, there is a marginally corresponding increase of
1.21 units in the impact of variants framing on vaccination intentions.
Table 3. Study-level moderation meta-regression results.
Efficacy
Moderators Experts a
Safety and Pro- Endemica Variants
Social
(1) (2) (1) (2)
COVID-19 vacc. rate in sample 0.003 -0.000 -0.004
(0.002) (0.005) (0.004)
Start month of experiment 0.031** 0.025 0.005 0.021 0.046*
(0.013) (0.016) (0.011) (0.021) (0.024)
Trust in healthcare workers -0.368 -0.421
(0.405) (0.340)
General perceived vaccine safety -0.248
(0.207)
General perceived vaccine -0.774 1.213*
effectiveness (1.363) (0.729)
Observations 16 13 10 14 7 5 4
Wellcome Global Monitora No Yes Yes No No Yes Yes
I2 86.0 85.6 37.4 67.7 78.3 71.2 72.0
Note: * p < .10, ** p < .05. a Because of missing data with trust in doctors, perceived vaccine safety, and perceived vaccine
effectiveness, for which data from the Wellcome Global Monitor 2018 were not available for Belize, Djibouti, Jamaica, Papua
New Guinea, and West Bank and Gaza, we report the results of two study-level moderator meta-regressions for experts and
endemic framings: (1) without the Wellcome Global Monitor indicators and all countries and (2) with the Wellcome Global
Monitor indicators, dropping countries with missing indicators.
20
3.4. Individual-Level Moderation: Subgroup Analyses
Overall, there is minimal individual-level heterogeneity across message framings by
demographics. For results at the individual level, see Table 4 and Figure 4. In most framings, we find no
significant moderating effects of demographic characteristics, indicating consistent message impacts
across these observable demographics. However, significant interactions were found between age and
expert messages, age and safety messages, and gender and efficacy and pro-social messages. For
detailed reporting on individual-level heterogeneity by study, please see Appendix H.
Figure 4. Heterogeneous effects of framing on intentions to vaccinate by individual-level moderators
Note: Circles represent effect sizes of specified interaction effects for individual studies included in the
analysis.
21
Table 4. Two-stage individual participant data (IPD) meta-analysis by intervention for the interaction
effect with gender, age, education level, and health worker status.
Education
Female Age 40+ Health worker
N secondary or below
(base: Male) (base: Age 39-) (base: rest)
Intervention (base: tertiary)
Log Odds Log Odds Log Odds Log Odds
p p p p
Ratio Ratio Ratio Ratio
55,762 -0.02 0.67 -0.09 0.04 -0.01 0.73 0.06 0.32
Experts (-0.10, 0.07) (-0.18, -0.00) (-0.10, 0.07) (-0.06, 0.18)
37,798 -0.07 0.14 -0.04 0.47 -0.08 0.13 0.08 0.22
Experts +
(-0.17, 0.02) (-0.14, 0.06) (-0.18, 0.02) (-0.05, 0.22)
Celebrities
Experts + 40,522 -0.02 0.73 -0.03 0.50 -0.03 0.52 -0.01 0.86
Religious (-0.11, 0.08) (-0.13, 0.07) (-0.13, 0.07) (-0.14, 0.12)
leaders
23,213 0.10 0.13 -0.15 0.03 0.07 0.32 -0.08 0.52
Safety (-0.03, 0.23) (-0.29, -0.01) (-0.06, 0.20) (-0.33, 0.17)
32,683 0.15 0.01 -0.11 0.12 0.05 0.36 -0.00 0.98
Efficacy and
(0.04, 0.27) (-0.25, 0.03) (-0.06, 0.17) (-0.20, 0.19)
Pro-Social
Dynamic 23,413 -0.04 0.57 -0.10 0.17 0.11 0.11 0.02 0.87
Social Norms (-0.17, 0.09) (-0.24, 0.04) (-0.03, 0.24) (-0.23, 0.28)
16,488 -0.04 0.68 0.28 0.02 -0.03 0.70 -0.18 0.17
Endemic
(-0.21, 0.14) (0.04, 0.51) (-0.21, 0.14) (-0.45, 0.08)
16,332 0.08 0.37 0.08 0.53 0.13 0.14 -0.25 0.06
Variants (-0.10, 0.26) (-0.17, 0.32) (-0.04, 0.31) (-0.52, 0.01)
Note. The model includes the intervention, the four demographic characteristics, and the interactions shown in the table. For
study level interactions, see Appendix H.
Age
The effect of expert messages was larger for people aged 39 or younger (vs. 40 years or older)
(interaction effects: LOR: -0.09, 95% CI: [-0.18, -0.00], p = 0.04). Subgroup analysis revealed a pooled
effect of expert messages for people aged 39 or younger of LOR: 0.26, 95% CI [0.21, 0.31], p = < 0.001).
In contrast, the pooled effect of expert messages for people 40 years or older in our subgroup analysis
was LOR = 0.18, 95% CI [0.11, 0.25], p < 0.001). See Figure 5.
Figure 5. Experts Message Impacts by Age
22
The effect of safety messages was also larger for people 39 years of age or younger (interaction
effects: LOR: -0.15, 95% CI: [-0.29, -0.01], p = 0.03). Subgroup analysis revealed a pooled effect of safety
messages for people under 40 years of age, which was LOR: 0.09, 95% CI [0.01, 0.17], p = 0.022).
Conversely, the pooled effect of safety messages for those 40 years old or older was LOR = -0.05, 95% CI
[-0.16, 0.06], p = 0.338). See Figure 6.
Figure 6. Safety Message Impacts by Age.
Gender
The effect of efficacy and pro-social messages was larger for women than men (interaction
effects: LOR: 0.15, 95% CI: [0.04, 0.27], p = 0.01). Subgroup analysis revealed a pooled effect of efficacy
and pro-social messages for women, which was LOR: 0.29, 95% CI [0.22, 0.37], p < 0.001).
Comparatively, the pooled effect of efficacy and prosocial messages for men was LOR: 0.18, 95% CI:
[0.10, 0.26], p < 0.001. See Figure 7.
Figure 7. Efficacy and pro-social message impacts by gender
23
4. Discussion
4.1. Findings Summary
To identify the effect of behaviorally informed vaccine messaging on vaccine intentions, we
conducted 28 randomized survey experiments about COVID-19 vaccination across 23 mostly low- or
middle-income countries and territories, recruiting participants through social media. In this individual
participant data meta-analysis, among the unvaccinated, we found that behaviorally informed messages
significantly increased the odds of having the intention to get vaccinated by 1.28 times overall, with
significant impacts of specific framings ranging from 1.16 times (variant framing) to 1.45 times (experts
and religious leaders framing), compared to the control group (i.e., no message). The messages using
experts and religious leaders showed the largest impacts, followed by endemic, messages with experts
and celebrities, efficacy and pro-social, messages with experts, variants, and dynamic social norms.
However, the safety message did not significantly increase vaccination intentions across studies. While
some framings exhibited larger effects than others, it is important to interpret these impacts considering
differences in implementation—as some framings (e.g., experts and safety) were used across more than
a dozen studies while other framings were tested in fewer studies as the pandemic evolved (e.g.,
variants). In addition, it should be reiterated that our findings represent effects for populations who are
active on social media and self-selected into a survey about COVID-19, and outcomes reflected self-
reported intentions immediately following message exposure.
Messages using experts and celebrities, messages using experts and religious leaders, and
dynamic social norms messages had consistent impacts across the countries and studies, indicated by
non-significant heterogeneity in effect sizes. Conversely, the endemic, safety, experts, efficacy and
prosocial, and variant messages showed moderate-to-high between-study heterogeneity in effect sizes,
with impacts varying across studies and countries. Still, none of our tested study-level moderators,
including country-level vaccination rates, intervention start dates, and GDP per capita PPP, significantly
explained the heterogeneity. Marginally, safety messages have larger impacts in countries with higher
vaccination rate, and expert messages have larger impacts when the intervention was conducted later.
Lastly, we found most messages have consistent impacts across observable individual demographics,
given the minimal individual-level heterogeneity by demographic characteristics, except larger impacts
of expert and safety messages for younger people as well as efficacy and pro-social messages for
women.
4.2. Implications for Behaviorally-Informed Communication in Practice
When addressing hesitancy, messages focusing on the underlying concerns and drivers of
vaccine acceptance can increase intentions to vaccinate. The observed heterogeneity in the effects of
specific message framings suggests that the effectiveness of messages may vary based on contextual
factors. Some messages, such as those involving experts and celebrities or religious leaders, showed
consistent impacts across studies and countries, while others, such as safety messages, were not
effective consistently across countries.
That some message impacts varied by context highlights the importance of testing messages
with the target population before implementing communications widely. This also underlines the
importance of developing messages tailored to specific contexts and populations to maximize their
24
effectiveness, aligning with prior research emphasizing the importance of message customization. This
study also demonstrates the potential for public health authorities to use rapid online survey
experiments to promptly identify vaccine hesitancy drivers (e.g., Badiani-Magnusson et al., 2022; Hoy et
al., 2022, 2023; Lim et al., 2024; Margolies et al., 2023) and test messages in fast-evolving environments
like the pandemic among a growing population within LMICs—social media users. There are over 5
billion smartphone users and over 3 billion Facebook daily users globally (Statista, 2023), underscoring
the relevance of these findings despite our study samples not being nationally representative on
dimensions like education. Furthermore, as we observe advances in the ability to deliver tailored
communication to individuals through technologies like online social media, future research can also
examine if and how messages tailored to personalized concerns and needs can increase vaccination
intentions.
Our results highlight the importance of identifying and working with multiple trustworthy
messengers, such as healthcare professionals, celebrities, and religious leaders, to convey messages
effectively, given the consistently significant message impacts using experts and celebrities or religious
leaders across countries. This aligns with prior research emphasizing the role of messengers in
influencing attitudes and behaviors (Alatas et al., 2024; Huang & Sundar, 2022; Lu et al., 2021; Ruggeri et
al., 2024). Interestingly, our results show that using multiple trusted messengers can bolster message
effectiveness, while using experts (e.g., “national and international experts”) alone did not result in the
consistent impacts on vaccination intentions across countries. Public health campaigns should consider
partnering with these trusted and popular figures to amplify their messaging and increase its impact
while future research can examine additional effects of consensus among multiple information sources.
Given the consistently significant impacts of dynamic social norms messages across countries,
public health authorities in low- and middle-income countries could benefit from leveraging these social
norms messages. Consistent with this, previous studies have shown that social norms messages focusing
on descriptive norms (e.g., widespread vaccine acceptance) can effectively increase vaccination
intentions (Ryoo & Kim, 2021; Sinclair & Agerström, 2021). Future research could explore strategies for
creating injunctive social norms (Martinez et al., 2023) to complement dynamic norms in contexts where
norms are not yet prominent to identify the best methods of promoting vaccination without triggering
unintended consequences such as boomerang effects.
Our study also adds to the literature on message fatigue and health risk communication in
dynamically evolving situations in that using diverse messages that shift contexts, such as endemic and
variant framings, may help mitigate the fatigue and keep messaging relevant. These messages
acknowledging the ongoing risks of COVID and highlighting that vaccination is protective even in
contexts where disease is endemic were effective in increasing intentions, suggesting that nuanced
information reflecting the experience of the public can be effective. This finding underscores the need
for ongoing message adaptation and evolution to maintain engagement and effectiveness over time,
especially in prolonged public health crises like the COVID-19 pandemic and slow-burning crises.
4.3. Study-Level and Individual-Level Moderators
While the message framing focusing on endemic, safety, experts, efficacy, and prosocial, and
variants showed moderate-to-high heterogeneity in message impacts across studies and countries, we
found that country-level vaccination rates, intervention start dates, and GDP per capita PPP do not
25
significantly explain the varying message impacts across studies and countries. We found only one case
where a moderator was significant: the intervention date for the expert messages. The intervention
date explained the differences in expert messages’ effects, indicating that expert messages were more
effective later than earlier. Marginally, intervention start date explained the differences in endemic
framing (p = 0.068), indicating that endemic messages were more effective earlier than later, and
perceived vaccine effectiveness was associated with the impact of the variants framing (p = 0.092),
indicating that variants framings were more effective when people perceived the vaccine’s
effectiveness. This may be because the remaining unvaccinated population in these countries might be
more concerned about vaccine safety and experts’ guidance as more people in the country become
vaccinated, or because the messages resonate with their personal experiences and become more
credible seeing others around them getting safely vaccinated and a great number of experts have
endorsed the vaccines. This also may be because in countries with high perceived vaccine effectiveness,
mentioning emerging variants reminded unvaccinated individuals of vaccines' roles and increased their
vaccination intentions.
The relative lack of significant individual-level heterogeneity in our study suggests that these
messages can have relatively consistent effects across diverse demographic populations active on social
media. Only larger effects of expert and safety messages for people younger than 40 years old (vs. 40
years or older) and larger effects for efficacy and pro-social messages for women were found. The
literature points to several plausible explanations. For example, messaging about personal efficacy of
vaccines may increase motivation among women who perceive men as particularly at risk of negative
complications from COVID-19 (Ackah et al., 2022) while enhanced effectiveness of prosocial messages
may reflect higher levels of prosocial orientations among women (Rand et al., 2016). Safety messages
might appeal to younger adults maybe because, unlike older adults, they were at relatively lower risk of
complications from COVID-19 and might have decided to wait and see due to lower perceived
susceptibility and impacts like in some countries (e.g., Hamel et al., 2021). Future studies can further
explore why these specific framings had differential impacts on younger (vs. older) adults and women
(vs. men).
The finding that many message framings were effective across different demographics is
particularly relevant for global health efforts, indicating behaviorally informed communications can be
broadly effective on users of social media. However, our findings might also reflect the relative
importance of individual characteristics beyond demographics when it comes to the impact of
behaviorally informed communications (Christy et al., 2022). Future studies should also examine
psychological variables, such as specific vaccine safety concerns, in their individual-level moderator
analyses.
Our findings have several implications for policy makers and public health authorities aimed at
increasing COVID-19 vaccination uptake. The significant increase in vaccination intentions with
behaviorally informed messages suggests that messaging designed with behavioral science principles
can be a powerful tool in addressing vaccine hesitancy. By leveraging specific message framings, public
health campaigns can tailor their messaging to different segments of the population, addressing varying
concerns and motivations.
26
4.4. Limitations
This study has limitations that are important to consider when drawing conclusions from our
evidence. Due to its rapid introduction and promotion on a large scale, our findings about COVID-19
vaccine hesitancy might not be applicable to other long-established vaccines (e.g., MMR, DTaP, flu) or
new vaccines (HPV). Recruitment and survey administration on Facebook, while innovative for mirroring
national demographics using Facebook targeting and quota sampling, cannot provide a representative
sample, leading to under-representation of older age groups, the poorest, and least educated, who
might be disproportionally impacted by the COVID-19 and future pandemics. Self-selection bias may
introduce systematic differences between respondents and non-respondents based on unmeasured
factors and affect the findings. As such, the generalizability of the findings from our meta-analysis to
offline populations remains to be tested and the impacts of our studies likely represent upper-bound
estimates of effects. Despite these limitations, the sampling strategy of stratifying by demographic
characteristics allowed for greater representation and timely data collection among key subgroups (e.g.,
women) who can be harder to reach in online settings.
The study measured self-reported vaccination intentions, not actual behavior. While behavioral
intentions are commonly used to predict behaviors in experiments, and one meta-analysis found that a
medium-to-large change in behavioral intention (d = 0.66) led to a small-to-medium change in behaviors
(d = 0.36) (Webb & Sheeran, 2006), the extent to which changes in intentions translate into increased
vaccinations across these international contexts remains unknown and warrants further investigation,
when feasible (e.g., Alhajji et al., 2023). Indeed, some research has identified differential impacts of
messaging interventions on vaccine intentions compared to behavior (Saccardo et al., 2024).
Lastly, we were limited in exploring additional study-level moderators (e.g., World Value
Surveys [2024]; Hofstede’s cultural dimensions [Culture Factor Group, 2024]) given scarcity of data
available for LMICs. While we were able to draw from the Wellcome Global Monitor Indicators to
explore the moderating role of country-level beliefs and trust in healthcare workers and beliefs about
vaccines, these concepts may be better explored at the individual level, as evidenced by modest
empirical evidence for aggregation to the country level described in Appendix D. Our analysis focused
on only demographic characteristics as individual-level moderators, as other potentially relevant factors
were omitted from surveys to keep them brief for participants. Future research could explore additional
moderating factors that might have more explanatory power.
4.5. Conclusions
This study is part of the World Bank's COVID-19 response, aimed at diagnosing vaccine hesitancy
and related factors, and informing and guiding real-time risk communication efforts across multiple
governments simultaneously. While the WHO declared the end of the COVID-19 pandemic in May 2023
(WHO, 2023), ensuring high immunization rates remains crucial for public health. Our findings
demonstrate that behaviorally-informed messages have the potential to nudge vaccine-hesitant
individuals towards vaccination. Impacts of such messages can represent significant public health gain,
as up to 6 million people were vaccine hesitant in these 23 countries and territories by the time of the
study. However, one lesson from our study is that a "one-size-fits-all" approach will not work. Our
research emphasizes the importance of testing and tailoring messages to specific audiences and
contexts, as the effectiveness of some messages varied across countries. This study highlights the value
27
of conducting rapid experiments in LMICs to inform their communication and policies to address
emerging health challenges.
Public health authorities must adapt and test their communication strategies to evolving
situations, following best practices, emphasizing transparency, and helping the public accept uncertainty
(Liu et al., 2021). For example, we initially tested the safety messages stating "no reported
hospitalizations in {country} due to vaccinations" at the time of study to the best of our knowledge.
However, rare severe reactions after COVID-19 vaccination have since been reported, despite the
benefits of COVID-19 vaccination continuing to outweigh any potential risks (US CDC, 2023). Real-time
research can adapt the messages tested.
To prevent future health emergencies and pandemics, public health authorities must expand
vaccination coverage in humans and animals (Rupasinghe et al., 2024). Our findings can guide future
global immunization efforts, including new vaccines like malaria and HPV. Looking ahead, leveraging
social media for data collection can facilitate timely research and adaptative messaging strategies. The
next step is to apply, transfer, replicate, and test findings in various online and offline real-world settings
with various target populations (Saccardo et al., 2024). By continuing to refine our understanding and
applying effective risk communication, we can build a robust evidence base to prepare for future health
emergencies and pandemics (World Bank, 2022), just as our research contributed to the fight against
COVID-19, particularly in low- and middle-income countries and to fortifying global health defenses.
28
References
Ackah, B. B., Woo, M., Stallwood, L., Fazal, Z. A., Okpani, A., Ukah, U. V., & Adu, P. A. (2022). COVID-19
vaccine hesitancy in Africa: a scoping review. Global Health Research and Policy, 7(1), 21, 1-20.
https://doi.org/10.1186/s41256-022-00255-1
Alatas, V., Chandrasekhar, A. G., Mobius, M., Olken, B. A., & Paladines, C. (2024). Do celebrity
endorsements matter? A Twitter experiment promoting vaccination in Indonesia. The Economic
Journal, 134(659), 913-933. https://doi.org/10.1093/ej/uead102
Allison, P. D. (2001). Missing data. Sage University Papers Series on Quantitative Applications in the
Social Sciences. Sage. https://doi.org/10.4135/9781412985079
Alhajji, M., Alzeer, A. H., Al-Jafar, R., Alshehri, R., Alyahya, S., Alsuhaibani, S., ... & Alsaawi, F. (2023). A
national nudge study of differently framed messages to increase COVID-19 vaccine uptake in
Saudi Arabia: A randomized controlled trial. Saudi Pharmaceutical Journal, 31(9), 101748, 1-7.
https://doi.org/10.1016/j.jsps.2023.101748
Ali, H. A., Hartner, A. M., Echeverria-Londono, S., Roth, J., Li, X., Abbas, K., ... & Gaythorpe, K.
A. (2022). Vaccine equity in low and middle income countries: a systematic review and
meta-analysis. International journal for equity in health, 21(1), 82, 1-30.
https://doi.org/10.1186/s12939-022-01678-5
Alsan, M., & Eichmeyer, S. (2024). Experimental evidence on the effectiveness of nonexperts for
improving vaccine demand. American Economic Journal: Economic Policy, 16(1), 394-414.
https://doi.org/10.1257/pol.20210393
Argote Tironi, P., Barham, E., Zuckerman Daly, S., Gerez, J. E., Marshall, J., & Pocasangre, O. (2021).
Messages that increase COVID-19 vaccine acceptance: Evidence from online experiments in six
Latin American countries. PloS one, 16(10), e0259059, 1-20.
https://doi.org/10.1371/journal.pone.0259059
Ashworth, M., Thunström, L., Cherry, T. L., Newbold, S. C., & Finnoff, D. C. (2021). Emphasize personal
health benefits to boost COVID-19 vaccination rates. Proceedings of the National Academy of
Sciences, 118(32), e2108225118. https://doi.org/10.1073/pnas.2108225118
Badiani-Magnusson, R. C., Lim, J. R., & Moscoe, E. E. (2022). Behaviorally informed interventions for
COVID-19 vaccine take-up: Insights from Romania family doctors. World Bank Group.
http://documents.worldbank.org/curated/en/099950210032237528/P1741810a530560c09c46
0404dfcd0bd2f
Bartoš, V., Bauer, M., Cahlíková, J., & Chytilová, J. (2022). Communicating doctors’ consensus
persistently increases COVID-19 vaccinations. Nature, 606(7914), 542-549.
https://doi.org/10.1038/s41586-022-04805-y
Batteux, E., Mills, F., Jones, L. F., Symons, C., & Weston, D. (2022). The effectiveness of interventions for
increasing COVID-19 vaccine uptake: a systematic review. Vaccines, 10(3), 386, 1-25.
https://doi.org/10.3390/vaccines10030386
29
Bavel, J. J. V., Baicker, K., Boggio, P. S., Capraro, V., Cichocka, A., Cikara, M., ... & Willer, R. (2020). Using
social and behavioural science to support COVID-19 pandemic response. Nature Human
Behaviour, 4(5), 460-471. https://doi.org/10.1038/s41562-020-0884-z
Bloom, D. E., & Canning, D. (2000). The health and wealth of nations. Science, 287(5456), 1207-1209.
https://doi.org/10.1126/science.287.5456.1207
Bodner, T. E. (2008). What improves with increased missing data imputations?. Structural equation
modeling: a multidisciplinary journal, 15(4), 651-675.
https://doi.org/10.1080/10705510802339072
Bokemper, S. E., Gerber, A. S., Omer, S. B., & Huber, G. A. (2021). Persuading US White evangelicals to
vaccinate for COVID-19: Testing message effectiveness in fall 2020 and spring 2021. Proceedings
of the National Academy of Sciences, 118(49), e2114762118, 1-3.
https://doi.org/10.1073/pnas.2114762118
Burke, D. L., Ensor, J., & Riley, R. D. (2017). Meta-analysis using individual participant data: One-stage
and two-stage approaches, and why they may differ. Statistics in Medicine, 36(5), 855–875.
https://doi.org/10.1002/sim.7141
Card, N. A. (2012). Applied Meta-Analysis for Social Science Research. Guilford.
Christy, K. R., Minich, M., Tao, R., Riddle, K., & Kim, S. (2022). To tailor or not to tailor: An investigation
of narrative tailoring for health communication. Journal of Health Communication, 27(3), 152-
163. https://doi.org/10.1080/10810730.2022.2068702
Chu, J., Pink, S. L., & Willer, R. (2021). Religious identity cues increase vaccination intentions and trust in
medical experts among American Christians. Proceedings of the National Academy of Sciences,
118(49), e2106481118, 1-3. https://doi.org/10.1073/pnas.2106481118
Davis, C. J., Golding, M., & McKay, R. (2022). Efficacy information influences intention to take COVID‐19
vaccine. British Journal of Health Psychology, 27(2), 300-319.
https://doi.org/10.1111/bjhp.12546
Dayton Eberwein, J., Edochie, I. N., Newhouse, D., Cojocaru, A., Bopahbe, G. D., Kakietek, J. J., ... &
Montes, J. (2023). How prevalent is COVID-19 vaccine hesitancy in low-income and middle-
income countries and what are the key drivers of hesitancy? Results from 53 countries. BMJ
Open, 13(11), e069152, 1-10. https://doi.org/10.1136/bmjopen-2022-069152
Debray, T. P. A., Moons, K. G. M., van Valkenhoef, G., Efthimiou, O., Hummel, N., Groenwold, R. H. H.,
Reitsma, J. B., & Group, on behalf of the G. methods review. (2015). Get real in individual
participant data (IPD) meta-analysis: A review of the methodology. Research Synthesis Methods,
6(4), 293–309. https://doi.org/10.1002/jrsm.1160
de Figueiredo, A., Simas, C., & Larson, H. J. (2023). COVID-19 vaccine acceptance and its socio-
demographic and emotional determinants: A multi-country cross-sectional study. Vaccine, 41(2),
354-364. https://doi.org/10.1038/s43856-021-00027-x
Dombrádi, V., Joó, T., Palla, G., Pollner, P., & Belicza, É. (2021). Comparison of hesitancy between COVID-
19 and seasonal influenza vaccinations within the general Hungarian population: a cross-
30
sectional study. BMC Public Health, 21(1), 2317, 1-13. https://doi.org/10.1186/s12889-021-
12386-0
Enders, C. K. (2022). Applied Missing Data Analysis. Second Edition. The Guilford Press.
Freeman, D., Loe, B. S., Yu, L. M., Freeman, J., Chadwick, A., Vaccari, C., ... & Lambe, S. (2021). Effects of
different types of written vaccination information on COVID-19 vaccine hesitancy in the UK
(OCEANS-III): a single-blind, parallel-group, randomised controlled trial. The Lancet Public
Health, 6(6), e416-e427. https://doi.org/10.1016/S2468-2667(21)00096-7
Funder, D. C., & Ozer, D. J. (2019). Evaluating effect size in psychological research: Sense and nonsense.
Advances in Methods and Practices in Psychological Science, 2(2), 156-168.
https://doi.org/10.1177/2515245919847202
Gelfand, M. J., Raver, J. L., Nishii, L., Leslie, L. M., Lun, J., Lim, B. C., ... & Yamaguchi, S. (2011).
Differences between tight and loose cultures: A 33-nation study. Science, 332(6033), 1100-1104.
Guan, M., Li, Y., Scoles, J. D., & Zhu, Y. (2023). COVID-19 message fatigue: How does it predict preventive
behavioral intentions and what types of information are people tired of hearing about?. Health
Communication, 38(8), 1631-1640. https://doi.org/10.1080/10410236.2021.2023385
Hamel, L., Lopes, L., Sparks, G., Stokes, M., & Brodie, M. (2021). KFF COVID-19 Vaccine Monitor: April
2021. KFF. https://www.kff.org/coronavirus-covid-19/poll-finding/kff-covid-19-vaccine-monitor-
april-2021/
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world?, Behavioral and
Brain Sciences, 33, 61-135. https://doi.org/10.1017/S0140525X0999152X
Holder, J. (2023, March 13). Tracking coronavirus vaccinations around the world. The New York Times.
https://www.nytimes.com/interactive/2021/world/covid-vaccinations-tracker.html
Hoy, C., Kanagavel, R., & Cameron, C. (2022). Intra-household dynamics and attitudes toward vaccines:
Experimental and survey evidence from Zambia. World Bank Policy Research Working Paper,
10136. https://hdl.handle.net/10986/37828
Hoy, C., Wood, T., & Moscoe, E. (2022). Addressing vaccine hesitancy in developing countries: Survey
and experimental evidence. PLoS One, 17(11), e0277493, 1-12.
https://doi.org/10.1371/journal.pone.0277493
Huang, Y., & Liu, W. (2022). Promoting COVID-19 vaccination: The interplay of message framing,
psychological uncertainty, and public agency as a message source. Science Communication,
44(1), 3-29. https://doi.org/10.1177/1075547021104819
Huang, Y., & Sundar, S. S. (2022). Do we trust the crowd? Effects of crowdsourcing on perceived
credibility of online health information. Health Communication, 37(1), 93-102.
https://doi.org/10.1080/10410236.2020.1824662
James, E. K., Bokemper, S. E., Gerber, A. S., Omer, S. B., & Huber, G. A. (2021). Persuasive messaging to
increase COVID-19 vaccine uptake intentions. Vaccine, 39(49), 7158-7165.
https://doi.org/10.1016/j.vaccine.2021.10.039
31
James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-group interrater reliability with and
without response bias. Journal of Applied Psychology, 69(1), 85-98.
Jenum, A. K., Brekke, I., Mdala, I., Muilwijk, M., Ramachandran, A., Kjøllesdal, M., ... & van Valkengoed, I.
G. (2019). Effects of dietary and physical activity interventions on the risk of type 2 diabetes in
South Asians: meta-analysis of individual participant data from randomised controlled trials.
Diabetologia, 62, 1337-1348. https://doi.org/10.1007/s00125-019-4905-2
Johns Hopkins University. (2023). COVID-19 dashboard. https://coronavirus.jhu.edu/map.html
Larson, H. J., Jarrett, C., Eckersberger, E., Smith, D. M., & Paterson, P. (2014). Understanding vaccine
hesitancy around vaccines and vaccination from a global perspective: a systematic review of
published literature, 2007–2012. Vaccine, 32(19), 2150-2159.
https://doi.org/10.1016/j.vaccine.2014.01.081
Lasswell, H. D. (1948). The structure and function of communication in society. In L. Bryson (Ed.), The
communication of ideas (pp. 37-51). Harper and Row.
LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and
interrater agreement. Organizational Research Methods, 11(4), 815-852.
https://doi.org/10.1177/1094428106296642
Lim, J. R., Kim, H., Altieri, E., Afif, Z., Nguyen, S. N., Kim, P., Ashcroft-Jones, S., Hoeflin, C., Hirneis, V.,
Kabitsis, P., Chadborn, T., de Almeida, J. R. (in press). The use of experiments to identify effective
behavioral interventions in COVID-19 response: A scoping review. World Bank.
Lim, J. R., Moscoe, E., Alqunaibet, A., Pinzon, D. A., Alruwaily, A., Chatila, M., ... & Algwizani, A. (2024).
Using social media surveys and interventions to address vaccine hesitancy in Saudi Arabia.
Eastern Mediterranean Health Journal, 30(3), 182-195. https://doi.org/10.26719/emhj.24.013
Liu, B. F., Lim, J. R., Shi, D., Edwards, A. L., Islam, K., Sheppard, R., & Seeger, M. (2021). Evolving best
practices in crisis communication: Examining US higher education’s responses to the COVID-19
pandemic. Journal of International Crisis and Risk Communication Research, 4(3), 313-346.
https://doi.org/10.30658/jicrcr.4.3.1
Louise, J., Poprzeczny, A. J., Deussen, A. R., Vinter, C., Tanvig, M., Jensen, D. M., ... & Dodd, J. M. (2021).
The effects of dietary and lifestyle interventions among pregnant women with overweight or
obesity on early childhood outcomes: an individual participant data meta-analysis from
randomised trials. BMC Medicine, 19, 1-15. https://doi.org/10.1186/s12916-021-01995-6
Lu, L., Liu, J., Yuan, Y. C., Burns, K. S., Lu, E., & Li, D. (2021). Source trust and COVID-19 information
sharing: the mediating roles of emotions and beliefs about sharing. Health Education &
Behavior, 48(2), 132-139. https://doi.org/10.1177/1090198120984760
Margolies, A., Moscoe, E., Pinzón, D., Mussini, M., Kim, P., & Di Giorgio, L. Behavioral insights for COVID-
19 vaccine uptake in the Caribbean: Rapid surveys from Belize, Haiti, and Jamaica. World Bank.
https://openknowledge.worldbank.org/handle/10986/40489
Marozzi, M. (2015). Measuring trust in European public institutions. Social Indicators Research, 123, 879-
895. https://doi.org/10.1007/s11205-014-0765-9
32
Martínez, D., Díaz, L. M., & Maldonado, S. (2023). Nudging the trendsetters: Increasing second-dose HPV
vaccination in Bogota, Colombia. IDB Working Paper, IDB-WP-1547. Inter-American
Development Bank. https://doi.org/10.18235/0005331
McGuire, W. J. (1989). Theoretical foundations of campaigns. In R. E. Rice & C. K. Atkin (Eds.), Public
communication campaigns (Vol. 2, pp. 43-65). Sage.
Motta, M., Sylvester, S., Callaghan, T., & Lunz-Trujillo, K. (2021). Encouraging COVID-19 vaccine uptake
through effective health communication. Frontiers in Political Science, 3, 630133.
https://doi.org/10.3389/fpos.2021.630133
Msemburi, W., Karlinsky, A., Knutson, V., Aleshin-Guendel, S., Chatterji, S., & Wakefield, J. (2023). The
WHO estimates of excess mortality associated with the COVID-19 pandemic. Nature, 613(7942),
130-137. https://doi.org/10.1038/s41586-022-05522-2
Nan, X., Iles, I. A., Yang, B., & Ma, Z. (2022). Public health messaging during the COVID-19 pandemic and
beyond: Lessons from communication science. Health Communication, 37(1), 1-19.
https://doi.org/10.1080/10410236.2021.1994910
Noar, S. M. (2006). A 10-year retrospective of research in health mass media campaigns: where do we
go from here?. Journal of health communication, 11(1), 21-42.
https://doi.org/10.1080/10810730500461059
Okuhara, T., Okada, H., & Kiuchi, T. (2023). Addressing message fatigue for encouraging COVID-19
vaccination. Journal of Communication in Healthcare, 16(3), 298-303.
https://doi.org/10.1080/17538068.2023.2207246
Orozco-Olvera, V.H., Malhotra, N. (2023). Media and Health Outcomes. In: Zimmermann, K.F. (eds)
Handbook of Labor, Human Resources and Population Economics. Springer, Cham.
https://doi.org/10.1007/978-3-319-57365-6_385-1
Our World in Data. (2024). What share of the population has received at least one dose of vaccine?.
https://ourworldindata.org/covid-vaccinations#what-share-of-the-population-has-received-at-
least-one-dose-of-vaccine
Palm, R., Bolsen, T., & Kingsland, J. T. (2021). The effect of frames on COVID-19 vaccine resistance.
Frontiers in Political Science, 3, 661257, 1-11. https://doi.org/10.3389/fpos.2021.661257
Pink, S. L., Chu, J., Druckman, J. N., Rand, D. G., & Willer, R. (2021). Elite party cues increase vaccination
intentions among Republicans. Proceedings of the National Academy of Sciences, 118(32),
e2106559118. https://doi.org/10.1073/pnas.2106559118
Rakotoniana, J. S., Rakotomanga, J. D. D. M., & Barennes, H. (2014). Can churches play a role in
combating the HIV/AIDS epidemic? A study of the attitudes of Christian religious leaders in
Madagascar. PloS one, 9(5), e97131. https://doi.org/10.1371/journal.pone.0097131
Rand, D. G., Brescoll, V. L., Everett, J. A., Capraro, V., & Barcelo, H. (2016). Social heuristics and social
roles: Intuition favors altruism for women but not for men. Journal of Experimental Psychology:
General, 145(4), 389–396. https://doi.org/10.1037/xge0000154
33
Rao, N., Donati, D., & Orozco-Olvera, V. (2020). Conducting Surveys and Interventions Entirely Online: A
Virtual Lab Practitioner’s Manual. World Bank.
https://documents1.worldbank.org/curated/en/819191635501106862/pdf/Conducting-
Surveys-and-Interventions-Entirely-Online-A-Virtual-Lab-Practitioner-s-Manual.pdf
Reeskens, T., & Hooghe, M. (2008). Cross-cultural measurement equivalence of generalized trust.
Evidence from the European Social Survey (2002 and 2004). Social Indicators Research, 85, 515-
532. https://doi.org/10.1007/s11205-007-9100-z
RevealBot. (2024). Facebook Ads Automation Software. https://revealbot.com/facebook-ads-
automation
Riley, R. D., Stewart, L. A., & Tierney, J. F. (2021). Individual Participant Data Meta‐Analysis: A Handbook
for Healthcare Research. John Wiley & Sons. https://doi.org/10.1002/9781119333784
Robbins, B. G. (2012). A blessing and a curse? Political institutions in the growth and decay of
generalized trust: A cross-national panel analysis, 1980–2009. PloS ONE, 7(4), e35120.
https://doi.org/10.1371/journal.pone.0035120
Ruggeri, K., Stock, F., Haslam, S. A., Capraro, V., Boggio, P., Ellemers, N., ... & Willer, R. (2024). A
synthesis of evidence for policy from behavioural science during COVID-19. Nature, 625(7993),
134-147. https://doi.org/10.1038/s41586-023-06840-9
Rupasinghe, N., Machalaba, C., Muthee, T., & Mazimba, A. (2024). Stopping the Grand Pandemic: A
Framework for Action. Addressing Antimicrobial Resistance through World Bank Operations.
World Bank. http://hdl.handle.net/10986/41533
Ryoo, Y., & Kim, W. (2023). Using descriptive and injunctive norms to encourage COVID-19 social
distancing and vaccinations. Health Communication, 38(4), 732-741.
https://doi.org/10.1080/10410236.2021.1973702
Saccardo, S., Dai, H., Han, M. A., Vangala, S., Hoo, J., & Fujimoto, J. (2024). Field testing the
transferability of behavioural science knowledge on promoting vaccinations. Nature Human
Behaviour, 1-13. https://doi.org/10.1038/s41562-023-01813-4
Saccardo, S., Dai, H., Han, M. A., Vangala, S., Hoo, J., & Fujimoto, J. (2024). Field testing the
transferability of behavioural science knowledge on promoting vaccinations. Nature Human
Behaviour, 1-13. https://doi.org/10.1038/s41562-023-01813-4
Santos, H. C., Goren, A., Chabris, C. F., & Meyer, M. N. (2021). Effect of targeted behavioral science
messages on COVID-19 vaccination registration among employees of a large health system: a
randomized trial. JAMA Network Open, 4(7), e2118702-e2118702.
http://doi.org/10.1001/jamanetworkopen.2021.18702
Sasaki, S., Saito, T., & Ohtake, F. (2022). Nudges for COVID-19 voluntary vaccination: How to explain peer
information?. Social Science & Medicine, 292, 114561, 1-13.
https://doi.org/10.1016/j.socscimed.2021.114561
Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological
Methods, 7(2), 147-177.
34
Sinclair, S., & Agerström, J. (2023). Do social norms influence young people’s willingness to take the
COVID-19 vaccine?. Health Communication, 38(1), 152-159.
https://doi.org/10.1080/10410236.2021.1937832
Smelt, A. F., Gussekloo, J., Bermingham, L. W., Allen, E., Dangour, A. D., Eussen, S. J., ... & Den Elzen, W.
P. (2018). The effect of vitamin B12 and folic acid supplementation on routine haematological
parameters in older people: an individual participant data meta-analysis. European Journal of
Clinical Nutrition, 72(6), 785-795. https://doi.org/10.1038/s41430-018-0118-x
So, J., Kim, S., & Cohen, H. (2017). Message fatigue: Conceptual definition, operationalization, and
correlates. Communication Monographs, 84(1), 5-29.
https://doi.org/10.1080/03637751.2016.1250429
Solís Arce, J. S., Warren, S. S., Meriggi, N. F., Scacco, A., McMurry, N., Voors, M., ... & Omer, S. B. (2021).
COVID-19 vaccine acceptance and hesitancy in low-and middle-income countries. Nature
Medicine, 27(8), 1385-1394. https://doi.org/10.1038/s41591-021-01454-y
Statista (2023) Facebook MAU worldwide 2023. In: Statista.
https://www.statista.com/statistics/264810/number-of-monthly-active-facebook-users-
worldwide/
Trueblood, J. S., Sussman, A. B., & O’Leary, D. (2022). The role of risk preferences in responses to
messaging about COVID-19 vaccine take-up. Social Psychological and Personality Science, 13(1),
311-319. https://doi.org/10.1177/1948550621999622
US Centers for Disease Control and Prevention [CDC]. (2023). Selected Adverse Events Reported after
COVID-19 Vaccination. https://www.cdc.gov/coronavirus/2019-ncov/vaccines/safety/adverse-
events.html
Wakefield, M. A., Loken, B., & Hornik, R. C. (2010). Use of mass media campaigns to change health
behaviour. The Lancet, 376(9748), 1261-1271. https://doi.org/10.1016/S0140-6736(10)60809-4
Weinberg, D., Stevens, G. W., Currie, C., Delaruelle, K., Dierckens, M., Lenzi, M., ... & Finkenauer, C.
(2021). Country-level meritocratic beliefs moderate the social gradient in adolescent mental
health: a multilevel study in 30 European countries. Journal of Adolescent Health, 68(3), 548-
557. https://doi.org/10.1016/j.jadohealth.2020.06.031
Wellcome Trust, & The Gallup Organization Ltd [Gallup]. (2019). Wellcome Global Monitor, 2018. [data
collection]. 2nd Edition. UK Data Service. SN: 8466. https://doi.org/10.5255/UKDA-SN-8466-2
White, I. R., Royston, P., & Wood, A. M. (2011). Multiple imputation using chained equations:
issues and guidance for practice. Statistics in medicine, 30(4), 377-399.
https://doi.org/10.1002/sim.4067
World Bank. (2022). Putting Pandemics Behind Us: Investing in One Health to Reduce Risks of Emerging
Infectious Diseases. https://hdl.handle.net/10986/38200
World Bank. (2024). World Development Indicators. GDP per capita, PPP (constant 2017 international
$) [Data file]. https://databank.worldbank.org/metadataglossary/world-development-
indicators/series/NY.GDP.PCAP.PP.KD
35
World Health Organization [WHO]. (2014). Report of the SAGE Working Group on vaccine hesitancy.
https://cdn.who.int/media/docs/default-source/immunization/sage/2014/october/sage-
working-group-revised-report-vaccine-hesitancy.pdf?sfvrsn=240a7c1c_4
WHO. (2021). Strategy to Achieve Global Covid-19 Vaccination by mid-2022.
https://cdn.who.int/media/docs/default-source/immunization/covid-19/strategy-to-achieve-
global-covid-19-vaccination-by-mid-
2022.pdf#:~:text=URL%3A%20https%3A%2F%2Fcdn.who.int%2Fmedia%2Fdocs%2Fdefault
WHO. (2023). WHO Director-General's opening remarks at the media briefing – 5 May 2023.
https://www.who.int/news-room/speeches/item/who-director-general-s-opening-remarks-at-
the-media-briefing---5-may-2023
Xia, S., & Nan, X. (2023). Motivating COVID-19 Vaccination through Persuasive Communication: A
Systematic Review of Randomized Controlled Trials. Health Communication, 1-24.
https://doi.org/10.1080/10410236.2023.2218145
36
Appendix A. Study-Level Moderators of Survey Experiment Sample Included in the Meta-Analysis
Table SA1. Descriptive statistics of study-level moderators.
Country's GDP per
COVID-19 COVID-19 capita, PPP Perceived
Start date Trust Perceived
vaccination vaccination (constant vaccine
Region Study of doctors and vaccine
rate in rate at time 2017 effectiveness
experiment nurses (1-4) safety (1-5)
sample of the international (1-5)
experiment $)
Lebanon R1 0.0% N/A Jan-21 17,172 3.2 4.5 4.6
Iraq R1 0.0% 0.0% Feb-21 10,299 2.7 4.6 4.6
West Bank and Gaza 0.0% N/A Feb-21 6,245
Tunisia R1 0.0% 0.0% Mar-21 11,101 3.2 4.4 4.4
Libya 0.0% N/A Mar-21 22,535 3.0 4.3 4.4
Middle Kuwait 44.8% 19.3% May-21 47,315 3.1 4.6 4.6
East & Jordan R1 44.8% 6.3% May-21 9,551 3.1 4.6 4.7
North Saudi Arabia 75.1% 40.2% May-21 44,955 3.1 4.4 4.4
Africa Djibouti 30.9% 1.1% Jun-21 4,768
Tunisia R2 38.0% 13.9% Jul-21 11,101 3.2 4.4 4.4
Jordan R2 85.9% 37.1% Nov-21 9,551 3.1 4.6 4.7
Tunisia R3 82.8% 53.5% Dec-21 11,101 3.2 4.4 4.4
Iraq R2 56.5% 23.8% Mar-22 10,299 2.7 4.6 4.6
Lebanon R2 65.9% 45.1% Apr-22 17,172 3.2 4.5 4.6
Congo, Rep. 62.4% 8.5% Nov-21 1,060 2.5 4.1 4.7
Sub-
Zambia 71.9% 14.5% Mar-22 3,372 3.3 4.4 4.4
Saharan
Chad 0.0% 0.2% Aug-21 1,562 2.7 4.3 4.4
Africa and
the Sahel Cameroon 0.0% 1.5% Sep-21 3,757 2.8 4.1 4.4
Gambia, The 60.9% 18.6% May-22 2,083 3.4 4.7 4.7
East Asia
Papua New Guinea 16.5% 0.5% Jun-21 3,948
& Pacific
Kosovo 28.2% 7.7% Jul-21 11,318 3.0 4.2 4.4
Europe &
Serbia 63.8% 39.2% Jul-21 18,307 3.1 4.1 4.3
Central
Asia North Macedonia 55.8% 23.6% Jul-21 16,773 3.0 3.9 4.1
Ukraine 62.8% 34.3% Dec-21 12,805 2.9 3.3 3.8
Latin Honduras 0.0% 0.4% Mar-21 5,614 3.0 4.6 4.6
America Haiti 26.2% 1.6% Apr-22 3,102 3.1 4.1 4.5
& Belize 89.3% 59.3% Apr-22 9,029
Caribbean Jamaica 49.5% 28.2% Apr-22 10,244
37
Study level moderators include the COVID-19 vaccination rate for the survey sample (Our World in Data, 2024), GDP_per_capita PPP (World Bank, 2024), and trust in healthcare
workers, perceived vaccine safety, and perceived vaccine effectiveness from Wellcome Global Monitor (Wellcome Trust & Gallup, 2018).
38
Appendix B. Summary of Sample Proportions Randomized to Study Arms
Dynamic Efficacy Experts +
Basic Experts +
Region Study Variants Endemic Social and Pro- Safety Religious Experts
framing Celebrities
Norms Social leaders
Lebanon R1 51.3 15.8 17.5 15.4
Iraq R1 49.2 17.0 17.1 16.7
West Bank and Gaza 49.6 17.2 16.7 16.5
Tunisia R1 50.6 16.9 16.5 16.0
Libya 50.0 16.7 16.6 16.7
Kuwait 39.6 15.2 16.0 15.2 14.1
Middle East & Jordan R1 39.5 14.7 15.3 15.6 14.9
North Africa Saudi Arabia 42.9 14.3 13.2 14.3 15.3
Djibouti 40.2 15.2 15.0 13.7 15.9
Tunisia R2 60.8 20.3 18.9
Jordan R2 49.2 16.4 17.1 17.3
Tunisia R3 53.9 30.5 15.5
Iraq R2 66.6 16.6 16.9
Lebanon R2 57.5 14.2 14.1 14.2
Congo, Rep. 59.7 20.5 19.9
Sub-Saharan Zambia 40.2 59.8
Africa and the Chad 67.6 17.0 15.4
Sahel
Cameroon 48.8 17.0 17.0 17.3
Gambia, The 50.6 25.2 24.1
East Asia &
Papua New Guinea 39.0 19.0 20.1 21.9
Pacific
Kosovo 40.0 14.8 15.2 15.0 15.1
Europe & Serbia 39.5 14.1 15.1 15.7 15.5
Central Asia North Macedonia 39.6 15.7 13.8 16.2 14.8
Ukraine 57.0 21.9 21.1
Honduras 50.1 16.7 16.5 16.8
Latin America &
Caribbean Haiti 62.0 12.4 12.3 13.3
Belize 67.1 9.0 11.8 12.1
39
Jamaica 49.1 17.5 17.0 16.4
Total 50.2 2.7 2.6 5.1 7.2 5.1 8.2 7.7 11.4
Percentage of the sample assigned to each intervention. Depending on the characteristics of each study, randomization included a larger proportion of the control group to have
a larger group unaffected by the intervention with respect to other diagnostic questions outside the scope of this study.
40
Appendix C. Summary of Message Framings
Framing Question content Country
Control If a COVID-19 vaccine becomes available, do you Zambia, West Bank and Gaza,
Framing plan to take it? Ukraine, Tunisia 1Libya,
(Behavioral Lebanon 1, Iraq 1, Honduras,
intentions)
KSA, Chad, Cameroon
Are you willing to get a COVID-19 vaccine? Kosovo, Jamaica, Honduras,
Haiti, Djibouti, Republic of
Congo, Cameroon, Tunisia 2
Do you plan to get the COVID-19 vaccine? Ukraine, Tunisia 3, Serbia,
Papua New Guinea, North
Macedonia, Lebanon 2,
Kuwait, Kingdom of Saudi
Arabia, Kosovo, Jordan 1/2,
Jamaica, Djibouti, Republic of
Congo, Belize, The Gambia,
Iraq 2
Experts If a COVID-19 vaccine is considered safe and West Bank and Gaza, Tunisia
effective by national and international experts, do 1, Libya, Lebanon 1, Iraq 1,
you plan to take the vaccine when it becomes Honduras, Chad, Cameroon
available?
The COVID-19 vaccines available in {country} ({type Serbia, Papua New Guinea,
of vaccines}) are considered safe and highly effective North Macedonia, Kuwait,
by national and international experts. Do you plan to Kingdom of Saudi Arabia,
get the COVID-19 vaccine? Kosovo, Jordan 1, Djibouti,
Experts + If a COVID-19 vaccine is considered safe and West Bank and Gaza, Tunisia
Celebrities effective by national and international experts, and 1, Libya, Lebanon 1, Iraq 1,
celebrities and famous athletes get it themselves, Honduras, Cameroon
would you also plan to take the vaccine?
Experts + If a COVID-19 vaccine is considered safe and West Bank and Gaza, Tunisia
Religious effective by national and international experts, and 1, Libya, Lebanon 1, Iraq 1,
Leaders religious leaders in your community get it Honduras, Chad, Cameroon
themselves, would you plan to take the vaccine?
41
Dynamic {Nationality} are getting vaccinated against COVID- Tunisia 2, Serbia, Papua New
Social Norms 19! More than {n1} have done it so far, with {n2} just Guinea, North Macedonia,
in the past 2 weeks alone. Do you plan to get the Kuwait, Kingdom of Saudi
COVID-19 vaccine? Arabia, Kosovo, Jordan 1/23,
Djibouti, Republic of Congo,
Ukraine
Safety COVID-19 vaccines are safe - there have been no Tunisia 2, Serbia, Papua New
reported hospitalizations in {country} due to Guinea, North Macedonia,
vaccinations compared to {the number of death Kuwait, Kingdom of Saudi
case} deaths due to COVID-19. Do you plan to get Arabia, Kosovo, Jordan 1/2,
the COVID-19 vaccine? Djibouti, Republic of Congo,
The Gambia
Efficacy and The latest studies from around the world confirm Serbia, North Macedonia,
Pro-social that the COVID-19 vaccines protect you, your friends Kuwait, Kosovo, Ukraine
and family from COVID-19 by substantially reducing
or eliminating hospitalizations and deaths. Do you
plan to get the COVID-19 vaccine?
The latest studies from around the world confirm Kingdom of Saudi Arabia,
that the COVID-19 vaccines protect you, your friends Jordan 1/2, Djibouti
and family from COVID-19 by reducing
hospitalizations and death to near 0%. Do you plan
to get the COVID-19 vaccine?
The latest studies from around the world confirm Lebanon 2, Jamaica, Haiti,
that the COVID-19 vaccines are safe and protect you, Belize
your friends and family from COVID-19 by reducing
hospitalizations and death. Do you plan to get the
COVID-19 vaccine?
Efficacy The best choice you can make to protect yourself Zambia
from COVID-19 is to get fully vaccinated. This greatly
reduces the risk of severe illness, hospitalization and
dying from COVID-19. Are you willing to get a COVID-
19 vaccine?
The best choice you can make to protect your Zambia
household and family from COVID-19 is to get fully
vaccinated. This will help to keep the people you
love healthy, safe and strong. Are you willing to get a
COVID-19 vaccine?
Endemic As we learn to live with COVID-19 it's likely we will Haiti, Belize, Jamaica,
all be exposed to the virus eventually. The best way Lebanon 2, Iraq 2, The
to prepare yourself is to get fully vaccinated because Gambia
it greatly reduces the risk of hospitalization and
death. Do you plan to get the COVID-19 vaccine?
COVID-19 is unlikely to be eradicated. This means Tunisia 3
everyone would be eventually exposed to the virus.
The best way to prepare yourself for such an
42
exposure is full vaccination because it greatly
reduces the risk of hospitalization and death. Do you
plan to get the COVID-19 vaccine?
Variant New variants like Omicron and Delta can be Tunisia 3, Lebanon 2,
worrying, but the best evidence so far indicates that Jamaica, Haiti, Belize
vaccines are still highly protective against serious
illness and death from COVID-19. Do you plan to get
the COVID-19 vaccine?
New variants of the COVID-19 virus can be worrying, Iraq 2
but the best evidence so far indicates that vaccines
are still highly protective against serious illness and
death from COVID-19. Do you plan to get the COVID-
19 vaccine?
Death COVID-19 is now the fourth leading cause of death in Ukraine
the world. Of all the patients with severe COVID-19
in hospitals in [country], [number]% are
unvaccinated. Do you plan to get the COVID-19
vaccine?
COVID-19 is now the fourth leading cause of death in Iraq 2, Lebanon 2
the world. Of all the patients with severe COVID-19
in hospitals in [country], [number]% are
unvaccinated. You can protect yourself by getting
vaccinated. Do you plan to get the COVID-19
vaccine?
Financial [country] recently announced that every vaccinated Ukraine
citizen will be eligible for [currency] [amount]
certificate in Diya. Do you plan to get the COVID-19
vaccine?
Requirement Many institutions and governmental & private Iraq 2
facilities demand that their employees and attenders
be vaccinated. Do you plan to get the COVID-19
vaccine?
43
Appendix D. Aggregation of Wellcome Monitor Moderators
Following best practices (e.g., Bliese, 2000), we empirically tested the rationale for aggregating
measures of trust in doctors and nurses, perceived vaccine safety, and perceived vaccine by examining
whether (a) individuals have an acceptably low level of variability in their perceptions within each
country, which is indicated by high within-group inter-rater agreement (rWG) values; and, (b) there is
significant between-country variance in the constructs, which is indicated by high ICC(1)).
Within-Group Inter-Rater Agreement
We first calculated the within-group inter-rater agreement (rWG) value for each country (James,
Demaree, & Wolf, 1984; LeBreton & Senter, 2007), which is an index of the extent to which individuals
within a given country have similar ratings for a given indicator. For each country, this index is calculated
by comparing the observed variance in a given measure to the variance that would be expected by
chance. For example, if there was no agreement in trust of doctors and nurses within a given country, all
response options for the trust measure would be selected with equal frequency. Thus, the distribution
that would be expected by chance is a uniform distribution.
The formula for rWG when there is only one item rated by many raters is given by:
(SD1)
������
������������������������������������2
������������������������������������ = 1 −
������������������������������������2
Where ������������������������������������2 is the mean of the observed variances of the J items and ������������������������������������2 is the expected variance of
the uniform distribution that would be expected if there was no agreement in the measures,
which is given by:
(SD2)
(������������ − 1)
������������������������������������2 =
12
Where A equals the number of response options for each of the J items in a scale.
We then report the mean rWG for each measure where general guidelines suggest an average rWG > 0.70
is good evidence for aggregation to the group level (James et al., 1984). As reported in Table SD1, mean
values of rWG across countries in our sample for our indicators of trust in doctors and nurses (rWG = 0.62),
perceived vaccine safety (rWG = 0.45), and perceived vaccine effectiveness (rWG = 0.57) were generally
weak-to-moderate in strength. As such, moderator results with these measures should be interpreted
with this in mind. Based on additional information provided by the ICC(1) estimates, however, we
proceeded with aggregation despite these somewhat weaker results for within-group inter-rater
agreement.
Intraclass Correlation Coefficient
ICC(1) estimates the proportion of the total variance in the measurements that is attributable to
differences between groups rather than differences within groups. Higher ICC(1) values indicate that a
relatively greater proportion of variance in an outcome is explained by between-country differences,
44
meaning that members of each group are relatively similar to each other compared to members of other
groups.
ICC(1) was calculated using two-level random intercept multilevel mixed-effects models with the mixed
package in STATA (version 18).
The two-level random intercept model can be written as:
(SD3)
������������������������������������ = ������������0 + ������������������������ + ������������������������������������
Where:
• ������������������������������������ is the outcome for individual ������������ in country ������������
• ������������0 is the overall intercept
• ������������������������ is the random intercept for country ������������, assumed to be normally distributed with mean 0 and
2
variance ������������������������
2
• ������������������������������������ is the residual error term, assumed to be normally distributed with mean 0 and variance ������������������������
And ICC(1) is given by:
(SD4)
2
������������������������
ICC(1) = 2 2
������������������������ + ������������������������
Guidelines suggest that an ICC(1) of 0.06 or greater indicates that the group-level analysis is appropriate
because a meaningful portion of the variance is attributable to group membership (Bliese, 2000). As
noted in Table SD1, calculated ICC(1) proportions for trust in doctors and nurses (ICC(1) = 0.07),
perceived vaccine safety (ICC(1) = 0.09) and perceived vaccine effectiveness (ICC(1) = 0.06) met or
exceeded this threshold, supporting aggregation to a country-level indicator.
45
Table SD1. Aggregation statistics for indicators from Wellcome Global Monitor Survey (2018).
Trust doctors and Perceived vaccine Perceived vaccine
Region Study nurses safety effectiveness
rWG rWG rWG
Lebanon 0.72 0.67 0.73
Iraq 0.48 0.67 0.72
West Bank and
Gaza - - -
Middle East Tunisia 0.66 0.56 0.64
& North
Libya 0.59 0.46 0.56
Africa
Kuwait 0.61 0.66 0.71
Jordan 0.64 0.73 0.76
Saudi Arabia 0.69 0.59 0.61
Djibouti - - -
Africa Congo, Rep. 0.39 0.07 0.69
Eastern and
Southern Zambia 0.64 0.40 0.35
Africa Chad 0.54 0.35 0.50
Western and Cameroon 0.59 0.18 0.43
Central Gambia, The 0.64 0.61 0.64
East Asia &
Papua New Guinea - - -
Pacific
Kosovo 0.59 0.46 0.51
Europe & Serbia 0.73 0.38 0.53
Central Asia North Macedonia 0.60 0.20 0.35
Ukraine 0.77 0.39 0.43
Honduras 0.59 0.63 0.64
Latin
Haiti 0.65 0.08 0.47
America &
Caribbean Belize - - -
Jamaica - - -
Average rWG 0.62 0.45 0.57
ICC(1) 0.07 [0.03 - 0.12] 0.09 [0.05 - 0.17] 0.06 [0.03 - 0.11]
Note. West Bank and Gaza, Papua New Guinea, Djibouti, Belize, and Jamaica were not included in the Wellcome
Global Monitor Survey (2018) and therefore have no values reported in the table.
46
Appendix E - Little's Missing Completely at Random (MCAR) test
Little's MCAR test Chi-square distance Prob > chi-square
Lebanon R1 52.14 0.00
Iraq R1 9.24 0.16
West Bank and Gaza 28.24 0.01
Tunisia R1 19.99 0.00
Libya 94.39 0.00
Kuwait 24.77 0.00
Jordan R1 12.43 0.05
Saudi Arabia 22.98 0.00
Djibouti 17.72 0.01
Tunisia R2 10.79 0.03
Jordan R2 34.36 0.00
Tunisia R3 6.20 0.18
Iraq R2 24.27 0.00
Lebanon R2 203.91 0.00
Congo, Rep. 9.08 0.17
Zambia
Chad 25.54 0.00
Cameroon 11.79 0.23
Gambia, The
Papua New Guinea 12.97 0.16
Kosovo 20.18 0.00
Serbia 47.93 0.00
North Macedonia 34.47 0.00
Ukraine 7.29 0.12
Honduras 20.63 0.00
Haiti 13.98 0.23
Belize 17.20 0.00
Jamaica 32.60 0.00
Little's Missing Completely at Random (MCAR) tests the assumption of missing completely at random for multivariate, partially
observed quantitative data. The test did not converge for the Gambia and Zambia studies.
47
Appendix F – Meta-Analysis Results Transformed to Odds Ratios
Study Intervention N `"Odds ratio"' (95% CI) P-value
Lebanon R1 Experts 3,137 1.31 (1.09, 1.56) 0.00
Iraq R1 Experts 6,916 1.34 (1.20, 1.50) 0.00
West Bank and Gaza Experts 8,691 1.37 (1.24, 1.51) 0.00
Tunisia R1 Experts 8,566 1.24 (1.12, 1.38) 0.00
Libya Experts 4,732 1.35 (1.18, 1.54) 0.00
Kuwait Experts 1,713 0.92 (0.69, 1.24) 0.59
Jordan R1 Experts 4,372 0.95 (0.83, 1.09) 0.44
Saudi Arabia Experts 445 0.80 (0.48, 1.34) 0.40
Djibouti Experts 1,150 0.98 (0.73, 1.31) 0.87
Chad Experts 2,619 1.28 (1.04, 1.58) 0.02
Cameroon Experts 1,834 1.53 (1.23, 1.91) 0.00
Papua New Guinea Experts 1,157 1.16 (0.85, 1.57) 0.35
Kosovo Experts 2,572 1.16 (0.97, 1.39) 0.11
Serbia Experts 2,936 1.36 (1.12, 1.65) 0.00
North Macedonia Experts 1,183 0.88 (0.67, 1.16) 0.37
Honduras Experts 3,739 1.52 (1.30, 1.78) 0.00
Experts (overall) Experts 55,762 1.26 (1.21, 1.31) 0.00
Lebanon R1 Experts + Celebrities 3,236 1.29 (1.09, 1.52) 0.00
Iraq R1 Experts + Celebrities 6,957 1.26 (1.13, 1.41) 0.00
West Bank and Gaza Experts + Celebrities 8,708 1.33 (1.21, 1.48) 0.00
Tunisia R1 Experts + Celebrities 8,620 1.36 (1.22, 1.51) 0.00
Libya Experts + Celebrities 4,728 1.25 (1.09, 1.43) 0.00
Cameroon Experts + Celebrities 1,825 1.46 (1.16, 1.82) 0.00
Honduras Experts + Celebrities 3,724 1.45 (1.24, 1.69) 0.00
Experts + Celebrities (overall) Experts + Celebrities 37,798 1.33 (1.26, 1.39) 0.00
Experts + Religious
Lebanon R1 leaders 3,158 1.56 (1.31, 1.85) 0.00
Experts + Religious
Iraq R1 leaders 6,938 1.41 (1.26, 1.58) 0.00
Experts + Religious
West Bank and Gaza leaders 8,781 1.54 (1.40, 1.71) 0.00
Experts + Religious
Tunisia R1 leaders 8,677 1.27 (1.15, 1.41) 0.00
Experts + Religious
Libya leaders 4,738 1.47 (1.28, 1.68) 0.00
Experts + Religious
Chad leaders 2,671 1.56 (1.27, 1.91) 0.00
Experts + Religious
Cameroon leaders 1,825 1.72 (1.38, 2.14) 0.00
Experts + Religious
Honduras leaders 3,734 1.43 (1.22, 1.67) 0.00
Experts + Religious
Experts + Religious leaders (overall) leaders 40,522 1.45 (1.38, 1.52) 0.00
Kuwait Safety 1,748 0.90 (0.68, 1.20) 0.49
48
Jordan R1 Safety 4,430 1.03 (0.90, 1.18) 0.65
Saudi Arabia Safety 437 0.98 (0.57, 1.69) 0.94
Djibouti Safety 1,104 1.09 (0.80, 1.47) 0.58
Tunisia R2 Safety 4,952 0.86 (0.75, 0.99) 0.04
Jordan R2 Safety 1,741 0.92 (0.74, 1.15) 0.48
Congo, Rep. Safety 540 1.50 (0.99, 2.26) 0.06
Gambia, The Safety 408 0.93 (0.61, 1.42) 0.73
Papua New Guinea Safety 1,123 1.93 (1.45, 2.59) 0.00
Kosovo Safety 2,568 0.98 (0.82, 1.17) 0.84
Serbia Safety 2,947 1.33 (1.09, 1.61) 0.00
North Macedonia Safety 1,215 1.17 (0.90, 1.50) 0.24
Safety (overall) Safety 23,213 1.05 (0.98, 1.11) 0.17
Kuwait Efficacy and Pro-Social 1,775 1.13 (0.84, 1.53) 0.41
Jordan R1 Efficacy and Pro-Social 4,408 1.24 (1.08, 1.42) 0.00
Saudi Arabia Efficacy and Pro-Social 429 0.73 (0.43, 1.23) 0.23
Djibouti Efficacy and Pro-Social 1,132 1.17 (0.88, 1.56) 0.29
Jordan R2 Efficacy and Pro-Social 1,735 1.35 (1.08, 1.68) 0.01
Lebanon R2 Efficacy and Pro-Social 4,223 1.12 (0.94, 1.33) 0.19
Zambia Efficacy and Pro-Social 2,924 1.59 (1.37, 1.85) 0.00
Kosovo Efficacy and Pro-Social 2,575 1.27 (1.06, 1.52) 0.01
Serbia Efficacy and Pro-Social 2,915 1.40 (1.15, 1.70) 0.00
North Macedonia Efficacy and Pro-Social 1,162 1.28 (0.98, 1.68) 0.07
Ukraine Efficacy and Pro-Social 707 1.41 (0.94, 2.12) 0.10
Haiti Efficacy and Pro-Social 6,878 1.36 (1.18, 1.55) 0.00
Belize Efficacy and Pro-Social 255 0.88 (0.37, 2.11) 0.78
Jamaica Efficacy and Pro-Social 1,572 0.79 (0.61, 1.01) 0.06
Efficacy and Pro-Social (overall) Efficacy and Pro-Social 32,690 1.27 (1.20, 1.34) 0.00
Kuwait Dynamic Social Norms 1,750 1.05 (0.78, 1.41) 0.75
Jordan R1 Dynamic Social Norms 4,355 1.11 (0.97, 1.27) 0.14
Saudi Arabia Dynamic Social Norms 437 1.64 (0.88, 3.07) 0.12
Djibouti Dynamic Social Norms 1,136 1.31 (0.99, 1.75) 0.06
Tunisia R2 Dynamic Social Norms 5,036 1.05 (0.91, 1.21) 0.53
Jordan R2 Dynamic Social Norms 1,716 1.20 (0.96, 1.50) 0.11
Congo, Rep. Dynamic Social Norms 544 0.98 (0.64, 1.49) 0.91
Papua New Guinea Dynamic Social Norms 1,102 1.11 (0.81, 1.53) 0.51
Kosovo Dynamic Social Norms 2,558 1.16 (0.97, 1.39) 0.11
Serbia Dynamic Social Norms 2,862 1.29 (1.05, 1.58) 0.02
North Macedonia Dynamic Social Norms 1,203 1.10 (0.85, 1.43) 0.48
Ukraine Dynamic Social Norms 714 1.73 (1.16, 2.56) 0.01
Dynamic Social Norms (overall) Dynamic Social Norms 23,413 1.15 (1.07, 1.22) 0.00
Tunisia R3 Endemic 1,163 1.20 (0.89, 1.63) 0.23
Iraq R2 Endemic 2,068 1.77 (1.38, 2.27) 0.00
Lebanon R2 Endemic 4,220 1.35 (1.14, 1.59) 0.00
Gambia, The Endemic 414 1.07 (0.71, 1.62) 0.75
49
Haiti Endemic 6,781 1.67 (1.45, 1.91) 0.00
Belize Endemic 254 2.08 (0.96, 4.48) 0.06
Jamaica Endemic 1,588 0.92 (0.72, 1.18) 0.52
Endemic (overall) Endemic 16,488 1.42 (1.31, 1.55) 0.00
Tunisia R3 Variants 1,414 1.11 (0.87, 1.42) 0.40
Iraq R2 Variants 2,061 1.34 (1.03, 1.73) 0.03
Lebanon R2 Variants 4,226 1.00 (0.84, 1.20) 0.96
Haiti Variants 6,789 1.35 (1.18, 1.55) 0.00
Belize Variants 245 1.19 (0.51, 2.80) 0.69
Jamaica Variants 1,598 0.90 (0.71, 1.15) 0.41
Variants (overall) Variants 16,333 1.16 (1.07, 1.27) 0.00
All combined (overall) All combined 1.28 (1.25, 1.31) 0.00
Two-stage individual participant data (IPD) meta-analysis by intervention for each study with intention to vaccinate as the
outcome. Estimated coefficients transformed to odds ratio. Model includes the intervention and gender, age, education, and
health worker status as covariates.
50
Appendix G. Bivariate Correlations Between Investigated Study-Level Moderators and Study Effect Sizes
Experts + Dynamic
Experts + Efficacy and
Experts Religious Safety Social Endemic Variants
Celebrities Pro-Social
Moderators leaders Norms
COVID-19 vaccination rate in sample 0.000 0.001 0.003*** 0.003*** 0.005** 0.002
(0.002) (0.001) (0.001) (0.001) (0.002) (0.001)
Start date of experiment (monthly) 0.031** 0.072*** 0.081*** 0.010 0.017*** 0.023*** 0.021*** 0.009
(0.013) (0.025) (0.025) (0.010) (0.006) (0.005) (0.007) (0.005)
GDP per capita, PPP (constant 2017 international $) -0.002 0.003 -0.001 -0.003 -0.001 0.005 -0.007 -0.017
(0.005) (0.019) (0.023) (0.005) (0.006) (0.004) (0.033) (0.019)
Trust healthcare workers -0.666* -0.507 -0.940 -0.596* 0.574 -0.124 -0.444 -0.520
(0.384) (0.631) (0.577) (0.306) (0.775) (0.404) (0.757) (0.550)
Perceived vaccine safety 0.044 0.521 0.434 -0.363* -0.252 -0.282* 0.279 -0.132
(0.359) (0.688) (0.939) (0.193) (0.245) (0.165) (0.844) (0.615)
Perceived vaccine effectiveness 0.219 1.040 1.085 -0.195 -0.229 -0.351 1.696** 1.213*
(0.512) (0.945) (1.316) (0.271) (0.378) (0.248) (0.830) (0.729)
Note: * p < .10, ** p < .05. Trust in doctors, perceived vaccine safety, and perceived vaccine effectiveness from the Wellcome Global Monitor 2018 were not available for Belize,
Djibouti, Jamaica, Papua New Guinea, and West Bank and Gaza.
51
Appendix H. Heterogeneous Effects at Individual Level With Only Coefficients For Interaction Effects Reported
Education
Female (base= Age 40+ (base= secondary or Health worker
Region Study Intervention N P-value P-value P-value P-value
Male) Age 39-) below (base= (base= rest)
tertiary)
Experts +
Religious
Lebanon R1 leaders 3158 -0.33 (-0.68, 0.02) 0.07 0.00 (-0.35, 0.35) 0.99 0.11 (-0.25, 0.47) 0.55 0.03 (-0.49, 0.55) 0.90
Experts +
Lebanon R1 Celebrities 3236 0.01 (-0.33, 0.35) 0.95 -0.02 (-0.36, 0.31) 0.89 0.21 (-0.13, 0.56) 0.23 -0.06 (-0.59, 0.46) 0.81
Lebanon R1 Experts 3137 -0.08 (-0.44, 0.28) 0.66 -0.09 (-0.45, 0.26) 0.61 0.02 (-0.34, 0.39) 0.90 0.47 (-0.06, 1.01) 0.08
Experts +
Religious
Iraq R1 leaders 6938 -0.01 (-0.23, 0.21) 0.90 0.09 (-0.14, 0.32) 0.44 -0.03 (-0.25, 0.20) 0.82 -0.13 (-0.46, 0.20) 0.43
Experts +
Iraq R1 Celebrities 6957 -0.15 (-0.37, 0.07) 0.18 -0.05 (-0.27, 0.18) 0.68 -0.13 (-0.36, 0.10) 0.25 -0.21 (-0.54, 0.11) 0.20
Iraq R1 Experts 6916 -0.14 (-0.36, 0.09) 0.23 -0.26 (-0.50, -0.03) 0.02 0.07 (-0.16, 0.30) 0.55 0.20 (-0.14, 0.55) 0.25
Experts +
West Bank Religious
and Gaza leaders 8781 0.25 (0.06, 0.45) 0.01 -0.07 (-0.28, 0.13) 0.50 -0.14 (-0.33, 0.06) 0.18 -0.10 (-0.38, 0.19) 0.50
Middle East West Bank Experts +
& North and Gaza Celebrities 8708 -0.02 (-0.23, 0.18) 0.82 -0.06 (-0.27, 0.15) 0.59 -0.20 (-0.41, -0.00) 0.05 0.17 (-0.12, 0.45) 0.26
Africa West Bank
and Gaza Experts 8691 -0.01 (-0.21, 0.20) 0.96 -0.01 (-0.21, 0.20) 0.96 -0.16 (-0.36, 0.05) 0.13 0.04 (-0.24, 0.32) 0.77
Experts +
Religious
Tunisia R1 leaders 8677 -0.12 (-0.33, 0.08) 0.24 -0.08 (-0.29, 0.13) 0.45 0.09 (-0.12, 0.30) 0.41 0.10 (-0.16, 0.36) 0.46
Experts +
Tunisia R1 Celebrities 8620 -0.08 (-0.29, 0.12) 0.42 -0.12 (-0.33, 0.09) 0.27 0.02 (-0.19, 0.23) 0.88 0.22 (-0.05, 0.48) 0.11
Tunisia R1 Experts 8566 -0.11 (-0.32, 0.10) 0.33 -0.01 (-0.23, 0.20) 0.90 -0.02 (-0.23, 0.19) 0.86 0.14 (-0.13, 0.41) 0.32
Experts +
Religious
Libya leaders 4738 0.05 (-0.23, 0.32) 0.74 0.17 (-0.11, 0.45) 0.25 0.01 (-0.28, 0.30) 0.95 0.04 (-0.32, 0.41) 0.83
Experts +
Libya Celebrities 4728 0.07 (-0.20, 0.35) 0.60 0.12 (-0.16, 0.40) 0.40 -0.07 (-0.35, 0.21) 0.64 0.17 (-0.17, 0.52) 0.33
Libya Experts 4732 0.05 (-0.22, 0.32) 0.71 0.07 (-0.21, 0.34) 0.63 -0.08 (-0.36, 0.20) 0.57 -0.13 (-0.48, 0.22) 0.46
Dynamic
Kuwait Social Norms 1750 0.22 (-0.39, 0.84) 0.47 -0.40 (-1.00, 0.20) 0.19 0.35 (-0.27, 0.97) 0.27 0.18 (-1.03, 1.38) 0.78
52
Efficacy and
Kuwait Pro-Social 1775 0.08 (-0.52, 0.69) 0.79 -0.01 (-0.61, 0.59) 0.98 -0.38 (-0.98, 0.22) 0.21 -0.02 (-1.23, 1.19) 0.98
Kuwait Safety 1748 -0.27 (-0.85, 0.32) 0.37 -0.45 (-1.03, 0.13) 0.13 -0.47 (-1.06, 0.12) 0.12 -0.23 (-1.14, 0.67) 0.61
Kuwait Experts 1713 0.63 (0.01, 1.24) 0.05 0.19 (-0.43, 0.80) 0.55 -0.07 (-0.67, 0.53) 0.82 -0.73 (-1.72, 0.26) 0.15
Dynamic
Jordan R1 Social Norms 4355 -0.16 (-0.44, 0.12) 0.27 0.00 (-0.29, 0.29) 0.99 0.07 (-0.21, 0.35) 0.63 0.59 (-0.06, 1.23) 0.07
Efficacy and
Jordan R1 Pro-Social 4408 0.28 (0.00, 0.56) 0.05 -0.27 (-0.55, 0.01) 0.05 0.18 (-0.10, 0.45) 0.21 -0.39 (-0.96, 0.19) 0.19
Jordan R1 Safety 4430 0.22 (-0.06, 0.49) 0.12 -0.20 (-0.48, 0.07) 0.15 0.10 (-0.17, 0.37) 0.46 0.03 (-0.54, 0.60) 0.91
Jordan R1 Experts 4372 0.15 (-0.13, 0.42) 0.29 -0.42 (-0.70, -0.14) 0.00 0.20 (-0.08, 0.47) 0.16 -0.02 (-0.61, 0.56) 0.93
Dynamic
Saudi Arabia Social Norms 437 1.10 (-0.12, 2.32) 0.08 0.58 (-0.75, 1.91) 0.39 1.67 (0.33, 3.02) 0.01 -1.36 (-3.40, 0.68) 0.19
Efficacy and
Saudi Arabia Pro-Social 422 -0.38 (-1.47, 0.70) 0.49 0.11 (-1.01, 1.22) 0.85 0.64 (-0.48, 1.77) 0.26
Saudi Arabia Safety 437 -0.00 (-1.11, 1.10) 0.99 0.93 (-0.29, 2.15) 0.14 0.88 (-0.31, 2.07) 0.15 -0.66 (-2.70, 1.37) 0.52
Saudi Arabia Experts 445 0.59 (-0.44, 1.63) 0.26 0.31 (-0.77, 1.40) 0.57 -0.25 (-1.30, 0.79) 0.63 0.52 (-1.83, 2.86) 0.67
Dynamic
Djibouti Social Norms 1136 -0.23 (-0.81, 0.34) 0.43 -0.41 (-1.34, 0.52) 0.39 -0.36 (-0.95, 0.22) 0.22 -0.16 (-0.98, 0.65) 0.70
Efficacy and
Djibouti Pro-Social 1132 0.38 (-0.21, 0.97) 0.20 -0.31 (-1.30, 0.69) 0.55 -0.63 (-1.22, -0.05) 0.03 -0.07 (-0.96, 0.82) 0.87
Djibouti Safety 1104 0.01 (-0.60, 0.62) 0.97 -0.51 (-1.47, 0.46) 0.31 0.03 (-0.59, 0.64) 0.94 -0.08 (-0.96, 0.80) 0.87
Djibouti Experts 1150 -0.10 (-0.69, 0.49) 0.74 -1.14 (-2.14, -0.14) 0.03 0.15 (-0.44, 0.74) 0.62 0.09 (-0.78, 0.96) 0.84
Dynamic
Tunisia R2 Social Norms 5036 -0.13 (-0.41, 0.16) 0.39 -0.11 (-0.42, 0.21) 0.51 0.32 (0.03, 0.60) 0.03 -0.10 (-0.74, 0.54) 0.76
Tunisia R2 Safety 4952 0.27 (-0.01, 0.56) 0.06 -0.17 (-0.48, 0.14) 0.29 0.12 (-0.18, 0.41) 0.44 -0.34 (-0.95, 0.27) 0.28
Dynamic
Jordan R2 Social Norms 1716 -0.14 (-0.64, 0.35) 0.57 0.28 (-0.22, 0.78) 0.27 -0.40 (-0.85, 0.05) 0.09 -1.48 (-2.80, -0.16) 0.03
Efficacy and
Jordan R2 Pro-Social 1735 -0.22 (-0.70, 0.27) 0.38 -0.10 (-0.57, 0.38) 0.69 0.30 (-0.16, 0.75) 0.20 -0.47 (-1.69, 0.75) 0.45
Jordan R2 Safety 1741 -0.24 (-0.73, 0.24) 0.33 -0.36 (-0.84, 0.13) 0.15 0.07 (-0.40, 0.53) 0.78 -0.58 (-1.89, 0.73) 0.39
Tunisia R3 Variants 1414 -0.02 (-0.51, 0.48) 0.95 0.15 (-0.43, 0.73) 0.62 0.47 (-0.02, 0.97) 0.06 -0.05 (-1.53, 1.43) 0.95
Tunisia R3 Endemic 1163 -0.04 (-0.65, 0.57) 0.89 0.23 (-0.53, 0.99) 0.55 -0.06 (-0.67, 0.55) 0.85 1.87 (-0.03, 3.77) 0.05
Iraq R2 Variants 2061 -0.20 (-0.74, 0.33) 0.46 -0.38 (-0.97, 0.21) 0.21 0.39 (-0.15, 0.92) 0.16 -0.27 (-1.29, 0.75) 0.61
Iraq R2 Endemic 2068 -0.27 (-0.78, 0.24) 0.29 0.02 (-0.52, 0.56) 0.95 -0.26 (-0.76, 0.24) 0.31 -0.59 (-1.49, 0.32) 0.20
Lebanon R2 Variants 4226 -0.10 (-0.45, 0.25) 0.58 0.22 (-0.18, 0.63) 0.28 -0.12 (-0.51, 0.26) 0.54 -0.18 (-0.82, 0.47) 0.59
Lebanon R2 Endemic 4220 -0.20 (-0.53, 0.13) 0.24 0.26 (-0.11, 0.63) 0.17 0.01 (-0.37, 0.38) 0.97 -0.27 (-0.97, 0.42) 0.44
Efficacy and
Lebanon R2 Pro-Social 4223 0.00 (-0.34, 0.35) 0.99 0.00 (-0.40, 0.40) 0.99 0.14 (-0.25, 0.53) 0.49 0.02 (-0.61, 0.66) 0.94
53
Dynamic
Africa Congo, Rep. Social Norms 544 0.02 (-0.85, 0.89) 0.96 0.03 (-1.15, 1.21) 0.95 -0.49 (-1.36, 0.37) 0.26 0.05 (-1.15, 1.25) 0.93
Eastern Congo, Rep. Safety 540 -0.30 (-1.17, 0.56) 0.49 -0.73 (-1.91, 0.45) 0.22 0.21 (-0.64, 1.05) 0.63 0.27 (-1.11, 1.64) 0.70
and Efficacy and
Southern Zambia Pro-Social 2924 0.13 (-0.18, 0.45) 0.41 -0.32 (-0.78, 0.13) 0.16 -0.10 (-0.41, 0.21) 0.53 0.41 (-2.79, 3.61) 0.80
Experts +
Religious
Chad leaders 2671 -0.35 (-0.80, 0.10) 0.13 0.49 (-0.21, 1.20) 0.17 -0.07 (-0.47, 0.34) 0.75 -0.15 (-0.60, 0.30) 0.52
Chad Experts 2619 -0.54 (-1.02, -0.06) 0.03 -0.07 (-0.75, 0.60) 0.83 0.03 (-0.39, 0.46) 0.88 -0.20 (-0.66, 0.25) 0.38
Experts +
Africa
Religious
Western
Cameroon leaders 1825 0.14 (-0.33, 0.62) 0.55 0.58 (-0.09, 1.25) 0.09 -0.38 (-0.82, 0.07) 0.10 -0.09 (-0.67, 0.49) 0.75
and Central
Experts +
Cameroon Celebrities 1825 -0.17 (-0.66, 0.32) 0.51 -0.26 (-0.92, 0.40) 0.44 -0.11 (-0.57, 0.34) 0.63 0.24 (-0.40, 0.88) 0.47
Cameroon Experts 1834 -0.08 (-0.55, 0.39) 0.74 0.32 (-0.33, 0.97) 0.34 0.06 (-0.39, 0.51) 0.81 0.30 (-0.30, 0.91) 0.32
Gambia, The Endemic 414 -0.23 (-1.13, 0.67) 0.62 -0.30 (-1.50, 0.90) 0.62 -0.28 (-1.16, 0.59) 0.52
Gambia, The Safety 408 -0.28 (-1.21, 0.64) 0.55 0.06 (-1.28, 1.40) 0.93 0.66 (-0.23, 1.54) 0.15
Papua New Dynamic
Guinea Social Norms 1102 0.61 (-0.13, 1.35) 0.11 0.15 (-0.65, 0.95) 0.72 0.75 (0.11, 1.39) 0.02 -0.49 (-1.48, 0.50) 0.33
East Asia & Papua New
Pacific Guinea Safety 1123 0.36 (-0.31, 1.04) 0.30 -0.49 (-1.22, 0.25) 0.19 0.32 (-0.26, 0.90) 0.27 -0.29 (-1.15, 0.56) 0.50
Papua New
Guinea Experts 1157 0.14 (-0.59, 0.87) 0.71 0.49 (-0.27, 1.24) 0.21 0.10 (-0.51, 0.72) 0.74 -0.20 (-1.14, 0.73) 0.67
Dynamic
Kosovo Social Norms 2558 0.03 (-0.34, 0.40) 0.87 -0.24 (-0.64, 0.16) 0.24 0.04 (-0.33, 0.41) 0.84 0.81 (0.12, 1.50) 0.02
Efficacy and
Kosovo Pro-Social 2575 0.22 (-0.14, 0.59) 0.23 -0.10 (-0.51, 0.32) 0.64 0.19 (-0.18, 0.55) 0.31 -0.21 (-0.86, 0.44) 0.53
Kosovo Safety 2568 -0.02 (-0.38, 0.34) 0.93 -0.17 (-0.56, 0.22) 0.40 0.12 (-0.24, 0.48) 0.51 0.60 (-0.10, 1.30) 0.09
Kosovo Experts 2572 0.11 (-0.25, 0.47) 0.55 -0.18 (-0.58, 0.22) 0.38 0.04 (-0.32, 0.40) 0.81 0.56 (-0.09, 1.22) 0.09
Dynamic
Europe & Serbia Social Norms 2862 0.08 (-0.34, 0.50) 0.71 -0.35 (-0.77, 0.06) 0.09 -0.10 (-0.52, 0.32) 0.64 -0.27 (-1.09, 0.54) 0.51
Central
Efficacy and
Asia
Serbia Pro-Social 2915 0.28 (-0.13, 0.69) 0.18 0.16 (-0.24, 0.56) 0.43 0.10 (-0.30, 0.51) 0.61 0.28 (-0.45, 1.01) 0.45
Serbia Safety 2947 -0.01 (-0.40, 0.39) 0.98 -0.12 (-0.51, 0.27) 0.54 -0.19 (-0.58, 0.21) 0.36 -0.46 (-1.20, 0.29) 0.23
Serbia Experts 2936 -0.05 (-0.44, 0.35) 0.81 -0.22 (-0.61, 0.17) 0.28 -0.03 (-0.43, 0.37) 0.89 0.23 (-0.44, 0.90) 0.50
North Dynamic
Macedonia Social Norms 1203 0.17 (-0.36, 0.69) 0.53 0.05 (-0.49, 0.59) 0.86 0.17 (-0.36, 0.70) 0.54 -0.74 (-1.71, 0.22) 0.13
North Efficacy and
Macedonia Pro-Social 1162 0.11 (-0.43, 0.65) 0.70 0.02 (-0.57, 0.61) 0.94 0.15 (-0.40, 0.70) 0.60 -0.44 (-1.37, 0.50) 0.36
54
North
Macedonia Safety 1215 0.45 (-0.08, 0.97) 0.10 0.70 (0.17, 1.23) 0.01 -0.08 (-0.61, 0.45) 0.77 0.27 (-0.80, 1.34) 0.62
North
Macedonia Experts 1183 0.12 (-0.44, 0.67) 0.68 0.46 (-0.11, 1.02) 0.11 -0.08 (-0.63, 0.47) 0.78 -1.58 (-2.67, -0.49) 0.00
Dynamic
Ukraine Social Norms 714 -0.71 (-1.51, 0.08) 0.08 -0.36 (-1.15, 0.43) 0.37 0.76 (-0.09, 1.61) 0.08 0.42 (-0.54, 1.37) 0.39
Efficacy and
Ukraine Pro-Social 707 0.26 (-0.60, 1.12) 0.55 -0.23 (-1.07, 0.61) 0.59 0.46 (-0.41, 1.34) 0.30 0.19 (-0.83, 1.21) 0.71
Experts +
Religious
Honduras leaders 3734 -0.19 (-0.50, 0.12) 0.23 -0.61 (-0.93, -0.29) 0.00 -0.01 (-0.37, 0.35) 0.95 0.16 (-0.28, 0.59) 0.48
Experts +
Honduras Celebrities 3724 -0.25 (-0.56, 0.07) 0.12 0.06 (-0.27, 0.39) 0.73 -0.15 (-0.53, 0.23) 0.43 -0.05 (-0.50, 0.39) 0.81
Honduras Experts 3739 0.02 (-0.29, 0.33) 0.88 -0.15 (-0.47, 0.18) 0.39 -0.21 (-0.58, 0.16) 0.27 0.03 (-0.39, 0.45) 0.90
Haiti Variants 6789 0.21 (-0.08, 0.49) 0.16 0.14 (-0.69, 0.97) 0.74 0.11 (-0.18, 0.39) 0.46 -0.18 (-0.50, 0.15) 0.29
Haiti Endemic 6781 0.12 (-0.17, 0.40) 0.42 0.24 (-0.62, 1.10) 0.58 -0.05 (-0.33, 0.23) 0.74 -0.05 (-0.37, 0.28) 0.78
Latin
Efficacy and
America &
Haiti Pro-Social 6878 0.09 (-0.19, 0.37) 0.52 0.11 (-0.75, 0.96) 0.81 -0.06 (-0.34, 0.22) 0.68 0.15 (-0.17, 0.47) 0.36
Caribbean
Belize Variants 244 0.48 (-1.48, 2.45) 0.63 0.13 (-2.42, 2.68) 0.92 0.35 (-2.40, 3.10) 0.80
Belize Endemic 254 0.03 (-1.53, 1.60) 0.97 1.15 (-0.52, 2.81) 0.18 -0.34 (-2.04, 1.37) 0.70 0.41 (-2.40, 3.23) 0.77
Efficacy and
Belize Pro-Social 255 0.45 (-1.26, 2.16) 0.61 0.57 (-1.29, 2.43) 0.55 -0.32 (-2.17, 1.53) 0.74
Jamaica Variants 1598 0.42 (-0.10, 0.95) 0.12 0.12 (-0.44, 0.68) 0.68 0.08 (-0.40, 0.57) 0.74 -1.11 (-2.06, -0.17) 0.02
Jamaica Endemic 1588 0.15 (-0.37, 0.67) 0.57 0.62 (0.08, 1.15) 0.02 0.27 (-0.22, 0.76) 0.28 -1.31 (-2.25, -0.36) 0.01
Efficacy and
Jamaica Pro-Social 1572 0.35 (-0.19, 0.89) 0.20 -0.23 (-0.83, 0.38) 0.46 0.13 (-0.38, 0.63) 0.62 0.19 (-0.71, 1.09) 0.68
Two-stage individual participant data (IPD) meta-analysis by study and intervention for the interaction effect with gender, age, education level, and health worker status. The
logistic model includes the intervention, the four demographic characteristics, and the interactions shown in the table.
55