WATER GLOBAL PRACTICE KNOWLEDGE BRIEF A Brief Summary of Global WASH Interventions What Works and What Doesn’t Luis Andres (Lead Economist, GWADR), Christian Borja-Vega (Economist, GWASP), Crystal Fenwick (Consultant, GWASP), Ronald Gomez-Suarez (Economist, GHNDR) and Jaime de Jesus Filho (Consultant, GWASP) MAY 2018 WASH practitioners and decision-makers lack evidence on the wider health and social effects of WASH interventions needed to create a paradigm shift in the sector. A global overview and meta-analysis on the effects ­ economic outcomes was undertaken. The results of different WASH interventions on different health and socio­ of this analysis show that evaluations of WASH interventions continue to focus predominantly on reducing diarrheal disease and there is a strong need for larger, more rigorously designed studies covering a broader scope ­ of outcome effects. Similarly, there is a need for greater geographical representation and finally, well-trained implementing agencies to achieve the desired results. Background The purpose of this study was two-fold: The effects of water supply, sanitation and hygiene (1) to collate existing evidence on global (WASH) interventions on the reduction of diarrheal WASH interventions into a single, publicly disease in children have been thoroughly documented, available repository and (2) to quantify the however, evidence evaluating the effects of WASH effectiveness of a broad range of WASH interventions on other outcomes of health and well-­ interventions on an array of outcomes being, such as school attendance and growth, is sparse. through a quantitative meta-analysis of WASH interventions, and consequently the studies impact evaluation (IE) studies. designed to evaluate their effectiveness, have similarly 1 focused on water quality, while relatively little is known about the effectiveness of other interventions, Meta-analysis is a statistical tool that such as sanitation and hygiene. Given the magni- pools results from individual studies into a tude of people impacted globally, time, and resource combined estimate of overall effects from ­ constraints, policy makers often face challenging different programs. The aggregate effects ­ decisions when  tasked with allocating resources to and conclusions of a meta-analysis are WASH programs.1 Understanding the evidence support- nonetheless based on the quality of the ing the effectiveness of different WASH interventions on individual studies included. different outcomes of health and socioeconomic well-­ being is imperative to optimizing results and improving intervention, accounting for 39 percent, followed by the performance and sustainability of WASH programs in percent), water supply (8 ­ hygiene (17  ­ percent), and the long run. sanitation (7 percent). Finally, more than half of all studies targeted incidence of diarrhea in children as an Sample outcome. An extensive search of published IEs was undertaken. A key word analysis was then applied and only papers Discussion using accepted IE methods were included, resulting in a final dataset of 136 IE studies. More than 50 percent To facilitate the meta-analysis,2 studies were divided of all studies were published after 2008. Geographical into five thematic groups based on expert review and representation appeared to mirror the lack of services according to outcome: (1) behavior change, hygiene, globally, and more than 80 percent of all studies were and sanitation; (2) growth, health, and mortality concentrated in three regions: South Asia (34 percent), in  children combined with other infectious diseases; Sub-Saharan Africa (27 percent), and Latin America and (3) cognitive skills and school absenteeism; (4) access the Caribbean (18 percent). Similarly more than two- to water and water quality; and (5) diarrhea and enteric thirds of all evaluations were conducted in rural areas. diseases. Water quality was the most commonly assessed WASH There has been a large increase in WASH-related IEs over the past decade. Studies have been concen- trated in the most underserved areas globally while An IE is an assessment tool for determining East Asia, Europe and Central Asia, and the Middle whether an intervention works while East and Northern Africa regions remain under repre- also assessing program design. Properly sented. Despite a large increase in studies evaluating conducted IE provide high quality evidence combined WASH interventions, few evaluations tar- that help orient investment decisions, get multi-country or multi-sector interventions. As improve design policies, adjust ongoing a whole, water quality interventions dominate IEs, interventions, and increase transparency especially in Sub-Saharan Africa and Latin America and accountability. To ensure the integrity and the Caribbean. This finding is not surprising given of this review, only evidence-based IE were the more quantitative nature of water quality interven- included, that is, studies that employed tions making them suitable for experimental methods. accepted research methods to rigorously measure impacts attributed to WASH Despite a relatively high number of unique outcomes (21) interventions. reported in the 136 studies, more than half of all evalua- tions focused on diarrhea and this review corroborates 2 A Brief Summary of Global WASH Interventions earlier findings (Cumming et al. 2014; Hutton and randomized controlled trials (RCTs)—the gold standard Chase 2017) that positive effects for diarrhea reduction for establishing a causal attribution between inter- are well-established and thoroughly documented, par- ventions and outcomes—tend to estimate larger and ticularly for hand washing with soap and water quality more precise effects. In some cases, study locale (e.g. trials. However, while experimental  designs resulted urban versus rural) was found to affect the reliability experimental designs did in significant effects, quasi-­ of results; however, this varied by outcome and could not produce the same results. 3 not always be explained by other study characteristics. For example, though WASH interventions targeting Evidence from the individual (non-pooled) studies for behavior change and sanitation produced more robust behavior change and other health-related outcomes, effects in urban areas, this was also a function of study e.g. child mortality, stunting, height, and weight is design, given the studies conducted in urban areas limited, and predominantly observed in single studies ­ applied experimental methods, which was shown to although small-scale studies evaluating combined improve results. On the contrary, WASH interventions interventions did report weak, but significant results targeting reduced incidence of diarrhea and enteric-­ for changes in child mortality and stunting. Moreover, related diseases conducted in rural areas produced while a range of WASH interventions were frequently more precise results than interventions in urban areas. employed to control cholera outbreaks, few programs This is despite the fact most interventions in urban have been evaluated using rigorous IE techniques areas had been implemented by NGOs, which were thereby limiting the ability to draw conclusive evi- shown to improve the accuracy of results when com- dence. Further, there is a clear distinction between pared to government agencies. This could be explained program effects from stand-alone WASH interven- by reinfection rates, which are typically higher and/or tions versus programs that are designed to target mul- impact more people in densely populated, urban areas. tiple WASH themes. For example, the evidence for However, given the potential bias in this group, results water quality in single interventions using e ­ xperimental should be interpreted with caution. designs is solid, yet less so in combined interventions, including sanitation and hygiene. Finally, IEs specifi- cally focused on child health outcomes showed Conclusions ­heterogeneous effects. In general, evaluations that encompass large numbers The pooled effects of WASH interventions on school of studies with rigorous (experimental) research meth- absenteeism were significant. The odds of missing ods produce more precise results. This was the case school were considerably reduced (30 percent) for stu- for studies evaluating effects on behavior and sanita- dents having received a WASH intervention. Similarly, tion outcomes. Improving study design specifically in ­ children were significantly more likely to use soap rural areas for these outcomes might be one area for (1.44 times) and less likely to develop Ascaris infections future research, while increasing the number of stud- (0.5 times), diarrhea (0.65 times), and die (0.91 times), ies in urban areas with high population density could as children not having received a WASH intervention. generate economies of scale. There is a wide range of Finally, child growth, hand washing, and latrine adop- qualitative approaches that can be employed in com- tion increased by 26, 8, and 22 percent respectively and bination with quantitative methods to strengthen water quality improved by 20 percent. effects. However, there is a trade-off4 between the internal and external validity of WASH IEs that should In certain circumstances, research methods can sig- be taken into account when designing studies. More nificantly influence outcome effects. In particular, research needs to be undertaken on hygiene and A Brief Summary of Global WASH Interventions 3 sanitation interventions overall and greater geo- have been published since this study began in 2013, this study remains the first of its kind to compare the effects of WASH interven- graphical representation is needed. Finally, additional tions on diverse health and non-health outcomes through a com- research is needed to better understand the impacts bined meta-analysis. of study locale on the results of WASH interventions 2. Aggregating outcomes ensured there was sufficient variation in sam- targeting reduced diarrhea and enteric diseases. The ple sizes, effects, and standard errors to conduct the meta-analysis, while increasing the statistical power. effects of multiple interventions, especially in com- bination with behavior change initiatives, would also 3. For instance, these studies reported that a recent meta-analysis of five randomized controlled trials found a mean difference of 0.08 in benefit from additional research. Specifically, there height-for-age z-scores of children under age five (95 percent CI: is  a need to ensure a consistent approach to under- 0.00–0.16) for solar disinfection of water, provision of soap, and taking IEs. For example, outcome effects—and not improvements in water quality (Dangour et al. 2013). just the combined effect—should be reported for each 4. See, for instance, Pritchett and Sandefur (2013), who state the trade- intervention as this has implications for the design, off with nonexperimental estimates of treatment effects comprise a causal treatment effect and a bias term due to selectivity. When non- composition of comparison groups number and ­ experimental designs and estimates vary across contexts, any claim (treatment and control). There are grounds to suggest of external validity must make the assumptions that (a)  treatment effects are not the same across contexts, and (b) selection processes capacity-building efforts  in government implement- vary according to contexts. Therefore, parameter heterogeneity will ing agencies would lead to more reliable results, which not come from economic or institutional factors that make external also supports the argument for a better, more cohesive validity implausible. approach to conducting IEs. Clearly, more evidence is needed to support the References emerging understanding of the wider health and Cumming, O., M. Elliott, A. Overbo, and J. Bartram. 2014. “Does Global Progress on Sanitation Really Lag behind Water? An Analysis of Global social effects of WASH interventions. In summary, all Progress on Community- and Household-Level Access to Safe Water and findings seem to point to the need for larger studies, Sanitation.” PLoS One 9 (12): e114699. with broader geographical representation and rigorous Dangour, A. D., L. Watson, O. Cumming, S. Boisson, Y. Che, Y. Velleman, research methods, in addition to well-trained trained S. Cavill, E. Allen, and R. Uauy. 2013. “Interventions to Improve Water Quality and Supply, Sanitation and Hygiene Practices, and Their Effects implementing agencies. on the Nutritional Status of Children.” Cochrane Database of Systematic Reviews 8: CD009382. Notes Hutton, G., and C. Chase. 2017. “The Knowledge Base for Achieving Sustainable Development Goal Targets on Water Supply, Sanitation and 1. Conducting a systematic review and meta-analysis on a single out- Hygiene.” International Journal of Environmental Research and Public come is a time-consuming endeavor. Expanding efforts to include the Health 13 (6): 536. doi:10.3390/ijerph13060536. array of global studies evaluating a diverse set of WASH outcomes is exponentially more laborious, as illustrated by the time lag between Pritchett, L., and J. Sandefur. 2013. “Context Matters for Size: Why the commencement of this study and the publication of results. External Validity Claims and Development Practice Don’t Mix.” Working Although many reputable impact evaluations and meta-analyses Paper 336, Center for Global Development. https://www.cgdev.org/sites​ evaluating the effects of WASH interventions on single outcomes /­default/files/context-matters-for-size_0.pdf. ­ ank. Some rights ­ © 2018 International Bank for Reconstruction and Development / The World B reserved. The find- ings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, represent. The World Bank does not guarantee the accuracy its Board of Executive Directors, or the governments they ­ work. This work is subject to a CC BY ­ of the data included in this ­ (https://creativecommons.org/ 3.0 IGO license ­ /­ licenses​ ­ ontent. It is your responsibility by/3.0/igo). The World Bank does not necessarily own each component of the c ­ wner. If you have to determine whether permission is needed for reuse and to obtain permission from the copyright o pubrights@worldbank.org. questions, email ­ 4  SKU W18024