Structured Paul Fenton Villar Literature Reviews Building Transparency and Trust in Standards of Reporting Evidence IEG Methods and Evaluation Capacity Development Working Paper Series © 2022 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org ATTRIBUTION Please cite the report as: Paul Fenton Villar. 2022. Structured Literature Reviews: Building Transparency and Trust in Standards of Reporting Evidence. IEG Methods and Evaluation Capacity Development Working Paper Series. Independent Evaluation Group. Washington, DC: World Bank. MANAGING EDITORS Jos Vaessen Ariya Hagh EDITING AND PRODUCTION Amanda O’Brien GRAPHIC DESIGN Luísa Ulhoa This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The bound- aries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. RIGHTS AND PERMISSIONS The material in this work is subject to copyright. Because The World Bank encourages dissem- ination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Structured Literature Reviews Building Transparency and Trust in Standards of Reporting Evidence Paul Fenton Villar Independent Evaluation Group November 2022 CONTENTS Authors������������������������������������������������������������������������������������������������������������������������ iv Abbreviations��������������������������������������������������������������������������������������������������������������� vi Acknowledgments�����������������������������������������������������������������������������������������������������viii Abstract������������������������������������������������������������������������������������������������������������������������ x Introduction�����������������������������������������������������������������������������������������������������������������xii Structured Literature Reviews: Background and Definitions�������������������������������������� 2 Review of Evidence of the Effects of Doing Business Reforms: Case Study�������������� 8 Background 10 Review Inclusion Criteria 12 Literature Search and Analysis 14 Summary of Key Findings 16 Conclusion������������������������������������������������������������������������������������������������������������������20 Bibliography��������������������������������������������������������������������������������������������������������������� 24 Appendix A | Summary of Doing Business Interventions and Reforms��������������������30 Appendix B | Selection of Literature Reviews Related to Doing Business��������������� 34 ii  AUTHORS Paul Fenton Villar1 Corresponding Author Paul Fenton Villar: pvillar@3ieimpact.org Author Affiliation 1 Consultant, World Bank Independent Evaluation Group Independent Evaluation Group | World Bank Group  v ABBREVIATIONS 3ie International Initiative for Impact Evaluation IEG Independent Evaluation Group Independent Evaluation Group | World Bank Group  vii ACKNOWLEDGMENTS This paper provides an overview of the use of structured literature reviews to promote greater reliability in the use and reporting of evidence in evaluations. The work draws from a case study used in an evaluation of the Doing Business project conducted by the Independent Evaluation Group (IEG) of the World Bank. Andrew Stone, lead evaluation specialist, led the evaluation under the guidance of José Candido Carbajo Martinez (former director, IEG Financial, Private Sector, and Sustainable Development) and Alison Evans (Director-General, Evaluation). The case outlined the methodology and analytical protocols used to synthesize evidence related to the impact of various business regulation reforms. This paper was produced as part of the Methodological Paper Series sponsored by the Independent Evaluation Group’s Methods Advisory Function. The authors are grateful for the feedback provided by the editor and the staff of the paper series: Ariya Hagh, Michael Harrup, and Jos Vaessen. A special thanks to Alexandre Hery, Amanda O’Brien, Rafaela Sarinho, Andrew Stone, and Luísa Ulhoa for their sup- port in editing, production, and graphic design. Although many people contributed to the preparation of this paper, the findings, interpretations, and conclusions expressed are entirely those of the authors and should not be attributed in any manner to the World Bank Group, to members of its Board of Executive Directors, or to the countries they represent. Independent Evaluation Group | World Bank Group  ix ABSTRACT This paper discusses using structured literature reviews to promote transparency and trust in the use and reporting of evidence in evaluations. Structured liter- ature reviews aim to provide a summary of the most impactful, innovative, and recent research on a specified topic using systematic procedures for identifying and synthesizing studies. The paper presents a case study of an application of the method from an Independent Evaluation Group evaluation of the World Bank’s Doing Business project. It highlights the evaluation’s key lessons, which suggest that the World Bank Group’s programs need to establish strong criteria for filter- ing and reporting evidence. Doing so will help safeguard the objectivity, accuracy, and validity of the Bank Group’s research, mitigating potential reputational risks associated with insufficiently validated evidence and ensuring that findings do not affect clients and stakeholders adversely. Independent Evaluation Group | World Bank Group  xi INTRODUCTION Staying abreast of the most recent developments in research is becoming an ever more challenging task. The sheer volume of evidence available is growing each year. Therefore, identifying the studies most relevant to a specific topic or ques- tion can be difficult (Snilstveit et al. 2016).1 Most evaluators have only a limited amount of time and resources to engage critically with research. Urgency to im- plement projects or make decisions about policies often amplifies this problem. A literature review can help address such issues by surveying existing research on a given topic. Surveys of this type may serve several purposes, such as providing an overview of current understanding related to a particular issue, appraising the relevance of existing theories and evidence, providing information to form the basis of future interventions (or improve existing ones), and guiding future research (Petticrew and Roberts 2005). However, literature reviews also have well-documented limitations. A literature review may not be comprehensive, and its authors may be unbalanced in their selection of literature to discuss. Authors may cherry-pick evidence favorable to their own opinions or vested interests, raising questions about the reliability and validity of their findings (Snyder 2019; White and Waddington 2012). Scientific understanding of systematic procedures for producing reviews of evidence has improved over the past decade, and use of such procedures has increased (Waddington et al. 2012; Waddington, Masset, and Jimenez 2018). Reviews of this type aim to minimize bias by ensuring transparency in experts’ efforts to identify relevant research and report their findings (Higgins et al. 2019). Along these lines, the Independent Evaluation Group is also promoting more rigorous standards for reporting existing evidence in its evaluations, mainly by encouraging the use of structured literature reviews. Its guide to evaluation methods notes that a structured literature review is intended to determine a given topic’s state of the art using systematic and transparent procedures for synthesizing existing evidence (Vaessen, Lemire, and Befani 2020). This paper   1 Researchers following the rise in impact evaluations for interventions in low- and middle-income countries have documented a tenfold increase in the annual production of research during the past two decades (Sabet and Brown 2018). More broadly, Google Scholar (currently the largest online database of scholarly work) also includes more than 400 million academic papers and other scholarly literature (Gusenbauer 2019). Independent Evaluation Group | World Bank Group  xiii provides an overview of various concepts and methods that researchers can use to systematically summarize evidence from a large collection of literature. This is sup- ported by a case study using a literature review to summarize evidence related to an evaluation of the World Bank’s Doing Business project. Chapter 1 looks at various literature review methodologies, defining them according to their scope and the practical considerations required to implement them. Chap- ter 2 shows how to apply the approach, using the recent Independent Evaluation Group evaluation of the World Bank’s Doing Business project as a case study. Chap- ter 3 concludes with closing remarks and practical reflections on the approach. xiv Structured Literature Reviews 1 STRUCTURED LITERATURE REVIEWS: BACKGROUND AND DEFINITIONS Systematic reviews identify and synthesize all research that meets explicit eligibility criteria Rapid reviews present answers based on the best available evidence Structured literature reviews of the Independent Evaluation Group (IEG) aim to synthesize existing research on a given topic using systematic and transparent procedures. The concept and principles of a systematic review inspire and guide the approach. A systematic review aims to identify, appraise, and synthesize all relevant research that meets explicit prespecified eligibility criteria (Higgins et al. 2019).1 Because systematic reviews are intended to be exhaustive, proper implementation can require a considerable amount of time, expertise, and resources. Snilstveit et al. (2017) report an average production time of 12–24 months (depending on the scope and resources available) for a systematic review involving a multi-person expert research team.2 Comprehensively identifying all relevant studies in large volumes of literature can be a particularly challenging and time-intensive task (Thomas, Newman, and Oliver 2013). A broad and fully systematic review often requires searching a variety of liter- ature sources, screening many thousands of studies to identify those relevant to the review. For example, searchable databases do not always use a common set of terms or keywords to index literature. Even when a standardized nomenclature for describ- ing an issue or topic is available, searches on websites such as Google Scholar or the World Bank eLibrary can still yield many irrelevant results. Moreover, such terms likely do not capture the full breadth or nuance of a concept perfectly (Cantrell, Booth, and Chambers, forthcoming). These factors can make it challenging to ensure that a search is comprehensive, and review teams may have to search very large vol- umes of literature to reach saturation of coverage for the phenomenon of interest. Many systematic reviews even fall short of the mark of conducting truly exhaustive searches (as also highlighted by Evans and Popova 2016). IEG’s structured literature review approach falls into a subset of more rapid evidence reviews inspired by the concept of a systematic review. Other common terms used for these types of literature reviews include rapid reviews, rapid evidence reviews, and rapid evidence assessments (Littell 2018; Tricco et al. 2015). For the sake of consis- tency, the type of literature review evaluated in this paper will be referred to as a rapid evidence review for the remainder of the paper. The Agency for Healthcare 1   Oya, Schaefer, and Skalidou (2018; agricultural certification), Snilstveit et al. (2015; education), Vaessen et al. (2014; microcredit), and Waddington et al. (2014; farmer field schools) provide some examples of systematic reviews. 2   New technologies are decreasing the time required to complete systematic reviews. However, a systematic review still requires some scoping to adequately delineate what should (and should not) be included in its coverage before pro- duction can begin. 4 Structured Literature Reviews | Chapter 1 Research and Quality categorizes such reviews according to the extent of synthesis applied to the material covered in review (AHRQ 2015). In its categorization, inven- tories provide a list of available evidence, along with other contextual information needed to help inform decisions related to the state of research on a given subject. However, inventories do not synthesize evidence or present conclusions related to the state of the literature. Rapid responses present the user with an answer based on the best available evidence but do not attempt to formally synthesize evidence into conclusions. Rapid reviews perform a synthesis (qualitative, quantitative, or both) to provide an answer about the direction of evidence and possibly its strength. Though ostensibly less nuanced than a full systematic review, these approach- es nonetheless abide by the same basic principles as a systematic review, such as adhering to prespecified criteria for including studies and transparently reporting on the analysis of all relevant studies identified. However, the methods used may streamline the general approach and procedures used. Many rapid evidence reviews aim to deliver results within six months or less (Ganann, Ciliska, and Thomas 2010; Snilstveit et al. 2017; Varker et al. 2015). They may also use a narrower range of search techniques or sources of literature (Haby et al. 2016; Harker and Kleijnen 2012). Such restrictions ensure that the review can be delivered within shorter time- frames and resource constraints or meet deadlines required to feed into policy and decision-making processes (Varker et al. 2015; Watt et al. 2008). Given that structured literature reviews may be limited in their coverage of the literature and depth of analysis, it is important to understand the caveats associated with applying this approach. Narrow searches of wide-ranging topics and omitting methods of critical appraisal and synthesis may limit what can reliably be said about the state of the literature and research on a particular subject. However, the process- es used to conduct structured literature reviews also vary greatly (Ganann, Ciliska, and Thomas 2010; Haby et al. 2016; Hunter et al. 2020; Varker et al. 2015). Each review inevitably establishes its own set of shortcuts and heuristics, delineating the review according to a unique set of project-specific objectives. To reflect the variety of adjustments a rapid evidence review may adopt, it is im- portant to treat each one as unique. Omitting certain sources of literature, search methods, and analytical approaches will affect different reviews unequally: some shortcuts might be more or less important, representing different levels of risk of bias in different reviews. For example, some sources of literature may be thematical- ly more important regarding some research topics (as would be the case for research on health-related topics, for which searching Medline and PubMed would be intui- tively more important). Alternatively, some studies may find citation and reference tracking (a method of searching discussed in “Literature Search and Analysis” in Independent Evaluation Group | World Bank Group  5 chapter 2) more important for identifying all relevant literature in some contexts (see Cooper et al. 2018; Linder et al. 2015; Papaioannou et al. 2010; and Wright, Golder, and Rodriguez-López 2014). The appropriate choice between the two can depend on the efficacy of search strategies using key terms to identify relevant liter- ature on websites and in databases. Furthermore, some rapid evidence reviews may also have very good coverage and depth: evidence indicates that a more thorough systematic review does not neces- sarily always yield different conclusions than a more abbreviated review of the same topic (for example, AHRQ 2015; Haby et al. 2016). Hence, broad-brush statements about the rigor and limitations of rapid evidence reviews do not necessarily reflect authors’ individual work or specific applications of the approach. This chapter discussed some of the underlying concepts and terminology; the next chapter presents an example of a structured literature review based on a case study assessing the effects of the World Bank’s Doing Business report. 6 Structured Literature Reviews | Chapter 1 2 REVIEW OF EVIDENCE OF THE EFFECTS OF DOING BUSINESS REFORMS: CASE STUDY Identifying relevant inclusion criteria for selection Analyzing and synthesizing findings from relevant literature This chapter presents an example of a structured literature review from an IEG evaluation of the Doing Business project for the Committee on Development Effec- tiveness of the World Bank Group’s Board of Executive Directors. The evaluation was conducted through the first half of 2021, and IEG shared its results with the World Bank’s management group for their factual review and comment on Septem- ber 8 of that year. The review followed the release of the 2020 Doing Business report (World Bank 2020), which included a literature review of evidence on the effects of business regulation reforms. The analysis preceded Bank Group management’s announcement on September 16 that it had decided to discontinue the report (World Bank 2022).1 Despite the cancellation of the Doing Business project, the evaluation highlighted several generalizable lessons about the need to ensure standards of transparency in research. The evaluation’s structured literature review contributed to these lessons, providing a textbook example demonstrating the need to ensure transparency in standards of reporting and use of evidence to continue to build trust in the Bank Group’s research. Background Since its inception in 2003, the Bank Group’s annual Doing Business report and indicators had aimed to provide objective annual measures of business regulations and their enforcement across most of the world’s economies. The Doing Business indicators included an overall measure of the burden of business regulations, which was also disaggregated across 10 categories informing the main score: starting a business, dealing with construction permits, getting electricity, registering property, getting credit, protecting minority investors, paying taxes, trading across borders, enforcing contracts, and resolving insolvency. Each category also featured several subindicators disaggregating these themes further (World Bank 2020).2 To enable international comparisons, the indicators were based on the idea of a “standardized firm.” This required a set of assumptions about the characteristics of a firm and the market in which it operated. These assumptions elicited considerable 1   The announcement followed an external audit by the law firm WilmerHale, which highlighted data irregularities and ethical concerns about the conduct of former World Bank officials involved in the project. 2   Indicators for two additional topics concerning employing workers and contract- ing with the government also featured in the 2020 Doing Business report (World Bank 2020), but they were not included in the indicators main score. 10 Structured Literature Reviews | Chapter 2 attention and prompted debates regarding indicator methodology,3 comprehensive- ness,4 representativeness,5 accuracy,6 and potential biases.7 Nevertheless, despite these issues, the Doing Business indicators were widely cred- ited with influencing business regulations worldwide, dominating market share among business climate indicators, and directly motivating many regulatory chang- es as countries vied for improved status in the rankings the project produced (Besley 2015; Doshi, Kelley, and Simmons 2019). For example, after his election as India’s prime minister, Narendra Modi’s administration set a target of achieving a rank of at least 50th place (an improvement of more than 100 places) in the country rank- ings based on Doing Business indicator scores. Similarly, in 2012, Russian Federa- tion president Vladimir Putin set a target of placing in the ranking’s top 20 by 2018 (Besley 2015; Fenton Villar 2021; Kelley and Simmons 2020). More than 70 countries established special policy committees and working groups dedicated to improv- ing their respective Doing Business indicator scores, and the World Bank received 3   For example, Cappiello (2014) discussed methodological issues with the way the indicators counted and aggregated the time associated with complying with regulations. 4   The Independent Evaluation Group’s 2015 investment climate evaluation consid- ered the indicators incomplete because they “do not cover all areas of regula- tion as identified in the best practice list” (World Bank 2015, xxxix). Specific areas identified as missing included accounting and auditing, contract laws, competition policy, consumer protection, environmental laws, intellectual property (including privacy, copyright, patent, trademark, and unfair business practices), investment policy and promotion, employment law, labor safety, and health and alternative dispute resolution. 5   Besley (2015) observed that the standardized firm might be more relevant to the economies of some countries than others. He provided as an example a firm in the agricultural sector, which the stylized assumptions in the indicators meth- odology would represent very inadequately. 6   Holden and Pekmezovic (2020) provided numerous examples from the Pacific islands demonstrating that changes in the indicators had little relationship with changes made to policies and regulations. Hallward-Driemeier and Pritchett (2015) also highlighted that the indicators did not summarize firms’ reported experience well.   7 For example, some arguments highlighted the indicators’ potential bias toward deregulation (Besley 2015). Independent Evaluation Group | World Bank Group  11 hundreds of requests every year from officials seeking advice on reforms that could benefit their countries’ rankings (Kelley and Simmons 2020). The Doing Business project’s popularity raised the question, was there evidence that the types of regulatory reforms the Doing Business project promoted had an impact on economic and other development outcomes? This issue became particularly salient among experts who were more skeptical about the popular narrative the project portrayed concerning the effects of reforms. For example, one study showed that the correlations between the Doing Business indicators and economic outcomes were unstable (Kraay and Tawara 2013); another showed that they suffered from omitted-variable bias (Pinotti 2008). Furthermore, changes in contextual factors that may have had nothing to do with the regulatory process could affect parts of the indicators, so it was sometimes impossible to attribute changes in the index sole- ly to regulatory adjustments or the outcomes of reforms Doing Business promoted (Branstetter et al. 2014; Chemin 2009). Review Inclusion Criteria The structured literature review in the Doing Business evaluation sought to assemble an inventory of evidence on the effects of changes in regulatory arrangements Doing Business promoted. Rather than focus on evidence examining correlations between the indicators and social and economic outcomes, it concentrated on econometric studies exploiting natural experiments or variations in the implementation of specific reforms (such as the variation in the timing or region of the reform) to evaluate the reforms’ effects. In doing so, it adhered to prespecified criteria defining the charac- teristics of an includable study and the method of data (information) extraction and synthesis, and it also reported a specific search strategy. The PICOS(LY) framework (representing the population, intervention, comparison, outcome, study design, lan- guage, and year of the research endeavor) used in the review specified the character- istics of eligible studies, outlining includable populations, interventions, comparison groups, outcomes, study designs, study language, and study year (table 2.1). Overall, the review included studies written in English on interventions identified in a framework based on the World Bank projects reform database (appendix A).8 The studies could include population groups in any country and be published in   8 The Doing Business reform framework presented in appendix A does not capture all possible types of interventions that may have been under implementation in each regulatory category. Interventions outside of the immediate focus of the Doing Business project were omitted from the structured literature review. 12 Structured Literature Reviews | Chapter 2 any format (journal, book chapter, working paper, institutional report, and so on), and studies were not excluded based on the outcome variable featured in the anal- ysis. One exception concerned studies of the effects of infrastructure interventions (for example, in the “getting electricity” domain), which were limited to studies including business outcomes. This restriction reflected the consideration that such interventions might also have broader social impacts unrelated to changes in the business environment, which were beyond the scope of the review. The review also did not include studies of interventions that did not meet the assumptions under- lying the stylized firm informing the indicators. For example, reforms to special investment or tax regimes and export processing zones were excluded. Studies were included if they used one of the following study designs: nclusion and Exclusion Criteria for Empirical Studies: PI- Table 2.1. I COS(LY) Summary Table Criterion Description Population There was no limitation on population covered; studies could focus on population groups in any country. Intervention The review included studies examining regulatory reforms related to the Doing Business project. The interventions and reforms in� cluded in the review were based on a review of the Doing Business reform database (see the framework presented in appendix A). The review excluded studies that did not examine the effects of a specific intervention or reform. Comparison Control groups could be subject to no intervention, on a waiting list for intervention, or part of an alternative intervention or condition. Outcome The review did not exclude studies based on their outcome except for when related infrastructure projects were considered. Studies examining the effects of infrastructure projects (for example, in the “getting electricity” domain) were limited to business outcomes. Study design The review included quantitative studies in which allocation to intervention and control groups was random, or selection bias had been addressed by design (see chapter text). It excluded simple before-and-after comparisons and simulation and forecast models. Language The review included only studies written in English. Year There was no limitation on the year the study was conducted, but the latest search date for studies was February 2021. Source: Independent Evaluation Group. ▪ Allocation of participants to intervention and control groups using a ran- domized or quasi-randomized mechanism at individual or cluster levels. Independent Evaluation Group | World Bank Group  13 ▪ Nonrandomized designs with selection on unobservables. This included natural experiments using a sharp or fuzzy regression discontinuity design, studies using panel or pseudo-panel data with estimation strategies to ac- count for time-invariant unobservables (for example, fixed-effects models), and cross-sectional studies using multistage or multivariate approaches to account for unobservables (including instrumental variable and Heckman two-step-type estimation approaches). ▪ Nonrandomized designs with selection on observables. This included cross-sectional studies that used a method to statistically match or weight observations in the intervention and comparison groups (for example, stud- ies using a form of propensity score matching or entropy balancing). These study designs are widely recognized as capable of establishing causal rela- tionships between interventions and outcomes when executed carefully (Aloe et al. 2017; Fenton Villar and Waddington 2019; Gertler et al. 2016; Reeves, Wells, and Waddington 2017; Waddington, Fenton Villar, and Valentine, forthcoming). The comparison condition of the control group of such studies could comprise observa- tions subject to no intervention, on a waiting list for intervention, or undergoing an alternative intervention or in an alternative condition. Studies that used simulation or forecast models and case studies that did not satisfy the methodological condi- tions described here were excluded. Literature Search and Analysis Identifying a comprehensive list of studies for the Doing Business evaluation proved challenging because the indicators covered such a broad variety of aspects of the regulatory environment. The evaluation required literature from all topics cov- ered in the Doing Business indicators, and early testing of searches using key terms in databases such as Web of Science and Scopus indicated a very large volume of literature would need to be searched. Reflecting the time and resources available to complete the literature search, the review adopted an alternative approach using two databases, forward and backward citation tracking of included studies (dis- cussed later in this section), and a selection of other relevant literature reviews on reforms related to Doing Business. The search approach was not intended to be fully exhaustive, and it refined the search to a sample of studies likely to be relevant to the evaluation (regarding topic, design, or both), encompassing studies from sources that potentially may not have featured in the Doing Business reports. The search started by screening the Doing Business team’s own database on stud- ies related to the project. This provided a good indication of the studies known to 14 Structured Literature Reviews | Chapter 2 the project team. At the time of the search (December 2020 to February 2021), this database included 426 topic-specific research papers. These papers had a strong bias toward studies from outlets ranked in the top 100 economic journals and working papers in Research Papers in Economics and consisted of a broad range of literature reviews, econometric studies, and case studies related to the topics covered by the Doing Business indicators. Each study was screened according to its relevance to the structured literature review, and each study was included or excluded based on the criteria described in the previous section. After this screening, the search screened a second online evidence portal provided by the International Initiative for Impact Evaluation (3ie). This source of liter- ature was independent of the Doing Business project and therefore less likely to consist of studies that simply supported the Doing Business team’s assumptions. At the time, the 3ie portal consisted of more than 3,500 research papers identified through a long-term project searching for studies meeting the study design crite- ria discussed in the preceding section. 3ie had screened more than 150,000 records from academic databases in health, economics, public policy, and the social scienc- es listed on platforms such as Ovid, EBSCOhost, and ProQuest. It had also searched library databases and websites from select research organizations, repositories, and academic institutions (such as the World Bank, Inter-American Development Bank, African Development Bank, Social Science Research Network, and Institute of Development and Economic Alternatives). Studies from sectors irrelevant to the evaluation (for example, education and health) were excluded, resulting in 1,206 records from relevant sectors for examination (for example, industry trade and services and public administration). To supplement the screenings of the Doing Business and 3ie databases, the search reviewed the reference lists of studies selected by the inclusion criteria outlined and a selection of other relevant literature reviews on Doing Business and the related reforms (also known as backward citation tracking; see appendix B). This screening intended to identify other known studies cited in the included literature but not listed in either database. It was complemented with citation tracking using Google Scholar’s electronic citation tracking system to identify recent studies referencing any of the included studies (forward citation tracking). Overall, citation tracking iden- tified an additional 11,842 records for examination. A single coder extracted data from each study that met the inclusion criteria: bib- liographic information (such as author names, study title, year of publication, and publication status) and study characteristics (main findings, type of intervention or regulatory reform, and method of analysis). The information was reported descrip- tively, highlighting areas with clusters of evidence, and compared with the evidence reported in the literature reviews featured in the Doing Business reports. Independent Evaluation Group | World Bank Group  15 Summary of Key Findings The review identified a growing body of evidence on the effects of business regu- latory reforms Doing Business promoted, including at least 103 studies. After du- plicate records in EndNote were removed, 9,221 records remained. These records were screened for relevance based on their title and abstract, and studies with no clear relevance to the topic being reviewed were removed (for example, studies not related to business regulation or the reforms identified in appendix A). Full-text screening of the 1,894 remaining studies removed additional studies that did not specifically meet the inclusion criteria outlined in “Review Inclusion Criteria” (for example, because they did not meet the study design criteria). In final, 103 records met the inclusion criteria.9 From the evidence identified, the review found an imbalance in the treatment of theory and evidence presented in the Doing Business reports. The reports mainly cited studies favoring the reforms they promoted, which made them more of an advocacy tool than even-handed research or an effective evaluation of the issues with which they were concerned. This finding highlighted that the Doing Business project required more transparent and systematic methods for reporting the available evidence, with more attention to nuance and complexity. The review revealed three points that had received almost no attention in Doing Business’s own literature reviews: ▪ Despite the positive outcomes reported in many evaluations and empha- sized in the annual Doing Business reports, 14 of the 103 identified studies also highlighted several unintended potential (adverse) effects of regulato- ry reforms. For example, findings related to the “starting a business” topic pointed to examples of the possible negative consequences of reforms in that area for issues such as the gender pay gap and environmental outcomes (for example, Castellaneta, Conti, and Kacperczyk 2020; Wang, Chen, and Yao 2019). The evaluation noted that these potential negative consequences required further research. ▪ The Doing Business indicators implied that reforms that improved the cost and speed of complying with regulations were always beneficial, but this implication neglected the fact that many regulations were also intended to serve socially valuable purposes that such reforms might jeopardize (for ex- ample, enhancing public health, safety, and the environment and reducing   9 World Bank (2022) offers additional information on the distribution of studies across Doing Business topics. 16 Structured Literature Reviews | Chapter 2 corruption). The findings from studies relating to the “enforcing contracts” topic provided an interesting example, highlighting that in some instances, judicial reforms might have been trading off judicial efficiency for the qual- ity of judicial decisions (Kondylis and Stein 2015). ▪ Cases in which evaluations of flagship interventions reported encouraging outcomes did not indicate that all similar reforms in other countries or con- texts would have the same or similar effects. Individual findings’ external validity needed to be given greater consideration before they were gener- alized to a broad range of contexts. In some instances, regulatory reforms might have also needed to be coupled with changes in other related factors the indicators did not cover to make a meaningful or material difference. For example, the evidence identified for the “starting a business topic” dis- cussed problems related to entrepreneurs’ land tenancy rights that were un- dermining efforts to encourage formality (de Mel, McKenzie, and Woodruff 2013). Doing Business provided little to no insight into how countries could identify or complement the reforms directly promoted by a particular proj- ect with broader reform priorities. Naturally, the review of literature related to the Doing Business project faced certain limitations imposed by time and capacity restrictions, the large scope of the topic under consideration, and the nature of the studies examined (see box 2.1). None- theless, even the abbreviated review conducted here uncovered new insights and relevant findings that contributed to the assessment of the subject’s overall validity. Box 2.1.  Doing Business Evaluation: Structured Literature Review Limitations „ The structured literature review screened over 9,000 studies, but because the Doing Business topics covered such a broad area, unidentified evidence may still exist. The review did not use an exhaustive search strategy. Future research should focus on continuing to record and expand on the set of studies the review identified. However, even if the coverage of the studies included in the review was not com- prehensive, the validity of the three key findings discussed in the chapter remains undiminished. „ The analysis for the review focused on coding descriptive information about each study. Further analysis would benefit from examining the magnitude of reform effects and the determinants of effect size heterogeneity. Such analysis might consider using formal statistical methods, meta-analysis, or narrative approaches (continued) Independent Evaluation Group | World Bank Group  17  oing Business Evaluation: Structured Literature Review Box 2.1. D Limitations (cont.) to synthesis. However, it should avoid making conclusions based purely on vote counting. Because it generally involves simply adding up (counting) the number of studies with positive, negative, and null findings, vote counting does not always account for the magnitude of studies’ estimated effect sizes or their precision (Waddington et al. 2012). „ The analysis did not assess the overall strength of the evidence base or the risk of bias associated with study findings (see Moyer and Finney 2005; Sterne et al. 2016; and Waddington et al. 2012). Ideally, two independent assessors and an expert adjudicator would complete critical appraisals. Source: Independent Evaluation Group. 18 Structured Literature Reviews | Chapter 2 3 CONCLUSION Adhering to prespecified inclusion and exclusion criteria Establishing clear standards for reporting literature Synthesizing findings based on the strength and relevance of evidence Overall, key lessons identified from IEG’s evaluation of Doing Business highlight that by favoring supportive evidence and not establishing strong criteria for filtering or reporting evidence, the Doing Business reports opened the door for critics of their objectivity and accuracy, posing a reputational risk to the Bank Group and poten- tially misleading clients and stakeholders. The Doing Business experience revealed the need for mechanisms and safeguards to ensure the accuracy and validity of Bank Group reports and related communications using robust and transparent standards of evidence (World Bank 2022). Because they aim to provide more robust procedures for synthesizing existing evi- dence (Vaessen, Lemire, and Befani 2020), structured literature reviews such as the one used for the Doing Business evaluation offer one possibility for addressing some of the shortcomings identified in the Doing Business project. Establishing clear con- ventions and standards for reporting literature, as such reviews do, appears essential to safeguarding and continuing to build trust in the Bank Group’s research. More broadly, establishing systematic practices for the identification and assessment of external literature provides a methodology for robustly integrating the wealth of knowledge and research that informs the topics and themes underlying evaluations. Structured literature reviews require reviewers to outline and adhere to prespeci- fied inclusion and exclusion criteria and methods of analysis; for this reason, they are less likely to be subject to expert bias (even if unintentional) in the selection of materials to review. In addition, they are more likely to promote transparency in ex- perts’ methodology for identifying research to review and in reporting their findings. Even when reviews of this type are more rapid and thus less exhaustive than a full systematic review, they need not be any less systematic in their application (Moons, Goossens, and Thompson 2021). The methods adopted by structured literature reviews may vary considerably de- pending on the context, time, and resources afforded to an evaluation, with no detrimental effects on the reviews’ validity. This argues against tightly defining the approaches used to conduct these reviews (which will inevitably vary). However, the basic steps outlined in the summary and the case study outlined in this paper provide a blueprint for the application of more systematic approaches to reviewing literature. Future efforts could build on the many very good practices established by other specialist organizations such as Cochrane, the Campbell Collaboration, the Collaboration for Environmental Evidence, Evidence for Policy and Practice Information, and 3ie. As it stands, the guidance outlined in this paper serves as a stepping-stone for the assessment of literature in evaluations, offering a basic scaffolding that can be altered or modified according to the needs and constraints of specific evaluations. 22 Structured Literature Reviews | Chapter 3 BIBLIOGRAPHY AHRQ (Agency for Healthcare Research and Quality). 2015. “EPC Methods: An Exploration of Methods and Context for the Production of Rapid Reviews.” Research White Paper 15-EHC008-EF, Agency for Healthcare Research and Quality, Gaithersburg, MD. Aloe, A. M., B. J. Becker, M. Duvendack, J. C. Valentine, I. Shemilt, and H. Wad- dington. 2017. “Quasi-experimental Study Designs Series—Paper 9: Collect- ing Data from Quasi-experimental Studies.” Journal of Clinical Epidemiology 89: 77–83. Besley, T. 2015. “Law, Regulation, and the Business Climate: The Nature and Influence of the World Bank Doing Business Project.” Journal of Economic Perspectives 29 (3): 99–120. Branstetter, L., F. Lima, L. J. Taylor, and A. Venâncio. 2014. “Do Entry Regulations Deter Entrepreneurship and Job Creation? Evidence from Recent Reforms in Portugal.” The Economic Journal 124 (577): 805–32. Cantrell, A., A. Booth, and D. Chambers. Forthcoming. “A Systematic Review Case Study of Urgent and Emergency Care Configuration Found Citation Search- ing of Web of Science and Google Scholar of Similar Value.” Health Informa- tion and Libraries Journal. Cappiello, A. 2014. “World Bank Doing Business Project and the Statistical Meth- ods Based on Ranks: The Paradox of the Time Indicator.” Rivista Italiana di Economia, Demografia e Statistica [Italian Journal of Economic, Demographic, and Statistical Studies] LXVIII (1): 79–86. Castellaneta, F., R. Conti, and A. Kacperczyk. 2020. “The (Un)Intended Conse- quences of Institutions Lowering Barriers to Entrepreneurship: The Impact on Female Workers.” Strategic Management Journal 41 (7): 1274–1304. Chemin, M. 2009. “The Impact of the Judiciary on Entrepreneurship: Evaluation of Pakistan’s ‘Access to Justice Programme.’” Journal of Public Economics 93 (1–2): 114–25. Cooper, C., R. Lovell, K. Husk, A. Booth, and R. Garside. 2018. “Supplementary Search Methods Were More Effective and Offered Better Value Than Bib- liographic Database Searching: A Case Study from Public Health and Envi- ronmental Enhancement.” Research Synthesis Methods 9 (2): 195–223. de Mel, S., D. McKenzie, and C. Woodruff. 2013. “The Demand for, and Conse- quences of, Formalization among Informal Firms in Sri Lanka.” American Economic Journal: Applied Economics 5 (2): 122–50. Independent Evaluation Group | World Bank Group  25 Doshi, R., J. G. Kelley, and B. A. Simmons. 2019. “The Power of Ranking: The Ease of Doing Business Indicator and Global Regulatory Behavior.” International Organi- zation 73 (3): 611–43. Evans, D. K., and A. Popova. 2016. “What Really Works to Improve Learning in De- veloping Countries? An Analysis of Divergent Findings in Systematic Reviews.” World Bank Research Observer 31 (2): 242–70. Fenton Villar, P. 2021. “The Extractive Industries Transparency Initiative (EITI) and the Technical Reforms Model: Insights from the Global Performance Assess- ments Literature.” The Extractive Industries and Society 8 (4): 100963. Fenton Villar, P., and H. Waddington. 2019. “Within Study Comparisons and Risk of Bias in International Development: Systematic Review and Critical Appraisal.” Campbell Systematic Reviews 15 (1–2): e1027. Ganann, R., D. Ciliska, and H. Thomas. 2010. “Expediting Systematic Reviews: Meth- ods and Implications of Rapid Reviews.” Implementation Science 5: 56. Gertler, P. J., S. Martinez, P. Premand, L. B. Rawlings, and C. M. J. Vermeersch. 2016. Impact Evaluation in Practice. 2nd edition. Washington, DC: Inter-American Development Bank and World Bank. Gusenbauer, M. 2019. “Google Scholar to Overshadow Them All? Comparing the Siz- es of 12 Academic Search Engines and Bibliographic Databases.” Scientometrics 118: 177–214. Haby, M. M., E. Chapman, R. Clark, J. Barreto, L. Reveiz, and J. N. Lavis. 2016. “What Are the Best Methodologies for Rapid Reviews of the Research Evidence for Evidence-Informed Decision Making in Health Policy and Practice: A Rapid Review.” Health Research Policy and Systems 14 (1): 83. Hallward-Driemeier, M., and L. Pritchett. 2015. “How Business Is Done in the De- veloping World: Deals versus Rules.” Journal of Economic Perspectives 29 (3): 121–40. Harker, J., and J. Kleijnen. 2012. “What Is a Rapid Review? A Methodological Explora- tion of Rapid Reviews in Health Technology Assessments.” International Journal of Evidence-Based Healthcare 10 (4): 397–410. Higgins, J., J. Thomas, J. Chandler, M. Cumpston, T. Li, M. Page, and V. Welch, eds. 2019. Cochrane Handbook for Systematic Reviews of Interventions. 2nd edition. Chichester, UK: Wiley-Blackwell. Holden, P., and A. Pekmezovic. 2020. “How Accurate Are the Doing Business Indi- cators? A Pacific Island Case Study.” Asia and the Pacific Policy Studies 7 (3): 247–61. Hunter, J., S. Arentz, J. Goldenberg, G. Yang, J. Beardsley, M. S. Lee, and S. P. Myers. 2020. “Choose Your Shortcuts Wisely: COVID-19 Rapid Reviews of Traditional, Complementary, and Integrative Medicine.” Integrative Medicine Research 9 (3): 100484. 26  Structured Literature Reviews | Bibliography Kelley, J., and B. Simmons, eds. 2020. The Power of Global Performance Indicators. Cambridge, UK: Cambridge University Press. Kondylis, F., and M. Stein. 2015. “Reforming the Speed of Justice: Evidence from an Event Study in Senegal.” Presentation, World Bank, Washington, DC. Kraay, A., and N. Tawara. 2013. “Can Specific Policy Indicators Identify Reform Prior- ities?” Journal of Economic Growth 18: 253–83. Linder, S. K., G. R. Kamath, G. F. Pratt, S. S. Saraykar, and R. J. Volk. 2015. “Citation Searches Are More Sensitive Than Keyword Searches to Identify Studies Using Specific Measurement Instruments.” Journal of Clinical Epidemiology 68 (4): 412–17. Littell, J. H. 2018. “Conceptual and Practical Classification of Research Reviews and Other Evidence Synthesis Products.” Campbell Systematic Reviews 14 (1): 1–21. Moons, P., E. Goossens, and D. R. Thompson. 2021. “Rapid Reviews: The Pros and Cons of an Accelerated Review Process.” European Journal of Cardiovascular Nursing 20 (5): 515–19. Moyer, A., and J. W. Finney. 2005. “Rating Methodological Quality: Toward Improved Assessment and Investigation.” Accountability in Research 12 (4): 299–313. Oya, C., F. Schaefer, and D. Skalidou. 2018. “The Effectiveness of Agricultural Certi- fication in Developing Countries: A Systematic Review.” World Development 112 (December): 282–312. Papaioannou, D., A. Sutton, C. Carroll, A. Booth, and R. Wong. 2010. “Literature Searching for Social Science Systematic Reviews: Consideration of a Range of Search Techniques.” Health Information and Libraries Journal 27 (2): 114–22. Petticrew, M., and H. Roberts. 2005. Systematic Reviews in the Social Sciences: A Prac- tical Guide. Malden, MA: Blackwell Publishing. Pinotti, P. 2008. “Trust, Honesty, and Regulations.” Bank of Italy, Rome, and Univer- sitat Pompeu Fabra, Barcelona. Reeves, B. C., G. A. Wells, and H. Waddington. 2017. “Quasi-experimental Study De- signs Series—Paper 5: A Checklist for Classifying Studies Evaluating the Effects on Health Interventions—A Taxonomy without Labels.” Journal of Clinical Epide- miology 89 (September): 30–42. Sabet, S. M., and A. N. Brown. 2018. “Is Impact Evaluation Still on the Rise? The New Trends in 2010–2015.” Journal of Development Effectiveness 10 (3): 291–304. Snilstveit, B., R. Bhatia, K. Rankin, and B. Leach. 2017. “3ie Evidence Gap Maps: A Starting Point for Strategic Evidence Production and Use.” 3ie Working Paper 28, International Initiative for Impact Evaluation, London. Snilstveit, B., J. Stevenson, D. Phillips, M. Vojtkova, E. Gallagher, T. Schmidt, H. Jobse, Maisie Geelen, Maria Grazia Pastorello, and John Eyers. 2015. “Interventions for Improving Learning Outcomes and Access to Education in Low- and Middle-In- Independent Evaluation Group | World Bank Group  27 come Countries: A Systematic Review.” 3ie Systematic Review 24, International Initiative for Impact Evaluation, London. Snilstveit, B., M. Vojtkova, A. Bhavsar, J. Stevenson, and M. Gaarder. 2016. “Evidence and Gap Maps: A Tool for Promoting Evidence Informed Policy and Strategic Research Agendas.” Journal of Clinical Epidemiology 79: 120–29. Snyder, H. 2019. “Literature Review as a Research Methodology: An Overview and Guidelines.” Journal of Business Research 104 (November): 333–39. Sterne, J. A., M. A. Hernán, B. C. Reeves, J. Savović, N. D. Berkman, M. Viswanathan, D. Henry, et al. 2016. “ROBINS-I: A Tool for Assessing Risk of Bias in Non-ran- domized Studies of Interventions.” BMJ 355: i4919. Thomas, J., M. Newman, and S. Oliver. 2013. “Rapid Evidence Assessments of Re- search to Inform Social Policy: Taking Stock and Moving Forward.” Evidence and Policy: A Journal of Research Debate and Practice 9 (1): 5–27. Tricco, A. C., J. Antony, W. Zarin, L. Strifler, M. Ghassemi, J. Ivory, L. Perrier, Brian Hutton, David Moher, and Sharon E. Straus. 2015. “A Scoping Review of Rapid Review Methods.” BMC Medicine 13: 224. Vaessen, J., S. Lemire, and B. Befani. 2020. Evaluation of International Development Interventions: An Overview of Approaches and Methods. Independent Evaluation Group. Washington, DC: World Bank. Vaessen, J., A. Rivas, M. Duvendack, R. P. Jones, F. Leeuw, G. van Gils, R. Lukach, N. Holvoet, J. Bastiaensen, J. G. Hombrados, and H. Waddington. 2014. “The Effects of Microcredit on Women’s Control over Household Spending in Developing Countries: A Systematic Review and Meta-analysis.” Campbell Systematic Re- views 10 (1): 1–205. Varker, T., D. Forbes, L. Dell, A. Weston, T. Merlin, S. Hodson, and M. O’Donnell. 2015. “Rapid Evidence Assessment: Increasing the Transparency of an Emerging Methodology.” Journal of Evaluation in Clinical Practice 21 (6): 1199–1204. Waddington, H., P. Fenton Villar, and J. C. Valentine. Forthcoming. “Can Nonrandom- ized Studies of Interventions Provide Unbiased Effect Estimates? A Systematic Review of Internal Replication Studies.” Evaluation Review. Waddington, H., E. Masset, and E. Jimenez. 2018. “What Have We Learned after Ten Years of Systematic Reviews in International Development?” Journal of Develop- ment Effectiveness 10 (1): 1–16. Waddington, H., B. Snilstveit, J. Hombrados, M. Vojtkova, D. Phillips, P. Davies, and H. White. 2014. “Farmer Field Schools for Improving Farming Practices and Farmer Outcomes: A Systematic Review.” Campbell Systematic Reviews 10 (1): 4–335. Waddington, H., H. White, B. Snilstveit, J. G. Hombrados, M. Vojtkova, P. Davies, A. Bhavsar, J. Eyers, T. P. Koehlmoos, M. Petticrew, J. C. Valentine, and P. Tugwell. 28  Structured Literature Reviews | Bibliography 2012. “How to Do a Good Systematic Review of Effects in International Develop- ment: A Tool Kit.” Journal of Development Effectiveness 4 (3): 359–87. Wang, Y., S. Chen, and J. Yao. 2019. “Impacts of Deregulation Reform on PM2.5 Con- centrations: A Case Study of Business Registration Reform in China.” Journal of Cleaner Production 235 (October): 1138–52. Watt, A., A. Cameron, L. Sturm, T. Lathlean, W. Babidge, S. Blamey, K. Facey, D. Hailey, Inger Norderhaug, and G. Maddern. 2008. “Rapid Reviews versus Full Systematic Reviews: An Inventory of Current Methods and Practice in Health Technology Assessment.” International Journal of Technology Assessment in Health Care 24 (2): 133–39. White, H., and H. Waddington. 2012. “Why Do We Care about Evidence Synthesis? An Introduction to the Special Issue on Systematic Reviews.” Journal of Develop- ment Effectiveness 4 (3): 351–58. World Bank. 2015. Investment Climate Reforms: An Independent Evaluation of World Bank Group Support to Reforms of Business Regulations. Washington, DC: World Bank. World Bank. 2020. Doing Business 2020. Washington, DC: World Bank. World Bank. 2022. The Development Effectiveness of the Use of Doing Business Indica- tors, Fiscal Years 2010–20: An Independent Evaluation. Independent Evaluation Group. Washington, DC: World Bank. Wright, K., S. Golder, and R. Rodriguez-López. 2014. “Citation Searching: A System- atic Review Case Study of Multiple Risk Behaviour Interventions.” BMC Medical Research Methodology 14 (1): 73. Independent Evaluation Group | World Bank Group  29 APPENDIX A SUMMARY OF DOING BUSINESS INTERVENTIONS AND REFORMS Doing Doing Business Business Doing Business Indicators Topic Category (What Was Measured) Related Reforms and Interventions Opening a Starting a Procedures, time, cost, and paid- Reduced business registration and post-registration costs (for business business in minimum capital to start a example, licensing, permits, publication, company seal, filing limited liability company for men fees); streamlined registration and post-registration procedures and women (for example, tax registration, social security registration, licens- ing); new (or improved) electronic services; one-stop shops for registration (or post-registration procedures or both); improved public infrastructure (for example, internal information systems and staffing); minimum capital requirement abolished (or re- duced, or initial capital deposits held at a bank before registra- tion); requirements to use a corporate seal abolished; improved notary systems (or reduced notary requirements); time limits on the registration process; and fast-track business registration process Getting a Dealing with Procedures, time, and cost to Streamlined permit application procedures, stricter qualification location construction complete all formalities to build requirements for architects and engineers, improved transpar- permits a warehouse and the quality ency and public access to information, reduced fees, one-stop control and safety mechanisms shops, and new (or improved) electronic services  in the construction permitting system Getting Procedures, time, and cost to get Tariff transparency interventions, streamlined connection proce- electricity connected to the electrical grid; dures, reduced connection costs, new (or improved) systems for the reliability of the electricity regulating outages and service restoration, training programs for supply; and the transparency of utility engineers, new (or improved) electronic services, power tariffs network modernization and capacity expansion, and compensa- Independent Evaluation Group | World Bank Group  tion programs for customers affected by blackouts  31 Doing Doing Business Business Doing Business Indicators Topic Category (What Was Measured) Related Reforms and Interventions Registering Procedures, time, and cost to Streamlined registration procedures; improved transparency property transfer a property and the and provision of information on property registration (includ� quality of the land administration ing land titles); reduced taxes or fees (for example, real estate system for men and women taxes and notary, property, and sale registration fees); new (or improved) electronic services; one-stop shops; improved public infrastructure and staffing (for example, internal information systems); and fast-track registration process  Financing Getting Movable collateral laws and Unified (or modernized) collateral registry for movable property; credit credit information systems introduction of a functional, secured transactions system; ex� panded range of movable assets that can be used as collateral; granting of absolute priority to secured creditors or allowing out-of-court enforcement; establishment or expansion of credit 32  Structured Literature Reviews | Appendix A bureau (or registry); offering of credit scores by bureau or regis� try; and legal guarantees of borrowers’ right to inspect relevant data Protecting Minority shareholders’ rights in Expanded shareholders’ rights and role in major corporate minority related-party transactions and in decisions, increased disclosure requirements for related-party investors corporate governance transactions, increased director liability, clearer (or simplified) rules governing ownership and control structures, introduction of independent reviews of related-party transactions, and pro� motion of independent directors Doing Doing Business Business Doing Business Indicators Topic Category (What Was Measured) Related Reforms and Interventions Dealing with Paying Payments, time, and total tax Streamlined filing and payment procedures, new (or improved) day-to-day taxes and contribution rate for a firm to electronic services, tax rate and social contribution reductions, operations comply with all tax regulations more inclusive tax-deductible expenses, more comprehensive and post-filing processes tax depreciation rules, reduced number of taxes (or merged taxes), streamlined value-added tax refund process, and streamlined tax audit processes  Trading Time and cost to export a prod� Streamlined documentation requirements (for example, licens� across bor� uct of comparative advantage es and certificates); new (or improved) electronic services; im� ders and to import a standardized proved border infrastructure; and improved custom clearance cargo of goods by sea (or inspection) systems  Operating Enforcing Time and cost to resolve a com� Streamlined civil procedures or enforcement rules, changes in a secure contracts mercial dispute and the quality to method of case management or rules, new (or improved) business en� of judicial processes for men electronic services, introduction (or expansion) of specialized vironment and women commercial court and public personnel, introduction (or expan� sion) of alternative dispute resolution mechanism, transparency and information interventions, and introduction of private bailiff system Resolving Time, cost, outcome, and recov� Improved (or simplified) restructuring procedures and insol� insolvency ery rate for commercial insol� vency process, improved provisions on treatment of contracts vency and the strength of the during insolvency, strengthened creditor rights, increased credi� legal framework for insolvency tor participation in insolvency proceedings, time limits during insolvency procedures, mandatory qualifications for insolvency administrators, increased regulation of administrators, and spe� cialized tribunal systems  Independent Evaluation Group | World Bank Group  33 APPENDIX B SELECTION OF LITERATURE REVIEWS RELATED TO DOING BUSINESS Aboal, D., N. Noya, and A. Rius. 2014. “Contract Enforcement and Investment: A Systematic Review of the Evidence.” World Development 64 (December): 322–38. Armour, J., A. Menezes, M. Uttamchandani, and K. van Zwieten. 2015. “How Do Creditor Rights Matter for Debt Finance? A Review of Empirical Evidence.” In Research Handbook on Secured Financing in Commercial Transactions, edited by F. Dahan, 3–25. Cheltenham, UK: Edward Elgar Publishing. Bruhn, M., and D. McKenzie. 2014. “Entry Regulation and the Formalization of Micro-enterprises in Developing Countries.” World Bank Research Observer 29 (2): 186–201. Cadot, O., A. Fernandes, J. Gourdon, and A. Mattoo, eds. 2011. Where to Spend the Next Million? Applying Impact Evaluation to Trade Assistance. Washington, DC: World Bank. Canare, T., A. Ang, and R. Mendoza. 2016. “Ease of Doing Business: International Policy Experience and Evidence.” http://dx.doi.org/10.2139/ssrn.2834757. Cirmizi, E., L. Klapper, and M. Uttamchandani. 2012. “The Challenges of Bank- ruptcy Reform.” World Bank Research Observer 27 (2): 185–203. Djankov, S. 2009. “The Regulation of Entry: A Survey.” World Bank Research Ob- server 24 (2): 183–203. Floridi, A., B. Demena, and N. Wagner. 2020. “Shedding Light on the Shadows of Informality: A Meta-analysis of Formalization Interventions Targeted at Informal Firms.” Labour Economics 67 (December): 101925. Floridi, A., B. Demena, and N. Wagner. 2021. “The Bright Side of Formalization Policies! Meta-analysis of the Benefits of Policy-Induced versus Self-Induced Formalization.” Applied Economics Letters 28 (20): 1807–12. Go, E. 2018. Contribution and Effectiveness of Trade Facilitation Measures: A Struc- tured Literature Review. Independent Evaluation Group. Washington, DC: World Bank. Grimm, M., and A. Paffhausen. 2015. “Do Interventions Targeted at Microen- trepreneurs and Small and Medium-Sized Firms Create Jobs? A Systematic Review of the Evidence for Low- and Middle-Income Countries.” Labour Economics 32 (January): 67–85. Hallward-Driemeier, M., and L. Pritchett. 2015. “How Business Is Done in the Developing World: Deals versus Rules.” Journal of Economic Perspectives 29 (3): 121–40. Independent Evaluation Group | World Bank Group  35 Jayachandran, S. 2020. “Microentrepreneurship in Developing Countries.” Working Paper 26661, National Bureau of Economic Research, Cambridge, MA. Jessen, J., and J. Kluve. 2021. “The Effectiveness of Interventions to Reduce Informal- ity in Low- and Middle-Income Countries.” World Development 138 (February): 105256. Kirkpatrick, C. 2012. “Economic Governance: Improving the Economic and Reg- ulatory Environment for Supporting Private Sector Activity.” Working Paper 2012/108, World Institute for Development Economics Research, Helsinki. Klapper, L. 2011. “Saving Viable Businesses: The Effect of Insolvency Reform.” View- point: Public Policy for the Private Sector Note 328, World Bank, Washington, DC. Macdonald, K. 2012. “The Impact of Business Environment Reforms on Poverty, Gen- der, and Inclusion.” K4D Helpdesk Report 898, Institute of Development Studies, Brighton, UK. Manuel, C. 2015. “Is There a Causal Link between Investment Climate and Growth? A Review of the Evidence.” Legal Assistance for Economic Reform Program Re- search Paper, UK Department for International Development, London. McKenzie, D. 2009. “Impact Assessments in Finance and Private Sector Develop- ment: What Have We Learned and What Should We Learn?” World Bank Re- search Observer 25 (2): 209–33. Menezes, A. 2014. “Debt Resolution and Business Exit: Insolvency Reform for Credit, Entrepreneurship, and Growth.” Viewpoint: Public Policy for the Private Sector Note 343, World Bank, Washington, DC. Oto-Peralías, D., and D. Romero-Ávila. 2017. “Literature Review on the Effect of the Ease of Doing Business on Economic and Financial Outcomes.” In Legal Tra- ditions, Legal Reforms, and Economic Performance: Theory and Evidence, 37–56. Cham, Switzerland: Springer. Piza, C., T. Cravo, L. Taylor, L. Gonzalez, I. Musse, I. Furtado, A. Sierra, and S. Abdel- nour. 2016. “The Impact of Business Support Services for Small and Medium Enterprises on Firm Performance in Low- and Middle-Income Countries: A Systematic Review.” Campbell Systematic Reviews 12 (1): 1–167. Rahman, A. 2014. “Investment Climate Reforms and Job Creation in Developing Countries: What Do We Know and What Should We Do?” Policy Research Work- ing Paper 7025, World Bank, Washington, DC. Ulyssea, G. 2020. “Informality: Causes and Consequences for Development.” Annual Review of Economics 12 (August): 525–46. White, S., and P. Fortune. 2015. Business Environment Reform and Poverty: Rapid Evi- dence Assessment. London: Coffey International Development. World Bank. 2017. “What Works? Examples of Empirically Proven Justice Reforms.” World Bank, Washington, DC. 36  Structured Literature Reviews | Appendix B The World Bank 1818 H Street NW Washington, DC 20433