Policy Research Working Paper 9935 Relevance of the World Bank Group’s Early Response to COVID-19 A Cross-Country Sector Analysis Dominik Naeher Raghavan Narayanan Virginia Ziulu Independent Evaluation Group February 2022 Policy Research Working Paper 9935 Abstract Evaluating the relevance of development interventions is indicators that have been argued in the economic litera- a complex task because many different dimensions must ture to proxy vulnerability to shocks at the country level. be considered. This study focuses on one particular, quan- Comparing the vulnerability score with the sector-specific tifiable aspect of relevance and proposes a method for allocation of support yields two empirical measures of rel- generating data-driven evidence that can be used to assess evance, one at the country level and one at the sector level the relevance of past interventions and guide decisions within each country. The proposed method is designed and about future strategic priorities. For the purpose of this applied to evaluate the relevance of the World Bank Group’s study, relevance is defined as the match between the types early response to COVID-19. At the same time, many of and scopes of provided support and the types and scopes the modeling insights are more broadly applicable and may of support that are most needed in each country. The latter also be useful in informing evaluations of development pro- is estimated based on a multidimensional vulnerability grams beyond the specific application considered here. score, which is constructed using data on various empirical This paper is a product of the Independent Evaluation Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at rnarayanan1@ifc.org, dnaeher@worldbank.org, and vziulu@ifc.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Relevance of the World Bank Group’s Early Response to COVID-19: A Cross-Country Sector Analysis Dominik Naeher ∗ Raghavan Narayanan + Virginia Ziulu‡ Keywords: Policy evaluation, Relevance, Covid-19, Vulnerability, Developing countries JEL Codes: H12, I15, I18, O19, O22 ∗ Corresponding author. University of Goettingen, Goettingen, Germany (e-mail: dnaeher@uni-goettingen.de). + Independent Evaluation Group, World Bank Group, Washington, DC, United States (e-mail: rnarayanan1@ifc.org). ‡ Independent Evaluation Group, World Bank Group, Washington, DC, United States (e-mail: vziulu@ifc.org). A previous version of this paper was prepared as a background paper for the World Bank’s Independent Evaluation Group’s evaluation study “Evaluation of the World Bank Group’s Early Response in Addressing the Economic Implications of COVID-19”. We thank Marialisa Motta, Željko Bogetić, members of the evaluation team, and World Bank internal referees for valuable comments and suggestions. The views expressed in this paper are solely those of the authors and do not necessarily reflect the views of the World Bank Group. 1. Introduction Measuring the relevance of development interventions to global shocks is a complex task because many different dimensions must be considered, including not just content and context, but also timing, technical proficiency, and beneficiary targeting. As a consequence, relevance assessments in practice often have to rely on qualitative, opinion-based data obtained from expert ratings or interviews with beneficiaries. In this study, we propose a complementary approach that focuses on one particular aspect of relevance and provides quantitative, data-driven evidence that can be used to assess the relevance of past interventions and to guide decisions about future strategic priorities. Specifically, for the purpose of this study, relevance is defined as the match between the types and scopes of provided support and the types and scopes of support that are most needed during a crisis in a given country. The proposed method is designed and applied to evaluate the relevance of the World Bank Group’s (WBG) response to address the economic implications of the Covid-19 pandemic during its first year. At the same time, many of the modeling insights are more broadly applicable and may also be useful in informing evaluations of development programs beyond the specific application considered here. The method focuses on providing empirical evidence on the relevance of development interventions at two levels: across countries and within countries (sector level). First, we study the degree to which countries that were more in need of support during the onset of the Covid-19 pandemic received stronger support, measured by the amount of WBG support in response to Covid-19 received per dollar of GDP. To operationalize this approach, we construct a multidimensional vulnerability score (Need Score) using data on various empirical indicators that have been argued in the economic literature to proxy vulnerability to shocks at the country level. 1 The Need Score and the measure of the scope of support for each country are then compared to obtain an empirical measure of the relevance of the provided support across countries. Importantly, the Need Score is constructed in a way that allows for a disaggregation of each country’s score into the individual contributions of different sectors, including (1) Education, (2) Health, (3) Social protection, (4) Public finance, (5) Financial system, (6) Economic fitness, (7) Agriculture, (8) Manufacturing, and (9) Services. The resulting sector-specific Need Score thus serves as a proxy measure in capturing the relative need for support of each sector in a given country (compared to the other sectors in that country), expressed as a percentage of the total need of the country. To evaluate the relevance of support within each country, the Need Score is set in relation to another normalized measure which captures the relative intensities of WBG support targeted at each sector in a given country (mapping projects to the same sector categories as considered in the Need Score). Specifically, by assigning projects amounts of WBG support in response to Covid-19 to the same sector-specific themes underlying the constructed Need Score, we obtain a normalized measure of the relative intensities of WBG support to each sector in each country. 1 The method we use in estimating the overall vulnerability and need of each country during the Covid-19 crisis builds on existing approaches in the literature for constructing country-level vulnerability measures (Adian et al. 2020, Apedo-Amah et al. 2020, Diop et al. 2021, Hu and Zhang 2021) and is similar to the methodology used in the construction of the World Bank’s vulnerability score (Moelders 2020) and firm liquidity scores commonly produced by rating agencies. Since we are interested in assessing the relevance of development intervention, we normalize and aggregate the sector-specific vulnerability scores in a particular way that allows for a quantitative comparison with the amounts and types of provided support. 2 Quantifying the match between the sector-specific Need Scores and the sector profile of the provided support in each country using the inverse of a distance metric yields a measure of the relevance of WBG support within each country (Relevance Score). Overall, the proposed approach yields two data-driven measures of the degree of relevance of the studied development program, one focusing on the allocation of amounts of provided support across countries, and one focusing on the achievements in addressing sector-specific needs within each country. 2 We apply the method to evaluate the relevance of the WBG’s early response to Covid-19 using a sample of 187 countries, of which 133 received WBG support (including 25 LICs, 47 LMICs, 47 UMICs, and 14 HICs). The construction of composite indices of economic vulnerability or resilience—such as our proposed Need Score—is not novel and several approaches have been proposed in the context of disaster response (Acharya et al. 2020, Briguglio et al. 2009, Fatemi et al. 2017, Noy et al. 2020). More specifically, in the case of Covid-19, it has been proposed the construction of an economic stimulus index which standardizes cross-country economic responses taken by governments (Elgin et al, 2021); a vulnerability index which examines the main factors that influence the resilience of economies to Covid-19 such as quality of health care, structure of the economy, and exposure and ability to respond to shocks (Davradakis et al. 2020); a state and district level vulnerability index for India based on indicators across the socioeconomic, demographic, housing and hygiene, epidemiological, and health system domains (Acharya et al. 2020); and vulnerability and resiliency indices using a global sample of 150 countries (Diop et al. 2021). While most studies focus on the public sector and the macroeconomic effects of Covid-19, our methodology for deriving the Need Score expands on previous approaches by also incorporating private sector indicators, which allow us to capture the short and medium-term effects of the pandemic on firms (Adian et al. 2020, Apedo-Amah et al. 2020, Hu and Zhang 2021). In addition, we mainly contribute to the literature by using the Need Score as an input to a new analysis to assess the relevance of policy interventions, and by applying the proposed method to evaluate the relevance of the WBG’s early response to Covid-19. It is important to note that the analysis provides insights on one particular aspect of the relevance of the studied support in response to Covid-19, namely how project amounts were distributed across sectors with different needs within each country and across countries. However, the “true” optimal portfolio of WBG support during the first year of the pandemic likely depended on other factors as well, including government demand in client countries or the WBG’s comparative advantages when international support was coordinated with other development organizations. The analysis should therefore be seen as providing one among several possible benchmarks for evaluating the relevance of the WBG’s response to Covid-19, and should be complemented by evidence based on other reference points and criteria. The rest of the paper is structured as follows. Section 2 explains the methodology and data underlying our analysis. Section 3 presents the results and discusses how the proposed method can be used to assess the relevance of past interventions and guide decisions about future strategic priorities. Section 4 concludes. 2 As explained below, the analysis is conducted in a consistent manner in each country so that the results are comparable across countries. At the same time, the analysis accounts for differences in fundamental characteristics across countries such as differences in economic structures and income levels. 3 2. Data and Methodology This section describes the three steps underlying the analysis. The first step focuses on the construction of a sector-specific Need Score for each country. The second step develops a normalized measure of the distribution of provided support across sectors within each country. In the third step, these two measures are compared to assess the relevance of the provided support based on the match between the types and scopes of provided support and the types and scopes of support most needed in each country. Step 1: Need Score - Construction of a Sector-Specific Need Score The Need Score for each country aggregates micro (firm-level) and macro (country-level) data across several sectors of the economy into a single composite index. At the aggregated (country) level it can be thought of as a proxy measure of the overall degree to which a country’s economy was adversely affected and in need of support during the Covid-19 pandemic in 2020. In addition, the Need Score is based on a set of sector-specific indicators and constructed in a way that allows for a disaggregation of each country’s score into the individual contributions of different sectors. Specifically, a disaggregation into the following nine sectors is considered: (1) Education, (2) Health, (3) Social protection, (4) Public finance, (5) Financial system, (6) Economic fitness, (7) Agriculture, (8) Manufacturing, (9) Services. These sectors, and the empirical indicators associated with each sector, are selected based on data availability and on existing similar approaches in the literature. In addition, we verify the robustness of our main results to changes in the selection of sectors and indicators by repeating the analysis for different subsets of sectors and indicators. For this purpose, the Need Score is calculated for subsets of sectors which successively exclude each individual sector. In particular, this helps to verify that the results are not driven by a single dimension (sector) or indicator used in the construction of the Need Score. The Need Score associated with each sector is calculated based on a set of empirical indicators which capture the vulnerability of each sector in each country to the Covid-19 pandemic. Ideally, we would like to measure both (i) the degree of ex-ante vulnerability (i.e., how good, or bad each sector in each country was prepared for the impact of the pandemic and subsequently imposed measures) and (ii) the severity by which each sector in each country was hit by the Covid-19 shock, since the need for support will be determined by both these aspects. However, due to limited data availability (particularly for low- and middle-income countries), this is not always possible for the various sector dimensions considered in the construction of the Need Score. For example, within the public and social domain, data are not yet available that would allow us to estimate the sector-specific impact of Covid-19 in each country. We therefore have to rely on indicators that capture the vulnerability of the state (such as government debt, interest expense, financing needs), and of the social protection system (e.g., indicators of the magnitudes of poverty, inequality, and unemployment) more generally. For the health system, data are available both for indicators of ex-ante vulnerability (such 4 as number of hospital beds and medical staff per population) and on indicators affected by the severity of the Covid-19 shock in each country (death rate). 3 More data capturing the impact of Covid-19 are available for the private sector. Specifically, we make use of a newly released harmonized data set of variables from the World Bank Business Pulse Surveys (BPS) and the World Bank Enterprise Survey follow-up on Covid-19 (WBES). This novel data set captures over 120,000 businesses across 60 countries and measures the impact of the pandemic across several important dimensions such as sales, labor adjustments, wages, liquidity and insolvency, and operations of the business. Importantly, these variables are observed for different industries (agriculture, manufacturing, services) so that our analysis can consider each of these industries separately. Table 1 provides a complete list of the variables used in the construction of the Need Score (all these variables come from publicly available data sources). Most of the variables are observed at the country level. The variables capturing changes in business performance (sales, arrears, workers, wages) are based on firm-level survey data which are aggregated at the country level. It should be noted that the analysis does not presume that the right type of support would necessarily have increased the values of these indicators (e.g., pupil-teacher ratios). It just assumes that pupil-teacher ratios together with the other considered indicators in this domain, such as government expenditure on secondary education per student (as a percentage of GDP per capita), provide a good proxy for the vulnerability of the education system to the Covid-19 shock. In other words, the specification relies on the assumption that countries with a higher pupil-teacher ratio and lower government expenditure on secondary education per student tend to be more vulnerable to such a shock, on average. To construct the Need Score for each sector and country, the indicators in Table 1 are aggregated in the following way. First, all indicators are standardized so that higher values correspond to greater need for support. This is achieved by taking the inverse of those variables for which higher values indicate less vulnerability (e.g., financial liquidity, number of hospital beds per 1,000 people). To normalize the ranges of values across indicators, we apply min-max rescaling, which ensures that all indicators range between 0 and 1. Since this normalization method is sensitive to outliers, we winsorize all variables at the 98- percent level beforehand. The variables underlying each sector (see Table 1) are then aggregated into a single number using the mean across the respective normalized indicators for which data are available for a given country. This gives a set of nine sector-specific scores for each country. Finally, taking the weighted average across the nine sector scores for each country gives the aggregated Need Score which is used in the cross-country analysis. To account for differences in the economic structures of countries, the weights used for the industry sectors (Agriculture, Manufacturing, Services) at this step of the aggregation are based on each industry’s relative size (captured by the industry’s added value as a percentage of GDP) in each country. 4 The so-obtained Need Score ranges between 0 and 1, with 3 We avoid working with Covid-19 death and infection rates as the literature widely considers these data to be systematically biased since testing strategies and the capacity to determine the real cause of death differ widely across countries (Fisher et al. 2020; Kobilov et al. 2021; Lau et al. 2021). 4 The other sectors are assigned equal weights, i.e., the analysis assumes that health, education, social protection, public finance, financial sector, and economic fitness are equally important across countries. 5 higher values indicating greater need for support relative to other countries. The disaggregated scores for each sector also range between 0 and 1, and capture the relative need for support of each sector (compared to the other sectors within the same country) expressed as a percentage of the total need of the country. Table 1. Need Score—Variables and Data Sources Sector Indicator Description Source Education Teachers Pupil-teacher ratio, secondary World Bank Expenditure Government expenditure per student, secondary (% of GDP World Bank per capita) Enrollment School enrollment, secondary (% net) World Bank Health Hospital beds Hospital beds (per 1,000 people) World Bank Medical staff Number of physicians and nurses per 1,000 people World Bank Deaths Average 14-day notification rate of reported deaths per European Centre million population for Disease Prevention and Control Social Protection Poverty Poverty headcount ratio at $5.50 a day, 2011 PPP (% of World Bank, WDI population) Inequality Gini index (World Bank estimate) World Bank, WDI Unemployment Unemployment, total (% of total labor force) World Bank Public finance Medium-term Composite index based on various factors such as World Bank, vulnerability government gross financing needs (% GDP), debt, interest Vulnerability expense, and debt service ratio. Index Financial System Regulatory Capital Regulatory Capital to Risk-Weighted Assets (%) IMF, FSI Loans Non-performing Loans Net of Provisions to Capital (%) IMF, FSI Liquidity Liquid Assets to Total Assets (%) IMF, FSI Foreign Exchange Net Open Position in Foreign Exchange to Capital (%) IMF, FSI Economic Fitness Recession Change in the GDP growth rate in 2020 compared to the World Bank average growth rate in 2017-2019. Agriculture, Sales Percentage of establishments with decreased monthly World Bank, BPS Manufacturing, sales compared to year before and ES Services Arrears Share of establishments in arrears or expect to fall in next 6 World Bank, BPS months and ES Workers Share of establishments that fired workers in the last 30 World Bank, BPS days and ES Wages Share of establishments that reduced wages in the last 30 World Bank, BPS days and ES Agriculture share Agriculture, forestry, and fishing, value added (% of GDP) World Bank, WDI Manufacturing share Manufacturing, value added (% of GDP) World Bank, WDI Services share Services, value added (% of GDP) World Bank, WDI Notes: FSI = Financial Soundness Indicators, BPS = COVID-19 Business Pulse Survey, EPS = Enterprise Surveys, WDI = World Development Indicators. 6 Step 2: WBG Support - World Bank Group Support for Each Sector We construct another normalized measure which captures the relative intensities of WBG support targeted at each sector in a given country. For this purpose, the sector codes of 1,112 projects across 133 countries which have been identified as forming part of the WBG’s early response to Covid-19 were mapped to the same sector themes considered in the construction of the Need Score (see Table 1). 5 The considered WBG projects originally carried 86 different sector labels. Most of these could be clearly mapped to one of the nine sectors listed in Table 1 (e.g., the project label “Education” was mapped to Education, “Financial Markets” was mapped to Financial System, and “Poverty and Equity” was mapped to Social Protection). The rest of the projects were mapped based on the information contained in the project name, and in a few instances also on information obtained from additional project documents. Projects that could not be attributed to a specific country (i.e., global, or regional projects) are excluded from the analysis. The project amounts associated with each sector in a given country are then summed up and expressed as a percentage of the total amount received by the country. This gives a set of nine normalized values for each country which capture the relative amount of WBG support targeted at each sector (compared to the amounts targeted at the other sectors within each country). Step 3: Relevance Score - Quantifying the Relevance of World Bank Group Support across and within Countries The relevance of the WBG’s early response to Covid-19 is assessed at two levels, across and within countries. For the country-level analysis, we study the degree to which countries that were more in need of support during the onset of the pandemic received stronger support from the WBG. The strength of received support is measured as the ratio of the country’s total amount of received support and its GDP (i.e., the support received per dollar of GDP). 6 The underlying data consists of the project amounts of the 1,112 projects across 133 countries forming part of the WBG’s early response to Covid-19 (see above). Intuitively, the relevance of the WBG’s overall response to Covid-19 may be perceived as higher if relatively more support was allocated to countries with greater need for support during the onset of the pandemic. In line with this view, we consider the empirical relationship between countries’ total need for support (measured by their aggregated Need Score) and the amount of WBG support received per dollar of GDP. If this relationship is significantly positive, this will be interpreted as an indication of high relevance of the WBG’s support in response to Covid-19 across countries. 5 Relevant projects were identified based on the portfolio review and analysis described in IEG (2021). Specifically, the evaluation covers relevant Covid-19-related projects from the WBG portfolio approved and refinanced, reprogramed, or restructured from March 2020 through April 2021. Within the Covid-19 response portfolio identified by IEG, the evaluation focuses on interventions aimed at improving macro fiscal, financial, and real sector policies, and at increasing private investment to save livelihoods and ensure more sustainable business growth. 6 Dividing received amounts of WBG support by GDP leads to extremely high values for some small states in our sample. To avoid any biases that these outliers may create in the analysis, we sometimes exclude countries with WBG support (amount/GDP) greater than 0.1. This affects the following countries: Central African Republic, Dominica, Micronesia, Tonga, and Tuvalu. 7 Second, we assess the relevance of the WBG’s support within each country. For this purpose, the information from steps 1 and 2 is combined to quantify the degree to which the WGB’s response to Covid- 19 was focused on those sectors with the greatest need for support within each country. If the match between the sector-specific Need Scores and the allocation of WBG support is large in a given country, then this is interpreted as “high relevance” of the WBG’s support in that country (and vice-versa for countries with a smaller match). The match between need and support can be measured using the inverse of the Minkowski distance, which is defined for each country as 9 �| (% ) − (% ) |, (1) =1 with ∈ {1, … ,9} indexing the sectors listed in Table 1. To convert this distance to a measure of relevance, we divide the distance calculated for each country by the maximum distance in the sample, and subtract the resulting value from one. This yields a normalized measure of the similarity (match) between the sector-specific need scores and the sector distribution of WBG support in each country. We refer to this similarity measure as the Relevance Score. Its values range between 0 to 1, with larger values indicating higher relevance. 3. Results 3.1 Descriptive Results Figure 1 provides a first impression of the constructed sector-specific Need Scores obtained for the countries with the highest Need Scores in the sample (a complete list is provided in Table A.1 in the appendix). The length of each stacked bar indicates the country’s overall Need Score. The colored segments indicate the contribution of each sector (note that some countries have missing data for some sectors; in these cases, the overall Need Score is calculated based on the available sectors and accounting for the smaller number of sectors). Most of the countries with the highest Need Scores are in Africa, including Liberia (with a Need Score of 0.62), Libya (0.60), Guinea (0.55), Sierra Leone (0.54), and Somalia (0.53). Countries with high Need Scores from other regions include Mongolia (0.50), the Philippines (0.49), Lebanon (0.49), and India (0.48). Figure 2 depicts the ranges of the country-level Need Scores by geographical region, ordered according to the mean score. These results suggest that there are large differences in the needs caused by the pandemic both across regions and between countries within the same region. The latter applies especially to the Middle East and North Africa region (with Need Score estimates ranging from 0.18 to 0.59), Sub-Saharan Africa (0.26 to 0.62), and East Asia and Pacific (0.14 to 0.49). 8 Figure 1: Countries with the Highest Need Score Notes: The Need Score is a normalized composite measure constructed using the methodology described in Section 2. If data on the indicators used in the construction of the Need Score for a particular sector is missing, then the Need Score is calculated based on the available sectors and accounting for the smaller number of sectors. Data are for the year 2020. Source: Authors’ analysis based on various data sources. Figure 2: Ranges of Countries’ Need Score Estimates by Region Notes: The boxes represent the range of values between the 25th and 75th percentile (including the median), the ends of the whiskers represent the lower and upper adjacent value, and the dots represent outliers. Countries with WBG Support (Amount/GDP) greater than 0.1 are excluded. The Need Score is a normalized composite measure constructed using the methodology described in Section 2. Data are for the year 2020. Source: Authors’ analysis based on various data sources. 9 Figure 3: Relationship between Countries’ Income Levels and Need Score Estimates Notes: The line in the left graph represents the fitted values of a bivariate linear regression of the variable on the y-axis on the variable on the x-axis. Countries with WBG Support (Amount/GDP) greater than 0.1 are excluded. The Need Score is a normalized composite measure constructed using the methodology described in Section 2. Data are for the year 2020. Source: Authors’ analysis based on various data sources. Figure 3 shows the relationship between countries’ Need Score estimates and income levels. The graph on the left depicts the relationship between the Need Score and per-capita GDP while the graph on the right shows the ranges of the Need Score estimates (including the median) for each income group. Both graphs show a clear negative relation between countries’ Need Score and income levels, indicating that poorer countries tended to be more vulnerable and in need of support than richer countries during the Covid-19 pandemic in 2020. At the same time, it is important to note that there is large variation in the Need Score estimates for poor countries, with estimates ranging from 0.26 to 0.62 in the group of LICs. This highlights that not all poor countries suffered equally under Covid-19. In fact, there are several examples of LICs with a Need Score well below the average scores observed in higher income groups. 3.2 Relevance of World Bank Group Support across Countries Figure 4 shows the relationships between WBG Support (expressed as the total amount of support per dollar of GDP) and countries’ per-capita GDP (in the left graph) and between WBG Support and the Need Score (right graph). The negative relation between WBG support and per-capita GDP indicates that the WBG’s early support in response to Covid-19 tended to be larger (relative to the size of the benefitting countries’ economies) in poorer countries. Given that poorer countries also featured greater vulnerability (recall Figure 3), this seems to suggest that the WBG’s early response to Covid-19 benefitted countries with greater need for support relatively more. 10 Figure 4: Relationship between WBG Support and Per-capita GDP (left) and Need Score (right) Notes: The line in each graph represents the fitted values of a bivariate linear regression of the variable on the y-axis on the variable on the x-axis. High-income (non-client) countries and countries with WBG Support (Amount/GDP) greater than 0.1 are excluded. The Need Score is a normalized composite measure constructed using the methodology described in Section 2. Data are for the year 2020. Source: Authors’ analysis based on various data sources. The right graph in Figure 4 provides more direct evidence that is consistent with this idea, showing a slightly positive relation between countries Need Score and received WBG support. At the same time, there is considerable variation along this relation and many countries that feature a high Need Score received only very little (in some cases zero) support. This points to the possibility that the observed positive correlation between WBG support and Need Score was mainly driven by the fact that LICs featured both higher WBG support and greater need. In other words, it is not clear from the (unconditional) correlations shown in Figures 3 and 4 whether the WBG was indeed successful in targeting individual countries with greater need. For example, it is thinkable that the WBG’s early response to Covid-19 was generally stronger in LICs (perhaps because it was expected that poorer countries are more vulnerable to the pandemic) but other than that the WBG’s response did not support those LICs with greater need relatively more compared to other LICs. To shed more light on this question, Table 2 reports the results of a set of linear regressions featuring WBG support (per dollar of GDP) as the dependent variable and the Need Score estimates as a regressor. In addition, some specifications also include per-capita GDP and region fixed effects (FE) as regressors. The results in column (1) correspond to the regression line depicted in the right graph in Figure 4 and show that the positive slope is indeed statistically significant. Column (4) replicates the results for countries with populations of at least one million, showing that the positive correlation between WBG support and the Need Score is not merely driven by small states. However, the positive link between WBG support and Need Score disappears once differences in countries’ income levels are accounted for by the inclusion of GDP per capita as a control variable (see columns 2 and 5 of Table 2). Further adding region fixed effects does not alter this result (see columns 3 and 6). 11 Table 2. Regression Results—Dependent Variable: WBG Support Population > 1 Million (1) (2) (3) (4) (5) (6) Need Score 0.028* -0.000 -0.010 0.036*** 0.015 0.009 (0.090) (0.981) (0.654) (0.006) (0.360) (0.588) GDP per capita -0.000** -0.000* -0.000** -0.000** (0.011) (0.065) (0.032) (0.045) Region FE no no yes no no yes Observations 126 124 124 107 105 105 R-squared 0.023 0.072 0.097 0.069 0.104 0.198 Notes: * p < 0.10, ** p < 0.05, *** p < 0.01. p-values in parentheses. Estimated via OLS (some specifications include region fixed effects). High-income (non-client) countries and countries with WBG Support (Amount/GDP) greater than 0.1 are excluded. The dependent variable is the amount of WBG support received per dollar of GDP. The Need Score is a normalized composite measure constructed using the methodology described in Section 2. Source: Authors’ analysis based on various data sources. Hence, the data do not appear to support the view that the WBG’s early response to Covid-19 was targeted specifically at those countries with greater needs (i.e., beyond a general tendency for the scope of support to be higher in LICs). In other words, there was an overall tendency for WBG support to be stronger in countries with greater need, because WBG support focused on LICs, and LICs also featured greater need, on average. However, once GDP per capita is controlled for, there is no statistically significant relation anymore between countries’ Need Scores and received WBG support. This implies that within the group of LICs, WBG support was not targeted at those countries with the greatest need. This finding is robust to moderate changes in the construction of the Need Scores. Specifically, Table 3 reports the results of estimating the regressions in columns (4) and (6) of Table 2 when the Need Score is constructed based on a subset of the considered dimensions. In columns (1) and (2) of Table 3 the Need Score is constructed without the Education dimension, in columns (3) and (4), it is constructed without the Health dimension, and so on (as indicated in the table header). Across all alternative specification, the results are very similar to the baseline estimates (with all dimensions) in columns (4) and (6) of Table 2. In particular, the results in the even-numbered columns in Table 3 indicate that the finding, that WBG support was not strongly targeted at countries with greater needs, is not driven by a single dimension of the constructed Need Scores. Importantly, this result on the relevance of the WBG’s response to Covid-19 applies only to the macro level, i.e., the allocation of provided amounts of support across countries. It is still possible that, at the micro level, the WBG’s early response to Covid-19 successfully targeted those sectors featuring the greatest need within each country. The extent to which this was the case is investigated in the next subsection. 12 Table 3. Robustness of Regression Results—Dependent Variable: WBG Support No Social No Public No Financial No Economic No Education No Health Protection Finance System Fitness (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) Need Score 0.035 0.010 0.025 -0.001 0.032 0.012 0.037 0.019 0.035 0.011 0.029 -0.001 *** ** ** *** *** *** (0.009) (0.522) (0.039) (0.921) (0.015) (0.419) (0.005) (0.254) (0.004) (0.482) (0.008) (0.968) GDP per -0.000 -0.000 -0.000 -0.000 -0.000 -0.000 capita ** ** ** * * ** (0.039) (0.022) (0.047) (0.057) (0.058) (0.026) Region FE no yes no yes no yes no yes no yes no yes Observations 107 105 107 105 107 105 107 105 107 105 107 105 R-squared 0.062 0.199 0.040 0.195 0.055 0.201 0.071 0.206 0.076 0.199 0.065 0.195 Notes: * p < 0.10, ** p < 0.05, *** p < 0.01. p-values in parentheses. Estimated via OLS (some specifications include region fixed effects). High-income (non-client) countries and countries with WBG Support (Amount/GDP) greater than 0.1 are excluded. The dependent variable is the amount of WBG support received per dollar of GDP. The Need Score is a normalized composite measure constructed using the methodology described in Section 2. Source: Authors’ analysis based on various data sources. 3.3 Relevance of World Bank Group Support within Countries Figure 5 provides an overview of the Need Score and Relevance Score estimates across all 133 countries that received support from the WBG’s early response to Covid-19, including labels for the nine exemplary country case studies. 7 The chart area is split into four quadrants according to the median Need Score and median Relevance Score of the sample. Most case study countries (Cabo Verde, Pakistan, Lao PDR, Ecuador, and Nigeria) feature Need Scores and Relevance Scores above the respective median. Senegal, Georgia, and the Philippines feature Need Scores above the median and Relevance Scores below the median. Serbia is the only country case study with a Need Score below the median and a Relevance Score above the median. Figure 6 depicts the ranges of the Relevance Scores by income group and geographical region, ordered according to the respective mean scores. These results indicate that the relevance of provided WBG support was not systematically higher in some particular income groups or regions than in others. Instead, all income groups and regions feature countries with small Relevance Scores and countries with large scores. Given that the need for support tended to be systematically higher in certain regions and income groups (recall the results in Figures 2 and 3), these results also indicate that higher average vulnerability in some groups of countries did not necessarily translate into higher achieved relevance within countries belonging to these groups. 7 The country case studies were selected by IEG based on a set of criteria identified during the initial Portfolio Review and Analysis (PRA), including country vulnerability level (based on 12 economic factors), country income level, and WBG Covid response commitment during the period March 2020-Feb 2021 (for more details, see IEG 2021). The country case studies are: Ecuador, Serbia, Cabo Verde, Georgia, Lao PDR, Nigeria, Pakistan, Philippines, and Senegal. Additional countries may be considered later. 13 Figure 5: Country Need Score and Relevance Score Notes: The lines in the graph represent the median values of the variables indicated on the x-axis and on the y-axis. Only countries with support from the WBG’s early response to Covid-19 are included. The Need Score and Relevance Score are normalized values constructed using the methodology described in Section 2. Data are for the year 2020. Source: Authors’ analysis based on various data sources. Figure 6: Ranges of Countries’ Relevance Scores by Income Group and Region Notes: The boxes represent the range of values between the 25th and 75th percentile (including the median), the ends of the whiskers represent the lower and upper adjacent value, and the dots represent outliers. Countries with WBG Support (Amount/GDP) greater than 0.1 are excluded. The Relevance Score is a normalized composite measure constructed using the methodology described in Section 2. Data are for the year 2020. Source: Authors’ analysis based on various data sources. 14 Figure 7 shows the sector-specific Need Score estimates and normalized distribution of WBG support for a selected set of country case studies (note that some countries have missing data for some sectors). Each graph also reports the Relevance Score which is calculated based on comparing the distribution of WBG support (indicated by the blue bars in Figure 7) with the Need Score (orange bars) as described in Section 2 (a complete list of the Relevance Score for all countries is provided in Table A.1 in the appendix). These results give rise to several interesting insights. First, there is considerable variation in the degree to which WBG support was aligned with the sector-specific Need Scores in each country. For example, Cabo Verde and Pakistan show a strong correlation between the Need Scores and the relative magnitudes of WBG support to each sector, corresponding to Relevance Scores of 78% and 69%. Given that the values of the Relevance Scores in the full sample of 133 countries that received support from the WBG in response to Covid-19 range between 0 and 78%, this indicates a very good match between the sector-specific needs and received support in Cabo Verde and Pakistan. In contrast, the match between sector-specific needs and WBG support is considerably weaker in Georgia, the Philippines, and Senegal, where the estimated values of the Relevance Score range only between 30% and 34%. For example, the results in Figure 7 show that the WBG’s support in response to Covid-19 in Senegal focused mainly on economic fitness and social protection. However, there were no projects with support for education or public finance as their primary target, although these sectors are estimated to have featured a relatively large need (or vulnerability) compared to other sectors in Senegal. The estimated Relevance Scores appear to be relatively robust to moderate changes in the selection of considered sectors and construction of the Need Scores. Specifically, when we calculate the Relevance Scores based on different subsets of the considered sectors, successively excluding different sectors (analogous to the specifications reported in Table 3), then the resulting Relevance Scores remain comparable in magnitude to the baseline estimates (with all sectors) in Figure 7. For example, the estimates for the Need Score of Cabo Verde then range between 0.73 and 0.81 (with a baseline score of 0.78), Ecuador between 0.42 and 0.61 (0.43), Serbia between 0.54 and 0.60 (0.57), Georgia between 0.30 and 0.44 (0.30), Lao PDR between 0.43 and 0.51 (0.43), Nigeria between 0.40 and 0.57 (0.40), Pakistan between 0.66 and 0.78 (0.69), Philippines between 0.30 and 0.46 (0.31), and Senegal between 0.34 and 0.49 (0.34). 15 Figure 7: Sector-Specific Need Score and WBG Support for Selected Countries Notes: Depicted values of WBG Support, Need Score, and Relevance are normalized values constructed using the methodology described in Section 2. If data on the indicators used in the construction of the Need Score for a sector is missing, then no (orange) bar is shown for that sector. Data are for the year 2020. Source: Authors’ analysis based on various data sources. 16 Figure 7 (cont.): Sector-Specific Need Score and WBG Support for Selected Countries Notes: Depicted values of WBG Support, Need Score, and Relevance are normalized values constructed using the methodology described in Section 2. If data on the indicators used in the construction of the Need Score for a sector is missing, then no (orange) bar is shown for that sector. Data are for the year 2020. Source: Authors’ analysis based on various data sources. 3.4 Where should the WBG conduct the next 100 interventions for the recovery phase? The results obtained from the constructed Need Scores also speak to the question regarding the most relevant types of support that future interventions could provide. Specifically, the results provide some rough guidance for identifying the countries with the greatest overall vulnerability, and the sectors within those countries that exhibit the greatest need for support (see Figure 1). At the same time, we stress that the analysis captures only one particular aspect of the relevance of the WBG’s support in response to Covid-19, and thus any resulting implications for the design of future interventions should be complemented by evidence based on other reference points and criteria. Instead of targeting countries with the highest overall Need Scores, future programs might aim at supporting particular sectors within those countries that feature the greatest relative need for a particular sector. The relevant results in this case are reported in Table 4, which lists the countries with the highest 17 Need Scores for specific sectors. For example, if a future program aimed at supporting the agricultural sector in those countries featuring the greatest need in this domain, then our analysis suggests that such a program may be most relevant to Côte d'Ivoire (with a Need Score for agriculture of 0.23), the Philippines (0.20), Afghanistan (0.20), Gabon (0.18), and India (0.16). Analogously, Table 4 also lists the countries with the highest Need Scores for the other sectors. One might also argue that future interventions should aim at supporting those countries that were strongly affected by the pandemic but received only limited support from the WBG so far (in 2020). Figure 8 replicates the graph from Figure 4 showing the relationship between countries’ Need Scores and received WBG support. This time the chart area is split into four quadrants according to the median Need Score and median WBG support. Of particular interest are the countries in the lower right quadrant. These countries feature high Need Scores (above the median) but received only limited WBG support in 2020 (below the median rate of support expressed per dollar of GDP). The countries with the highest Need Scores in this group are Gabon (with a Need Score of 0.51), Angola (0.51), Zimbabwe (0.51), South Africa (0.50), Nigeria (0.49), Mongolia (0.49), the Philippines (0.49), Lebanon (0.49), India (0.48), and Bangladesh (0.47). Table 4. Countries with the Highest Need Score for each Sector Sector Country (Need Score) Education Papua New Guinea (0.32), Solomon Islands (0.36), Central African Republic (0.37), Eritrea (0.37), South Sudan (0.39) Health Benin (0.20), Somalia (0.21), Peru (0.21), Niger (0.23), Mali (0.28) Social protection Botswana (0.42), Tuvalu (0.44), Guinea-Bissau (0.45), Guyana (0.47), Micronesia (0.68) Public finance Ukraine (0.44), Iran (0.46), Belarus (0.51), Venezuela, RB (0.56), Syrian Arab Republic (0.58) Financial system Botswana (0.25), Grenada (0.26), Vietnam (0.26), Bhutan (0.27), Dominica (0.28) Economic fitness Thailand (0.34), Cabo Verde (0.34), Belize (0.35), Libya (0.42), Maldives (0.57) Agriculture India (0.16), Gabon (0.18), Afghanistan (0.20), Philippines (0.20), Côte d'Ivoire (0.23) Manufacturing Gabon (0.20), Philippines (0.20), Bangladesh (0.20), Moldova (0.21), Algeria (0.22), Mongolia (0.26) Services Morocco (0.21), Zimbabwe (0.21), Honduras (0.22), Liberia (0.23), Mongolia (0.25) Notes: The Need Score is a normalized composite measure constructed using the methodology described in Section 2. HICs are excluded. Data are for the year 2020. Source: Authors’ analysis based on various data sources. 18 Figure 8: Country Need Score and WBG Support Notes: The lines in the graph represent the median values of the variables indicated on the x-axis and on the y-axis. Only countries with support from the WBG’s early response to Covid-19 are included. The Need Score is a normalized composite measure constructed using the methodology described in Section 2. Data are for the year 2020. Source: Authors’ analysis based on various data sources. 4. Conclusion This study proposes a new method for generating data-driven evidence on the relevance of development interventions in response to global shocks. Our method is based on combining a multidimensional vulnerability score with the sector-specific allocation of provided support, and quantifying the match between the provided types of support and the types of support most needed in each country. This approach yields two empirical measures which allow to evaluate relevance both across and within countries. Applying the proposed approach to evaluate the relevance of the World Bank Group’s early response to Covid-19 generates a rich set of insights that help to demonstrate the usefulness of our method. For example, we find that, while there was a general tendency for WBG support to be stronger in poorer countries (which, on average, featured greater vulnerability than richer countries), among the groups of LICs the support was not targeted specifically at those countries with the greatest need. At the same time, there exists large variation in the estimated relevance of the provided support within countries; that is, when considering the allocation of support towards projects in those sectors of the economy featuring the greatest need for support in each country. In particular, this implies that focusing only on evaluating the relevance at the country level (based on the amounts of received support per dollar of GDP in each country) would provide only a very partial picture of how relevant the WBG’s support in response to Covid- 19 was to each country. 19 We stress again that the analysis is subject to several limitations, including those resulting from limited availability and quality of data that can be used to capture the degree to which each country was affected by the Covid-19 crisis. Moreover, our analysis focuses on providing insights on one particular aspect of relevance, namely how project amounts were distributed across sectors with different needs within and across countries. Since the relevance of development interventions comprises many more aspects than captured here, our analysis should be seen as providing only one among several possible benchmarks for evaluating the relevance of development interventions. More research on ways to assess the relevance of development interventions and guide policy decisions is clearly warranted. Our modeling insights may provide a point of departure for such efforts and inform the development of more advanced methods, including for relevance assessments in contexts beyond the specific application considered here. References Acharya, R., & Porwal, A. (2020). A vulnerability index for the management of and response to the COVID-19 epidemic in India: an ecological study. The Lancet Global Health, 8(9), e1142-e1151. Adian, I., Doumbia, D., Gregory, N., Ragoussis, A., Reddy, A., and Timmis, J. (2020). Small and Medium Enterprises in the Pandemic: Impact, Responses and the Role of Development Finance. Policy Research Working Paper 9414, Washington, DC: World Bank Group. Apedo-Amah, M. C., Avdiu, B., Cirena, X. et al. (2020). Unmasking the Impact of COVID-19 on Businesses. Firm Level Evidence from Across the World. Policy Research Working Paper 9434, Washington, DC: World Bank Group.Briguglio, L., Cordina, G., Farrugia, N., & Vella, S. (2009). Economic vulnerability and resilience: concepts and measurements. Oxford Development Studies, 37(3), 229-247. Davradakis, E., Zwart, S., Marchitto, B., & Santos, R. (2020). The EIB COVID-19 Economic Vulnerability Index: An analysis of countries outside the European Union. Diop, S., Asongu, S. A., and Nnanna, J. (2021). COVID‐19 economic vulnerability and resilience indexes: Global evidence. International Social Science Journal. Elgin, C., Basbug, G., & Yalaman, A. (2020). Economic policy responses to a pandemic: Developing the COVID-19 economic stimulus index. Covid Economics, 1(3), 40-53. Fatemi, F., Ardalan, A., Aguirre, B., Mansouri, N., & Mohammadfam, I. (2017). Social vulnerability indicators in disasters: Findings from a systematic review. International Journal of Disaster Risk Reduction, 22, 219-227. Fisher, D., Teo, Y. Y., and Nabarro, D. (2020). Assessing national performance in response to COVID-19. The Lancet, 396(10252), 653-655. Hu, S. and Zhang, Y. (2021). COVID-19 Pandemic and Firm Performance: Cross-country evidence. International Review of Economics & Finance, 74, 365-372. IEG (2021). Evaluation of the World Bank Group’s Early Response in Addressing the Economic Implications of COVID- 19: Approach Paper. Washington, DC: World Bank Group. Kobilov, B., Rouen, E., and Serafeim, G. (2021). Predictable country-level bias in the reporting of COVID-19 deaths. Journal of Government and Economics, 2(100012), 1-10. 20 Lau, H., Khosrawipour, T., Kocbach, P., Ichii, H., Bania, J., and Khosrawipour, V. (2021). Evaluating the massive underreporting and undertesting of COVID-19 cases in multiple global epicenters. Pulmonology, 27(2), 110– 115. Moelders, F. (2020). Vulnerability Heat Maps – July Update. Washington, DC: World Bank Group. Noy, I., Doan, N., Ferrarini, B., & Park, D. (2020). Measuring the Economic Risk of COVID‐19. Global Policy, 11(4), 413- 423. World Bank Group (2020). Saving Lives, Scaling-up Impact and Getting Back on Track: World Bank Group COVID-19 Crisis Response: Approach Paper. Washington, DC: World Bank Group. 21 Appendix Table A.1: Need Score, Relevance Score, and WBG Support for each Country Need Relevance WBG Support Need Relevance WBG Support Country Country Score Score (Amount/GDP) Score Score (Amount/GDP) Liberia 0.618 0.375 0.035 Suriname 0.417 0.362 0.008 Guinea 0.547 0.437 0.018 Venezuela, RB 0.416 0.500 Sierra Leone 0.538 0.555 0.018 Congo, Dem. Rep. 0.416 0.435 0.012 Somalia 0.532 0.450 0.029 Togo 0.414 0.445 0.020 Zambia 0.522 0.570 0.012 Cabo Verde 0.412 0.779 0.040 Gabon 0.512 0.011 0.001 Tunisia 0.411 0.537 0.032 Angola 0.512 0.148 0.008 Panama 0.410 0.464 0.027 Zimbabwe 0.509 0.209 0.000 Colombia 0.409 0.504 0.005 South Africa 0.501 0.448 0.001 Montenegro 0.407 0.077 0.006 Nigeria 0.498 0.402 0.005 Djibouti 0.406 0.310 0.016 Chad 0.496 0.454 0.016 Georgia 0.403 0.297 0.018 Mongolia 0.495 0.213 0.008 Senegal 0.403 0.340 0.012 Philippines 0.491 0.309 0.007 Mauritania 0.398 0.408 0.017 Lebanon 0.488 0.169 0.010 Eswatini 0.396 0.352 0.015 India 0.486 0.436 0.002 Madagascar 0.396 0.585 0.058 Niger 0.485 0.315 0.036 Côte d'Ivoire 0.395 0.215 0.013 Haiti 0.478 0.692 0.014 Timor-Leste 0.394 0.265 0.000 Mozambique 0.475 0.463 0.023 Bahamas 0.394 0.705 0.020 Bangladesh 0.467 0.509 0.006 Burundi 0.393 0.101 0.017 Malawi 0.465 0.165 0.012 Guatemala 0.392 0.260 0.001 Yemen, Rep. 0.463 0.414 0.008 Lesotho 0.392 0.141 0.006 Honduras 0.456 0.386 0.009 Fiji 0.389 0.496 0.048 Sudan 0.454 0.333 0.026 Argentina 0.388 0.674 0.004 Nepal 0.453 0.216 0.013 Nicaragua 0.386 0.259 0.002 South Sudan 0.452 0.003 Pakistan 0.385 0.694 0.006 St. Lucia 0.449 0.354 0.031 Benin 0.385 0.365 0.021 São Tomé and Príncipe 0.445 0.393 0.074 Lao PDR 0.380 0.425 0.003 Ecuador 0.442 0.429 0.017 Armenia 0.380 0.360 0.010 West Bank and Gaza 0.440 0.442 0.005 Kosovo 0.380 0.284 0.035 Ethiopia 0.431 0.311 0.028 Dominica 0.378 0.575 0.110 El Salvador 0.429 0.058 0.001 Barbados 0.378 0.343 0.025 Myanmar 0.424 0.491 0.003 Ghana 0.377 0.462 0.006 Afghanistan 0.423 0.455 0.039 Mali 0.377 0.457 0.005 Sri Lanka 0.420 0.115 0.003 Jordan 0.375 0.451 0.059 Central African Rep. 0.420 0.235 0.167 Morocco 0.372 0.537 0.013 22 Table A.1 (continued): Need Score, Relevance Score, and WBG Support for each Country Need Relevance WBG Support Need Relevance WBG Support Country Country Score Score (Amount/GDP) Score Score (Amount/GDP) St. Vincent and the 0.361 0.500 0.052 Seychelles 0.294 0.503 0.038 Grenadines Botswana 0.360 0.505 0.022 North Macedonia 0.286 0.349 0.022 The Gambia 0.359 0.706 0.055 Ukraine 0.285 0.385 0.012 Kenya 0.356 0.405 0.016 Iraq 0.285 0.177 0.002 Cameroon 0.354 0.188 0.006 Romania 0.284 0.313 0.004 Brazil 0.353 0.440 0.001 Samoa 0.278 0.427 0.065 Bolivia 0.352 0.296 0.016 Bosnia and 0.272 0.452 0.029 Herzegovina Papua New Guinea 0.352 0.285 0.005 Vietnam 0.272 0.494 0.002 Albania 0.351 0.304 0.025 Moldova 0.271 0.181 0.010 Maldives 0.349 0.293 0.028 Tajikistan 0.264 0.257 0.001 Grenada 0.347 0.327 0.025 Mauritius 0.260 0.163 0.035 Rwanda 0.346 0.560 0.031 Malaysia 0.259 0.321 0.000 Costa Rica 0.342 0.338 0.012 Belarus 0.256 0.263 0.004 Dominican Republic 0.341 0.403 0.004 Trinidad and 0.254 0.098 0.001 Tobago Mexico 0.339 0.326 0.002 Russian Federation 0.247 0.504 0.000 Burkina Faso 0.339 0.435 0.011 Croatia 0.242 0.241 0.011 Jamaica 0.336 0.371 0.012 Kazakhstan 0.238 0.042 0.000 Paraguay 0.333 0.338 0.004 Poland 0.235 0.192 0.000 Egypt, Arab Rep. 0.332 0.551 0.003 Bulgaria 0.234 0.584 0.000 Kyrgyz Republic 0.330 0.447 0.021 Serbia 0.230 0.575 0.013 Uganda 0.329 0.311 0.011 Azerbaijan 0.228 0.265 0.000 Turkey 0.324 0.454 0.002 Cambodia 0.225 0.596 0.014 Peru 0.318 0.287 0.004 Micronesia 0.220 0.088 0.116 Tanzania 0.318 0.141 0.001 Tuvalu 0.210 0.277 0.447 Chile 0.317 0.347 0.000 Thailand 0.205 0.339 0.000 Solomon Islands 0.316 0.500 0.002 Tonga 0.203 0.600 0.115 Bhutan 0.314 0.292 0.018 China 0.203 0.126 0.000 Vanuatu 0.311 0.121 0.078 Saudi Arabia 0.197 0.151 0.000 Indonesia 0.311 0.665 0.004 Uzbekistan 0.197 0.163 0.006 Uruguay 0.310 0.524 0.008 United Arab 0.196 0.379 0.000 Emirates Iran, Islamic Rep. 0.301 0.127 0.000 Malta 0.179 0.084 0.000 Comoros 0.297 0.286 0.063 23