Policy Research Working Paper 10241 A Proxy Means Test for Targeted Social Protection Programs in Sudan Alvin Etang Paul Ervin Suleiman Namara Poverty and Equity Global Practice November 2022 Policy Research Working Paper 10241 Abstract This study seeks to support the efforts of Sudan in the target- 61.1% of the population in 2015. In light of these events, ing of the Sudan Emergency Safety Nets Project (SESNP) there is a growing consensus on the need of social safety and any other future safety net programs in Sudan to those net programs in the country. The proposed program, the most in need. Sudan has faced numerous challenges for SESNP, is expected to provide unconditional cash and food several years. Since its separation from South Sudan in 2011, transfers to nearly 2 million Sudanese people (i.e., about 5% the economy has experienced macroeconomic imbalances of the population). To support this program in targeting that have lasted for over a decade. In addition, the country beneficiaries to improve the poverty impact of the program, has faced political instability, internal conflicts, and chal- we develop a Proxy Mean Tests (PMT) for Sudan based lenging climatic conditions. All of these challenges pose on the National Household Budget and Poverty Survey direct consequences to the poor. More recently, the onset (NHBPS) 2014/2015. The results indicate that the use of a of the COVID-19 pandemic and the Russia-Ukraine crisis PMT could considerably improve the program in reaching have only exacerbated the economic and social situation those most in need, while reducing expenditure towards in the country. In this context, the number of poor and those with adequate resources. This could improve both the vulnerable is expected to have increased considerably since poverty impact, as well as the sustainability of the program. the last reported official poverty rate, which accounted for This paper is a product of the Poverty and Equity Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at aetangndip@worldbank.org; paervin@gmail.com; snamara@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team A Proxy Means Test for Targeted Social Protection Programs in Sudan Alvin Etang, Paul Ervin, and Suleiman Namara1 World Bank2 Keywords: Proxy means test; poverty; consumption expenditure; welfare; targeting. JEL: I32, I38, H53. 1 Alvin Etang is a Senior Economist in the Poverty and Equity Global Practice of the World Bank; Suleiman Namara is a Lead Social Protection Specialist in the Social Protection & Jobs Global Practice of the World Bank. Paul Ervin is a Hazard Risk Economist at North Carolina Emergency Management. This paper was prepared as part of the Sudan Poverty Analytics ASA (P174168) under the overall guidance of Pierella Paci (Practice Manager, EAEPV). The authors are grateful to Franck Adoho (Senior Economist, EAWPV) for feedback and suggestions on an earlier draft. All remaining errors and omissions are the responsibility of the authors. The authors may be contacted at aetangndip@worldbank.org, paervin@gmail.com, and snamara@worldbank.org. 2On October 25, 2021 the World Bank paused disbursements in all of its existing operations in Sudan and stopped processing any new operations due to the developments in the country. These measures were adopted in accordance with the World Bank Operational Policy 7.30 “Dealing with De Facto Governments”’. The present paper is a Bank-financed product and was concluded before October 25, 2021. However, consultations with the client were not feasible due to the aforementioned pause. 1. Introduction Sudan has been experiencing multiple macroeconomic shocks, such as low growth, a wide trade deficit, and the emergence of multiple exchange rates since its separation from South Sudan in 2011. The loss of three-quarters of its oil reserves, which were located in South Sudan territory, led to a shortage of fuels and food staples, and the country became a net oil importer. Given the high fuel prices, the government implemented subsidies to protect the poor and vulnerable. However, the subsidies contributed to a further deterioration of the fiscal accounts and an increase in inflation, which has sustained macroeconomic imbalances over the last decade. The social, economic, and political discontent led to the removal of President Omar al-Bashir in April 2019 and to a transitional civilian government in September of 2019. They latter implemented an ambitious agenda of reforms, including the removal of various subsidies and unification of the exchange rate, in an effort to reverse years of structural imbalances. In this context, the civilian government deemed social safety net programs crucial to protect the poor and vulnerable. Indeed, the transitional government implemented a quasi-universal cash transfer program – the Sudan's Family Support Program (SFSP), parallel to its agenda of economic reforms. The SFSP program was specifically designed to mitigate the expected short-term adverse effects of the economic reforms on the population and to help generate political and economic space for the reforms. The SFSP –a quasi- Universal Basic Income (UBI) cash transfer program, was expected to reach 80% of the population. It consisted of delivering cash transfers to the Sudanese families in the recipient's territory expected to be affected by economic reforms and other short-term shocks. The SFSP sought to exclude the top 20% using administrative data and other sets of exclusion criteria.3 The SFSP social safety net program was paused after a military takeover in October 2021. With the onset of the COVID-19 pandemic, and the adverse economic and social effects that it brought to the country, political discontent increased in Sudan. This ultimately resulted in a military takeover in October 2021. The military government in office since October 25, 2021 has continued the efforts to achieve fiscal 3For the SFSP, beneficiary eligibility determination and enrollment modality may differ in urban and rural areas. In urban areas, program eligibility was determined through exclusion of 20 percent of households using administrative data to verify income and asset ownership. A simple set of exclusion criteria were most likely to include (i) voluntary self-exclusion; (ii) mobile phone usage of voice services above certain threshold; and (iii) holding a public service position (with the possibility for waiving this criterion for low-level public servants). In rural areas, with higher concentrations of poverty, the program considered using a geographic targeting approach to identify areas that may be universally included. In such localities, families were registered and automatically enrolled without the application of exclusion criteria, given targets of reaching 80 percent of the population. 2 consolidation and to fight inflation, however, the effects of these reforms are not expected to be seen in the short-term. Since 2019, Sudan has been experiencing soaring rates of inflation, which were partially driven by the higher energy and staple food prices (Figure 1). More recently, the Russia-Ukraine crisis that started in March 2022, has further exacerbated the country’s social and economic problems, through Sudan’s higher import costs of wheat,4 given the high dependency of the country on imported wheat and food and the country’s trade connections to Russia and Ukraine.5 In particular, Sudan is expected to face significant negative food security implications as food prices reach historic peaks due to the Russian invasion of Ukraine. All of the economic and social problems that have negatively impacted Sudan are expected to have increased the number of poor in the country, which represented about 61.1% of the population based on the latest available household survey in 2014/2015. Figure 1 – Evolution of inflation rate (%), 2019 - 2022 450 400 350 300 Percent 250 200 150 100 50 0 May-21 May-19 May-20 July-19 September-19 July-20 September-20 July-21 September-21 January-19 January-20 January-21 January-22 November-19 November-20 November-21 March-19 March-20 March-21 Inflation rate (%) Source: Authors’ elaboration based on data obtained from the Central Bureau of Statistics (CBS) of Sudan, and World Bank Currently, there are efforts in Sudan to rollout a new Social Safety Net Program, which seeks to assist the most food insecure households in the most affected states and localities, by providing unconditional cash and food transfers to beneficiary families in 11 states of Sudan. The Sudan Emergency Safety Nets Project 4 Sudan Monthly Update, March 2022, World Bank, Macroeconomics, Trade and Investment (MTI) Global Practice 5 Sudan imports 50 percent of its wheat from Russia and 4 percent from Ukraine. 3 (SESNP), for which the World Food Program is both a recipient of the grant and an implementing agency, involves two components: i) Safety Net Transfers to provide unconditional cash transfers, and ii) Delivery Systems, Monitoring and Evaluation, and Learning and Project Management. The estimated total beneficiaries are expected to be about two million people in Sudan (which is around 4.6 percent of the total population).6 Under the first component, the project will provide cash or food transfers. The cash transfers of US$7 per person per month will be provided for a period of four months to targeted vulnerable households in localities with functioning food markets. The food transfers equivalent to US$7 per person per month will be provided to households for a period of four months in localities with poorly functioning food markets. Given the limited number of beneficiaries, in this context, an efficient targeting tool will be crucial for the selection of beneficiaries of the program. The objective of this study is to support the efforts of Sudan in the targeting the Sudan Emergency Safety Nets Project (SESNP) to those most in need and to improve the implementation of the SESNP to reach its intended beneficiaries. To meet this objective, we highlight the feasibility of using a Proxy Means Test (PMT) for targeting social programs in Sudan by developing a PMT and presenting how a PMT can be used to improve program targeting to reach those most as need. A PMT is a regression-based index of observable and verifiable household characteristics that serves as a proxy for household welfare, such as consumption expenditure or income. The resulting PMT score is commonly used to determine household eligibility for public assistance programs. We focus solely on analyzing the PMT method. We note that a number of alternative targeting approaches exist, such as targeting based on self-selection (Coady and Parker 2009), nutritional risk (Reber et al. 2019), social worker evaluation (Rossell et al. 2018), multidimensional poverty (Alkire and Seth 2013), and Poverty Scorecards (Skoufias et al. 2020), as well as community-based targeting (Alatas et al. 2012; Sabates‐ Wheeler et al. 2015) and geographical targeting (Van Domelen 2007), to name a few. However, PMTs have been widely used as a targeting system in several countries, due to their efficacy at accurately targeting the poor, as well as their relatively low cost and ease of implementation compared to alternative targeting methods. 7 Furthermore, a PMT can be combined with other targeting methods based on program goals, for example nutritional risks, environmental or hazard risks, or educational risk, to identify individuals who 6As of 2020, Sudan’s population is estimated to be 43.8 million according to statistics from the World Development Indicators (WDI). Link here. 7 We direct the reader to Grosh (1994), Coady, Grosh, and Hoddinott (2004), Grosh et al. (2008), Kidd and Wylde (2011), and del Ninno and Mills (2015) for additional reviews of various program targeting approaches and the pros and cons of the PMT. 4 are most in need both in terms of resources (e.g. have a low PMT score) and program support (e.g. are at a higher risk of not meeting some program objective, such as having adequate nutritional intake). We develop a PMT for Sudan based on the latest household survey, the National Baseline Household Budget and Poverty Survey (NHBPS) 2014/2015. We first identify a set of candidate variables that best suit the country’s situation and that we believe serve as good proxies for welfare in Sudan. Based on this set of variables, we then perform statistical analyses to determine the final candidate variables to be included in the PMT. This step includes the estimation of different model specifications. Later, we evaluate the out-of- sample performance of the model across different PMT cutoff scores and use several evaluation metrics to evaluate the targeting performance of the PMT. Given the recent and rapid changes of the Sudan economy, this PMT methodology can readily be applied to new household survey data when it becomes available. Compared to social program targeting based on a uniform allocation, where everyone has the same chance of being included, the results show that a PMT should be considered for targeting social protection programs in Sudan.8 The PMT would assist social programs to reach a higher share of the poor and vulnerable, while reducing expenditure going towards households that are less in need, improving both the poverty impact, as well as the sustainability of the program. We explore various specifications of the PMT and find that our proposed Sudan national PMT performs as well as a rural and urban specific PMT, making it easier to implement and manage in practice. Furthermore, costly and time-consuming employment information does not improve the accuracy of the PMT in a meaningful way. Finally, our proposed Sudan PMT relies on about 20 variables that could be collected by a short survey in the field on a large portion of the population. The rest of the study proceeds as follows. Section 2 briefly describes Proxy Means Tests and provides an overview of program targeting considerations. Section 3 presents the data used in the study and the methodology followed. Section 4 presents the main results of the study, while Section 5 presents robustness analyses. Section 6 concludes. 2. Proxy Means Tests (PMT) and Program Targeting A PMT is an index of observable and verifiable household characteristics that serves as a proxy for household welfare, such as consumption expenditure or income. In general, a PMT can be any weighted 8 Uniform allocation is a natural counterfactual for assessing a PMT and has been used as a baseline of comparison in other PMT studies (Brown, Ravallion, and Van de Walle 2018). We discuss this further in the methodology section. 5 function of household characteristics, such as asset ownership, housing quality, education levels, etc., that produces a score used to rank households by their estimated level of welfare, when verifiable consumption or income data are not readily available and costly to obtain. Various methods exist for identifying and aggregating household characteristics to produce a PMT score (Coady, Grosh, and Hoddinott 2002; Chen and Schreiner 2009). However, it has become common to use statistical, regression-based methods to identify weights in the PMT from regression coefficients of household consumption or income using household survey data (Brown, Ravallion, and Van de Walle 2018). The PMT weights are then applied to information collected from program applicants to construct a household specific PMT score, which is an out-of-sample prediction of consumption expenditure or income, depending on the welfare variable the National Statistics Office measures. The resulting PMT score is thus an estimate of the household welfare variable, which national poverty and welfare measures are typically based. The PMT score is then used to determine household eligibility for social programs based on a predetermined cutoff (i.e., the PMT score cutoff, similar to a poverty line) and may also be used to determine the level of benefit. While universal social protection programs that cover the entire population may be desired, they are often not feasible given limited budgets, particularly in developing countries. The use of a PMT to target social programs has been shown to improve targeting incidence and reduce the inclusion of people unintended for public program support, improving program efficiency (Grosh, 1994). Furthermore, because the PMT ranks households by their estimated welfare level, the same tool can be used for targeting various welfare programs. A common targeting approach for the various social programs, based on a single PMT score, can improve consistency across programs, reduce overlap and duplication of benefits, and promote future harmonization of public programs (Sebastian et al. 2018). Due to these benefits, PMTs are now widely used in administering various social programs in numerous countries throughout the world (Grosh, 1994; Mills, Del Ninno, and Leite 2015; Brown, Ravallion, and Van de Walle 2018; Sebastian et al. 2018). Program eligibility determined by a PMT, however, is susceptible to inaccuracy, because it is an estimate or prediction of a household’s welfare. Errors may arise both by including ineligible applicants whose PMT score predicts them to be poor when in fact they are not (i.e., inclusion errors), and excluding eligible applicants whose estimated PMT score predicts them to be not poor when in fact they are (i.e., exclusion errors). These inaccuracies are due to prediction error arising primarily from unmeasured and costly to collect socioeconomic factors and idiosyncratic behavior, as well as model selection and potential measurement error in the data (Gazeaud, 2020). 6 It is generally the case that verifiable information on program applicants’ level of consumption or income will not be available due to the difficulty and cost of collecting such information. Careful data collection and statistical modeling are required to ensure an accurate prediction of household welfare. Nevertheless, due to these inaccuracies, it is important for programs to include a grievance or appeals process available to excluded program applicants who may like to provide additional information in support of their application or have their case reviewed for potential errors (Sebastian et al. 2018). Even with careful data collection and statistical modeling, the PMT score is still a prediction of a households’ actual welfare and subject to inaccuracies. The magnitude of inclusion and exclusion errors depends on the program eligibility threshold (the PMT score cutoff) and the true level of poverty rate in a country. Considering a fixed poverty rate in a country, at one extreme, a program that does not include any beneficiaries will have zero inclusion error, but an exclusion error equal to the poverty rate. Likewise, at the other extreme, a program that includes everyone will have zero exclusion error, but an inclusion error equal to percentage of the population not living in poverty. In practice, the PMT score cutoff can be selected to balance these errors based on policy and program objectives. 3. Data and Methods The development of the Sudan PMT proceeds as follows: First, we identify a set of candidate variables from household survey data that we believe could serve as a good proxy for a household’s welfare—in the case of Sudan this is household consumption expenditure. Second, we perform statistical analyses to determine a subset of variables to be used in the final PMT. In this step, we also explore different model specifications. Third, we evaluate candidate models’ out-of-sample predictions across different PMT cutoff scores using several evaluation metrics. Finally, we explore the performance of hypothetical targeting methods and transfer schemes using the final PMT. Data and Variables That data used to develop the Sudan PMT comes from the National Household Budget and Poverty Survey (NHBPS 2014/2015) conducted by the Sudan Central Bureau of Statistics.9 The survey includes modules on consumption, household characteristics, education, employment, housing, and asset ownership. The survey, representative at the national level, as well as states and rural and urban areas, includes 11,953 households. 9 This is the most recent household survey data available at the time of this study. 7 Candidate variables were selected from the NHBPS 2014/2015 based on a review of the existing literature, hypothesized correlations to consumption, and the following criteria important for the field implementation of the PMT (Coady, Grosh, and Hoddinott 2004; Brown, Ravallion, and Van de Walle 2018; Sebastian et al. 2018): 1. It should be feasible and cost-effective to collect the required data on a variable for a significant share of the population in a timely manner. 2. Each variable in the PMT should be easy to measure and observe. 3. Each variable in the PMT should be difficult for the household to manipulate. While it may be the case that surveyed household consumption is a more accurate measure of welfare, it is often difficult and costly to obtain such information from a large number of program applicants.10 Additionally, if variables in the PMT are not observable and verifiable, households may have an incentive to hide or understate their assets and wealth in order to qualify for public assistance. Based on these criteria, a broad range of candidate variables were selected from the household survey covering household head characteristics, household population characteristics, asset ownership, and employment. Table A1.1 in Annex 1 provides a list of the candidate variables included in this analysis along with their definitions. Summary statistics of for all candidate variables are presented in Table A1.2 in Annex 1.11 In this paper, we propose a PMT and show how the use of a PMT could improve program targeting. The final design of the PMT targeting system and final administration, including the final choice of variables and survey forms, should be developed in consultation with the government agency tasked with managing the targeting system. Certain variables may be more easily manipulatable or difficult to verify in different country contexts. Guided by the expected program design in Sudan and robustness analyses, we exclude state and area (e.g., rural and urban) indicator variables, as well as employment variables, since welfare programs in Sudan will be administered at the state level,12 and employment information is often difficult to obtain and potentially manipulatable by households. The impact of these decisions on the PMT are presented in Section (5). 10 For example, the purchases and consumption module in the Sudan NHBPS 2014/2015 is approximately 20 pages long and consists of hundreds of questions. 11 A small percentage of observations were missing for several variables. We explore model results for dropping missing observations and imputation. We found minor changes and high parameter stability between the two methods (Annex 2, Table A2.1). Therefore, all missing observations are imputed based on modes and means, where applicable. 12 Geographic targeting will be a first tool to identify areas with a high prevalence of potential beneficiaries, e.g., areas with high poverty rates or nutritional risks, and allocate program budget shares based on program goals. 8 Furthermore, we do not include candidate variables that are applicable to only one segment of the population, such as farming assets and crop production, that only apply to household farmers. Although we do include categorical variables on the sources of livelihood (e.g., farming, aid, wages, etc.). Finally, we avoid variables that capture a temporary state of the household, such as a recent sickness or other shock to the household. However, a PMT could be designed for specific use cases or combined with additional information to target such households, e.g., households impacted by recent hazardous weather and that have a low PMT score. A limitation of this analysis is that the 2014/15 household survey is outdated, particularly with the multiple shocks Sudan has experienced in recent years. With support from the World Bank, the Sudan Central Bureau of Statistics has started preparing a new household budget and poverty survey to update poverty and other socio-economic statistics. The survey was due to be fielded in 2022, but has been paused following the military takeover in October 2021. Survey preparations should resume once things return to normalcy. The PMT methodology presented in this paper can readily be applied to new household survey data when it becomes available. Methodology In an effort to avoid overfitting on specific in-sample characteristics of the data, we split the household survey data into a training and a testing set using a stratified random sample based on the outcome variable of log per capita household consumption expenditure (Kuhn and Johnson 2019, Chapter 3).13 All modeling decisions are based on the training data, before arriving at a small number of potential models that are applied to the testing data to explore model performance on unseen, out-of-sample information. Statistical analyses are performed using the statistical programming language R and the Tidymodels package (R Core Team 2022; Kuhn et al. 2020). To select a subset of variables from all candidate variables, we use the Least Absolute Shrinkage and Selection Operator (LASSO) estimator (Tibshirani 1996). This is done to develop a parsimonious PMT formula that can be more easily implemented in the field. The LASSO estimator selects a subset of candidate variables by minimizing the sum of squared errors conditional on a tuning parameter that penalizes the number of parameters in the model. The tuning parameter is chosen to minimize the mean squared prediction error using split sample cross validation (CV) to reduce overfitting.14 This is a primary advantage 13We use a 75% and 25% split of household survey data for the training and testing set, respectively. 14Julia Silge provides an excellent example of developing a LASSO model and tuning the penalty parameter using CV on her blog at https://juliasilge.com/blog/lasso-the-office/ 9 to the LASSO method over common stepwise variable selection methods, which may overfit on in-sample information. We then use Ordinary Least Squares (OLS) regression of the log of per capita household consumption on the subset of variables identified by the LASSO estimator to develop the PMT. This OLS post-LASSO estimator performs at least as well as the LASSO estimator, but with less bias (Belloni and Chernozhukov 2013) and has been applied in analyses of other topics, such as wage regressions (Böheim and Stöllinger 2021). We have chosen this method, because OLS is common in the PMT literature due to its ease of interpretation and implementation (Brown, Ravallion, and Van de Walle 2018; Sebastian et al. 2018). While additional statistical and machine learning methods may be able to improve prediction accuracy (McBride and Nichols 2018), many of these models are often difficult to implement and interpret, especially as their complexity grows (Petch and Nelson 2021). Furthermore, it is important that the determination of beneficiaries is defensible and transparent or there is a potential for social unrest and erosion of credibility in public administration (Cameron and Shah 2014; Brown, Ravallion, and Van de Walle 2018).15 The performance of the resulting PMT models is evaluated based on a number of accuracy criteria for PMT cutoff scores set to include the bottom 20%, 40%, and 60% of the observed household consumption distribution. These PMT cutoff scores were chosen to evaluate the PMT’s predictive ability to identify households living in a range of poverty—from approximately extreme poverty to total poverty.16 The PMT evaluation criteria include total accuracy, poverty accuracy, undercoverage, and leakage commonly found in the PMT literature (McBride and Nichols 2018; Brown, Ravallion, and Van de Walle 2018). Table 1 below presents a confusion matrix for the possible poverty prediction outcomes based on the PMT. These include true positives (TP), where the PMT correctly identifies households living in poverty; false negatives (FN), where the PMT predicts a household to be non-poor when in fact they are; false positives (FP), where households are predicted to be in poverty when in fact they are not; and true negatives (TN), where households are correctly identified as non-poor. Table 2 presents a list of the PMT evaluation criteria and their definitions. Total accuracy (TA) is the sum of correctly predicted poor (TP) and the correctly predicted non-poor (TN) as a percentage of all households in the sample (TP+FP+FN+TN). Poverty accuracy (PA) is the correctly predicted poor (TP) as a percentage 15 The PMT formula and weights may intentionally be kept secret from the public to reduce applicants’ incentives to inaccurately report household information in order to improve their chances of receiving public assistance. In this case, transparency remains important so that independent agencies may audit and verify social program targeting methods. 16 The World Bank’s Poverty & Equity Brief (Oct. 2021) reports 2014 poverty rates in Sudan at 12.2% under a $1.90 (2011 PPP) per day per capita poverty line, 44% under a $3.20 (2011 PPP) per day per capita poverty line, and 79.3% under a $5.50 (2011 PPP) per day per capita poverty line (World Bank, 2021). 10 of the true observed poor (TP+FN). Undercoverage (UC) is the incorrectly predicted non-poor (FN) as a percentage of the true observed poor (TP+FP). Lastly, leakage (LE) is the incorrectly predicted poor (FP) as a percentage of the true observed non-poor (FP+TN). Table 1. PMT Poverty Prediction Outcomes Observed Poor Non-Poor Predicted Poor True Positive (TP) False Positive (FP) Non-Poor False Negative (FN) True Negative (TN) Table 2. PMT Evaluation Metrics Evaluation Criteria Metric Definition Total Accuracy (TA) TA = (TP+TN)/(TP+FP+FN+TN) Poverty Accuracy (PA) PA = TP/(TP+FN) Undercoverage (UC) UC = FN/(TP+FN) Leakage (LE) LE = FP/(FP+TN) Additional metrics, for example inclusion error and exclusion error rates, as well as metrics that apply (normative) penalties to poverty accuracy based on leakage and undercoverage rates, such as the Balanced Poverty Accuracy Criterion (IRIS Center 2005); are common in the PMT literature (McBride and Nichols 2018; Brown, Ravallion, and Van de Walle 2018). We do not present these metrics, because they are very similar to our chosen metrics in Table 2. Finally, we note that leakage is defined in various ways in the PMT literature (McBride and Nichols 2018; Brown, Ravallion, and Van de Walle 2018). We have adopted a definition of leakage known as the false positive rate in the statistics literature. This allows to understand the relative contribution of poverty accuracy (PA) and leakage (LE) to our sample of identified beneficiaries (B), where B=N*[PA*p+LE*(1-p)], given a population (N) and a poverty rate (p). For our baseline of comparison, we compare our PMT results to a uniform allocation targeting strategy, where each household has the same probability of being targeted. Uniform allocation has been used as a baseline of comparison in other PMT studies (Brown, Ravallion, and Van de Walle 2018). To assess targeting performance under different model specifications, we compare targeting performance metrics across models. It was not possible to compare PMT targeting performance to the previous SFSP social safety net 11 allocation method, because the targeting strategy was being refined up until the program was stopped in 2021. The final PMT was chosen based on the above evaluation criteria and program considerations, namely a tradeoff between model accuracy and complexity. We find that, in general, more complex models (i.e., including more variables) only marginally improve targeting accuracy. However, the inclusion of more information raises the cost of obtaining information and administering the program. Therefore, we propose a parsimonious model (i.e., one that uses less variables) that provides high targeting accuracy. 4. Results To arrive at a final model for the Sudan PMT, we split the data into a training and testing set using a 75% training and 25% testing split. In an effort avoid overfitting on in-sample characteristics of the sample data, all modeling decisions, including any variable transformations or groupings, were based on the training data before arriving at a small number of potential models that were applied to the testing data to explore model performance on unseen, out-of-sample observations. This is done in an effort mimic how the PMT would be applied in practice and to understand out-of-sample predictive performance of the PMT. After exploring model performance on out-of-sample data, the top performing models were selected and estimated using the complete sample of data. In an effort to improve predictive accuracy of the Sudan PMT, we analyzed the raw household survey data and selected and transformed variables to better capture the relationship with per capital household consumption and the country context.17 For all categorical variables, we explored each category for intuitive groupings that were related to household income using cross-tabulations and boxplots. For continuous variables, we explored non-linear relationships and, if non-linear relationships were present, we applied transformations, so that the relationship between continuous predictors and the outcome variable of log per capita household consumption was approximately linear. Examples of these analyses are found in Annex 4. Table 3 below presents the main results of the Sudan national PMT. During the modeling process, we found that a more parsimonious model, i.e., one with less variables and parameters, performed similar to the complete model, which included all candidate variables. Thus, the final model presented below retains the 17For example, the level of schooling exhibited a stronger relationship with household consumption than years of schooling, due to grade repetitions increasing years of schooling, but not the level earned. 12 most important variables identified during the modeling process (See Table A3.1 in Annex 3 for an evaluation of the two models). Table 3. National PMT OLS Regression of Log Per Capita Household Consumption National Variables Estimate Std. Error Household size: 2 members -0.432*** (0.037) Household size: 3 members -0.700** (0.036) Household size: 4 members -0.902*** (0.036) Household size: 5 members -1.061*** (0.037) Household size: 6 members -1.181*** (0.037) Household size: 7+ members -1.379*** (0.037) Dependency ratio -0.090*** (0.015) Max. school level: less than Secondary 4 -0.027*** (0.009) Max. school level: less than Primary -0.042*** (0.01) Livelihood source: farming, aid, other -0.028*** (0.007) Head: female -0.077*** (0.009) Rooms per household member 0.055*** (0.007) House material: mud, wood, straw, tent, or incomplete -0.093*** (0.009) Floor material: earth -0.066*** (0.013) Lighting source: grass, firewood, or none -0.063*** (0.008) Drinking water source: not common network -0.029*** (0.009) Cooking fuel: cow dung, grass, or does not cook -0.134*** (0.009) No motorcycle -0.055*** (0.017) No motor vehicle -0.221*** (0.016) No radio -0.055*** (0.008) No computer -0.174*** (0.022) No refrigerator -0.073*** (0.012) No television -0.051*** (0.011) No AC -0.055*** (0.016) No mobile phone -0.147*** (0.008) Intercept 10.727*** (0.046) Obs. 11,947 Adj. R-Squared 0.579 Notes: Variable definitions and reference categories for categorical variables are presented in Table A1.1. in Annex 1. A small percentage of missing categorical and continuous observations were imputed by modes and means, respectively. *** p<0.01, ** p<0.05, * p<0.10. Furthermore, state and area (e.g., rural and urban) indicator variables, as well as employment variables, were excluded from the final model, since welfare programs in Sudan will be administered at the state 13 level, and employment information is often difficult to obtain and potentially manipulatable by households. The impact of the decision to exclude this information is explored further in Section (5). The final Sudan national PMT presented in Table 3 has an adjusted R-Squared just above 0.57, which is in the range of 0.40 to 0.65 commonly reported in other PMT studies (Brown, Ravallion, and Van de Walle 2018; Sebastian et al. 2018). The number of household members is the largest contributor to the PMT score, with larger households having a lower score. This is typical of PMT models because welfare is defined in per capita terms and, thus, household size tends to be an important predictor of welfare (Sebastian et al. 2018). Additional socioeconomic and demographic characteristics of household members contribute to the PMT score. For example, the dependency ratio, measured as the ratio of the sum of household members younger than 15 and 65-years-old and older to total household members, contributes negatively to the PMT score. Lower schooling levels among household members are also associated with a lower household PMT score and households that rely on external aid or farming have a lower PMT score. Households headed by a female also have a lower PMT score. Housing and asset ownership contribute to the household’s PMT score in expected ways. Houses with more rooms per household member have higher PMT scores, while houses made of mud, wood, and straw, as well as with dirt floors, have lower PMT scores. Lack of access to household services, such as lighting, drinking water, and improved cooking fuels, was also associated with lower PMT scores. Additionally, not having assets, such as a motorcycle, motor vehicle, radio, computer, refrigerator, television, air conditioner (AC), or mobile phone, was also associated with a lower PMT score. To evaluate the performance of the national PMT in identifying the poor for inclusion in welfare programs, we explore the PMT’s performance in identifying the bottom 20%, bottom 40% and the bottom 60% of the population. To do this, we set the PMT cutoff score equal to the observed per capita household consumption that identifies this group in the bottom of the consumption distribution, although in practice this PMT cutoff score could be set to any value based on program size and program objectives. For comparison purposes, we evaluate model performance relative to randomly drawing beneficiaries from the sample of households surveyed in the NHBPS 2014/2015.18 Table 4 below presents targeting results based on a random sample of household beneficiaries. In this case each household has a probability of being included in the social program equal to the household poverty 18It may be the case that in practice a program not using a PMT could do better than randomly admitting applicants from the population, but a program could also do worse by admitting wealthy applicants not intended for the program. 14 rate. Therefore, poverty accuracy and leakage will approximately be equal to the household poverty rate used to determine beneficiaries. Table 4. Evaluation of a National Random Sample (Uniform Allocation Strategy for Baseline of Comparison) Poverty Line / PMT Cutoff Evaluation Criteria Bottom 20% Bottom 40% Bottom 60% Poverty headcount (observed) 20.00% 40.02% 60.00% Household poverty rate (observed) 19.08% 37.20% 55.50% Household poverty rate (predicted) 19.42% 37.31% 55.53% Total accuracy 68.64% 53.35% 50.62% Poverty accuracy 18.69% 37.44% 55.54% Undercoverage 81.31% 62.56% 44.46% Leakage 19.59% 37.23% 55.52% Notes: The predicted household poverty rate is determined by randomly selecting a proportion of households equal to the observed household poverty rate. Table 5 presents targeting results based on the Sudan national PMT. Here program beneficiaries are determined by the PMT and a PMT cutoff score based on the observed household consumption that identifies the bottom percent of the distribution. We see that choosing this poverty line often underpredicts true poverty at low levels and overpredicts at higher levels. However, compared to randomly accepting beneficiaries, we see that poverty accuracy and leakage are significantly improved. For example, poverty accuracy with the PMT increases by over 15 to 30 percentage points depending on the targeted population living in poverty, with larger increases in performance associated with a larger population living in poverty and/or eligible for the social program. Likewise, leakage falls by a similar magnitude, again depending on the targeted population living in poverty. However, in this case, leakage increases with stricter PMT score cutoffs. It is also worth noting that in a random sample the total accuracy is much lower (50% to 68%) to the total accuracy of the PMT, which ranges between 78% and 83% over various PMT cutoff scores. Table 6 presents the results of a rural and urban PMT. The same modeling process as the national PMT was used to create individual rural and urban PMT models. The results are similar for both rural and urban areas. The urban model has a higher adjusted R-squared (0.65) than the rural model (0.52), although parameter estimates between the two models do not vary much where they coincide. The main difference between the rural and urban models is that several variables, conditional on the other variables, were estimated to not be statically significantly related to the PMT score in one of the areas. 15 Table 5. Evaluation of the National PMT (OLS Regression) Poverty Line / PMT Cutoff Evaluation Criteria Bottom 20% Bottom 40% Bottom 60% Poverty headcount (observed) 20.00% 40.02% 60.00% Household poverty rate (observed) 19.08% 37.20% 55.50% Household poverty rate (predicted) 11.38% 37.01% 58.92% Total accuracy 83.19% 78.29% 80.25% Poverty accuracy 35.76% 70.57% 85.29% Undercoverage 64.24% 29.43% 14.71% Leakage 5.63% 17.14% 26.03% Notes: The poverty line and PMT cutoff score are set at the observed consumption expenditure that identifies the bottom 20%, bottom 40%, and bottom 60% of the population. The PMT cutoffs are the log of these values. For example, in urban areas, the gender of the household head, waste disposal, motorcycle ownership, and television ownership did not exhibit a strong relationship with per capita household income. While in rural areas, the age and marital status of the household head, floor material, and ownership of an air conditioner or satellite dish was not strongly related to per capita household income. Therefore, these variables were omitted from their respective PMTs. To explore if an area specific PMT improves targeting performance, we evaluate the targeting performance of rural and urban specific models compared to the national model applied to rural and urban areas. Table 6 presents the rural and urban targeting evaluation by model. The total accuracy, poverty accuracy, and leakage of the models are nearly identical. The main difference in the models appears to be in the improved poverty accuracy of the urban model in identifying the bottom 20%, with a poverty accuracy of 47% compared to 40%. However, because the target population (the bottom 20%) is relatively small, the model performance is minimal in absolute number terms. For example, in a sample of 100 households, if 20 are poor (20%), then the urban model would identify 9 people (20*0.47) as poor versus 8 people (20*0.40) using the national model. Likewise, the slightly larger leakage of the urban model would result in including nearly an extra non-poor in the program per 100 people in the population, because the non-poor comprises a large portion of the sample (the top 80%). 16 Table 6. PMT OLS Regression of the Log of Per Capita Household Consumption by Area Rural Urban Variables Estimate Std. Error Estimate Std. Error Household size: 2 members -0.442*** (0.041) -0.491*** (0.082) Household size: 3 members -0.721*** (0.041) -0.727*** (0.081) Household size: 4 members -0.93*** (0.041) -0.908*** (0.081) Household size: 5 members -1.08*** (0.042) -1.086*** (0.081) Household size: 6 members -1.203*** (0.042) -1.198*** (0.081) Household size: 7+ members -1.395*** (0.042) -1.397*** (0.081) Dependency ratio -0.092*** (0.019) -0.107*** (0.025) Max. school level: less than Secondary 4 -0.028** (0.012) -0.022* (0.014) Max. school level: less than Primary -0.041*** (0.012) -0.046*** (0.016) Livelihood source: farming, aid, other -0.03*** (0.009) -0.027* (0.015) Head: female -0.093*** (0.012) Head: age -0.001*** (0.000) Head: not married -0.067*** (0.016) Rooms per household member 0.033*** (0.009) 0.121*** (0.012) House material: mud, wood, straw, tent, or incomplete -0.079*** (0.013) -0.099*** (0.012) Floor material: earth -0.06*** (0.015) Lighting source: grass, firewood, or none -0.075*** (0.01) -0.088*** (0.017) Drinking water source: not common network -0.032** (0.013) -0.027** (0.013) Cooking fuel: cow dung, grass, or does not cook -0.155*** (0.012) -0.098*** (0.016) Waste disposal: heap, burning, other -0.067*** (0.017) No motorcycle -0.064*** (0.023) No motor vehicle -0.255*** (0.023) -0.184*** (0.022) No radio -0.07*** (0.01) -0.027** (0.012) No computer -0.152*** (0.049) -0.178*** (0.023) No refrigerator -0.054*** (0.018) -0.096*** (0.015) No television -0.039*** (0.015) No AC -0.074*** (0.018) No mobile phone -0.145*** (0.01) -0.139*** (0.019) No satellite dish -0.052*** (0.015) Intercept 10.697*** (0.07) 10.779*** (0.09) Obs. 8,357 3,590 Adj. R-Squared 0.523 0.652 Notes: Variable definitions and reference categories for categorical variables are presented in Table A1.1. in Annex 1. A small percentage of missing categorical and continuous observations imputed by modes and means, respectively. *** p<0.01, ** p<0.05, * p<0.10. 17 Table 7. Evaluation of Different Models by Area (OLS Regression) Performance by Area using the National Model Rural Urban Poverty Line / PMT Cutoff Poverty Line / PMT Cutoff Bottom Bottom Bottom Bottom Bottom Bottom Evaluation Criteria 20% 40% 60% 20% 40% 60% Poverty headcount (observed) 20.00% 39.97% 59.99% 20.04% 40.02% 59.95% Household poverty rate (observed) 18.89% 37.11% 55.14% 16.77% 34.96% 53.82% Household poverty rate (predicted) 8.50% 37.56% 59.51% 9.58% 32.79% 56.52% Total accuracy 82.18% 76.33% 77.83% 87.19% 80.84% 81.20% Poverty accuracy 25.33% 68.72% 83.85% 40.37% 69.48% 85.04% Undercoverage 74.67% 31.28% 16.15% 59.63% 30.52% 14.96% Leakage 4.57% 19.18% 29.58% 3.38% 13.06% 23.28% Model Performance by Area using Rural and Urban Models Rural Urban Poverty Line / PMT Cutoff Poverty Line / PMT Cutoff Bottom Bottom Bottom Bottom Bottom Bottom Evaluation Criteria 20% 40% 60% 20% 40% 60% Poverty headcount (observed) 20.00% 39.97% 59.99% 20.04% 40.02% 59.95% Household poverty rate (observed) 18.89% 37.11% 55.14% 16.77% 34.96% 53.82% Household poverty rate (predicted) 7.78% 37.12% 59.90% 11.75% 34.09% 56.49% Total accuracy 82.23% 76.63% 78.08% 87.35% 81.03% 81.84% Poverty accuracy 23.56% 68.53% 84.44% 47.34% 71.63% 85.61% Undercoverage 76.44% 31.47% 15.56% 52.66% 28.37% 14.39% Leakage 4.10% 18.59% 29.74% 4.59% 13.92% 22.56% Notes: The poverty line and PMT cutoff score are set at the observed consumption expenditure that identifies the bottom 20%, bottom 40%, and bottom 60% of the population. These values are 3,181, 4,173, 5,278 SDG in rural areas and 3,958, 5,259, 6,734 SDG in urban areas. The PMT cutoffs are the log of these values. The national model is presented in Table 3 and the rural and urban models are presented in Table 6. The results show that a PMT could help social programs in Sudan reach a higher share of the poor and vulnerable, while reducing expenditure going towards households that are less in need. The national PMT, which relies on 19 variables, performs nearly as well as a rural and urban specific PMT, making it easier to implement and manage in practice. Additionally, due to requiring less information than individual rural and urban models, the information needed for the national PMT could more easily and rapidly be collected by a short survey in the field on a large portion of the population. 5. Robustness To develop the PMT a number of decisions have to be made on the statistical model to be used and the variables to be included. In this section, we explore how different models and variables effect the targeting 18 performance of the PMT. We explore a binary dependent variable model, which models the probability of a discrete outcome, in this case household poverty status. Then we explore, several specifications that add a complete set of state and rural and urban indicator variables (i.e., dummy variables), as well as employment information, in the form of the proportion of number of household members who have ever worked for income to the total household members. Table 7 presents the evaluation metrics of the PMT using a binary dependent variable logistic model compared to the OLS regression model used in Section 4. 19 For comparison purposes, the PMT cutoff score for the logistic model was set so that the predicted household poverty rate was similar under both models. The results for total accuracy, poverty accuracy, undercoverage, and leakage, are very similar and always within 2 percentage points. Therefore, it does not appear to make much difference whether OLS or logistic regression is used to develop the PMT on targeting performance. However, in practice, OLS is easier to implement and interpret due to its linear functional form. Table 8. Evaluation of a Binary Dependent Variable Model (Logistic Regression) Binary Dependent Variable (Poverty Status) Model versus OLS Regression of Household Per Capita Consumption Logistic Model OLS Model Poverty Line / PMT Cutoff Poverty Line / PMT Cutoff Bottom Bottom Bottom Bottom Bottom Bottom Evaluation Criteria 20% 40% 60% 20% 40% 60% Poverty headcount (observed) 19.96% 39.95% 59.89% 19.96% 39.95% 59.89% Household poverty rate (observed) 19.36% 37.59% 54.92% 19.36% 37.59% 54.92% Household poverty rate (predicted) 12.88% 39.40% 59.40% 12.81% 38.76% 59.60% Total accuracy 83.41% 78.39% 80.00% 83.01% 77.89% 79.26% Poverty accuracy 40.41% 73.67% 85.87% 39.21% 72.15% 85.38% Undercoverage 59.59% 26.33% 14.13% 60.79% 27.85% 14.62% Leakage 6.26% 18.76% 27.15% 6.47% 18.65% 28.19% Notes: The logistic model is estimated using observed poverty status in the NHBPS 2014/2015 based on the national poverty line. For comparisons purposes, the PMT cutoff was selected to identify a similar share of households as the National Post-LASSO OLS model. Both models include the same variables presented in Table 3. The results are based on out-of-sample prediction on a random sample of 25% of households. Table 9 presents the results of the Sudan national PMT with the inclusion of a complete set of state and rural and urban indicator variables. The inclusion of state and area indicator variables improves model accuracy by less than 2% with the largest gains observed when targeting the bottom of the consumption distribution. Similarly, poverty accuracy, undercoverage, and leakage all show modest improvements with 19 The regression results are available upon request. 19 the inclusion of location indicator variables. This improvement leads to one additional correctly identified poor household and the exclusion of one additional correctly identified non-poor households per 100 program applicants at the national level when those in the bottom 20% are targeted. However, in practice, the PMT will likely not be used to administer funds to states, rather the states will administer a program given a pre-determined program budget. Therefore, within a state, the ranking of households based on the PMT score will not change with or without state indicator variables and may not be necessary to include in the PMT. Table 9. Evaluation of the National PMT with Additional State and Area Variables (OLS Regression) Evaluation of Including a Complete Set of Location Indicators in the PMT OLS Model with Location Indicators OLS Model without Location Indicators Poverty Line / PMT Cutoff Poverty Line / PMT Cutoff Bottom Bottom Bottom Bottom Bottom Bottom Evaluation Criteria 20% 40% 60% 20% 40% 60% Poverty headcount (observed) 19.96% 39.95% 59.89% 19.96% 39.95% 59.89% Household poverty rate (observed) 19.36% 37.59% 54.92% 19.36% 37.59% 54.92% Household poverty rate (predicted) 12.71% 38.76% 58.73% 12.81% 38.76% 59.60% Total accuracy 85.32% 79.63% 80.40% 83.01% 77.89% 79.26% Poverty accuracy 44.91% 74.47% 85.63% 39.21% 72.15% 85.38% Undercoverage 55.09% 25.53% 14.37% 60.79% 27.85% 14.62% Leakage 4.98% 17.26% 25.96% 6.47% 18.65% 28.19% Notes: This model includes all variables in Table 3, as well as location indicators for states and rural and urban areas where noted. The results are based on out-of-sample prediction on a random sample of 25% of households. Table 10 presents the results of the Sudan national PMT with the inclusion of employment information in the form of the proportion of household members that have ever worked for income to total household members. Although this information is strongly and statistically significantly associated with per capita household consumption. Including this information does not impact the targeting performance of the PMT in a meaningful way. Employment information is generally difficult and time consuming to collect in household surveys, consisting of a series of questions to capture the employment status of each household member of working age and the type of employment, for example, salaried or non-paid, family workers. 20 Table 10. Evaluation of the National PMT with Employment Information (OLS Regression) Evaluation of Including Employment Information in the PMT OLS Model with Employment OLS Model without Employment Information Information Poverty Line / PMT Cutoff Poverty Line / PMT Cutoff Bottom Bottom Bottom Bottom Bottom Bottom Evaluation Criteria 20% 40% 60% 20% 40% 60% Poverty headcount (observed) 19.96% 39.95% 59.89% 19.96% 39.95% 59.89% Household poverty rate (observed) 19.36% 37.59% 54.92% 19.36% 37.59% 54.92% Household poverty rate (predicted) 13.04% 39.16% 59.70% 12.81% 38.76% 59.60% Total accuracy 83.18% 77.96% 79.43% 83.01% 77.89% 79.26% Poverty accuracy 40.24% 72.78% 85.63% 39.21% 72.15% 85.38% Undercoverage 59.76% 27.22% 14.37% 60.79% 27.85% 14.62% Leakage 6.51% 18.92% 28.12% 6.47% 18.65% 28.19% Notes: This model includes all variables in Table 3, as well as the proportion of household members who have worked for income to the total household members where noted. The results are based on out-of-sample prediction on a random sample of 25% of households. In this section, we explored the impact of different model specifications and the inclusion of location and employment variables on the targeting performance of the PMT. The results of a binary dependent logistic regression of the poverty status of the household were nearly identical to the results of the PMT based on OLS regression. Employment information did not impact the performance of the PMT in a meaningful way. And the inclusion of a complete set of location indicator variables marginally improved targeting performance of the PMT. However, in practice, social programs are likely to be administered at the state level with pre-determined budgets, rather than by using the PMT to determine the allocation of beneficiaries across states. Due to this, along with the minor improvements of the performance of the PMT with the inclusion of location variables, the Sudan national PMT excluding location indicators appears to perform well to meet program objectives. 6. Conclusion In this paper we develop and propose a PMT to support the efforts of Sudan in the targeting of the Sudan Emergency Safety Nets Project (SESNP) to those most in need to improve the poverty impact and sustainability of the program.20 Currently, there are efforts in Sudan to implement a new social safety net 20Given that targeting can be politically sensitive (Del Ninno and Mills, 2015) and, particularly in FCV environments (Fragility, Conflict, and Violence) such as Sudan, political economy considerations and the costs of adopting a PMT should be examined. In this regard, when operationalizing the PMT, it will be important to coordinate with the Central Bureau of Statistics of Sudan and other government stakeholders on this work. 21 program, which seeks to assist the most food insecure households in the most affected localities and states, by providing unconditional cash transfers to beneficiary families in all 18 states of Sudan. Sudan has been experiencing multiple macroeconomic shocks, such as low growth, a wide trade deficit, and the emergence of multiple exchange rates since its separation from South Sudan in 2011. The country has implemented an ambitious agenda of reforms, which included the removal of various subsidies, in an effort to reverse years of structural imbalances and high inflation. Parallel to such reforms was the introduction of a quasi-Universal Basic Income cash transfer program (SFSP) that sought to reach the bottom 80% of the population. The SFSP program was eventually discontinued, as, with the onset of COVID- 19, political discontent ultimately resulted in a military takeover and a renewed effort by the military-led government to achieve fiscal consolidation. More recently, the Russia-Ukraine crisis that started in March 2022, has further exacerbated the country’s social and economic problems. In particular, Sudan is expected to face significant negative food security implications as food prices reach historic peaks due to the Russian invasion of Ukraine. All of the economic and social problems that have negatively impacted Sudan are expected to have increased the number of poor in the country. The worsening situation in Sudan has led to renewed efforts to develop a new social safety net program. To support the efforts of Sudan in the targeting of the Sudan Emergency Safety Nets Project (SESNP) to those most in need, we develop and propose a PMT for Sudan based on the latest household survey, the National Baseline Household Budget and Poverty Survey (NHBPS) 2014/2015. We find that a PMT should be considered for targeting social protection programs in Sudan. The PMT would assist social programs to reach a higher share of the poor and vulnerable, while reducing expenditure going towards households that are less in need, improving both the poverty impact, as well as the sustainability of the program. We explore various specifications of the PMT and find that our proposed Sudan national PMT performs as well as a rural and urban specific PMT, making it easier to implement and manage in practice. Furthermore, costly and time-consuming employment information does not improve the accuracy of the PMT in a meaningful way. Finally, our proposed Sudan PMT relies on 19 variables that could be collected by a short survey in the field on a large portion of the population. 22 References Alatas, V., Banerjee, A., Hanna, R., Olken, B. A., & Tobias, J. (2012). Targeting the poor: evidence from a field experiment in Indonesia. American Economic Review, 102(4), 1206-40. Alkire, S., & Seth, S. (2013). Selecting a targeting method to identify BPL households in India. Social indicators research, 112(2), 417-446. Belloni, A., and V. Chernozhukov. (2013). Least Squares after Model Selection in High-dimensional Sparse Models. Bernoulli 19 (2): 521–547. Böheim, R., & Stöllinger, P. (2021). Decomposition of the gender wage gap using the LASSO estimator. Applied Economics Letters, 28(10), 817-828. Brown, C., Ravallion, M., & Van de Walle, D. (2018). A poor means test? Econometric targeting in Africa. Journal of Development Economics, 134, 109-124. Cameron, Lisa, and Manisha Shah. (2014). Mistargeting of Cash Transfers, Social Capital Destruction, and Crime in Indonesia. Economic Development and Cultural Change 62(2): 381-415. Chen, Shiyuan, and Mark Schreier. (2009). A Simple Poverty Scorecard for Bangladesh. Microfinance Risk Management, LLC, https://www.microfinance.com/#Bangladesh Coady, David; Grosh, Margaret; Hoddinott, John. (2004). Targeting of Transfers in Developing Countries: Review of Lessons and Experience. Washington, DC: World Bank. © World Bank. https://openknowledge.worldbank.org/handle/10986/14902 License: CC BY 3.0 IGO. Coady, D. P., & Parker, S. W. (2009). Targeting performance under self-selection and administrative targeting methods. Economic Development and Cultural Change, 57(3), 559-587. Del Ninno, C., & Mills, B. (2015). Safety nets in Africa: Effective mechanisms to reach the poor and most vulnerable. Washington, DC: World Bank; and Agence Française de Développement. Gazeaud, J. (2020). Proxy Means Testing vulnerability to measurement errors? The Journal of Development Studies, 56(11), 2113-2133. Grosh, M. E. (1994). Administering targeted social programs in Latin America: From platitudes to practice (Vol. 94). World Bank Publications. 23 IRIS Center (2005). Developing poverty assessment tools project: note on assessment and improvement of tool accuracy. Morrill Hall, Md, USA: IRIS Center, University of Maryland. Kidd, S., & Wylde, E. (2011). Targeting the Poorest: An assessment of the proxy means test methodology. AusAID Research Paper, Australian Agency for International Development, Canberra, Australia. Kuhn, M., & Johnson, K. (2019). Feature engineering and selection: A practical approach for predictive models. CRC Press. Kuhn et al., (2020). Tidymodels: a collection of packages for modeling and machine learning using tidyverse principles. https://www.tidymodels.org McBride, Linden; Nichols, Austin. (2018). Retooling Poverty Targeting Using Out-of-Sample Validation and Machine Learning. Published by Oxford University Press on behalf of the World Bank. © World Bank. https://openknowledge.worldbank.org/handle/10986/33525 License: CC BY-NC-ND 3.0 IGO. Mills, B., Del Ninno, C., & Leite, P. (2015). Effective Targeting Mechanisms i n Africa: Existing and New Methods. Safety Nets in Africa: Effective Mechanisms to Reach the Poor and Most Vulnerable, 19- 37. Petch, J., Di, S., & Nelson, W. (2021). Opening the black box: the promise and limitations of explainable machine learning in cardiology. Canadian Journal of Cardiology. R Core Team (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/. Reber, E., Gomes, F., Vasiloglou, M. F., Schuetz, P., & Stanga, Z. (2019). Nutritional risk screening and assessment. Journal of Clinical Medicine, 8(7), 1065. Rossell, N., Salaverria, C., Hernandez, A., Alabi, S., Vasquez, R., Bonilla, M., ... & Reis, R. (2018). Community resources support adherence to treatment for childhood cancer in El Salvador. Journal of Psychosocial Oncology, 36(3), 319-332. Sabates‐Wheeler, R., Hurrell, A., & Devereux, S. (2015). Targeting social transfer programmes: Comparing design and implementation errors across alternative mechanisms. Journal of International Development, 27(8), 1521-1545. Sebastian, A.R., Shivakumaran, S., Silwal, A.R., Newhouse, D.L., Walker, T.F. and Yoshida, N., (2018). A proxy means test for Sri Lanka. World Bank Policy Research Working Paper, (8605). 24 Skoufias, E., Diamond, A., Vinha, K., Gill, M., & Dellepiane, M. R. (2020). Estimating poverty rates in subnational populations of interest: An assessment of the Simple Poverty Scorecard. World Development, 129, 104887. Tibshirani, R. (1996). Regression Shrinkage and Selection via the LASSO. Journal of the Royal Statistical Society. Series B (Methodological) 58 (1): 267–288. Van Domelen, J. (2007). Reaching the poor and vulnerable: Targeting strategies for social funds and other community-driven programs. Network HD. Washington DC: World Bank, 1-52. World Bank (2021) Sudan Poverty & Equity Brief, Africa Eastern & Southern, Washington DC 25 Annex 1. Variable Definitions and Descriptive Statistics Table A1.1. List of Candidate Variables and Definitions Variable Name Definition Consumption PC Annual per capita (pc) household consumption expenditure (Sudanese Pound, SDG) Household poverty rate Household poverty rate National poverty line National poverty line Consumption expenditure cutoff for bottom 20% of pc household consumption Cutoff bottom 20% distribution (population). Consumption expenditure cutoff for bottom 40% of pc household consumption Cutoff bottom 40% distribution (population). Consumption expenditure cutoff for bottom 60% of pc household consumption Cutoff bottom 60% distribution (population). Rural Proportion of households living in rural areas. Head age Age of household head. Head female Household head is female. Head illiterate Household head is illiterate. Head not married Household head is not married. Includes widowed, divorced, and never married. (reference) Head married Reference: Household head is married. Household's primary livelihood source is from crop farming, animal husbandry, aid, Livelihood source: farming, aid, other or others. (reference) Livelihood source: wages, Reference: Household's primary livelihood source is from wages, salaries, business salaries, business, remittances, pension, owner, property, remittances, pension, or transfers from household members. or household transfers. No motor vehicle Household does not own a motor vehicle. No motorcycle Household does not own a motorcycle. No television Household does not own a television. No satellite dish Household does not own a satellite dish. No radio Household does not own a radio. No mobile phone Household does not own a mobile phone. No computer Household does not own a computer. No refrigerator Household does not own a refrigerator. No fan Household does not own a fan. No AC Household does not own an air conditioner. Rooms per household member Rooms per household member. House of mud, wood, straw, tent, or House is made of mud, wood, straw, a tent, or is incomplete. incomplete (reference) House of brick or concrete, Reference: House is made of brick or concrete, is a flat or apartment, villa, or multi- flat or apartment, multi-story house story house. Floor material earth Floor of house is earth. 26 (reference) Floor material ceramic, tile, Reference: Floor of house is not earth. Includes ceramic, tile/bulat, bricks, cement, bricks, cement, or other and other. Drinking water source not from common Household drinking water not from a common network. Includes hand pumps, deep network boreholes, wells, etc. (reference) Drinking water source from Reference: Household drinking water from a common network. common network Lighting source from grass, firewood, or Household lighting source from grass or firewood, or has no source of lighting. no source of lighting (reference) Lighting source from electric Reference: Household lighting source from electric grid, fuel, or solar power. grid, fuel, or solar power Cooking fuel cow dung, grass, or does not Household cooking fuel from cow dung, grass, or does not cook. cook (reference) Cooking fuel gas, electricity, Reference: Household cooking fuel from gas, electricity, etc. etc. Waste disposal heap, burning, other Household waste disposal uses heap, burning, or other. (reference) Waste disposal disposable Reference: Household waste disposal uses disposable bags, skip bin, or pit. bags, skip bin, or pit Household size: 1 member Only 1 person lives in household. Household size: 2 members 2 people live in household. Household size: 3 members 3 people live in household. Household size: 4 members 4 people live in household. Household size: 5 members 5 people live in household. Household size: 6 members 6 people live in household. Household size: 7+ members 7 or more people live in household. Highest level of schooling completed by any household member is higher than Max. school level less than secondary 4 incomplete primary, but less than secondary 4. Highest level of schooling completed by any household member is less than Max. school level less than primary primary. (reference) Max. school level higher than Reference: Highest level of schooling completed by any household member is secondary 4 secondary 4 and higher. The ratio of the sum of children aged 0 to 14 and adults aged 65 and older to total Dependency ratio household members. Only adults 65+ The proportion of households consisting of only adults aged 65 and older. The number of household members who have ever worked and earned income, i.e., Income earners pc are not unpaid workers (for family or others), per total household members. A complete set of indicators for the states of Al-Gezira, Northern, River Nile, Red Sea, Kassala, Al-Gadarif, Khartoum, White Nile, North Kordufan, South Kordufan, Location indicators West Kordufan, North Darfur, West Darfur, South Darfur, Central Darfur, and East Darfur, as well as urban and rural areas. 27 Table A1.2. Household Descriptive Statistics of Candidate Variables Obs. Variable Name Obs. Mean St. Dev. Missing Consumption PC 11,947 6,324.7951 3,985.3110 0.05% Household poverty rate 11,947 0.5650 0.4958 0.05% National poverty headcount 11,947 0.6090 0.4879 0.05% National poverty line 11,947 5,815.0000 0.0000 0.05% Cutoff bottom 20% 11,953 3,401.0000 0.0000 0.00% Cutoff bottom 40% 11,953 4,521.0000 0.0000 0.00% Cutoff bottom 60% 11,953 5,738.0000 0.0000 0.00% Rural 11,953 0.6995 0.4585 0.00% Head: age 11,953 46.4927 14.7069 0.00% Head: female 11,953 0.1487 0.3559 0.00% Head: illiterate 11,925 0.3896 0.4877 0.23% Head: not married 11,944 0.1184 0.3231 0.08% (reference) Head: married 11,944 0.8816 0.3231 0.08% Livelihood source: farming, aid, other 11,596 0.4580 0.4983 2.99% (reference) Livelihood source: wages, salaries, business, remittances, pension, 11,596 0.5420 0.4983 2.99% or household transfers. No motor vehicle 11,852 0.9558 0.2056 0.84% No motorcycle 11,853 0.9641 0.1859 0.84% No television 11,870 0.6624 0.4729 0.69% No satellite dish 11,833 0.6725 0.4693 1.01% No radio 11,812 0.7414 0.4379 1.18% No mobile phone 11,860 0.2472 0.4314 0.78% No computer 11,720 0.9738 0.1597 1.95% No refrigerator 11,794 0.7939 0.4045 1.33% No fan 11,787 0.7506 0.4327 1.39% No AC 11,757 0.9277 0.2590 1.64% Rooms per household member 11,887 0.5299 0.3536 0.56% House material: mud, wood, straw, tent, 11,920 0.7992 0.4006 0.28% or incomplete (reference) House material: brick or concrete, flat or apartment, multi-story 11,920 0.2008 0.4006 0.28% house Floor material: earth 11,415 0.8996 0.3005 4.50% (reference) Floor material: ceramic, tile, 11,415 0.1004 0.3005 4.50% bricks, cement, or other Drinking water source: not common 11,894 0.6758 0.4681 0.50% network (reference) Drinking water source: 11,894 0.3242 0.4681 0.50% common network Lighting source: grass, firewood, or none 11,764 0.4133 0.4924 1.58% 28 (reference) Lighting source: electric grid, 11,764 0.5867 0.4924 1.58% fuel, or solar power Cooking fuel: cow dung, grass, or does 11,914 0.5568 0.4968 0.33% not cook (reference) Cooking fuel: gas, electricity, 11,914 0.4432 0.4968 0.33% etc. Waste disposal: heap, burning, other 11,921 0.8393 0.3673 0.27% (reference) Waste disposal: disposable 11,921 0.1607 0.3673 0.27% bags, skip bin, or pit Household size: 1 member 11,947 0.0090 0.0947 0.05% Household size: 2 members 11,947 0.0711 0.2569 0.05% Household size: 3 members 11,947 0.1101 0.3130 0.05% Household size: 4 members 11,947 0.1383 0.3452 0.05% Household size: 5 members 11,947 0.1515 0.3586 0.05% Household size: 6 members 11,947 0.1441 0.3512 0.05% Household size: 7+ members 11,947 0.3760 0.4844 0.05% Max. school level: less than Secondary 4 11,953 0.2512 0.4337 0.00% Max. school level: less than Primary 11,953 0.4487 0.4974 0.00% (reference) Max. school level: higher 11,953 0.3002 0.4584 0.00% than secondary 4 Dependency ratio 11,947 0.4539 0.2424 0.05% Only adults 65+ 11,953 0.0092 0.0955 0.00% Income earners pc 11,947 0.2598 0.2002 0.05% Note: Statistics are unweighted. 29 Annex 2. PMT Results and Imputation Table A2.1. OLS Regression Results for Log Per Capita Household Consumption Drop Missing Impute Missing Variables Obs. Obs. Household size: 2 members -0.454*** -0.432*** (0.04) (0.037) Household size: 3 members -0.729*** -0.7*** (0.04) (0.036) Household size: 4 members -0.928*** -0.902*** (0.039) (0.036) Household size: 5 members -1.081*** -1.061*** (0.04) (0.037) Household size: 6 members -1.209*** -1.181*** (0.04) (0.037) Household size: 7+ members -1.407*** -1.379*** (0.04) (0.037) Dependency ratio -0.088*** -0.09*** (0.017) (0.015) Max. school level less than secondary 4 -0.026** -0.027*** (0.01) (0.009) Max. school level less than primary -0.042*** -0.042*** (0.01) (0.01) Livelihood source: farming, aid, other -0.026*** -0.028*** (0.008) (0.007) Head female -0.074*** -0.077*** (0.01) (0.009) Log rooms per household member 0.055*** 0.055*** (0.008) (0.007) House of mud, wood, straw, tent, or incomplete -0.085*** -0.093*** (0.01) (0.009) Floor material earth -0.078*** -0.066*** (0.014) (0.013) Lighting source from grass, firewood, or no source of lighting -0.067*** -0.063*** (0.009) (0.008) Drinking water source not from common network -0.025** -0.029*** (0.01) (0.009) Cooking fuel cow dung, grass, or does not cook -0.128*** -0.134*** (0.01) (0.009) No motorcycle -0.052*** -0.055*** (0.019) (0.017) No motor vehicle -0.217*** -0.221*** 30 (0.018) (0.016) No radio -0.055*** -0.055*** (0.008) (0.008) No computer -0.164*** -0.174*** (0.024) (0.022) No refrigerator -0.065*** -0.073*** (0.013) (0.012) No television -0.049*** -0.051*** (0.011) (0.011) No AC -0.049*** -0.055*** (0.017) (0.016) No mobile phone -0.156*** -0.147*** (0.009) (0.008) Intercept 10.728*** 10.727*** (0.05) (0.046) Obs. 10251 11947 Adj. R Squared 0.58 0.579 Note: Categorical variables are imputed with mode of the categories. Continuous variables are imputed with sample means. 31 Annex 3. Tuned LASSO and Restricted Model Out-of-Sample Comparison Table A3.1. An out-of-sample comparison of the tuned post-LASSO OLS model and restricted Post-LASSO OLS model. National Model Out-of-Sample Evaluation Tuned Post-LASSO OLS Model Restricted Post-LASSO OLS Model Poverty Line / PMT Cutoff Poverty Line / PMT Cutoff Evaluation Criteria Bottom 20% Bottom 40% Bottom 60% Bottom 20% Bottom 40% Bottom 60% Poverty headcount (observed) 19.96% 39.95% 59.89% 19.96% 39.95% 59.89% Household poverty rate (observed) 19.36% 37.59% 54.92% 19.36% 37.59% 54.92% Household poverty rate (predicted) 12.81% 39.06% 59.53% 12.81% 38.76% 59.60% Total accuracy 83.14% 77.99% 79.20% 83.01% 77.89% 79.26% Poverty accuracy 39.55% 72.69% 85.26% 39.21% 72.15% 85.38% Undercoverage 60.45% 27.31% 14.74% 60.79% 27.85% 14.62% Leakage 6.39% 18.81% 28.19% 6.47% 18.65% 28.19% Note: The Tuned Post-Lasso OLS Model excludes only the variable Only adults 65+. The Restricted Post-Lasso OLS Model (Table 3 in Results) excludes an additional six variables. These include Head: married, Waste Disposal, Head: age, Head: literate, No satellite dish, and No fan. 32 Annex 4. Categorical Variable Groupings and Continuous Variable Transformations. Table A3.1. Categorical variable grouping and continuous variable transformation examples. Example of Floor material: earth’s association with low income. Example of the approximate linear relationship between log rooms per capita and log per capita household consumption expenditure. 33