JOBS WORKING PAPER Issue No. 78 Shaping Better Jobs Policies Through Measurement: Findings from a Pilot Program to Estimate Indirect Jobs Theresa Osborne and Jose Manuel Romero SHAPING BETTER JOBS POLICIES THROUGH MEASUREMENT: FINDINGS FROM A PILOT PROGRAM TO ESTIMATE INDIRECT JOBS Theresa Osborne and Jose Manuel Romero © 2023 International Bank for Reconstruction and Development / The World Bank. 1818 H Street NW, Washington, DC 20433, USA. Telephone: 202-473-1000; Internet: www.worldbank.org. Some rights reserved This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions This work is available under the Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO) http://creativecommons.org/licenses/by/3.0/igo. Under the Creative Commons Attribution license, you are free to copy, distribute, transmit, and adapt this work, including for commercial purposes, under the following conditions: Attribution—Please cite the work as follows: Theresa Osborne and Jose Manuel Romero. “Shaping Better Jobs Policies through Measurement: Findings from a Pilot Program to Estimate indirect Jobs” World Bank, Washington, DC. License: Creative Commons Attribution CC BY 3.0 IGO. Translations—If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by The World Bank and should not be considered an official World Bank translation. The World Bank shall not be liable for any content or error in this translation. Adaptations—If you create an adaptation of this work, please add the following disclaimer along with the attribution: This is an adaptation of an original work by The World Bank. Views and opinions expressed in the adaptation are the sole responsibility of the author or authors of the adaptation and are not endorsed by The World Bank. Third-party content—The World Bank does not necessarily own each component of the content contained within the work. The World Bank therefore does not warrant that the use of any third-party-owned individual component or part contained in the work will not infringe on the rights of those third parties. The risk of claims resulting from such infringement rests solely with you. If you wish to re-use a component of the work, it is your responsibility to determine whether permission is needed for that re-use and to obtain permission from the copyright owner. Examples of components can include, but are not limited to, tables, figures, or images. All queries on rights and licenses should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Images: © World Bank. Further permission required for reuse. 2 Abstract Quantifying the jobs impacts of development interventions is a challenge, largely because much of the impact is indirect. Yet, given the tremendous jobs challenges in developing countries, these impacts are fundamental to designing and prioritizing more effective policies and investments. How can development institutions do a better job of taking them into account? We Shaping Better Jobs Policies through Measurement: Findings from a piloted some approaches and share lessons here. Pilot Program to Estimate indirect Jobs Theresa Osborne and Jose Manuel Romero World Bank Jobs Group 3 ACKNOWLEDGEMENTS This paper synthesizes energetic and dedicated work of a team consisting of Sabah Abdulla, Laura Nelima Barasa, Tuukka Castrén, Alassane Drabo, Santosh Ram Joshi, Mathilde Lebrand, Hugo Lucatelli, and Francisco Juan Alberto Meneses Ponzini, and Camilo Mondragon-Velez, as well as outside efforts undertaken by DIME, Mathematica, SANEM, 3ie, and others. Theresa Osborne managed the work of the team and provided technical support. Jose Romero provided expert technical and process support. Input was also provided by Veronica Michel Gutierrez, Christopher Delgado, Eric Jospe, and other members of the DIME team. Ian Walker (Practice Manager, Jobs Group) and Federica Saliola supervised the program, facilitated funding and provided valuable comments. Funding was provided by the Jobs Multi-Donor Trust Fund Donors, the UK’s Foreign, Commonwealth & Development Office/UK AID, the Governments of Austria, Germany, Italy, Norway, the Austrian Development Agency, and the Swedish International Development Cooperation Agency. We thank project teams and host country counterparts for the collaboration and inputs into this agenda. World Bank Group project teams involved, including Boban Varghese Paul, Andres Garcia, Anna Aghababyan, Zuzana Dobrotkova, Sheoli Pargal, Siow Chew Kuek Skuek, Casey Torgusson, Franck Taillandier, Nargis Ryskulova, Noor Ibrahim Mohamed, Maimouna Gueye, Nina Doetinchem, Iretomiwa Olatunji, Vinay Kumar Vutukuru, James Muli Musunga, Ross Hughes, Hisham Osman, Lesya Verheijen, Stephen Ling and others. The views expressed in this note represent those of the authors alone and do not reflect the views of the World Bank Group, its funders, or its partner governments. The estimation results discussed and reported in this paper are part of a learning exercise on jobs measurement and are not meant as assessments of project performance. 4 ACRONYMS AGPI Agricultural Growth Project 1 AIMM Anticipated Impact Measurement and Monitoring BRT Bus Rapid Transport CGE Computable General Equilibrium CSM Computable Structural Model DPO Development Policy Operation EDGE Enhancing Digital Government and Economy FDI Foreign Direct Investment FTE Full Time Equivalent GCCN Government Common Core Network GDP Gross Domestic Product GIZ Gesellschaft für Internationale Zusammenarbeit (Germany’s main development agency) GE General Equilibrium IDA International Development Association IDA19 19th Capital Replenishment of International Development Association IFPRI International Food Policy Research Institute IE Impact Evaluation IFC International Finance Corporation ILO International Labour Organization IT Information Technology JET Jobs and Economic Transformation JToC Jobs-Focused Theory of Change MSME Micro, Small, and Medium Enterprise M&E Monitoring and Evaluation NOFBI National Optic Fiber Backbone Infrastructure OLS Ordinary Least Squares PE Partial Equilibrium PIR Policy and Institutional Reform PM Parameterized Model RF Reduced Form R&D Research and Development: SAM Social Accounting Matrix SEZ Special Economic Zone STP Special Technology Park(s) ToC Theory of Change 5 2SLS Two-Staged Least Squares VC Value Chain WBG World Bank Group WBES World Bank Enterprise Survey WFM Westfalia Mozambique 6 Contents 1. Introduction ........................................................................................................................ 9 2. Foundations ...................................................................................................................... 10 2.1. Definitions and Classification of Indirect Jobs Channels ............................................ 10 2.2. Key Contextual Information ....................................................................................... 13 2.3. Minimum Feasibility Requirements ........................................................................... 14 3. Overview of Pilots, Methods, and Findings ...................................................................... 15 3.1. The Pilot Portfolio ....................................................................................................... 15 3.2. Overview of Estimation Approaches .......................................................................... 18 The Choice of Reduced Form Versus Parameterized Model ............................................ 19 3.3. Main Estimation Findings ........................................................................................... 21 Importance of the Estimates ............................................................................................ 21 The Job Efficiency of Public Spending ............................................................................... 21 4. Feasibility, Reliability, and other Lessons ......................................................................... 24 4.1. Feasibility Limits ......................................................................................................... 24 Theories of Change ........................................................................................................... 24 Comprehensiveness .......................................................................................................... 26 Survey Realities ................................................................................................................. 27 Resources .......................................................................................................................... 27 Time and Timing................................................................................................................ 27 4.2. Reliability .................................................................................................................... 29 Impact Evaluation ............................................................................................................. 30 Qualitative Methods ......................................................................................................... 30 Input-Output Models ........................................................................................................ 30 Computable Structural Models (CSM) .............................................................................. 32 Econometric Estimation and Extrapolation ...................................................................... 33 Value Chain Mapping and Surveys ................................................................................... 33 4.3. Methodological Lessons ............................................................................................. 33 Standardization ................................................................................................................. 38 5. Recommendations ............................................................................................................ 38 REFERENCES .............................................................................................................................. 41 ANNEX A: Definition of “job” ...................................................................................................... i ANNEX B: Background and Rationale for Channels of Job Impact Terminology .........................ii ANNEX C: Pilot Summaries .......................................................................................................... xi 7 Angola: Strengthening the National Social Protection System Project Cash Transfer Program.............................................................................................................................. xii Bangladesh: Enhancing Digital Government and Economy (EDGE) Project ..................... xiii Bangladesh Private Sector Development Project ............................................................. xiv Bangladesh Private Investment & Digital Entrepreneurship (PRIDE) Project ................... xv Cameroon-Chad Transport Corridor (Structural modelling Estimation) .......................... xvi Cameroon-Chad Transport Corridor (P167798) (Reduced form empirical estimates) .. xviii Ethiopia Second Agricultural Growth Project (Direct measurement Method ................. xix Ethiopia Second Agricultural Growth Project (Structural modelling method) .................. xx Ghana: Ceramic Tile IFC Additional Financing – (KEDA) (Ex ante estimation) ................. xxi Ghana: Ceramic Tile IFC Additional Financing – (KEDA) (Ex post estimation) ................ xxii Kenya: National and Rural Inclusive Growth Project (P153349) .................................... xxiii Kenya Digital Economy Acceleration project .................................................................. xxiv Lesotho Transport Infrastructure and Connectivity Project (LTIC) .................................. xxv Mali: Promote Access to Finance, Entrepreneurship and Employment......................... xxvi Mozambique: Westfalia IFC Project (WFM) (Ex ante estimation) ................................ xxvii Mozambique: Westfalia IFC Project (WFM) (Ex post estimation) ................................ xxviii Rwanda Rural Feeder Roads Development Program ..................................................... xxix Rwanda: Energy Reform DPO Series ................................................................................ xxx Tanzania: Dar es Salaam Urban Transport Project (Balboni, et al. 2020)...................... xxxii Tonga: Pathways to Sustainable Oceans Project (Value chain-based estimation) ...... xxxiii Tonga: Pathways to Sustainable Oceans Project (AIMM Real Sector Assessment Model Estimation) .................................................................................................................... xxxiv Uganda: Investing in Forests and Protected Areas for Climate-Smart Development Project ............................................................................................................................ xxxv Uzbekistan: Livestock Sector Development Project ..................................................... xxxvi ANNEX D: Spotlight on Structural Computable Model Versus Reduced Form Results: Chad- Cameroon Transport Corridor ..............................................................................................xxxvii 8 “The gold standard is the best method for the question at hand” – Martin Ravallion 1. Introduction A key rationale of many development interventions is to boost economic activity and productivity and —whether explicitly or implicitly— generate more and/or better jobs.1 The reasons are clear: it is through rising labor income that most poverty reduction occurs, and labor is a motor for an economy that grows, reinvests, and generates broader opportunities. In areas ranging from skills and human capital, infrastructure, finance, and agriculture, to regulatory, trade, and tax policy, the oft-stated goal of inclusive growth, however defined,2 involves the creation of better work opportunities for more people. Therefore, under its Jobs and Economic Transformation (JET) agenda, the World Bank Group (WBG) is paying more explicit attention to this aim. Often, the more sizeable jobs impacts are those that are indirect rather than those occurring within the entities implementing or directly benefitting from an intervention. 3 Employment impacts of economic reforms, investments in infrastructure, financial systems, the environment, governance and regulatory systems, and other public goods are almost by definition indirect. Yet these economic benefits, so core to development and poverty reduction, are typically neither factored into decisions on prioritization across a portfolio of potential interventions nor in their design. This is in part because of measurement difficulties. Thus, the World Bank Group committed to IDA19 funders to conduct 20 pilot exercises to estimate the indirect jobs impacts of IDA interventions. 4 With more experience, guidance, and accumulating evidence on all significant jobs impacts, policymakers can make better informed decisions. The social value of jobs improved could be included in economic analyses of interventions and increase the accuracy of distributional analysis. It could also inform more comprehensive monitoring and evaluation and, in relevant cases, support dialogue around potential policy reforms. All told, a key step in pursuing more effective jobs policies is to measure these impacts. The jobs created or improved indirectly by public interventions are challenging to measure because they may span a wide geographic area and/or range of sectors or enterprises.5 Impacts 1 See Annex A for a discussion of how a “job” is defined. 2 Ianchovichina and Lundstrom (2009) define inclusive growth as that which is broad-based across sectors and inclusive of the large part of the country’s labor force. The OECD defines inclusive growth as economic growth that is distributed fairly across society and creates opportunities for all. 3 Beneficiaries can include individuals, enterprises, network sectors, and/or institutions among others. 4 More precisely, the commitment states: “IDA will conduct 20 pilots in economic transformation IDA projects to estimate indirect and/or induced jobs.” See https://documents1.worldbank.org/curated/en/459531582153485508/pdf/Additions-to-IDA-Resources- Nineteenth-Replenishment-Ten-Years-to-2030-Growth-People-Resilience.pdf 5 We define a “public intervention” for the purposes of this report to be any investment, expenditure, activity, or policy measure undertaken and/or financed by a government, a donor, or another institution that is ultimately funded, owned, or whose funding is guaranteed or backed by government(s) and their agents (including MDBs) using (its) their taxing authority. This definition is based on the purpose of estimating social impacts, which is to improve public policy and enhance the efficiency of public spending. Therefore, we do not consider investments or portions 9 may occur and be observable only after the project’s monitoring and evaluation period. They can also take many forms, from job quality changes and wage increases to job creation per se. And, as with any impact assessment, attribution is a difficult issue: a multitude of factors can affect these outcomes, and economies are constantly changing. Despite these difficulties, as demonstrated in this paper, with a clear cognizable theory of change supported by economic logic, evidence to support key assumptions, and baseline and/or endline data on the relevant outcomes, estimation of approximate indirect jobs impacts is usually feasible. To our knowledge, no comprehensive guidance exists on the full range of means for measuring an intervention’s indirect jobs impacts. This paper extracts lessons from the IDA pilot program, first reviewing the variety of pilots undertaken, the methodologies chosen, and feasibility and reliability issues, as well as possible avenues for improvement. A companion paper provides a decision framework and technical guide to estimation methods for practitioners.6 Together these pieces provide a significant step forward on this agenda. In the next section (2), we outline the conceptual and practical foundations for conducting indirect jobs estimation. In the subsequent section (3), we provide an overview of the pilots conducted and tentative findings on the magnitude of indirect jobs. We report a crude proxy for jobs-impact efficiency — the number of jobs created or improved per million dollars of WBG funding.7 In Section 4, we discuss lessons on the limits to feasibly reliably estimating indirect jobs effects. Finally, we provide recommendations based on the experience and potential operational value of further efforts to measure jobs impacts (Section 5.) One-page summaries of each pilot are contained in Annex C and cross referenced in the report. 2. Foundations To build the foundations for the rest of this report, we first define and classify broad channels of indirect jobs impact. We then discuss the role of context and the minimum feasibility requirements for estimating them. 2.1. Definitions and Classification of Indirect Jobs Channels We consider the effects of public interventions of all types, whether these are directed to governments or other public entities, non-governmental organizations, individuals, or private companies. We propose a classification of channels of jobs impact to serve as a framework for estimation and a common language for discussion. The relationships are depicted in Figure 1. First, jobs effects can be either direct or indirect: of investments ultimately financed or undertaken by private actors to meet this definition. For consistency, we consider the IFC portion to be the “public intervention.” IFC, on the other hand, defines the “project” to be the whole investment program undertaken by a private company in connection with its (partial) financing for the purposes of estimating jobs impacts.. 6 Limestone Analytics, 2023/forthcoming. 7 The IFC uses an efficiency measure of the number of jobs created per million dollars of total investment required ( only a portion of which is financed by IFC) in its AIMM development impact assessments. In the IFC’s case, the total investment amount is readily available. 10 • Direct Jobs: Jobs created, destroyed, or whose quality or terms of employment change within the entity(ies) or sector(s) being “treated” directly, and which arise as a direct result of it. This includes jobs to operate and maintain any project assets / services over the anticipated life of the investment or new/improved service or function. • Indirect Jobs: Any jobs created or whose quality is impacted through other than direct channels. To illustrate, consider an intervention in a country’s power sector which expands and improves the reliability of the national electrical grid (the “treated” entity). The direct jobs impacts would be those entailed in delivering the project outputs (additional generation capacity, transmission lines and new electricity code, for example) and those to operate the grid on an ongoing basis. Indirect jobs effects would be those occurring outside the power sector that are nonetheless caused by the intervention. To take another example, consider an intervention to provide direct financing (such as by IFC) to a cement company. In this case, the direct jobs are those occurring within that company. The indirect jobs impacts would be those outside the cement company. Within the category of “Indirect Jobs,” one can distinguish three broad channels of possible impact, which have been previously defined, but here with a clearer nomenclature, as detailed in Annex B. • (Forward) Factor Usage Jobs: These are jobs outcomes that occur when a change in the available supply, quality, or cost of an input, productive factor, or condition causes a change in either (i) the supply of labor by workers who utilize or rely on the factor or condition; 8 or (ii) demand for labor by producers who utilize or rely on the factor or condition (e.g., through forward value chain linkages). In our power sector example, factor usage jobs would be those occurring within the production entities consuming electrical power. With better access to reliable power, enterprises (such as cement factories but also many others) may expand or contract their operations and hire more (or less) labor.9 In the case of direct support to a cement factory, possible factor usage jobs would be those occurring within customers of the cement company, such as construction companies or other fabricators. Improved pricing or availability may make it profitable for their businesses to expand and employ more workers. 8 Most jobs outcomes are due to shortfalls in demand relative to supply; however, there are instances when labor supply is the limiting factor to improved jobs outcomes. This is more often the case for certain segments of the labor force, such as for women who face social or legal barriers or ethnic minorities that may lack threshold skills. Factor usage jobs impacts can also occur when an intervention changes the cost (including the social and opportunity cost) and/or benefit of working and thereby induces a shift in labor supply. 9 Electricity and labor are generally found to be gross complements in the literature, so typically these would be increased jobs. 11 • (Backward) Supply Chain jobs: Backward supply chain jobs include those impacts that arise due to changes in the demand for domestically produced inputs by entities directly impacted by the intervention. They also include the jobs impacts from changes in demand for domestically produced inputs that are complementary to the good or service by treated firms experiencing “forward factor usage” effects. For example, in our power sector project, such jobs impacts could accrue within producers of electrical poles, wires, switches, and other equipment sold to power companies. Beyond this core supply chain effect, there could be impacts within producers of inputs (other than electricity) used by the enterprises that have increased production due to cheaper or more accessible electricity (the factor using entities). In a cement factory, this would include labor demand in suppliers of inputs such as limestone, chalk, sand, or transport services. It would also include demand for inputs complementary to cement by construction companies (e.g., pipes, roofing materials). • Consumption Spillover Jobs: Consumption spillover jobs impacts are those due to changes in the demand for goods and services on the part of the people experiencing a change in income from direct jobs, forward factor usage jobs, and backward supply chain jobs impacts. These occur in other markets, beyond those impacted through the other channels.10 Consumption spillover jobs could arise, for example, as employees of productive consumers of power or cement and their suppliers experience increased income and demand more domestically produced goods and services. Although the logic of these channels appears to be predicated on a changed demand for labor, some interventions may also seek to enhance either the supply of labor or the skills of workers. If the intervention “treats” the laborers directly through training, skills testing/certification, or other means, then the impacts on them would be considered direct jobs outcomes. If, on the other hand, the intervention changes a policy, available service, or institution with the aim of making it more worthwhile or feasible to take up employment, then the workers are using the factor (the changed service, for example) and the jobs would be indirect factor usage jobs. 10 These are sometimes called “induced” jobs impacts. 12 Figure 1: Channels of Possible Jobs Impacts among Economic Entities CORE Jobs channels In estimation, it is also helpful to distinguish between “core” and “equilibrium or multiplier” channels of impact for the purposes of selecting estimation methods (see Limestone Analytics, forthcoming.) As shown in the figure, the first round, or core jobs channels are those reaching backward and forward one step from the “direct jobs” icon and lie inside the gold ovals. The other potential supply chain, factor usage, and additional consumption spillover jobs are multiplied through the economy. 2.2. Key Contextual Information The value and plausibility of potential jobs impacts from an intervention depend upon the labor market context. In principle, for an intervention to result in positive social welfare from newly created or improved jobs now or in the future, there must be under- or un-employment of labor (or the relevant segment of it) in the economy. This means there must be a significant distortion or friction either in the labor market or another market affecting the economy’s investment and production patterns that results in labor being under- or inefficiently utilized.11Otherwise, any 11 Here, “efficiency” is meant in a comprehensive social welfare sense, including the dynamic, non -pecuniary and other social benefits of jobs. Distortions can therefore be textbook market failures or policy failures, but also social- behavioral biases that may result in adverse political or social consequences. Textbook distortions in labor markets can arise from many sources, including taxation, monopsony power, some forms of labor market regulation, asymmetric information (and high costs of identifying and monitoring workers), or distortions in other markets for 13 jobs created in association with a given intervention would simply displace others with equivalent or greater economic value.12 Moreover, the labor market context will affect how benefits are likely to accrue, whether in the form of wage increases, reduced under-employment, or through distributional impacts.13 Thus, estimators would ideally ascertain the extent to which workers are un- or under-employed, using easily accessible labor market data. 14 This would aid in predicting whether a potential increase in demand for or supply of labor arising from a public intervention would result in displacement from less economically productive jobs to higher productivity ones, higher wages, and/or more jobs for those un- or under- employed. In fact, developing economies are typified by an array of distortions and frictions in markets, and these result in a significant misallocation of resources in both private and public sectors. 15 Indeed, there are often reserves of un- or under-employed labor, and employment in low productivity jobs is pervasive. Therefore, interventions that transfer workers into more productive sectors and organizational modes — such as larger, more complex firms and organized value chains — are likely to have positive welfare effects even when there is no (net) job creation in the economy, as long as the intervention does not exacerbate allocative inefficiencies in the economy.16 2.3. Minimum Feasibility Requirements Experience suggests that the minimum feasibility requirements for estimating the effects of an intervention are (i) a well-defined intervention, including a measure of intervention inputs and/or outputs; (ii) a clear jobs-focused theory of change (JToC) based on logic and/or evidence linking projected or realized inputs and/or outputs to jobs outcomes; and (iii) usually, some baseline or endline data on jobs outcomes. Project Definition: First, the intervention’s scope must be clearly defined, with inputs and/or outputs quantified so that estimators can logically link changed jobs outcomes to the scale and scope of the intervention. For example, for a project to train mechanics and artisans, at a factors of production that are complementary with labor. Alternatively, sub-optimal labor market outcomes may also result from distortions in investment decisions, due to imperfect capital markets, technology or agglomeration spillovers, economies of scale and coordination failures, as well as policy and institutional barriers. 12 Worker displacement is often not accounted for in impact evaluations of programs where it can be expected to occur. For example, in a recent survey of the evaluation literature for active labor market policies, McKenzie (2017) found that all of the evaluations measured private returns to vocational training assuming the treatment group was not crowding out non-program participants. 13 IFC seeks to refine its modeling to permit a split of job creation estimates by formal and informal sectors. It is also working with the Poverty and Equity Global Practice to estimate the distributional impacts of its financing based on a profile of workers that would be expected to fill the jobs created. 14 The World Development Indicators, ILO, and the Global Labor Databases are all possible sources. 15 For example, in countries where public services are under-provided, adding more public sector jobs would increase social welfare while also providing “good” jobs in public service. In some contexts, in contrast, the public sector may employ too many workers or mis-allocate them among public services. 16 By revealed preference, workers would not take up new jobs unless they improved their welfare. However, if the intervention steered resources to activity that is only economically viable due to a high level of distortion (or subsidization), net social welfare could still decline. 14 minimum one would need to know how many of each would receive training. If an investment were (to be) made into the electrical grid or road system, one would need to know the placement and breakdown of improved transmission, distribution, and generation assets and the effects on supply and outages; or kilometers of improvements to which road segments, and to what degree they will be improved. Theory of Change: A JToC maps from the intervention (inputs) to outputs and higher order outcomes, including jobs. A JToC is needed so that estimators can find an entry point and ascertain which channels of impact to estimate, as well as what types of data, evidence, or other information could be used to conduct the estimation. A sound theory of change is typically supported, at least in part, by evidence. For example, if there is evidence that better tourism attractions increase tourism receipts in similar contexts, this could support a quantifiable ToC linking wildlife management to jobs. Or if there is evidence that a simpler tax code sparks private investment, this could form a link in the JToC. A strong JToC would also ensure that every actor or institution whose changed behavior is necessary to achieve the projected outcomes has an incentive consistent with that change (known as “incentive compatibility”). For example, for a project predicated on farmers’ adopting a water-saving technology, a thorough TOC should be subjected to a (risk-adjusted) profitability test from the farmers’ perspective. Baseline and/or Endline Data on Relevant Job Outcomes: Typically, to estimate a change in the jobs outcome of interest requires data on that outcome either at baseline or endline upon which to base an estimate. For example, for a forestry or fisheries project, one would need to know how many jobs were currently in the sector or relevant entities, in order to predict (ex-ante) or infer (ex post) the change resulting from the project as set against this starting or endpoint. 3. Overview of Pilots, Methods, and Findings 3.1. The Pilot Portfolio Table 1 lists the IDA interventions that were part of the pilot program. It provides a brief description of the intervention, the channels of indirect impact estimated (Section 2.1), and other information discussed in this report. The goal at inception was to capture as much variety in sector themes and methods as feasible. At the start, there were already some estimations of indirect jobs impacts planned or ongoing. However, special efforts to recruit cases were required to expand and round out the portfolio. Transport, digital/ IT, and electricity sectors were all represented, with ten pilot estimations in infrastructure, two of which comprised different approaches to the same transport project. There were nine pilot estimations for interventions providing support to specific real sectors or firms. For some of these, there were two methods of estimation to compare. There was also one financial sector intervention and one cash transfer program included. Although no projects specifically in the education sector opted to participate, training programs are the main subject of the estimates for a digital program in Bangladesh. There were no interventions from the health, nutrition, or water and sanitation themes. Of the total 21 pilot estimations done, eleven were ex ante (or early/mid-implementation) and ten were ex post. 15 All channels of potential indirect jobs impact were represented in the portfolio, as shown in Table 1 (column 2). Some patterns emerged regarding the relevant channels of indirect jobs impacts by intervention type. Most infrastructure-related pilots emphasized the (forward) factor usage channel, given that a primary role of infrastructure is as an input to production. This channel was also captured in financial and agricultural interventions. Backward supply chain effects were important for interventions to support commercial sectors and firms. And whereas consumption spillovers were only central to one jobs estimation, they were included for others. For some pilots all three channels were estimated. 16 Table 1: List of Indirect Jobs Estimation Pilots under IDA19 Project Timing Intervention assessed Main approach (PM, RF, or Hybrid / GE or PE) Channels* Infrastructure, Related Skills, Capacity, and Policy Cameroon-Chad Transport Corridor Ex ante - F Improved road and rail corridor Structural geospatial modeling (PM/GE) Cameroon-Chad Transport Corridor Ex ante - F Improved road and rail corridor Econometric estimation (RF + transport network model / GE) Bangladesh Private Sector Development Ex post - F Special Economic Zones and Social Accounting Matrix- based Simulations Parks with labor module (PM/PE) Bangladesh Private Investment & Digital Ex post - F Special Economic Zones and Social Accounting Matrix based Simulations Entrepreneurship (PRIDE) Project Parks with labor module (PM/PE) Tanzania: Dar es Salaam Urban Transport Ex post - F Bus Rapid Transport Impact Evaluation (RF/ GE) Lesotho Transport Infrastructure and Ex ante – F and C Foot bridges Econometric estimation & extrapolation Connectivity (RF/PE) Rwanda Rural Feeder Roads Development Ex post - F Improved rural feeder roads Impact Evaluation/mid-line econometric Program estimation (RF/GE) Kenya Digital Economy Acceleration Ex ante – F, C, S Broadband expansion Econometric estimation & extrapolation (RF/GE) Bangladesh: Enhancing Digital Government and Ex ante - F Training and training capacity in Econometric estimation & extrapolation Economy (EDGE) Project digital skills (RF/PE) Rwanda: Energy Reform DPO Series Ex post - F Enable fiscally sustainable Before-after: Qualitative and quantitative. (RF- expansion of electricity sector Event study/GE) Indirect or Public Goods/Service Support to Specific Industries Tonga: Pathways to Sustainable Oceans Ex ante – F S Fisheries management, value Activity-specific input-output coefficients chain inputs, and market access. (PM/PE) Tonga: Pathways to Sustainable Oceans Ex ante – F S C Fisheries management, value AIMM (SAM-multiplier and broad sector chain inputs, and market access. coefficients) (Hybrid/PE) Ethiopia: Second Agricultural Growth Ex post: - S F Ag Support Services, Research; Direct measurement of trend. (RF/PE) Small Scale Irrigation Ethiopia: Second Agricultural Growth Ex post – S F C Ag Support Services, Research; Dynamic Recursive CGE model (created by Small Scale Irrigation IFPRI) (PM/GE) Kenya: National and Rural Inclusive Growth Ex ante – S F Selected value chain Product-specific input-output coefficients (NARIG) coordination and community (PM/PE) investments. Uzbekistan: Livestock Sector Development Ex post S F C. Expanded access to finance to Document reviews, interviews and focused sector operators. discussions (N/A). Uganda: Investing in Forests and Protected Ex ante S F C Matching grants and enhanced Tourism: Social Accounting Matrix Areas for Climate-Smart Development resource management. Wood sector: value chain analysis (PM/PE) Direct Support to Private Firms Mozambique: Westfalia IFC Project Ex-ante – S C Loan to company Hybrid: SAM-multiplier (PM/PE) and estimated employment elasticities (RF/Hybrid) Ex post – S C Hybrid: Survey of first round suppliers (PM/PE) and estimated multipliers and elasticities (Hybrid) Ghana: Ceramic Tile IFC AF Ex ante – S C Loan to company Hybrid: SAM-multiplier (PM/PE) and estimated employment elasticities (Hybrid) Ex post – S C Hybrid Survey of first round suppliers (PM/PE) and estimated multipliers and elasticities (Hybrid) Finance and Social Protection Mali: Promote Access to Finance, Ex ante – F C Credit Guarantees and Econometric estimation & extrapolation Entrepreneurship and Employment employment in public goods. (PM/PE). Angola: Strengthening the National Social Ex ante - Direct Cash transfers Econometric evidence & extrapolation Protection System Project and C calibrated to Angola (Hybrid/PE). Notes: For channels, Factor Usage=F, Consumption Spillover=C, Supply Chain=S. For main approach, PM is parameterized model, RF is reduced form model, GE indicates a general equilibrium approach, and PE partial equilibrium. 3.2. Overview of Estimation Approaches To aid understanding of the range of possible approaches, we classify general approaches by two characteristics. The first is how much parameterization of the JToC is (to be) done as part of the estimation. Some estimations entail the quantification of most or all of the causal linkages in a ToC between a project input and jobs outcomes, thus producing a “parameterized model” (PM). Others instead estimate the relationship between an intervention or output and jobs outcomes without quantifying the intermediate linkages – a reduced form (RF) approach. Estimates can also be a hybrid of the PM and RF approaches for different parts of the ToC. The second characteristic is the degree of change permitted on other related markets and equilibrium prices as part of the estimation. When estimators allow factor and output markets and prices to change and carry feedback effects on the market directly impacted as well, so that a new equilibrium is reached in all markets, these are “general equilibrium” (GE) effects. Alternatively, estimators may take a partial equilibrium (PE) approach, does not allow for feedback effects on factor and output prices, with prices and quantities changing only in the directly impacted market (e.g., for electricity, cement, or transport) and the labor market.17 Table 2 shows a general taxonomy of these general approaches used in the pilots. The matrix is based on the distinctions between (i) general equilibrium (GE) and partial equilibrium (PE) approaches in the first instance; and (ii) parameterized models (PM) and reduced form models (RF) in the second. It shows that the two characteristics are distinct choices. For both PE and GE, approaches also vary in the degree to which links in the theory of change are quantified. Although general equilibrium estimation is generally more complex for ex ante estimation using a PM approach, that is not the case when RF approaches are possible. RF estimations can capture market- or population-level outcomes data, and real-world outcomes are “general equilibrium” in nature. Therefore, very often, unless data are based on a limited sample or firm-level data, GE effects would be implicitly captured in RF estimates. Table 2: General Taxonomy of Indirect Jobs Estimation Approaches GE (General Equilibrium) PE (Partial equilibrium) Hybrid PM (parameterized • Structural model with • Partial equilibrium model, Partial equilibrium PM for model) endogenous prices using input output some linkages in ToC, RF • Observational survey-based relationships estimated GE effects (with methods possible extrapolation) for Pilot estimations: 6 others. Pilot estimations: 2 Pilot estimations: 4 RF (reduced form) • Impact Evaluation • Estimation (and possible • Estimation (and possible extrapolation to future or extrapolation) another context) Pilot estimations: 5 Pilot estimations: 3 17 Actual price, quality, quantity and/or shadow price. The Choice of Reduced Form Versus Parameterized Model In principle, one could utilize either RF, PM, or hybrid approaches for almost any type of intervention. For example, a hypothetical intervention to make a country’s investment code less restrictive could have a JToC starting with the impact on the entry of formal firms’, then, in turn, competition; this could then affect firms’ productivity, the allocation of ca pital, and ultimately jobs and wages across the economy. Estimators could utilize a structural (GE or recursive) model of the economy which specified these linkages and then use this type of parameterized model to predict outcomes to capture the counter-factual ex post or extrapolate to other projects or contexts ex ante. Alternatively, for the same intervention, it may be more opportune to utilize a reduced form (RF) GE approach, which uses existing data or empirical evidence to estimate the full effects of the intervention – e.g., a more liberal investment code – on jobs or a proximate outcome, bypassing intermediate linkages. In an ex ante setting, this could then be used to extrapolate impacts for the intervention in question.18 Similarly, for a road improvement project, a PM estimate could be based on the full effects of reduced travel time and vehicle operating cost on transport service price and availability, its effect on the price of transport services, inputs and output transported along the road. The model could then quantify the effect on the profitability of economic activity, investment and output of enterprises, and ultimately on labor used in the affected activities. In contrast, the reduced form (RF) approach would simply estimate the effect of improved roads on jobs outcomes. Among pilot cases, estimators used such a RF approach for a bus rapid transit (BRT) project in Dar es Salaam, as well as a Rwanda rural feeder road program. In the Chad-Cameroon Transport Corridor Project, both a PM (spatial structural GE model with endogenous prices and labor mobility) approach and a quasi-RF estimate were used for ex ante estimation. For two IFC cases, a hybrid approach was used. That is, some linkages are parameterized, and these are then paired with estimates of the employment-output elasticity by country and sector, which are general equilibrium estimates (RF GE). How structural or parameterized an approach pilot estimations used depended most on precisely what kind of data and evidence were available. 19 In some instances, microdata (e.g., from household surveys) were available to estimate the causal link from the intervention to jobs outcomes. In other cases, evidence and data were available for some links of the JToC, which then was supplemented through modelling. The choice also depended on time and cost constraints. In this case, an IDA19 policy commitment deadline (which shifted from June 30, 2023 to June 30, 2022) 20 made it difficult to pursue purpose-built structural modelling and to add impact evaluations to the portfolio. Interventions that were not confined to a specific real sector, such as an intervention that aims to increase access to finance for SMEs regardless of sector (see 18 In principle, this can be done using time series, panel, or cross country regressions. Proximate outcomes to jobs such as GDP could also be estimated, then extrapolation done using other parameters, and still classify as a RF or hybrid method. 19 The questions and factors to consider when deciding on the appropriate approach to estimation are captured systematically in Limestone Analytics (forthcoming) to inform future efforts. 20 In addition, the phase of the project cycle made it more likely to use certain approaches, but as pointed out in Limestone Analytics (forthcoming), the distinction between ex ante and ex post is not as clean as one may initially suppose. Even when projects are completed and end line data are available, impacts in the future require a model or set of assumptions in the same manner as an ex ante estimation does. 19 the Mali Finance and Entrepreneurship project), would not be suitable for a value-chain PM approach; whereas an assessment of the banana value chain would be. The decision on whether to utilize a partial or general equilibrium approach would also depend on the data available, as well as the magnitude and timeframe of intervention impacts. The more impactful the intervention on the economy, the wider the geographic area and broader the sectors impacted, the more relevant GE approaches are. In the long run, for a cross-cutting intervention which stimulates more domestic and international trade, investment, and input price adjustments, a GE approach is more suitable. The pilot program illustrates the potential range of approaches. As shown in Error! Reference source not found., there were pilots covering all combinations of the PM/RF and PE/GE matrix. Complexity ranged from simple extrapolations based on existing input-output relationships, such as for the National and Rural Inclusive Growth Project (NARIG) project in Kenya, to complex economic modeling, such as for an agricultural growth project in Ethiopia. Low-cost methods involved the use of existing data, in a few cases to estimate key parameters of the theory of change econometrically. In others, simple input-output relationships or jobs elasticities of production were utilized to estimate forward and/or backward usage/supply chain jobs. Sometimes multiple approaches were combined for a given project or component. The pros and cons of the selected approaches for each pilot case are listed in the one page summaries. Broadly speaking, the methods used depended on the timing within the project cycle (i.e., whether it was ex post) and the preexistence of an appropriate structural model (such as that developed by IFPRI for agricultural interventions), empirical evidence, and/or available data (e.g., Lesotho). Most estimations that adopted a RF approach did so because it was possible to utilize existing data to econometrically estimate key impact parameter(s) and extrapolate to future periods or to another context. For example, in the Mali project on finance and entrepreneurship, estimators used enterprise survey data to estimate the link between enterprise-level access to finance and their firm job creation – a partial equilibrium effect; then the regression estimates were used to calculate the number of new jobs based on the number of firms expected to be supported under the project. In another case, the Kenya Digital Economy project, estimators found a repeated (cross sectional) FinAccess survey of households that permitted the estimation of district level impacts on wages of expanding internet access.21 Since the real world data would capture any GE effects, this is a RF/GE case. For more industry-specific support programs, econometric estimation was less relevant. In those cases, estimates were built up using parameters (PM) from the same or similar contexts and sector(s). Some took advantage of existing SAMs or IO tables which map the entire economy to capture indirect effects on other sectors; others, such as the Kenya NARIG project sought to map and quantify jobs in the value chain without considering all possible multipliers. Ex post surveys were used to directly observe parts of the value chain and measure at least one linkage (such as for a ceramic factor project in Ghana). Either way, whether ex ante or ex post, some then estimated additional indirect effects using SAM-based multipliers (PM PE approach). 21 County and year effects are included to help address simultaneity bias issues. 20 3.3. Main Estimation Findings Importance of the Estimates The full extent of indirect job impacts is often impossible to directly observe, and the counterfactual is never directly observable. Therefore, no estimate is 100 percent accurate. All estimates reported here should be taken as approximations with differing levels of reliability (discussed in Section 4.3, below). With this caveat, in many cases indirect jobs impacts —changes occurring beyond the entity(ies) receiving the project intervention — appear quantitatively important. Although direct jobs impacts were not estimated for all pilots, in some cases the level of indirect job impact looks to exceed the direct impact. This conclusion is based on estimates for Bangladesh PRIDE as well as some transport, digital economy, agriculture, and natural resource management interventions (e.g., that in Uganda.) In many cases, indirect jobs impacts are, at least in theory, also more lasting than direct ones. These findings, although tentative, nonetheless underscore the importance of estimating such impacts when setting policy priorities, project scope, and project development objectives.22 The Job Efficiency of Public Spending The ideal metric of the jobs efficiency of public interventions would weigh the social value of all impacts, including jobs improvements, against the total opportunity costs of economic resources expended to elicit them.23 As a first step, these pilots were designed to “merely” learn how to enumerate indirect jobs created or improved and therefore did not entail a social valuation of these effects. Moreover, on the cost side for many projects it is difficult to capture all public and private resources devoted to achieving the outcomes captured.24 We use the number of ‘better’ jobs created per US$ of World Bank funding as a crude proxy for jobs-efficiency, as displayed in Table 3. This metric is not perfectly comparable across estimations which have different degrees of comprehensiveness and/or rigor. Nonetheless, the orders of magnitude, are suggestive. If we take them as broadly correct, the types of interventions delivering more indirect jobs per USD 1 million of WBG funding would fall into either agriculture, eco-tourism, or digital sectors, with over 1,000 jobs per USD 1 million spent. Agricultural and tourism value chains tend to be relatively labor-intensive in the applicable contexts (Sub-Saharan 22 This point was also made by IFC (2013). 23 The social value of a job is the net change in the welfare to all members of the society. It may differ from the private value of a job if there are spillovers, whether positive -- such as through the enhanced welfare of the worker not counted in employers’ decision-making (an externality) or the increased productivity of other workers and capital, or negative (such as when the job created for a given worker is created at the expense of others on net). Ideally, to take the social value of a change in jobs outcomes into account is to include any externalities them in the economic analysis / rate of return calculation. See Robalino and Walker (2017). Considerations of social costs may also depend upon the grant element of the publicly financed portion of the intervention. 24 For the NARIG project in Kenya, some illustrative rate of returns calculations based on the indirect jobs benefits were calculated. 21 Africa in these cases); these and digital technology investments leverage private capital to deliver benefits, thereby boosting this metric. 25 And jobs impacts were not always a project development objective (PDO) for cases with significant estimated indirect jobs impacts. For example, the cash transfer program in Angola is estimated to result in between 169 and 422 direct and indirect jobs per USD 1 million, respectively. Overall, the estimated impacts of infrastructure on jobs varied widely, given the diversity of contexts, of types of infrastructure, and in the timing and estimation methods. In some ex ante cases, better job creation is estimated in the hundreds per million dollars spent. Estimates suggest that transport investments can have important effects on sectors of employment and quality of jobs. However, on a per-project-dollar basis, these impacts appear more modest than for other sectors. This could be due to the high public expenditures involved and/or the limited margins of improvement over counterfactual transport conditions. 25 For tourism and agriculture, estimates assumed that current technologies, production patterns, and factor prices would persist, and that constraints, frictions, and market demand would not limit further expansion of value chains. They may therefore be seen as upper bounds. The estimate for a digital technologies project utilized international evidence of broad / macroeconomic enabling effects to predict large job gains. 22 Table 3: Tentative (Unverified) Estimated Impacts on Jobs Created or Improved per $1 Million of World Bank Funding Project Indirect Jobs created or improved at peak impact per $1 million of World Bank funding Cameroon-Chad Transport Corridor, Structural Model For Cameroon, 194 and 97 people change their main sector ($538mn)* of employment when considering transport combined with border facilitation and transport policies only, respectively. For Chad, 332 and 18 people changing their main sector when considering transport plus border facilitation and transport policies only, respectively. Cameroon-Chad Transport Corridor, Reduced Form Estimation ($538 82 people shifting out of agriculture in Cameroon and 213 in mn) Chad. Bangladesh Private Sector Development ($168mn) 63 direct jobs and (during 2016-2021) and 405 indirect jobs. Bangladesh Private Investment & Digital Entrepreneurship (PRIDE) 521.8 (total) Project (P170688) ($500mn) 406.4 (indirect only) Bangladesh: Enhancing Digital Government and Economy (EDGE) 440-853 indirect jobs Project (P161086) ($295mn) Tanzania: Dar es Salaam Urban Transport Project P150937($425mm) 0 Lesotho Transport Infrastructure and Connectivity Project (LTIC) [foot Jobs created: 0. Jobs improved: bridges] Paid Jobs (Mountain Areas): 248 Written Contracts: 164. Permanent Jobs: 185 Pension Fund: 72. Paid Sick Leave: 171 Rwanda Rural Feeder Roads Development Program ($170mn) 47 Kenya Digital Economy Acceleration project N/A (Quality improvements were found) Rwanda: Energy Reform DPO Series No effect on jobs detected (possibly due to weakness of empirical test) Tonga: Pathways to Sustainable Oceans Project ($3.9mn) 3.5 (indirect only) Tonga: Pathways to Sustainable Oceans Project ($3.9mn) 2.8 (indirect only) Ethiopia: Second Agricultural Growth Project ($455mn, before 80mn 1,431 additional financing) Ethiopia: Second Agricultural Growth Project: P148591 ($455mn, 3,076 before 80mn additional financing) Kenya: National and Rural Inclusive Growth Project 1,035 Uzbekistan: Livestock Sector Development Project $150mn, US$243.8 169/104 total) Mozambique: Westfalia IFC Project ($7.5mn IFC $2.8mn ) Ex ante: (2025): 1,738 | (2021): 650 Mozambique: Westfalia IFC Project ($7.5mn IFC $2.8mn Ex post: (2025): 1,061-1,536 | (2021): 400-570 Ghana: Ceramic Tile IFC AF – Project ($44.7mn IFC $25mn) Ex-ante (2021) ~21 jobs Ghana: Ceramic Tile IFC AF – Project ($44.7mn IFC $25mn) Ex-post (2021) ~40 jobs Uganda: Investing in Forests and Protected Areas for Climate-Smart Wood sector: 733 Development (USD 85.2mn) Total project US$ 148.2mn. Tourism: 1707 to 2285 Mali: Promote Access to Finance, Entrepreneurship and Employment. Direct: 393.3 Indirect: 39.4 (Through credit: 5.7/19.9. Consumption spillover: 19.5)Total: 432.7 Angola: Strengthening the National Social Protection System Project 169-422 Cash Transfer Program ($260mn) NOTE: World Bank budgeted amounts do not represent the full social cost of the interventions. Some estimates are for certain components of projects only, and some pursue only a subset of possible indirect jobs channels. For the Lesotho transport project estimates provided are considering the funding designated for the first 19 project's footbridges (US$ 9.2 million). *In the structural model used for the Cameroon-Chad Transport project, it was only possible to measure the shifts in employment shares between agricultural, manufacturing, and services. 23 4. Feasibility, Reliability, and other Lessons 4.1. Feasibility Limits In almost all cases attempted, across multiple sectors and project types, it was possible to conduct estimations of indirect jobs outcomes within time and budget limits. Nonetheless, the pilot estimations tested feasibility limits related to theories of change, survey realities, time, and resources. The practical lessons are discussed here. Theories of Change For most interventions the original ToC did not encompass jobs impacts, but it was usually straightforward to either extend it or create a new, purpose-built JToC. For example, for the Angola Social Protection Project the original ToC centered around improved social safety nets (Figure 2.A). However, project supervisors noticed during implementation that the recipients had used some of the support for investments that could improve their own-account jobs and possibly employ others. This sparked interest in defining a revised theory of change and estimating jobs impacts. In this case, a new ToC was established linking cash transfers to the financing of household investments and ultimately jobs impacts (Figure 2.B.) In another case, the purpose of a financial Figure 2: Theories of Change for Angola Social Protection Project sector intervention A. Project Theory of Change at Design Stage in Mali was to expand financing to micro, small, and medium enterprises. If successful, although not part of the initial theory of change, this greater access to credit could enable an expansion B. Project Theory of Change for Estimating Jobs Effects of borrowing Direct enterprises’ recipient/ beneficiary Intermediate Direct jobs Indirect jobs workforces, so the Constraint Intervention of support outcome outcomes outcomes original ToC could be Inefficient Credit Increase Investment in self- Increase self- Increase hired extended to Markets Capital Families owned firms employment workers encompass jobs No collateral/low Increase Investment in self- Increase self- Increase hired impacts. investment Capital Families owned firms employment workers 24 Nonetheless, it is not always possible to establish a ToC that supports indirect jobs estimation. First, reasonably proximate logical linkages between the elements of the intervention and jobs outcomes are required. For example, a project to collect labor force data and train statisticians could in theory improve jobs outcomes by enabling data-driven, pro-jobs policy decisions. However, linking the project’s outputs (i.e., more available data) to jobs outcomes would be highly contingent on there being appropriately conducted analysis of the data and on policy makers’ applying those analytical findings in a way that affects jobs. This ToC would be too tenuous for a credible estimation. Similarly, an intervention to reduce fiscal obligations on the part of a government-owned utility, digitize public records, improve monitoring and evaluation, or some other types of public sector improvement, while important, may have impacts too diffuse to clearly link to jobs. In other cases, there may be a more proximate link between a broad category of intervention and jobs outcomes, but the intervention may be too narrowly gauged to draw on available evidence for quantifying it. Under many World Bank Development Policy Operations (DPOs), this can occur. For example, evidence pinned to a specific change to a sub- clause of a water user law, to the specific timing or extent of officials’ financial disclosures, say, or to a utility regulator’s board selection process may simply not exist to aid in quantification, even if evidence on impacts of broader reform programs they support may. Box 1: Estimating the Jobs Impacts of Policy and Institutional Reform Just as with more concrete project outputs, the jobs impacts of changes in development policies or institutions can be estimated, provided that minimum conditions obtain. For example, in an early effort (2010), the Millennium Challenge Corporation (mcc.gov), as part of its economic analysis, estimated the forward factor user jobs arising from prospective reforms to the governance of Malawi’s electricity sector. The theory of change was that more independent power sector regulation, which had been shown empirically to foster greater investment in generation capacity (see Cubbin & Stern, 2006) in developing countries, would result in greater supply and reliability; that this would impact producers through a decline in unit costs of power with a shift away from own-generated fuel and an expansion of power consumption; this would cause an outward supply shift in production and a commensurate shift in the demand for labor (with parameters taken from the WBES). Scenarios regarding the success of reforms were modeled. After discounting wage income to account for reservation wages, these benefits were added to the flow of economic benefits of the project economic analysis, including a distributional analysis. (Source: T. Osborne, then-MCC Lead Economist for Malawi and co-author of this report). These issues came to the fore with one attempted pilot estimation. Although jobs impact estimation can often be done for policy and institutional reforms (PIR), the World Bank’s Development Policy Operations (DPO) instrument per se can bring additional estimation difficulties. (See Box 1 for a discussion of one more straightforward PIR case). 26 To test feasibility limits in estimating the jobs effects of DPO’s the Bank team aimed to include at least one such 26 In this case, the set of prior actions may have opposing effects on jobs and seem to have been combined or selected without an explicit ToC. 25 case and to use a parameterized GE model capable of capturing complex linkages between drivers of productivity and jobs.27 However, it proved difficult to enlist and implement such a pilot in the time allowed. The estimation team tried various ways to elicit interest; general calls for proposals and other forms of outreach were not successful. A management nomination resulted in a candidate DPO case in Senegal. However, in the end, it proved infeasible to associate the specific prior actions (PA’s) to the modeling cited as supporting analyses (Country Economic Memorandum) for the DPO, which emphasized the dynamics of firm entry, competition, and productivity.28 The PA’s were also too specific to permit a link to available quantitative evidence, which – where it exists – tends to concern broader policy shifts. Some prior actions may also have had offsetting effects on jobs, since jobs impacts were not the main objective. As a result, the implicit theory of change lacked a sufficiently clear line of sight to jobs.1 And ultimately, the effort to marry this case to an existing structural modeling effort failed. Because there was no match between subsets of the DPO’s prior actions and any known existing structural model , it would likely have been necessary to build a model for this special purpose. Yet the resources and time were lacking to develop such a model. Overall, feasibility comes down to having a measurable project outcome and a mapping from that outcome to jobs impacts. Comprehensiveness Pilot estimators did not always deem it practical, relevant, or worthwhile to conduct an estimation for each project activity. This was particularly the case if the linkage to jobs of an activity was considered remote (due to the ToC issues discussed above) or quantitatively unimportant.29 For this reason, estimators declined to estimate impacts for activities designed to improve governance of the digital economy or to improve government safety nets systems. In other cases, circumstances complicated efforts to observe and estimate all effects. For the Kenya: National and Rural Inclusive Growth Project (P153349), there were other projects in the area that affected the same target population, making attribution of some elements or channels of indirect jobs impact impossible. Moreover, even for those elements where estimation was done, it was not always practical to capture all possible channels of indirect jobs impact. For several pilots, only the main channel was estimated. The jobs impacts potentially deriving from consumption demand spillovers were not always included, presumably because the estimation effort was deemed to be high relative to the likely impact. The one pagers in Annex C provide more details on each case. 27 A DPO loan is a budget support instrument in which the World Bank lends on the basis of specific actions taken pursuant to policy and institutional reforms. Such actions could be aimed at, for example, strengthening public financial management, improving the investment climate, addressing bottlenecks to improve service delivery, improving the health sector, or diversifying the economy. DPOs support such reforms through non-earmarked general budget financing that is subject to the borrower's own implementation processes and systems. 28 Some PA’s could have resulted in a change in the gender split of jobs impacts and may have been possible to derive estimates for, but this was not the aim. 29 In all cases, components relating to supervision and monitoring and evaluation are considered part of the larger project, without their own separate benefit streams. 26 Survey Realities Although surveys are key to ground-truthing estimates, sometimes survey aims came up against real-world obstacles. For the Forestry and Tourism project in Uganda, the plan was to conduct a survey to aid in ex ante jobs projections. However, in practice it proved impossible with the resources available to draw a representative sample of operators in the sectors most likely to be impacted. Thus, the data collected were deemed reliable only for certain indicators. Identifying and sampling firms adjacent to sectors like tourism, which typically involves a range of services and enterprises, can be especially challenging. Non-response, a classic issue with business surveys, was particularly problematic in the IFC cases. Within the cost and time constraints of the IDA pilots, these surveys were also unequal to the task of measuring more than one round of value chain impact (see Mozambique: Westfalia IFC Project (WFM) (Ex post estimation) and Ghana: Ceramic Tile IFC Additional Financing – (KEDA) (Ex post estimation). Nonetheless, IFC was able to pair direct observation (survey data) with SAM-based models to estimate other channels of impact more fully. Resources The financial costs of estimating indirect jobs varied according to method and type of intervention. Some of the pilots were funded and coordinated through the Jobs Group Multi Donor Trust Fund (MDTF); others were separately instigated and funded. Those using existing data averaged approximately USD 50,000 dollars (including consultants, coordination, review, and oversight costs). Others entailed special purpose data collection and carried a significantly higher cost, of around $185,000. One separately funded pilot (Rwanda energy DPO series) cost less than USD 50,000. And impact evaluations typically cost hundreds of thousands, or even millions, of dollars as they require repeated mobilization of experts and several rounds of data collection. The other key resource involved in estimating indirect jobs is that of sufficient expertise. It was not a trivial matter to identify and recruit enough economists who were both available and able to conduct the estimations, and two of those recruited were soon employed elsewhere full time. Time and Timing Timing and time requirements are key variables in determining whether and how to estimate indirect jobs impacts. In the pilot cases, estimations were done too late to factor indirect jobs impacts into portfolio or project design decisions where they could have the most value. In some ex ante cases, projects had been underway for some time, and estimates could be considered mid-line and could prove helpful to any restructuring or additional financing decisions. In other cases, ex post estimates may have been done too early to be conclusive. Except for IFC investments, estimating the indirect jobs impacts of interventions is not part of the early phase 27 of IDA project identification, design, or assessment.30 Ideally, estimates of jobs and other social benefits (and costs) would be integrated with project scoping and design, so that insights from early estimates could be used to improve design and potential results. Aligning timing to this objective is not easy. If estimates are not timely, decisions cannot benefit from them. Moreover, a sufficient period of time is needed after an intervention is defined to construct and verify the estimates before decisions are made. For ex ante estimates utilizing existing data, based on the pilot experience, once analytical resources are in place final estimations could be done within several weeks, including steps involving writing and incorporating feedback. Those based on a field survey took longer. Contracting a survey firm and survey preparation took time even if the data collection itself could be done relatively quickly. 31 Strategic timing also matters for learning and accountability purposes. First, to build confidence in ex ante estimations, it would be valuable to conduct both ex ante and ex post estimations for the same intervention(s), including ex post estimates for the ex-ante pilot cases. Because of the IDA timing constraints, the pilots contained only two ex post cases where ex-ante estimates had been done, and many of the pilots were done during implementation rather than before or after. Currently, such full project cycle jobs measurement rarely occurs. Indeed, this was the first time that IFC was able to obtain direct ex post survey data to compare with its ex ante projections. For ex post analysis, the main obstacles are a lack of planning and funding for data collection after project closure, and with the lag needed for impacts to materialize. For ex ante estimation, without a mandate and standing resources (financial and analytical) it can be difficult to phase jobs estimation into the scoping and design process. In many cases, ex post jobs impact estimates could be obtained using monitoring indicators and secondary data. However, for greater confidence in results, final outcome measures would be done at the appropriate time after intervention completion. For one impact evaluation included in the pilot cases, delays occurred along with project implementation delays (rural roads project in Rwanda.) In this case, jobs impacts were estimated using interim data, without the full specification planned for the ultimate evaluation. The final results will likely differ from those obtained under the pilot estimation. 30 Ex ante jobs impact estimations are done systematically under AIMM for relevant IFC interventions to support decision-making. In this case, for both IFC projects, the ex-ante economy-wide impact was estimated for AIMM assessment using the earlier SAM multiplier (GTAP database version 9). Under the pilot program, the team re-ran the 2019 estimates using the latest SAM multiplier, i.e. based on GTAP database version 10, to compare with the ex- post. 31 Estimations where existing data were used were all completed according to the IDA19 deadline of June 30, 2022, which had shifted from June 30, 2023 without a change in the pilot target of 20. Cases where special purpose surveys were used – whether of value chain actors, households, or firms -- faced timing issues, and in several cases were not completed according to the original timeline set. This was in part due to COVID-19, but also in part due to contracting delays. 28 4.2. Reliability Estimation of any intervention’s impact inevitably raises the issue of reliability. There is no guarantee of accuracy for any estimation exercise which is based on partial data, especially where confounding factors intrude. Yet policymakers are continually called upon to make decisions with imperfect information, and the same is true for interventions that may have indirect jobs benefits. Given this, perhaps the right standard of reliability for an indirect jobs estimation is generally that: (1) estimates be done objectively, without optimism bias;32 (2) methods used be appropriate to the purpose and case at hand. ▪ This means that approaches align with the ToC, use the best available data, evidence, and method given the purpose(s) of the estimation. The time and cost constraints would tend to be determined by the purpose;33 and that (3) the main sources of possible bias be discussed. ▪ In this way, when assessing their options, policymakers may place more weight on information that comes from more rigorous evidence, and/or that is confirmed from multiple sources of information. This does not mean that all methods confer the same level of reliability. Although it is impossible to compare estimates with the “truth,” the aggregate experience of the pilots does provide some insight on the level of confidence to place in various methods. In general, as with all impact studies, reliability increases with better data, better estimation methods (econometrics), and the reasonableness of assumptions used. Ex-post estimates by definition can factor in more recent data and thus are should generally be considered more reliable. Ex ante estimation also tends to privilege theories of positive impact. Ex-ante estimators may expect difficult conversations surrounding a null estimate, despite the instruction to maintain objectivity. Optimism bias may also be an issue. Thus, whereas none of the ex-ante pilots estimated there to be no significant effect on indirect jobs, two ex post estimations were unable to reject the hypothesis of no impact. One of these was a robust impact evaluation of a bus rapid transit project in Dar es Salaam, which tested the theory that improved transport would provide access to better jobs and labor-job matches for employers. Another, albeit much weaker, ex post test of aggregate impacts on jobs of electricity sector reforms, was also unable to detect any evidence of indirect jobs impact. However, in this case, there is a relatively high risk of a false negative result. Another factor differentiating estimation methods is that for some, there is no applicable standard of statistical significance applied, whereas for others, estimators may set the probability of a false positive result at a low level. 34 This means that it is not straightforward to compare a positive point estimate – set to zero due to a lack of statistical significance – to a positive estimate not subjected to statistical criteria. One cannot consider ex ante estimates based on a theory, without empirical evidence or any statistical significance standard, to be as reliable as a rigorously done impact 32 See a collection of findings on optimism bias here: https://www.sciencedirect.com/topics/neuroscience/optimism-bias 33 If data collection is infeasible, then the standard is to utilize available data in as rigorous and unbiased way as possible. 34 Typically, researchers use a p value (probability of a false positive) of no greater than 10 percent. 29 evaluation that convincingly addresses all key sources of potential bias. We discuss the various methods used and their likely reliability with respect to some of the pilot cases, below. Impact Evaluation The most reliable method for impact estimation is typically considered to be impact evaluation (IEs). 35 Different IE methods provide different levels of reliability, however. Rigorous observational studies might be more appropriate or reliable in some cases. Still, the IE’s includ ed in the pilots appear well designed and executed and therefore likely to provide reliable estimates. However, IE methods are not always available and may not always be suited to the purpose. First, although previous IE’s can inform ex ante estimation, IE’s are exclusively an ex post approach. Because IE’s generally take years to complete, they are not always suitable for frequent or near term reporting to stakeholders. Moreover, many interventions, such as broad policy and regulatory reforms, large infrastructure investments affecting many areas of the economy, and other similar interventions are generally not amenable to IE.36 For the two IE’s included in the pilots, that for the Dar es Salaam urban transit is well designed and has the statistical power to be considered reliable. The same is true of the Rwanda Rural roads IE, but the full specification and final results were not available by the IDA deadline. Qualitative Methods The use of simple ad hoc modeling based on qualitative surveys or expert opinion can be adequate for some purposes. Information from objective, expert sources can be more finely tuned to the sector, geographic area, and economic conditions than some other, more “rigorous” methods. In general, blending qualitative with quantitative methods can improve reliability. Still, a purely qualitative approach to answering quantitative questions, such as that used in the Uzbekistan Livestock Sector Development Project, is more subject to credibility challenges than other approaches. Ultimately, reliability declines with each link in the ToC that is based on assumptions, opinion or educated guesses and not backed by additional data. Input-Output Models The use of Social Accounting Matrices (SAM) and other input-output relationship data to parameterize backward supply chain and forward factor usage jobs was a frequent choice among the pilots. It was the go-to approach for interventions into a specific real sector or enterprise. Since these estimates usually draw upon on country-specific empirical relationships, they would provide at a minimum a sound starting point for estimation. However, they rely on the assumptions of constant technology choice by producers and abstract from any constraints in factor markets or limits to demand. 35 Advanced econometric techniques using non-experimental data can also be as reliable as IE’s if they adequately address selection and simultaneity bias. 36 If not designed to do so, reduced form IE’s may not elucidate “why” impacts are either low/non -existent or high, and therefore may hold little external validity or lessons for future projects. See, e.g., Ravallion (2020) and Deaton (202) for discussions of pros and cons of randomized IE’s in particular. 30 The pilots included both ex ante and ex post estimations for two IFC financing projects, an avocado firm in Mozambique and a ceramics manufacturer in Ghana. The ex-ante estimations were according to the IFC’s Anticipated Impact Measurement and Monitoring (AIMM) framework. This draws upon a SAM to estimate all backward value chain and consumption spillovers across detailed sub sectors; it then applies estimates of the elasticity of employment- to-value added (estimated by Bürgi et al., 2020), separately by country and broad sector (agriculture, services, or manufacturing) to compute the indirect jobs impacts anticipated. The ex post estimations update the assumptions on the sales, employment, and other variables for the firms receiving IFC financing using ex post data. They also capture and update parameters linking borrower results to employment by surveying a sample of borrowers’ first round suppliers. These first round effects were then augmented using the same SAM-based model used ex ante. Thus, estimates based on the surveys can be compared with the results for each using IFC’s Real Sector Impact Assessment methodology. In both cases, the ex-ante and ex post results were of a similar order of magnitude to each other. In the Mozambique case, all estimates were well within a factor of two of each other: the ex-ante estimate was 1,820, and the ex post estimate was lower, at 1,120-1,596. For Ghana’s ceramics company, however, the ex post estimate (1,810-1,850) was almost double that of the ex-ante estimate of 950 indirect jobs. The estimation report ascribes this to an unanticipated rise in demand for the firm’s products. There is further discussion of the role of demand and attribution in Box 2. Two ex-ante estimations were also done for the Tonga project for comparative purposes. One was a bespoke IO study by Meneses; the other, by the IFC, used the AIMM framework real sector model discussed above. First, Meneses analyzed the Mabe pearl production and jewelry value chain impacted by a component of the project, using input-output relationships in the pearl sector from Chile and jewelry producers in Tonga. For the Offshore Fisheries: Tuna Longline, Snapper, and Others, and investing in Sustainable Management Areas (SMA) component, he then used Tonga-specific information on the sector to estimate increased output deriving from the project and combined this with a jobs-to-output elasticity estimate for fisheries from Latin America to compute indirect jobs. The IFC approach contrasts with Meneses in three ways: First, for the fishing industry, it is more comprehensive in that it estimates consumption spillover linkages. Second, it was unable to use Tonga-specific data or sector-disaggregated detail within the fisheries sector or to model the Mabe pearl sector. Third, for broad sector IO linkages, it used a SAM from Fiji (in the absence of a SAM from Tonga) capturing these at a more aggregated sector level. For the specific components and channels estimated by both Meneses and the IFC, the estimates of indirect jobs created were almost twice as high at 11 jobs in the former, versus 6 in the latter. Based on these comparisons, one can generally conclude that the accuracy of the IO method depends on (i) the specificity and match of the “sectors,” “inputs,” “economies” and “agents” captured in the data; (ii) the magnitude of impacts on the sector(s) treated. If large relative to the overall economy, the intervention could change factor price relationships and violate the constant technology assumption; and (iii) the mobility of, or frictions and constraints affecting 31 the supply of factors of production.37 In many cases, relative to an economy’s initial input-output relationships, the true dynamic impact will be constrained by either supply-side frictions or demand limitations, in which case IO-driven impact estimates will tend to over-estimate impacts. In other cases, agent heterogeneity that is not factored into the analysis can skew estimates. For example, emerging evidence suggests that the local economy linkages and indirect jobs effects of investments in large foreign-owned firms are over-estimated using I-O tables by a factor of at least 2, because these firms tend to source proportionately more from abroad (Alfaro-Ureňa et al., 2022 and forthcoming). Computable Structural Models (CSM) Computable structural models can be utilized to capture a range of possible dynamics, spatial factors, and market interactions. Structural modeling —so called because the structure of the economy is specified through a series of inter-related equations —is helpful for studying theoretical questions and the possible mechanisms behind the data observed. Computable General Equilibrium (CGE) models, which allow flexible factor and output prices, are one example of this approach. Structural models can elucidate issues not captured in reduced form empirical approaches, such as which areas of a country, which types of workers, or which factors may experience a change in returns. They can be tailored to the question posed, the economy in question, and the available data. However, if they are not anchored in rigorously estimated parameters from the same context, and/or if they fail to build in key aspects of the economy – including possible frictions in mobility, in supply responses, and in price adjustments – they are not well suited to prediction. This is illustrated by the comparison between two pilot estimations for the same project, the Chad-Cameroon Transport Corridor. One estimate used a reduced form approach and another a spatially disaggregated computable structural model. As discussed in Annex D, under the structural model, the transport project would increase the share of workers employed in agriculture in both countries, but using the RF approach, which is based on historical impacts, the opposite is true. This could be at least in partly due to the use of model parameters from a different context (Ethiopia), in addition to the difficulty of capturing key frictions in the model. Rigorously estimated empirical relationships from actual data likely furnish a more reliable approach to estimating average or aggregate impacts. See Annex D for a detailed discussion of the issues encountered and likely reliability of the two approaches for the Cameroon-Chad Transport Corridor case. In another case with two estimates, the Ethiopia agricultural growth project, a CSM was used, which assumed that skilled labor is fully employed; but studies have highlighted the fact that the rapid expansion of higher educational programs has led to high unemployment of college graduates (Wobse, Menuta, and Liga 2022). Ultimately, results are only as reliable as the parameters used and the real-world complexities and constraints captured. 37 If used to estimate the impacts of an intervention directing public financing to private commercial enterprise, an additional difficulty arises, in establishing the degree of attribution to that financing. See Box. 32 Econometric Estimation and Extrapolation Econometric methods can also provide a reliable basis for estimation in cases where suitable observational data exist. Estimates based on econometric modeling are easier to deploy than a specially designed impact evaluation: they allow econometric techniques to substitute for an actual control group. Estimators can utilize existing econometric studies (ex-ante) or estimate their own models. These approaches also vastly expand the range of intervention types that can be assessed, to include broad economic policy or governance issues, which are not amenable to impact evaluation (such as when conducted on cross-country data.) Econometric estimation and extrapolation was used for ex ante pilots to estimate consumption spillover jobs in Angolan villages receiving large cash transfers and in Bangladesh’s project to enhance the digital economy. In the case of the Kenya Digital Economy project, not only were the data from the same context, but it was also possible to utilize techniques (such as fixed effects) to address key sources of simultaneity bias. Nonetheless, there are typically shortcomings associated with the data available for such estimations, since they are not specially designed to answer the question of interest. For the Mali: Promote Access to Finance, Entrepreneurship and Employment and the Lesotho pedestrian bridges estimations, for example, the data were less than ideal. In Mali, there were relatively few observations in the enterprise dataset used, and unobserved factors affecting a firm’s creditworthiness are likely an issue.38 For Lesotho, nationally representative survey data were used to estimate the impacts of a small number of footbridges, but more data on the local contexts would have been valuable for helping to address any program placement bias. While the reliability of estimates from these methods can vary considerably case by case, they are fairly straightforward to implement. Some additional estimates, modeling, or assumptions may be needed to arrive at jobs outcomes, as econometric studies may concern a jobs-proximate outcome, such as production levels or investment rates. Ex ante, extrapolation is also usually required, to another context, intervention, and/or into the future. And external validity issues (arising with the use of imported parameters) complicate matters just as they do with IE results. Value Chain Mapping and Surveys Field survey-based methods in principle provide direct measurement, but they are more costly and can suffer from data quality issues. Nonetheless, there is an emerging body of practice, among these pilots and previous value chain estimations. These involve multiple steps in order to achieve a suitable sampling frame and representative sample. These were challenges in both the Uganda eco-tourism and forestry and the IFC loan cases. Moreover, unless combined with suitable econometric techniques, these surveys do not necessarily elucidate the counterfactual. 4.3. Methodological Lessons Although some of the pilots used creative approaches in a data- and time- limited environment, hindsight provides some lessons on possible avenues for improvement. In practice, it proved 38 Various attempts to control for unobserved, endogenous determinants of access to credit were made, but were not successful. 33 difficult in observational studies to conduct unbiased estimation in a manner that addressed all possible sources of bias. Surveys ran into serious sampling difficulties. Causality was not always fully considered, and the baseline was often used as the counterfactual. The issue of causal attribution for interventions involving direct financial support to private enterprise (in whether in the form of equity, grants or loans) presented a unique causality issue. Estimating the additionality of investment, jobs, or other outcomes due to such support is especially tricky, as discussed in Box 2. But there are some potentially promising techniques available that were not attempted. Improved efforts to prepare sampling frameworks could improve reliability. In the pilot cases where special surveys were undertaken, it proved challenging to draw a representative sample. For cases such as the IFC financings, it made sense to map specific value chains centered on a given client company. The idea then would be to use sequential chain sampling, wherein suppliers or customers are listed and sampled at each of a number of value chain linkages. However, in practice more basic issues such as high non-response rates and incomplete client cooperation compromised representativeness at the first chain link, so subsequent linkages were not pursued through surveys. More efforts to enlist the cooperation of the important enterprises may have been possible. Another frequent sampling challenge relates to sectors with a high degree of informality, such as tourism, agriculture, and forestry. Relying on a firm registry as a sampling frame will often mean missing significant actors in the value chain. For such cases, sampling can be done along spatial axes (say, by radial distance from the forest) and/or from a spatial block grid.39 In general, the options of more extensive use of sector data and of low-cost phone surveys can provide additional empirical input to estimations. In the Rwanda Electricity Development Policy Series, in part due to limited resources, estimators used macroeconomic, labor force, and aggregate sector data in an event study approach to ex post estimation. This was combined with qualitative indicators from the World Bank Enterprise Survey regarding the degree to which electricity was constraining to firms. The approach taken was reasonable given the resources available but left a relatively high risk of a false null result. If more resources had been available, it may have been possible to utilize more disaggregated data from the national utility on availability and outage times by location, customer lists pre and post, and a phone survey of electricity utility customers stratified by electricity usage, to include retrospective questions on employment.40 Establishment of the counterfactual – clearly central to causal inference, remained a challenge for several pilots. In the absence of a better method, estimators typically used the baseline for the counterfactual (see, e.g., the Bangladesh Private Sector Development Project.) This approach 39 See, e.g., the World Bank Jobs Group’s Value Chain Mapping and Sampling tools. 40 There will be bias due to selection into the existing customer cohort that would need to be addressed, just as with using the WBES, but if changes to service experienced by the customer were small, one could make the simplifying assumption that electricity service is not a significant determinant of firm entry and exit. If such a survey were done prior to and after the interventions(s), this would also help deal with compositional / attrition and recall issues. 34 implicitly assumes that there would be no progress or deterioration in outcomes in the absence of the intervention, no shocks and no economic fluctuations. Yet in some cases, more credible assumptions could have been made. Simple trend analysis (as was done in the Rwanda Energy DPO), synthetic controls, or other econometric methods using ancillary data may often be feasible for constructing a counterfactual when impact evaluation or structural economic modeling are not. 35 BOX 2: Causation and Public Funding for Private Goods Causation is an especially thorny issue whenever public funding is provided for private goods, such as an enterprise’s productive assets. This is because economic actors are more likely to make the same investments without such support, even if more slowly (such as by using retained earnings or borrowing elsewhere) because they are privately profitable. In contrast, in more classic areas of public sector intervention involving public goods and other market failures, special concessions or higher public financing are usually necessary to spur investment; in infrastructure, education, health, environment, legal, and other typically public spheres, market failures are more pronounced and typically cannot be addressed with direct public funding. Thus, for additionality of private investment and economic activity is an issue, which has been explored in a variety of evaluation efforts, some using randomized controlled trials to capture the counter-factual of “no grant” (see, e.g., Shroj et al. 2020). Among the pilot cases, this issue pertains in particular to the Uganda forestry and tourism project, some elements of the Tonga blue economy project, and the two IFC interventions to support the Ghana ceramics company and Mozambique avocado investments. For its part, the IFC estimates jobs impact of the total private investment associated with its financing, rather than the intervention of financing itself. For a matching grant program under IDA, the “project” is the matching grant program, whereas for IFC, the “project” is the total private investment made by the client. To address additionality issues, the IFC reports at a more aggregate portfolio level using separate methods, rather than when estimating the its impact through specific activities. Therefore, there was no attempt to establish the “no IFC support” counterfactual for these two interventions. This makes comparison of impacts across pilot cases or individual interventions more difficult. There are ways to factor in additionality for cases of direct public support for private goods (grants, financing, and asset or input provision). First, one could consider how constrained the specific client or beneficiary’s access to external capital has been and its recent historical investment rate. One could also use a firm-specific financial model to estimate an investment and employment trajectory with and without the intervention and with realistic demand-side constraints. In the case of the Ghana ceramics financing, such an exercise could help clarify how the firm could have responded to an exogenous doubling of demand without the IFC financing, whether the IFC financing was sufficient to enable this expanded capacity, or whether it would have had to undertake further investment to meet this demand; Some similar modeling is done as part of the AIMM framework, but here we argue for a specific ex post effort for these two activities. 36 Causation is an especially thorny issue whenever public funding is provided for private goods, such as an enterprise’s productive assets. This is because economic actors are more likely to make the same investments without such support, even if more slowly (such as by using retained earnings or borrowing elsewhere) because they are privately profitable. In contrast, in more classic areas of public sector intervention involving public goods and other market failures, special concessions or higher public financing are usually necessary to spur investment; in infrastructure, education, health, environment, legal, and other typically public spheres, market failures are more pronounced and typically cannot be addressed with direct public funding. Thus, for additionality of private investment and economic activity is an issue, which has been explored in a variety of evaluation efforts, some using randomized controlled trials to capture the counter- factual of “no grant” (see, e.g., Shroj et al. 2020). Among the pilot cases, this issue pertains in particular to the Uganda forestry and tourism project, some elements of the Tonga blue economy project, and the two IFC interventions to support the Ghana ceramics company and Mozambique avocado investments. For its part, the IFC estimates jobs impact of the total private investment associated with its financing, rather than the intervention of financing itself. For a matching grant program under IDA, the “project” is the matching grant program, whereas for IFC, the “project” is the total private investment made by the client. To address additionality issues, the IFC reports at a more aggregate portfolio level using separate methods, rather than when estimating the its impact through specific activities. Therefore, there was no attempt to establish the “no IFC support” count erfactual for these two interventions. This makes comparison of impacts across pilot cases or individual interventions more difficult. There are ways to factor in additionality for cases of direct public support for private goods (grants, financing, and asset or input provision). First, one could consider how constrained the specific client or beneficiary’s access to external capital has been and its recent historical investment rate. One could also use a firm-specific financial model to estimate an investment and employment trajectory with and without the intervention and with realistic demand-side constraints. In the case of the Ghana ceramics financing, such an exercise could help clarify how the firm could have responded to an exogenous doubling of demand without the IFC financing, whether the IFC financing was sufficient to enable this expanded capacity, or whether it would have had to undertake further investment to meet this demand; Some similar modeling is done as part of the AIMM framework, but here we argue for a specific ex post effort for these two activities. Finally, for impact evaluations or other ex post methods, parameterizing as much of the JToC as practicable can expand the external validity and scope for improving future project designs. Data collection on more outputs and outcomes along the theory of change (as well as contextual factors) can enhance learning about the key determinants of impact (or lack thereof) and what 37 mechanisms to be cognizant of in future interventions.41 For example, in our hypothetical roads project (section 3.2), by measuring transport, farmgate, other prices, traffic levels and their composition, as well as ultimate jobs outcomes, one could parameterize each linkage and understand where the ToC may break down (or not). This would also help inform project and M&E design in future roads projects. Additional study of market relationships and prices is being done in conjunction with the ongoing DIME IE on Rwanda’s rural roads. Standardization Efforts to standardize and streamline estimation approaches for interventions with the same theory of change and similar data and information availability can be worthwhile for some types of common intervention. Standardized tools such as those under the IFC’s AIMM Framework can reduce estimator (optimism/pessimism) bias, make the work more efficient, and incorporate new knowledge and data more systematically. However, in general terms, such standardization in modeling is not feasible or appropriate. Project designs, JToC’s, contexts, and data availability vary so much across the full range of interventions that approaches will usually need to be adapted to these specific situations. 5. Recommendations The experience of the JET IDA19 Policy Commitment 12 pilot program illustrates the potential magnitude of the indirect impacts of public interventions on jobs outcomes. Given the core role of labor as an input to the economy and jobs in delivering social welfare, quantifying these impacts is important for informing policy. If important social impacts such as improved jobs outcomes are either omitted from or exaggerated in cost benefit analysis, policymakers may make prioritization errors. Policymakers may also miss opportunities to achieve greater social impact. For example, they may focus too many resources on direct service provision to households or workers rather than interventions that cause broader, less direct, jobs benefits. They may conceive of “jobs projects” in too limited a sense, with a focus on direct jobs benefits, when other interventions with indirect ones would be more cost effective. For example, they may intervene directly at the enterprise level rather than enabling a broader range of enterprises to create jobs. They may utilize the notion of indirect jobs benefits as a justification without testing this assumption with evidence. They may fail to estimate the benefits indirectly accruing to relatively poor segments of the labor force. Ultimately, to make better policy decisions requires an estimation of social costs and benefits that include indirect jobs. More comprehensive analysis can support policy dialogue not only by informing policymakers of important benefits but also by educating stakeholders. In addition, monitoring and ex post estimation of indirect jobs benefits is essential to maintaining a focus on jobs outcomes during implementation as well as for developing the evidence base needed to improve ex ante estimation and project/intervention design. 41 This pertains to the issue of external validity as well, which is much discussed with respect to IE. External validity questions are not confined to IE, however, and are pertinent to almost all empirical studies conducted using data from a specific context or population. 38 Yet much more experimentation and experience are needed to validate, refine, and enrich the available lessons on methodologies and, ultimately, on policy. Based on the experiences from this pilot program and other similar efforts, we recommend the following: 1. Aim to derive portfolio benefits by aiming to conduct more fulsome jobs impact estimation for a wide range of relevant interventions. a. Estimation should be done with due care to protect objectivity and be part of portfolio and design decision-making. b. Some threshold level of jobs and total economic rates of return is typically the main decision metric, but returns could also be compared across intervention ideas when there are insufficient resources to pursue all interventions meeting an institutional threshold for cost effectiveness. 2. Because these estimations require resources and scarce staff time, devise and implement a strategy to expand this practice and its benefits. When the World Bank Group committed to IDA funders, there was no corporate effort to optimize the exercise for learning purposes. Although the team attempted to balance efforts with strategic aims in mind, without a sufficient number of pilot cases initially, the focus was on completing the requisite number. A strategy should be established with a view to: a. Apportioning resources in a manner that will optimize learning. b. Establishing meaningful and comparable metrics; c. Valuing jobs impacts on par with other economic benefits of interventions, as an input to program decision making, learning, and results reporting. 3. The strategy should aim to optimize learning and better decision-making. a. Ideally, both ex ante and ex post estimates would be done for relevant interventions to build the stock of lessons on reliability and likely project impacts on jobs. Where rigorous estimates are available, a “blind” effort to conduct “low cost” comparator estimates could generate early lessons. b. In addition, planning early to follow up existing ex ante cases with ex post estimation will usually increase the evidence base for further estimation. Institutional constraints related to budgeting and project closure issues should be re-examined. c. Purposive piloting of new methods, including more advanced econometric techniques using observational data, should be pursued. d. As feasible and relevant, greater effort to capture jobs outcomes could be pursued across more of the WBG’s relevant Impact Evaluation agenda. 4. Put in place institutional measures to strike the right balance between timeliness, integration with policy or project team deliberations, and objectivity. If assessments such as these are not seen as credible, they are not likely to affect prioritization or design decisions. a. To enable frank problem identification and discussion, peer review – probably blind peer review – or another similar mechanism may be needed to counter incentives to skew analyses more favourably. 39 5. Standing capacity would facilitate efficient deployment. Project teams generally have little incentive to embark upon additional analysis, so the barriers – the need to identify estimators, possibly econometrician/advisers, and funds, as well as the time involved in engaging with them – need to be reasonably low. IFC has established special capabilities to expedite and standardize estimations and this has enabled it to extend jobs estimation to 80 percent of its portfolio within a few years. Further learning and refinement of these and other tools would help in lowering barriers while building credibility and reliability. 6. To enhance methodological soundness: a. All estimations would ideally start with a consideration of the economy’s labor market conditions, with an examination of unemployment, under-employment, informal self- employment, and wage employment rates.42 b. Estimations should attempt to factor in what is known regarding which constraints to jobs are among the most severe, as these are important background conditions for assessing impact. Though it may not have been known at the time, if, for example, travel time to work was not a key constraint to improving jobs outcomes in Dar es Salaam, then ex ante estimates would be less likely to over-estimate these impacts. c. Ideally, ex post estimation efforts should aim to understand why a theory of change proved incorrect or results were not as anticipated (see discussion on ex ante vis-à- vis ex post results). This would require collecting data that would permit a parameterized JToC. d. All estimations, whether ex ante or ex post, should include an explicit statement of the counter-factual, how the “intervention” being assessed is defined, including whether it includes private contributions, and a justification for its underlying assumptions. Estimations for direct financing support to for-profit firms should address, at least qualitatively, the issue of likely additionality of investment or employment outcomes.43 Those that provide support to a number of enterprises as part of a grant or financing program should seek to establish a realistic counter-factual of positive private investment as well, based on the available empirical evidence. 7. To balance costs with reliability, jobs impact estimations should endeavor in the first instance to utilize existing data and studies, deploy advanced econometric techniques as available, such as via academic partners, and utilize supplementary data collection creatively as needed. 42 The Global Labor Database, currently being built and updated, will provide these indicators in a convenient public place. The IFC also provides some of this in a streamlined tool. 43 In cases of IFC support, particularly that provided on a commercial basis, additional elements from the AIMM framework and general report findings could suffice. 40 REFERENCES Alonso Alfaro-Ureňa, A., I. Manelici, and A. J. P. Vasquez (2022). “The Effects of Joining Multinational Supply Chains: New Evidence from Firm-to Firm Linkages.” Quarterly Journal of Economics. Vol. 137, Issue 3, August. Balboni, G. Bryan, M. Morten, and B. Siddiqi, 2020. Impact Evaluation of the Dar es Salaam Bus Rapid Transit System (BRT) Tanzania. https://www.3ieimpact.org/sites/default/files/2020- 03/IE110-DPW1.1029-Tanzania-BRT.pdf Bürgi, C., Hovhannisyan, S., and C. Mondragon-Velez, 2020. “GDP-Employment Elasticities across Developing Economies”, Forthcoming, International Finance Corporation. Canadian Energy Center, 2020. “Understanding the ripple effect of oil and gas jobs in Canada . How one job in oil and gas spurs employment across multiple sectors.” https://www.canadianenergycentre.ca/understanding-the-ripple-effect-of-oil-and-gas- jobs-in-canada/ Cubbin, J., & Stern, J. (2006). The Impact of Regulatory Governance and Privatization on Electricity Industry Generation Capacity in Developing Economies. The World Bank Economic Review, 20, 115-141. https://doi.org/10.1093/wber/lhj004 Dasso, R. and F. Fernandez, 2015. “The effects of electrification on employment in rural Peru,” IZA Journal of Labor & Development (2015) 4:6. Deaton, A., 2020. “Randomization in the tropics revisited: a theme and eleven variations” http://www.princeton.edu/~deaton/downloads/Deaton%20Randomization%20revisited% 20v7%202020.pdf Donaldson, D., 2018. “Railroads of the Raj: Estimating the Impact of Transportation Infrastructure,” American Economic Review, April 2018, 108 (4-5), 899–934. GIZ, 2020. “Monitoring of employment effects: Workbook for practitioners.” July. Garsous, G., D. Corderi, M. Velasco, and A. Colombo, 2017. “Tax Incentives and Job Creation in The Tourism Sector of Brazil’s SUDENE Area.” World Development 96: 87-101. Gertler, P., S. Martinez, P. Premand, L. Rawlings, C. Vermeesch, 2016. Impact Evaluation in Practice: Second Edition. World Bank. Grimm, M. and A. L. Paffhausen, 2015. “Do interventions targeted at micro-entrepreneurs and small and medium-sized firms create jobs? A systematic review of the evidence for low and middle income countries. Labour Economics 32. Ianchovichina, E. and S. Lundstrom, 2009. “Inclusive Growth Analytics: Framework and Application.” World Bank Policy Working Paper Series No. 4851. March. IFC, 2013. Jobs Study: Assessing private sector contributions to job creation and poverty reduction. January. IFC, 2020. “Ground Work towards a World Bank Group Jobs Measurement Framework IFC Proposal for the Supporting Effective Jobs Lending at Scale Program (SEJLS) under the Jobs Multi-Donor Trust Fund Phase 1.” Draft for Comment January 22. ILO. Employment Impact Assessment: https://www.ilo.org/global/topics/employment- intensive-investment/themes/empia/lang--en/index.htm. Jouanjean, M. and D. Willem te Velde, 2013. “The role of development finance institutions in promoting jobs and structural transformation: a quantitative assessment.” Overseas Development Institute. Krishnan, S.B., R. Karlen, T. Peterburs, and S. Tokle 2017. Monitoring and evaluation of jobs operations (English). Jobs M&E Toolkit; no. 1 Washington, D.C. : World Bank Group. http://documents.worldbank.org/curated/en/530811506021536510/Monitoring-and- evaluation-of-jobs-operations Limestone Analytics (forthcoming). Measuring Indirect Jobs Impacts: A Decision Framework and Available Methods. McKenzie, D., 2017. How effective are labor market policies in developing countries? A critical review of recent evidence. IZA Discussion Paper Series, No. 10655, March. http://ftp.iza.org/dp10655.pdf Moneke, N., 2020. “Infrastructure and Structural Transformation: Evidence from Ethiopia.” Dissertation, London School of Economics Moneke, N., 2020. “Can Big Push Infrastructure Unlock Development? Evidence from Ethiopia,” unpublished working paper/forthcoming. Ngai, L. Rachel and Christopher A. Pissarides, “Structural Change in a Multisector Model of Growth,” American Economic Review, March 2007, 97 (1), 429–443. Osborne, T. 2004. “Market News in Commodity Price Theory: Application to the Ethiopian Grain Market, The Review of Economic Studies, Volume 71, Issue 1, January 2004, Pages 133–164, https://doi.org/10.1111/0034-6527.00279 Schwartz, J., L. Andres, and G. Dragoiu, 2009. Crisis in Latin America: Infrastructure Investment, Employment and the Expectations of Stimulus. Journal of Infrastructure Development. 1(2) 111–131. Ravallion, M. 2020. « Should Randomistas (Continue to) Rule ? » NBER Working Paper 27554. https://www.nber.org/papers/w27554. Robalino, D., and I. Walker (2017) “Economic Analysis of Jobs Investment Projects” Jobs Working Paper No. 7. https://openknowledge.worldbank.org/handle/10986/28219. Srhoj, S., M. Lapinski, M. & J. Walde (2021). “Impact evaluation of business development grants on SME performance.” Small Bus Econ 57, 1285–1301 (2021). https://doi.org/10.1007/s11187-020-00348-6. Simas, M., & Wiebe, K. (2021). Assessing the job impact of stimulus investments in Latin America and the Caribbean. SINTEF. Steward Redqueen, 2020. “Joint Impact Model: Methodology Paper.” November. https://jointimpactmodel.com/doc/JIM_Methodology_-_JIM_1.3.pdf Victoria Transport Policy Institute, 2020. Generated Traffic and Induced Travel: Implications for Transport Planning. https://www.vtpi.org/gentraf.pdf. Wobse, Belete Adelo, Yohannes Haile Menuta, and Abebge Debu Liga (2022). “Modeling of waiting time to first employment of graduates at wolkite university, Ethiopia: Application of accelerated failure time models.” Cogent Education, Volume 2022, Issue 1. https://www.tandfonline.com/doi/full/10.1080/2331186X.2022.2143032?scroll=top&nee dAccess=true&role=tab World Bank 2008. Applying the HDM-IV Model to Strategic Planning of Roadworks. (Sept.) https://openknowledge.worldbank.org/handle/10986/17419 42 ANNEX A: Definition of “job” To discuss jobs impacts requires first a definition of a “job:” Job: A set of tasks and duties performed, or meant to be performed, by one person for a single economic unit. Persons may have one or several jobs. Those in self-employment will have as many jobs as the economic units they own or co- own, irrespective of the number of clients served. The above definition is based on the current ILO definition, with one key difference. 44 This definition is silent on the degree to which the tasks or duties performed are for an economic unit producing for own consumption versus for remuneration (in cash or kind), since the ILO relatively recently changed its definition of “employment,” “job,” “labor force,” and related terms to require that production not be mainly for own consumption.45 How one considers work that may not fit this criterion can be determined based on the country context and available data. However, in particular in more developed contexts, and where economic transformation is a key goal of the World Bank’s engagement, the definition of a job could be further narrowed to include only work for an economic unit engaged in activities for pay or profit, rather than mainly for subsistence or own consumption. Employment (current ILO Definition): Persons in employment are defined as all those of working age (15 and over) who, during a short reference period, were engaged in any activity to produce goods or provide services (mainly) for pay or profit.46 In contexts where it is either difficult to distinguish among economic units producing primarily for own consumption and those that are not; or where it could be misleading in productivity or welfare terms to draw a sharp distinction between them, that requirement may be dropped but should in any case be complemented with a measure of job quality. 44 The ILO’s glossary is found at: https://www.ilo.org/ilostat-files/Documents/Statistical%20Glossary.pdf. 45 https://www.ilo.org/global/about-the-ilo/newsroom/news/WCMS_647540/lang--en/index.htm. 46 Although the ILO makes the categories of subsistence work and employment distinct and mutually exclusive, it fails to explicitly include self-employment when it refers to employment as comprising work performed for others in exchange for pay or profit (emphasis added). ANNEX B: Background and Rationale for Channels of Job Impact Terminology Discussion of indirect jobs issues is sometimes complicated by the fact that there are currently multiple nomenclatures in use to classify jobs impact channels.47 This Annex lays out definitions for World Bank Group usage, which are designed to improve the clarity and consistency in accounting for, quantifying, and communicating about the jobs impacts that may result from WBG and other interventions, whether policy or regulatory reforms, investments, or other programs.48 The next section of this Annex provides a brief review of jobs impact definitions currently in use. It is followed by a section containing proposed definitions for the World Bank Group, their rationales, and graphics illustrating concepts. Definitions currently in use A brief review of recent studies and donor documents reveals variation in the terms used to describe the possible jobs impacts of public and private interventions.49 This is to be expected given that some of the terms lack an established technical definition. All authors distinguish in some way between “direct” and “indirect” impacts, but they do not always define these terms in the same way. Some authors add a third category to these two and some include a fourth. Some utilize terms such as “primary” and “secondary” to classify impacts as well, and in ways that may not always be consistent. Relevant impact evaluations (IE), for example, are primarily concerned with quantifying employment impacts, without necessarily attempting to estimate each channel of impact separately. 50 IE’s sometimes refer to “direct” impacts on jobs. 51 For example, in a systematic review of studies of the impact of support to micro, small, and medium sized businesses, Grimm and Paffhausen (2015) use the term “direct” to describe employment effects. In addition, IE’s must often consider “spillover” effects, whether positive or negative, intended or unintended, particularly as they must consider possible contamination of control groups. Otherwise, impact evaluations do not adhere to a standard lexicon regarding jobs impacts.52 Policy analyses also sometimes distinguish between direct and indirect jobs effects. For example, in considering the employment effects of a large infrastructure investment program in the US, Romer and Bernstein (R&B, 2009) defined “direct” jobs impacts as those coming from the direct hiring by the program, plus those “hired to produce the goods demanded as the result of 47 For example, among the IDA19 JET Theme the WBG committed to estimating “direct” and “indirect” jobs impacts, but it is not clear which definitions to use. 48 “Public intervention” means any intervention financed, undertaken, or instigated and coordinated by a publicly financed entity, even if the intervention is in the private sector. 49 We do not attempt an exhaustive review of all uses of such terms, but merely to demonstrate the variation in usage. 50 For example, the employment effects of rural electrification, rural roads and other area-specific interventions may be estimated with reference to an untreated rural area. See, e.g., Dasso and Fernandez, 2015. 51 Garsous et al. (2017) exemplifies this terminology in a paper finding that fiscal incentives led to a substantial increase in tourism employment in the SUDENE area. 52 See, e.g., Gertler et al., 2016. The “spillover” concept is central to impact evaluation, given the possible problem of spillovers onto a control group. However, a sample of impact evaluations published by 3ie shows that there is no standard definition of “spillover” in use in the IE literature. ii subsidies (or investments) targeted to specific activities (such as smart electrical meters and software systems for health IT).” They considered indirect jobs impacts to be those caused by the increased spending by newly employed workers. Table A1: Summary of Definitions in Use to Describe Indirect Jobs Impacts Caused by increased Caused by increased Caused by improved consumption demand demand for suppliers access to infrastructure due to higher and distributors employment income Schwartz et al., Tertiary Secondary None 2009 IFC, 2013 Induced Indirect Secondary (infrastructure only) ILO Induced Indirect Spin-off/development (a developed or maintained asset) GIZ (employment Indirect or induced, Indirect or induced, None programs only) depending on target depending on target beneficiaries beneficiaries Steward Indirect Supply Chain Power or finance-enabling Redqueen (only) Usage also varies among donor organization authors, as summarized in Table A1. Some have proposed clearer definitions. Schwartz et al. (2009), in considering possible employment effects of infrastructure projects, break direct effects as R&B define them into “primary” and “secondary” impacts, the latter of which they call “indirect,” and add another channel. This typology remains in use today and is as follows:53 • Primary Impact: Those directly employed on site to undertake the task at hand; • Secondary Impact: Those indirectly employed in the manufacture of materials and equipment as a result of the initial investment (considered direct by R&B); and • Tertiary Impact: The induced employment generated by the direct and indirect jobs created, including the jobs supported by consumer expenditures resulting from wages in the two previous levels (considered “indirect” by R&B). None of the above definitions explicitly considers a key channel of impact: for many interventions, the primary economic justification is the effect of improvements to a productive factor —whether to the quality, supply, reliability, and/or user cost— on producers and enterprises that may use it. IFC (2013) addresses this omission, setting out four categories of jobs 53 See, e.g., Canadian Energy Center, 2020. iii impacts. In addition to direct jobs impacts, which they implicitly define as jobs created by direct recipients of program interventions, they adopt the following:54 • Indirect jobs: employment changes in suppliers and distributors. (This is considered part of “direct” jobs by R&B; and called “secondary (indirect) impacts” by Schwartz et al., 2009. See Table 1) • Induced employment: jobs resulting from direct and indirect employees spending more and increasing consumption (called “indirect” by R&B and “tertiary/induced” employment by Schwartz et al., 2009) • Secondary effects: job creation through benefits of improved access to infrastructure, such as access to more reliable power allowing enterprises to produce more, and more efficiently (not previously defined). (In chapter six of the same report, this is expressed as “Second-order or growth-related jobs.”) The ILO, in outlining the potential jobs impacts of public investment, utilizes a similar set of definitions to the IFC’s, as follows: • Direct employment is created directly by construction, operation or maintenance activities (including workers directly recruited by contractors and subcontractors, technicians, supervisors and other skilled professional staff). • Indirect employment is created in the backward-linked industries, supplying tools, materials, plant and equipment for construction and maintenance activities. • Induced employment is created through forward linkages when households benefiting from direct and indirect employment spend some of their additional income on goods and services in the economy. • Spin-off/development impacts comprise secondary employment created as a result of an improved or maintained asset within the areas of influence. 54 These definitions are used in some subsequent evaluation(s) of development finance organization (DFI) interventions (e.g., Jouanjean and te Velde, 2013). iv Figure A1: GIZ Definitions of Direct, Indirect, and Induced Effects of Employment Programs A more recent draft paper by the IFC (2020), which describes approaches for modeling of jobs impacts of interventions, bridges two of the above terminologies, stating that “employment generation can be decomposed into direct, indirect, and induced effects” (as in Schwartz et al., 2009), while also noting differences between “upstream” and “downstream” effects. But these three categories are not adequate for framing their methodological contributions, and IFC (2020) also utilizes “secondary growth impacts” as introduced in IFC (2013).55 The Joint Impact Model developed by Steward Redqueen (2020) for a group of Development Finance Institutions, has three similar categories to those of Schwartz et al., but uses different terminology, and adds two special categories for finance and energy.56 Other donors and agencies have adopted their own definitions as well. For example, the German Development Agency (GIZ, 2020) classifies the jobs impacts of employment and training programs into three categories. It refers to direct effects as all changes in outcomes among its beneficiaries (including employment) that are caused by the intervention, adopting the definitions shown in Figure A1. It considers indirect effects to be spillovers to a target population from direct beneficiaries of the program, with “target population” defined by each program. For GIZ, “induced” effects are spillovers among individuals and firms that are not part of the target group. However, as shown in Figure 1, the distinctions between “indirect” and “induced” are determined by the project’s definitions of “target 55 It states, for example, that “in the case of infrastructure, the methodologies employed estimate the increase in the capacity of real sectors to operate more efficiently or grow at a faster pace. Therefore, the focus of the infrastructure models lies on appropriately representing its enabling effects on the real sectors.” 56 They define them as follows: Direct: impacts at the client company/ project; Supply chain: impacts at the client company/ project’s suppliers and their suppliers; Induced: impacts associated with the spending of wages earned by employees of the client company/ project, its suppliers and their suppliers; Finance enabling: impacts at companies, suppliers of companies, and their suppliers associated with the financial intermediary’s lendin g; Power enabling: impacts associated with the additional output created by companies that use the additional power generated by the client company/ project, as well as by non-power using firms in their supply chain (e.g. small-scale agriculture). v population” and “direct beneficiaries.” In addition, both terms include possible multipliers and spillovers, as well as substitution and displacement effects. A.1 Principles of our Nomenclature Our nomenclature builds on and refines existing classifications by: (1) retaining all distinctions between channels of impact previously identified; (2) broadening applicability to all relevant types of interventions in a logically consistent manner; (3) adhering broadly to common English usage, while being as precise and descriptive as possible; and (4) avoiding terms connoting a magnitude or importance of impacts that may be misplaced. And (5) adopting as descriptive and intuitive terminology as possible. We first classify all possible effects as belonging to the categories “direct” or “indirect.” Under standard English usage, these two terms exhaust all possibilities. However, we break “indirect” impacts into sub-categories to retain distinctions previously introduced. We propose a more specific definition for previous usage of “direct” jobs impacts and a broader definition of “indirect” ones. All categories of jobs impacts could encompass both temporary and more lasting jobs, and definitions are as presented in Section 2, above. Under these definitions, jobs created to construct a bridge, school, or hospital would be classified as direct jobs, as would jobs to operate, maintain, and staff them in the future. Changes in any jobs outcomes in the energy sector which are caused by an intervention designed to shift the sector from fossil-fuel to renewable-energy technologies, such as a technology-specific subsidy or tax, public/private investment program, or other direct intervention would also represent “direct” jobs effects. Similarly, if a program to provide financing, subsidies, training, or other private support directly to firms led those firms to adjust their staffing or employment terms, those would also be counted as direct jobs impacts, because those firms are the “treated” entities. One can also consider any changes to policies that directly tax or subsidize employment to be direct jobs impacts, since the workers and firms involved are the treated entities. Reforms to working hours restrictions or to minimum wages would cause jobs impacts directly as well for all entities subject to and compliant with these policies. Direct jobs impacts can also be negative, for example in a project that aims to reduce excess staffing costs for a public service; or when labor-saving technologies are adopted, and staff downsized. Many sector interventions result in both direct and indirect jobs impacts. To illustrate, financial sector jobs impacts resulting from reforms to the financial sector would be direct ones; whereas jobs resulting from the change in availability of financing to other sectors of the economy would be indirect. A program to provide childcare would result in direct jobs in the childcare sector and could generate indirect ones to the extent that labor supply expands. Similarly, the effects on staffing levels at skills training centers or teacher training colleges that resulted from building, expanding, or improving those centers would count as direct jobs impacts; but impacts on future trainees of these institutions would be indirect ones. vi The “directness” of impacts should not be confused with their magnitude or importance. For instance, in the case of a typical public investment project, such as a bridge or training center mentioned above, the direct jobs created would usually not be the primary justification for the project. Typically, small differences in the direct jobs associated with various alternative investments would not be sizeable relative to investment costs to support investment in one sector or project over another.57 Although there are cases where direct jobs impacts can be an important rationale, such as when projects are undertaken to provide macro-stimulus (to deal with cyclical unemployment or an economic crisis), often, the indirect effects are greater and more aligned with an intervention’s purpose than the direct jobs impacts. In the terminology proposed, we retain these channels, but extend beyond interventions in infrastructure or finance for one of the categories. We have checked that our classification works across a vast array of intervention types and consulted with multiple donors and WBG stakeholders with an interest in the naming convention used. Our terminology is based on common English usage. For example, since the term “induced” is used in conflicting ways, often in ways that contravene standard English usage, we avoid utilizing it. We also propose to avoid terms such as “primary” or “secondary,” as these imply a ranking of importance or magnitude which is often unwarranted. 58 Our proposed nomenclature is more descriptive of the channels of impact than earlier ones and can thereby facilitate comprehension, even for those not familiar with these precise definitions. The primary purpose of many public interventions is to provide an economic value from the improved supply of a productive input, factor (such as electricity, skills, water, access to finance or transport infrastructure), or condition (such as improved regulatory performance, property rights, reliable contract enforcement, or efficient skills matching in the labor market). These enabling conditions are delivered by entities such as a regulatory authority, court system, or employment bureau. Although these entities may expand direct employment, the jobs associated with the improved business conditions are indirect. Simple theory of forward factor usage jobs: According to standard microeconomic theory, when an intervention lowers the cost of a factor or a condition faced by producers, this would induce a response by entities using or reliant on that factor or condition. For example, as the user cost of energy or port services falls, a profit-maximizing firm which uses these inputs will expand production until the potential for increased profit arising from improved factor supply has been exhausted. The effect of this on labor demand depends on whether the improved factor is a “gross complement” of labor. As shown in Figure , if that is the case, a producer will utilize more of both that factor and labor, moving from point L, to M and N. In the long run, firms may also 57 The modest impact at the project level would represent a shift in employment with little net positive effect. In addition, if there is full employment, the direct jobs created will not provide an additional social benefit. In standard cost benefit analysis (CBA), the valuing of public investments is premised on the assumption of full employment. 58 IFC (2013) chapter 6, for example, underscores that “second-order growth impacts” are often the most important impacts. vii invest more capital and so, further adjust their demand for labor. If the project induces an expansion of labor demand, more jobs and/or better remunerated jobs could be an important benefit of the project. These adjustments can arise through (dis-) investment, firm entry or exit, adjustments to capacity utilization, or changes in technology. The “factor usage” jobs effect is potentially empirically tractable so long as the productive consumers/users of the factor can be identified and the level at which they employ and compensate workers measured. 59 Figure A2: Profit-Maximizing Producer’s Factor Choice Labor Employed Input, Factor or Condition Note: This depiction assumes fixed technologies and levels of capital. Most jobs outcomes are due to shortfalls in demand relative to supply; however, there are instances when labor supply is the limiting factor to improved jobs outcomes. This is more often the case for certain segments of the labor force, such as for women who face social or legal barriers or ethnic minorities that may lack threshold skills. Factor usage jobs impacts can also occur when an intervention changes the cost (including the social and opportunity cost) and/or benefit of working and thereby induces a shift in labor supply.60 Based on standard English usage, one could alternatively term this channel of impact as “induced”, but this is not ideal. “Induced jobs” continues to be used in a variety of conflicting ways in discussing jobs impacts, which do not concord well with standard English usage of the word, and aligning usage now is unlikely to prove successful. Moreover, the term “(forward) factor usage jobs” is more descriptive of the cause-and-effect relationship we have in mind. Finally, we strategically avoid the term “secondary” (per IFC, 2013) for this channel to obviate potential misreading of this channel’s importance.61 59 Analysts should consider such potential effects, as they would any economic benefits, with caution and verify them with evidence, first assessing whether: (a) there is evidence of under- or unemployment in the relevant economy; and/or (b) the shift of employed workers between sectors or firms would result in increased labor productivity and/or higher wages. In addition, the size of those impacts should be anchored empirically in the usage levels by existing firms of the expanded or improved factor, the incremental employment patterns observed for such firms, and aligned with the scale of the intervention. 60 In theory, absent a change in labor demand, this would tend to reduce wages. 61 “Induce” has no precise definition in economics, but in common parlance it means to cause, persuade, or make something happen, such as in “inducing participation,” “sleep -inducing tonic” or “medically induced coma.” As such, viii Figure summarizes the relationships between jobs impact channels in our terminology. Figure A3. Categories of Possible Jobs Impacts Jobs Direct Jobs Indirect Jobs Jobs in sector or entities of (Forward) Factor (Backward) Consump- tion direct Usage Jobs Supply Spillover Chain Jobs intervention Jobs Our typology for describing jobs impacts also aligns with feasible paths for monitoring outcomes. Direct jobs results are typically the most straightforward to monitor,62 since the treated entities are readily identifiable and can report changes in employment levels and other relevant outcomes. For economywide treatments, such as a decrease in labor taxes, a representative survey of treated entities (enterprises or individuals) can be carried out. Backward supply chain jobs impacts, such as to provide luggage belts or dining services to a new airport, can be traced by listing the suppliers of directly treated entities and surveying those suppliers. Forward factor user jobs impacts can be tracked by listing and surveying the users of the relevant input, factor or condition. For example, customers of an electricity utility or a port whose service had improved could be identified by the utility or port operator and surveyed to track their employment outcomes. Such methods can provide outcome data relative to a baseline, and if control groups it connotes a relatively proximate impact of a measure, similar to that envisioned in this channel. In transport economics, the effects of a roadway project on traffic and the economic activity associated with it (including employment income) are said to be induced when a lower user cost of travel results in a movement along the (possibly latent) demand curve for travel (“induced traffic”). In contrast, a multiplier, externality or spillover effect occurs when an activity, behavior, or exchange between two parties (such as producer and consumer or producer and energy supplier) has an impact on another party who is outside or ‘external’ to the exchange . That third party does not use, experience, or necessarily observe the changed factor or condition but experiences a cost or a benefit. For example, an improved roadway may induce greater trade in goods and services as well as more trips along the road. That same improvement might also affect congestion experienced by others using an alternate road, causing a spillover effect. Similarly, an enterprise accessing newly available financing may be induced to expand its operations and, in the process, could adjust its use of labor (“induced” jobs). 62 For example, Krishnan et al (2017), a monitoring and evaluation toolkit for jobs outcomes developed by the World Bank, provides well defined metrics and guidance on data collection and processing that covers jobs created directly by beneficiaries of jobs-focused interventions, but not for indirect jobs impacts. ix or counterfactuals can be constructed, impacts can also be estimated. Of the four types, consumption multiplier jobs tend to be the least tractable to measure through survey methods, but a variety of methodologies also exist to estimate such linkages economywide. x ANNEX C: Pilot Summaries This Annex provides one-page overviews of each of the pilot estimations. Pros and cons to the methods used are listed where these are clear. All findings must be treated as tentative, albeit with increasing reliability assigned for cases using methods more robust to selection, program placement, or other sources of bias. Angola: Strengthening the National Social Protection System Project Cash Transfer Program WBG Project Code: P169779 Budget: US$260mn (total project value US$410mn) Pros of approach: Simple to Type: Ex ante implement Components Included in Estimation: (1) Cash Cons of approach: Assumptions or transfer program (known as Kwenda) to poor imported parameters required on the households (US$260 million plus a government share of cash transferred devoted to financing of US$100million). Not included is investment, elasticity of labor use, and degree of local production to component (2) the creation and implementation of meet demand. an institution that provides a permanent safety net system in the country (US$50 million). Noteworthy: Indirect effects are appreciable, but were not the aim of Channels: Direct (treated households’ enterprises) the program, which was initially to and Consumption Spillover. alleviate the effects of new taxes and support households through COVID- Theory of Change: Cash transfers alleviate 19. household capital constraints and risk. This increases household investment and, with it, labor demand. Increased household income has demand spillovers in a context where much of village receives transfers. Methods and data: Econometric evidence from multiple countries (in particular, Ghana), calibrated to Angola. Findings: 62,000-110,000 total full time equivalent jobs created of which 21,000 are indirect. xii Bangladesh: Enhancing Digital Government and Economy (EDGE) Project WBG Project Code: P161086 Budget: $295mn (subcomponents focused on training individuals) Type: Ex ante Pros of approach: Reliable as long as Components Included in Estimation underlying studies and assumptions are, and context is sufficiently similar. Easy to Components 3.2 and 3.5 below. Other components implement. strengthen demand for the trainees and thus support the estimated impact but may also have additional Cons of approach: Evidence used is not indirect impacts not estimated. Project Components from the same country context and not are: Component 1: Enabling Environment for Digital rigorous with respect to estimation of counter-factual / impacts. Government and Digital Economy (US$44 million); Component 2: Transforming Digital Government (US$138.5 million); Component 3: Developing Digital Enablers (US$83.5 million); Subcomponent 3.1. Digital Enablers Coordination (US$15 million); Subcomponent 3.2. Hire and Train Program for Youth (US$28 million); Subcomponent 3.3. Strengthening and Promoting the IT Industry (US$15 million); Subcomponent 3.4. Digitalization of Small and Medium Enterprises (US$3.5 million); Subcomponent 3.5. Establish Training, Research, and Innovation (US$22 million) Channels: Factor Usage Theory of Change and Channels: With a shortage of IT jobs in the country, training individuals for the IT sector will enable more sector development (and service expansion in the economy). Although not explicit, other project components also contribute to expanding the IT sector so that there is demand for a proportion of newly trained individuals. Methods and data: Reviewed literature to establish a causal empirical link between IT penetration and growth (especially from East Asia) and between IT skills training and IT sector expansion (using monitoring data / project “results” linking training to total/incremental IT jobs.) Used a range of plausible impact parameters from this evidence and previous project indicators. Findings: The analyzed subcomponents are expected to train 125,000 individuals, generate 33,000-64,000 jobs and 22,000-42,666 indirect jobs xiii Bangladesh Private Sector Development Project WBG Project Code: P120843 Budget: US$168mn Pros of approach: Use of economywide Type: Ex post model, using actual SAM plus labor market data from Bangladesh, takes into Components Included in Estimation: Support to Special account possible spillovers outside of zones/parks. Detailed data by economic Economic Zones (SEZs) and Special Technology Parks activity and by worker characteristics are (STPs). used. Channels: Factor Usage Cons of approach: No counterfactual established. Theory of Change and Channels: Project sought to address constraints identified in the areas of: (i) access to serviced land, (ii) power provision, (iii) access to skilled labor and gender segregation, (iv) a cumbersome bureaucracy and burdensome contract enforcement procedures, and (v) access to finance. The newly envisaged SEZ policy sought to simultaneously address land, infrastructure, and regulatory constraints to job-creating private investment; enhance potential indirect benefits of zones from spillover effects on local firms through FDI and the adoption of international environmental and social practices; and be less reliant on fiscal subsidies but instead further engage the private sector in the development and operation of zones. Investment-friendly environment for foreign and domestic companies and enhanced efficiencies increase private investment and job creation in the economy as a whole. Methods and data: Estimates are based on an integrated modelling system comprising four linked components. First, the level of SEZs and STPs-specific investment amounts (direct SEZs and STPs-specific) was identified from secondary sources (i.e. BEZA and BHTPA). An economywide model based on a social accounting matrix was used to estimate the economy- wide impacts. Using the LFS 2016-17 microdata, an employment matrix covering gender, skills and locations was constructed. Then an integrated model was developed interlinking the investment module into the SAM-employment integrated model to assess the total employment impacts of SEZs and STPs investments. Findings: The WB project mobilized a total investment of US$3.95 billion, which is associated with 41,000 direct jobs and 1.6 million indirect (actual or prospective) jobs. xiv Bangladesh Private Investment & Digital Entrepreneurship (PRIDE) Project WBG Project Code: P170688 Budget: $500mn Pros of approach: Utilization of available Type: Ex ante country-specific data and project level investment data. Components Included in Estimation Cons of approach: The counter-factual is essentially baseline investment and jobs, Channels: Factor Usage which in a growing economy is unlikely to be correct. Some “impacts” are likely not Theory of Change: additional, therefore. Project seeks to address constraints identified in the areas of: (i) access to serviced land, (ii) power provision, (iii) access to skilled labor and gender segregation, (iv) a cumbersome bureaucracy and burdensome contract enforcement procedures, and (v) access to finance. The newly envisaged SEZ policy sought to simultaneously address land, infrastructure, and regulatory constraints to job-creating private investment; enhance potential indirect benefits of zones from spillover effects on local firms through FDI and the adoption of international environmental and social practices; and be less reliant on fiscal subsidies in the development and operation of zones. Investment-friendly environment for foreign and domestic companies and enhanced efficiencies increase private investment and job creation in the economy as a whole. Methods and data: Social Accounting Matrix-based Simulations, extended with an Employment Module. These are calculated by applying the sectoral composition of actual investments (which will materialize in the timeframe of PRIDE from 2021-2026); it does not consider the investments that will continue to be triggered as a consequence of the project, but which will materialize after the project ends. For the public EZ called Bangabandhu Sheikh Mujib Shilpa Nagar development project (BSMSN), the study used the sectoral distribution of pipeline investments proposed for BSMSN (provided by BEZA) whereas for the Private EZs, the study applied the same sectoral composition used for pipeline investments under PSDSP. Findings: A total investment of US$2.02 billion during PRIDE (and the World Bank share) is estimated to be associated with 233,000 (57,673) direct jobs and 821,000 (203,218) actual and prospective indirect jobs. xv Cameroon-Chad Transport Corridor (Structural modelling Estimation) WBG project code: P167798 Pros of approach: Allows for spatial Budget: US$538mn heterogeneity and labor mobility, as well Type: Ex ante as equilibrium wages in each locality. Components Included in Estimation: Rail infrastructure Cons of approach: Parameters used in investment and signaling modernization, rehabilitation modeling are not estimated within the of bridges and railroad crossings; road rehabilitation, model. No frictions are modeled, so pavement strengthening, and maintenance transition to new static equilibrium not investments for selected sections along Douala captured. Counterfactual road conditions N’Djamena Transport Corridor. Trade facilitation (without project) are simply equivalent to baseline but could have been modeled activities and reforms to remove some of the non- using engineering model. Complex and physical barriers in the trade between Chad and time consuming to implement. Cameroon, building on the achievements of the CEMAC TTFP. Noteworthy: Based on a World Bank working paper, Lebrand (2022). Not included: safety improvements, investments in rail/road transfer platform rehabilitation. Channels: Factor Usage Theory of Change: Improved connectivity improves access to markets, boosts local economic activity and trade, which fosters off-farm employment. Firms and workers respond by switching their location and sector. Locations that are not directly affected by the infrastructure investments are affected through spillovers from economic activity, employment and wages in nearby connected locations. Equilibrium adjustments occur through the equalization of household utility across locations in response to wages and land prices. Forward factor usage channels include a reduction in transport (and therefore production) costs and greater labor demand across multiple sectors. This causes higher non-agricultural employment. Other channels include backward supply chain jobs via suppliers of producers using the road network, as well as consumption demand spillovers. Methods and data: A static, three-sector spatial general equilibrium model (Lebrand, 2022) with multiple locations, varying transport costs, and labor mobility is solved, using parameters from other Sub Saharan African countries. This choice of model reflects a long run perspective with endogenous prices and market clearing. It uses georeferenced household survey data, infrastructure data, and district characteristics from Cameroon. New information was collected on road network expansions, access to the electricity network, and access to Internet fiber xvi backbone, as well as estimates of existing (baseline) travel speeds (reflecting road conditions and border delays). Findings: The model-based estimation was implemented under the scenarios with and without improvements to border crossing times. In the scenario where border crossing times are improved (as, in principle will occur under the project), in Cameroon, the project is predicted to lead to an increase in agricultural and manufacturing sector employment (about 10,000 and 50,000, respectively ) and a reduction of the share of workers in services (about 60,000). Without border facilitation, the model predicts even greater increases in agricultural jobs (about 30,000) and reductions in the share of workers in manufacturing and services (about 10,000 and 20,000 respectively). Thus, border facilitation is key to protecting or expanding manufacturing sector jobs in Cameroon. In Chad, the model with both transport and border policies also predicts increased employment in agriculture (about 74,000) and a reduction in manufacturing and service sectors while that considering only transport predicts labor moving out of manufacturing to agriculture and services. Thus, the parameterized theory (CSM) is not consistent with the JToC. See subsequent page and Annex D. xvii Cameroon-Chad Transport Corridor (P167798) (Reduced form empirical estimates) WBG project code: P167998 Pros of approach: Uses IV for some Budget: US$ 538 million outcomes, for Cameroon (to support non- Type: Ex ante IV evidence). Cons of approach: data available for Components Included in Estimation: Rail infrastructure Chad, one round of a household survey, investment and signaling modernization, rehabilitation and lack of instrumental variables limits of bridges and railroad crossings; road rehabilitation, reliability of results for the country. pavement strengthening, and maintenance Noteworthy: Based on a World Bank investments for selected sections along Douala working paper, Lebrand (2022). N’Djamena Transport Corridor. Not included: safety improvements, improved trade facilitation at border, investments in rail/road transfer platform rehabilitation. Channels: Factor usage Theory of change: Decreased transport costs and times and improved safety and service along corridor will improve access to markets and enhance economic integration within the CEMAC area, boost local economic activity, and foster off-farm employment opportunities. Methods and Data: Household and satellite data are used to generate two sets of estimates: Individual logit regressions with district-level and individual-level control variables: impact of improved transport services on the likelihood of working in various occupations. District level regressions: impact on employment shares in different occupations with the same district level controls. Impacts on market access are extrapolated from a simulation of the existing and improved transport network using a network model, and the reduced form estimates are applied to this change in market access to yield estimates of the impacts of improvements on employment shares by sector. Both aggregate and district impacts are estimated. Two stage least squares estimation is attempted using the optimal road network as an instrument as in Lebrand, 2022, but appears to be a poor instrument for market access, the project output used for jobs estimation. Findings: In Cameroon, the project will lead to a reduction in the share of workers in agricultural occupations (about 25,800), with most of the jobs being relocated to skilled occupations. In Chad, there will be an increase in unskilled jobs in agriculture and non- agriculture (about 47,910) and a decrease in skilled and high-skilled jobs. See Annex D for more details. xviii Ethiopia Second Agricultural Growth Project (Direct measurement Method WBG project code: P148591 Budget: $455 million, before US$80mn additional Pros of approach: Data are from relevant financing. context. Type: Ex post Cons of approach: Difficulty establishing counterfactual. Components Included in Estimation: Better public services and structures, irrigation services, access to agricultural technologies and marketing structures Channel: Supply Chain and factor usage. Theory of Change and Channels: Better public services and structures, irrigation services, and access to agricultural technologies and marketing structures, will lead to improved commercialization through developed value chains (VCs). Commercialization will increase demand for labor / job opportunities along the value chain from input provision to marketing. Higher agricultural and other incomes may also have consumption spillover effects. Productivity shocks can have effects outside project impacted zones through these channels. Methods and data:). Direct Measurement Method: Key stakeholder interviews and trend analysis using customized Jobs in Value Chains survey instrument, with three main modules for employment, production and marketing & sales issues. No explicit assumption for counter- factual growth. Findings: 707,246 temporary jobs plus 227,148 permanent jobs (or 651,495 FTE) in value chain. Out of the total permanent jobs created, a larger share come in low skilled and part time. The program has created about 396,000 jobs for women and 505,000 jobs for youth. xix Ethiopia Second Agricultural Growth Project (Structural modelling method) WBG Project Code: P148591 Budget: $455mn, before 80mn additional financing Pros of approach: Takes account of important responses and interaction Type: Ex ante effects. Recursive modeling simplifies estimation. Components Included in Estimation: Better public Cons of approach: Key shock to services and structures, irrigation services, access to productivity was not rigorously estimated, agricultural technologies and marketing structures though 10 percent was consistent with baseline differences. No matching of outcomes to actual data was done to Channels: Factor Usage and Consumption Spillover calibrate the model. Theory of Change and Channels: Better public services and structures, irrigation services, access to agricultural technologies and marketing structures, will lead to improved commercialization through developed value chains (VCs) and commercialization will increase demand for labor / job opportunities along the value chain from input provision to marketing. Higher agricultural and other incomes may also have consumption spillover effects. Productivity shocks can have effects outside project impacted zones through these channels. Methods and data: Dynamic Recursive CGE economywide model (created by International Food Policy Research Institute) based on parameters estimated using preliminary data from an (incomplete) Impact Evaluation. Labor supply across sectors and locations is perfectly mobile and output and input prices (including wages except unskilled wage), input/factor proportions, and the accumulation of physical capital are all treated as endogenous. A range of productivity changes were assumed for a few agricultural crops: cereals (Teff, maize, wheat, sorghum, barely, and rice), pulses (peas, beans, and haricot beans), oilseeds (sesame, and Niger seed), vegetables (cabbage, carrot, and reddish), cash crops (coffee, tea, and chat), and livestock (cattle, shots, dairy, and poultry), chosen carefully based on their importance and relevance for project areas (proxied by area, volume, and value) as is reported the country’s Agricultural Sample Survey data. Area-heterogeneity in productivity was assumed away given the lack of data, and productivity improvements were based on plausible ranges. Findings: 1.4 million additional/better jobs by end 2026 (using 10 percent productivity boost scenario, based on initial productivity differentials from Agricultural Growth Project 1 (AGPI). xx Ghana: Ceramic Tile IFC Additional Financing – (KEDA) (Ex ante estimation) WBG project code: 42507 Budget: (Total financing $44.7 million of which IFC Pros of approach: Convenient use of financing $25 million) available data through standardized tool. Type: Ex ante Cons of approach: SAM-based multipliers may overstate impacts in the presence of Components Included in Estimation: Loan financing. important supply-side production constraints. Channels: Backward supply chain and consumption Noteworthy: Ideally, comparison would spillovers. be by channel. Primary challenge remains to attribute expansion of demand or manufacturing capacity fully to IFC Theory of Change and Channels: Loan financing to financing per se (to make results comparable to other interventions in expand the ceramic tiles manufacturing facility, terms of project definition and diversify into new product lines and substitute for attribution. imports (93 percent of consumption). This would create more jobs in supplier firms and in the country due to consumption demand spillovers. Methods and data: SAM-multiplier and broad national-sector coefficients (IFC’s real sector estimation methodology within the AIMM framework). Findings: For 2021, total jobs created = 950, including consumption spillover jobs. xxi Ghana: Ceramic Tile IFC Additional Financing – (KEDA) (Ex post estimation) WBG project code: 42507 Pros of approach: Utilizes actual ex post Budget: (Total financing $44.7 million of which IFC data from client and first round suppliers. financing $25 million) Cons of approach: Difficulty reaching and Type: ex post obtaining cooperation from suppliers, as well as a key partner/distributor reduces Components Included in Estimation: Financing for above representativeness and reliability of estimates. Relationship between the referenced company. investments financed and any further investment or expansion of production capacity beyond the level of increased Channels: Backward supply chain and consumption sales anticipated ex ante is unclear. spillovers. Noteworthy: Demand for company’s products multiplied, and ex post estimates Theory of Change and Channels: Loan financing to expand exceed the ex-ante ones. More context on the ceramic tiles manufacturing facility, diversify into new this would be valuable, including whether this was COVID related. product lines and substitute for imports (93 percent of consumption). This would create more jobs in supplier firms and in the country due to consumption demand spillovers. Methods and data: Special Purpose survey of client and a sample of key first-round suppliers. Additional linkages are estimated using AIMM SAM-based model, as in the ex-ante estimation (previous page). Findings: In 2021, total jobs created are between 1810 – 1840. This resulted from an approximate doubling of demand for ceramic tiles. xxii Kenya: National and Rural Inclusive Growth Project (P153349) WBG project code: P148591 Budget: Fill Pros of approach: Context and good- Type: Ex ante specific data are used. Cons of approach: Assumes no constraints Components Included in Estimation to demand or supply responses in the Component 1, organizing growers into groups (i.e., value chains. Not all project components and channels could be assessed given vulnerable and marginalized groups-VMGs and other related interventions occurring in common interest groups -- CIGs), capacity building to the same target areas. enable the groups to implement livestock and crop Noteworthy: Illustrative rate of returns community micro-projects (e.g. milk production, local calculations were conducted for each poultry, bee keeping, production of mangoes and value chain based on (just) the indirect tissue-culture banana). In Component 2 VMGs and CIGs jobs benefits estimated. form into producer organizations, which serve as an entry point to value chains for many growers. This is complemented by the construction of priority infrastructure in their communities funded under Component 3. The infrastructure in Component 3 construction will be done through labor-intensive public works, which likely also generate positive indirect job effects. Five of 17 value chains included in estimation. Channels: Supply Chain, Factor Usage Theory of Change and Channels: Community driven development (including infrastructure), strengthened producer organizations and value chain (VC) development enhance agricultural production and stimulate increased demand for labor within the value chain. Methods and data: First, Value Chain Mapping was done, then product-specific input-output (I- O) coefficients applied for selected value chains (dairy, avocado, poultry, mango, and banana) based on available producer organization and sector data. Findings: Direct jobs to be created number 138,486 and indirect ones at 68,554 for the products considered. xxiii Kenya Digital Economy Acceleration project Pros of approach: Based on econometric WBG project code: P170941 estimation using Kenyan household data, Budget: US$390mn with theory of change supported by Type: Ex ante existing econometric studies. Utilized a variety of specifications. Components Included in Estimation: Component 1: Cons of approach: Estimation bias is Enhanced Broadband Access. possible due to unobserved factors. Noteworthy: Projects positive effects on The project full components are: Component 1: wages and non-farm employment. Broadband Access, Affordability and Quality: To increase access to high-speed internet for individuals, industry and government and increase the capacity and reliability of the National Optic Fiber Backbone Infrastructure (NOFBI); Component 2: Digital Government Services Infrastructure, Architecture, Data Privacy and Security: to better utilize digital technologies and enhance connectivity to improve public service delivery. This includes: (a) Government Cloud Infrastructure, Unified Communications System and Shared Services Architecture; (b) Upgrading and expansion of the Government Common Core Network (GCCN) (c) Establishment the Office of the Data Protection Commissioner; Component 3: Digital Capabilities aims to build digital skills by supporting the digital Learning Program (digital labs, teacher training, educational content) and to connect individuals to digitally enabled employment generating opportunities through the Ajira platform. Channels: Factor usage and consumption spillovers. Theory of change: Usage and speed of broadband internet are improved (through upgrading of backbone, digital skill training) resulting in improved job matching, start-ups, and e-commerce, expanding markets, including for business processing services, and shift labor out of agriculture, increasing the productivity and wages of the workforce. Method and data: Using Kenya FinAccess (Household) Surveys a pooled ordinary least squares model with year and county fixed effects is used to econometrically estimate the effect of internet usage on employment participation and wages. This estimation strategy captures unobserved heterogeneity across households that is fixed over time. Findings: Impacts on individual employment outcomes are found, but examined at the district level only wages are impacted, suggesting and individuals sort into different/better jobs with a net improvement in productivity rather than increased employment. Average wages increase in all impacted districts 3-10 percent, depending on scenario. xxiv Lesotho Transport Infrastructure and Connectivity Project (LTIC) Pros of approach: Opportunistic use of existing data, combining spatial WBG project code: P155229 information and household level Budget: US$18.3mn outcomes data. Type: Ex ante Cons of approach: Simultaneity bias issues may remain with respect to Components Included in Estimation: Construction of placement of current bridges. pedestrian bridges. Channel: Factor usage and consumption spillovers. Theory of change: Reduced time to walk to markets enhances sales of goods and services as well as access to jobs. Increased income causes local demand spillovers. Method and data: Econometric estimation of jobs outcomes using radial zone of influence is used to assign households to groups that were either previously treated or untreated by a footbridge. Household data from Lesotho Continuous Multipurpose Household Survey 2017- 2018 and bridge data from Environmental and Social Management Plan Report are used. Estimates are based on the effect of footbridge proximity (defined as being within 5 kms) on various household outcomes, controlling for 17 demographic and geographic variables, including household’s distance to closest district capital, which at least partially address non - random household and footbridge placement. Statistically significant estimates from reduced- form estimates are used to directly compute ex-ante expected long-run incremental impacts of the bridges to be built on the specified outcomes. Findings: The main estimates show a reduction in the travel time to essential social services, a higher probability of access to superior quality remunerated jobs, poverty reduction, an increase in per capita consumption, and an increase in the agricultural production share destinated to sales. The estimated number of new jobs created was zero, but a number of quality improvements were projected, such as permanency and benefits. xxv Mali: Promote Access to Finance, Entrepreneurship and Employment WBG project code: P168812 Budget: $60 million Pros of approach: Relatively rigorous Type: Ex ante approach using data from Mali. Components Included in Estimation: Loan Guarantee Cons of approach: Estimation approach does not fully correct for selection bias Fund, Cash Transfers into the borrower group as it does not address unobserved heterogeneity. The Channels: Factor Usage (credit), Consumption spillover borrowing group may also differ from those captured in WBES, which does not Theory of Change and Channels: Indirect channel 1: include micro enterprises and likely comprises fewer female borrowers. Credit Guarantee program expansion increases access to credit and new banking relationships for MSME’s, especially for un- and under-served actors (including women) and leads to higher firm investment and/or production (factor usage channel). Cash Grants component: Cash grants for income generating activities will enhance self-employment (direct jobs). Public goods component: Employment in creating public goods increases incomes directly. 2: (consumption spillover channel) local demand through the consumption multiplier from above. Methods and data: For indirect channel 1: Calculated the number of loans likely available through project, based on project information and reasonable assumptions on lending and repayment flows; applied an additionality factor from evidence from another setting (Chile). Then used data from the World Bank Enterprise Survey (WBES) for Mali to estimate the causal impact of credit access to firms on their employment growth, controlling for multiple factors. Propensity score matching is used to address the potential bias in OLS estimates due to the non-random allocation of credit. Multiplying the estimated impact of credit access on employment by the expected increase in the number of loans (under a range of alternative assumptions) yields an estimate of the total impact of the intervention on employment. For the cash transfer component, indirect channel (2): Using estimates of the total “transfer” of income to program recipients, estimates of the indirect consumption-spillover impact are obtained using an accounting approach – that is, by applying the marginal propensity to consume out of income, and the impact of an increase in household consumption on productive activities. Findings: Jobs created are as follows: Direct: 23,600. Through credit: between 342 and 1,195. Through demand spillovers 1,168. Total Indirect: 1,510 - 2,363 xxvi Mozambique: Westfalia IFC Project (WFM) (Ex ante estimation) WBG project code: IFC Project ID # 42280 Pros of approach: Convenient, consistent Budget: Total financing $7.5 million of which IFC model-based estimates using country specific SAM. financing $2.8 million Type: Ex-ante Cons of approach: Assumes no technology or factor price changes. Components Included in Estimation: Loan financing Noteworthy: Estimates from the two estimation (ex-ante and ex post) of this Channels: Backward supply chain and consumption project were of similar order of magnitude. The latter are assumed to demand spillovers result in an equal increase in the intermediate inputs by sector demanded in the model. Theory of Change: Direct loans to avocado producer and aggregator for small scale avocado growers. Extended loan tenor allows the creation of new direct jobs, including picking and bulking; improved hiring of female staff, increased training, and increased wage premiums for employees. Backward supply chain: Increased demand for services of seed traders and distributors, and producers of fertilizer and farming equipment (though a significant fraction of chemical inputs are imported). For the out-grower farmers, who supply WFM, the short-term benefits include increased access to international markets via an avocado hub to be integrated into the global avocado value chain. These should result in higher farm incomes. Consumption Spillover: Impact of higher incomes from direct, forward and backward channels impact consumption demand for all goods/services. Methods and data: Social Accounting Matrix-multiplier approach using and broad national- sector coefficients using IFC’s real sector estimation methodology. Findings: Projected 2025 (at full production) total jobs at full production in 2025: 4,870. In 2021: 1,820. xxvii Mozambique: Westfalia IFC Project (WFM) (Ex post estimation) WBG project code: IFC Project ID # 42280 Budget: Total financing $7.5 million of which IFC Pros of approach: Direct employment and financing was $2.8 million input expenditures are provided by the Type: Ex post Client. Supplier surveys are to actual suppliers. Components Included in Estimation: Loan financing Cons of approach: Survey-based estimates could be biased due to the lack of representativeness of survey Channels: Backward supply chain, and consumption respondents and partial estimation of spillover. value chain (larger, only first round suppliers). Theory of Change and Channels: Direct loans to Noteworthy: Estimates from the two estimation (ex-ante and ex post) of this avocado producer and aggregator for small scale project were of similar order of magnitude avocado growers. Extended loan tenor allows the and the survey-based estimates could be creation of new direct jobs, including picking and biased due to the partial estimation of value chain (larger, only first round bulking; improved hiring of female staff, increased suppliers). Direct employment and input training, and increased wage premiums for employees. expenditures are provided by the Client. Backward supply chain: Increased demand for services The latter are assumed to result in an equal increase in the intermediate inputs of seed traders and distributors, and producers of by sector demanded in the model. fertilizer and farming equipment (though a significant fraction of chemical inputs are imported). For the out-grower farmers, who supply WFM, the short-term benefits include increased access to international markets via an avocado hub to be integrated into the global avocado value chain. These should result in higher farm incomes. Consumption Spillover: Impact of higher incomes from direct, forward and backward channels impact consumption demand for all goods/services. Methods and data: Primary data on employment and input expenditures collected from (WFM’s / KEDA’s) client financials. Ex post Survey of direct recipient and first round suppliers combined with further indirect jobs using IFC’s real sector jobs estimation methodology. Expenditures are assumed to result in an equal increase in the intermediate inputs by sector demanded in the model, out-grower farmers (WFM), with no adjustment to input prices, and employment and related data are collected from the main first round suppliers in a survey. The survey employment elasticities were derived from responses by suppliers collected in May/June 2022, with responses/recall from 2019 & 2021. Findings: Projected at full production in 2025: between 2970 – 4300. In 2021: 1,120-1,596. xxviii Rwanda Rural Feeder Roads Development Program WBG project code: P126498 Pros of approach: Planned strategy can Budget: $170mn produce robust estimates. Integrated into Type: Ex post wider evaluation research agenda. Cons of approach: Implemented strategy Components Included in Estimation: All: at mid-line did not use difference-in- Rehabilitation, Upgrading, and Maintenance of difference approach, as planned in final evaluation. Involves several rounds of Selected Rural Feeder Roads. Implicitly, benefits from data collection and years of engagement. support for developing strategy for rural access and transport mobility improvements support the Noteworthy: Many outcomes are benefits of the project. captured in the evaluation, so a more fulsome understanding of channels of Channels: All impact is possible. Theory of Change: Enhanced all-season connectivity to market centers will raise agricultural productivity and production, ensuring food security, and enhancing agricultural marketing. The project is designed as an integral part of the agricultural support initiatives in Rwanda. Jobs impacts are not the main objective of the project but are measured as one of the outcomes of interest for the IE. Methods and data: Impact Evaluation using difference-in-difference estimates from household baseline (2015) and follow up survey (2021), from being within 2 km of an improved feeder road (for outcome of producer prices). At mid-line, used simple RF regression of the effect of remoteness on paid employment. Data were special household surveys in impact and control zones. Findings: at Midline: No significant effects on producer prices or the fraction of produce sold at market (negative, insignificant coefficients). A 3.9 percentage point increase in the probability of a household member having paid employment due to improved connectivity, translating to 8,057 paid jobs. xxix Rwanda: Energy Reform DPO Series Pros of approach: Simple to implement, opportunistic use of available Labor Force, WBG project code: P162671, P166458 and P169040 Firm Census, World Bank Enterprise, and Size of Loan: US$ 475mn sector data, to check trends in outputs Type: Ex post and outcomes along the ToC. Cons of approach: Approach not powerful Components Included in Estimation: enough to detect modest impacts. Approach depends on electricity being a A series of prior actions was designed to: binding constraint rather. In other cases, • Lower the cost of electricity service delivery the approach may detect “impact” through adoption of the Least-Cost Power spuriously. There was no attempt to Development Plan (LCPDP) and facilitate a condition on other factors and take into account possible simultaneity issues. transition to a low-carbon energy mix. Noteworthy: More data from the • Increase utility revenues from electricity service electricity provider and the quality of jobs delivery. The Energy Utility responsible for in various sectors and sub-sectors could maintaining electricity infrastructure (such as power have permitted a more comprehensive plants and transmission/distribution networks) and assessment or parameterization of the electricity service delivery, was expected to have ToC. implemented a series of measures to enhance transparency, reduce non-technical losses, and facilitate recovery of operating costs through an updated pricing system. • Enhance affordability for low-income consumers. By the end of the series, the reduction in the cost-of-service delivery in combination with a strategy to increase access to low-income consumers was expected to boost demand and incentivize electricity consumption. The adoption of international quality standards for solar products was also expected to help create a conducive business environment for private-sector actors in the off-grid energy market and enhance access of lower-income consumers to energy services. Theory of Change: There are three distinct channels for job impacts in the economy: (1) opening fiscal space for GoR to increase public spending directly linked to employment (e.g., road construction) or indirectly (e.g., via social service delivery) [generally difficult to estimate]; (2) improved quality of grid electricity leads to expanded production, with firms hiring under- or un-employed labor; and (3) expanded access to more affordable power to households induces productive use, with concomitant employment potential. Method and data: Simple trend analysis of aggregate outcomes over time using Rwanda Labor Force Surveys, as well as WBES, Establishment censuses, and electricity provider data. Findings: Impacts not clearly detected. Improvements in the sector continued and were sustained, but had begun before the DPO series. Overall, the timing of improvements did not support a conclusion of positive impact on jobs. Although the estimators revised the ToC to include jobs impacts, these were not originally explicit objectives of the series. Trends observed xxx in the data could be interpreted in a variety of ways. Reliability improved between first two of series, and pricing to productive users fell slightly (in USD terms). Firm establishment rates stayed relatively constant. Broad sector employment shares are the opposite of what one would expect, with the share of employment in the more electricity-intensive sectors falling and that in agriculture and forestry rising (and accounting for 75 percent of job growth). Although GDP growth was faster in manufacturing than other sectors, employment did not respond commensurately. Power supply to rural areas may have made larger scale agriculture and forestry more attractive, and more reliable; affordable power for large producers could have caused a shift towards energy-intensive technologies that are also more capital-intensive/labor- saving. Ultimately, the estimators assessed that there was no impact detectable from analyzing these trends, in part due to limitations of the approach and possibly because it is too early to detect them. xxxi Tanzania: Dar es Salaam Urban Transport Project (Balboni, et al. 2020) WBG project code: P150937 Budget: IDA USD 425 million (disbursement ongoing) Pros of approach: Rigorous, ex post Type: Ex post estimation with high reliability, robust to differences in control and treatment groups, as well as different trends. Components Included in Estimation All Bus Rapid Transit (BRT) investment (phase 2 of an ongoing Cons of approach: May be under- powered to detect small changes. project) Multiple rounds of special purpose data collection required. Channel: Factor Usage. Theory of Change: The construction and operation of a BRT line is expected to reduce commuter travel times and urban congestion. Job seekers are expected to find jobs that better match their skill sets, as reductions in commute time increase the number of employers they can access. Conversely, employers benefit from finding more skilled employees. Ultimately, a better skill match in the labor market is expected to increase overall productivity. Firms located close to the BRT would have access to a larger markets and workers, making the area more attractive to businesses and increasing the number of firms in the area. Overall, these three forces would lead to a potential increase in land prices, changes in land use and changes in the mix of people living in the area; better jobs matches, and higher employment levels. Method and data: Impact Evaluation. Triple difference using the intensity of treatment/proximity to BRT. The most recent population census is used for sampling purposes. Three rounds of household data collection were done according to a geographical sampling strategy to cover equal intervals along 12 arcs at radii increasing at 1.5km intervals from the central business district. In addition, travel time data were collected. Findings: No evidence found of impact on employment status, wages, or self-employment income (or indeed income or consumption). Travel time to jobs and usage of cars fell. There is also no differential impact found on the labor market outcomes for female versus male respondents. xxxii Tonga: Pathways to Sustainable Oceans Project (Value chain-based estimation) WBG project code: P164941 Budget: $3.9mn Pros of approach: Simple to implement. Type: Ex ante Does not require detailed company financial data. Activity-level specificity (i.e., pearls) is possible to estimate using Components Included in Estimation bespoke approach and may provide more All: (1) expanded public hatchery capacity for Mabe reliability than broader activity categories. pearl spats; training, input provision, and business Cons of approach: Some coefficients are support services (market access) for pearl and jewelry from another context (Latin America). producers; (2) outshore (longline) fisheries: support for Assumption of constant technology and labor productivity may not hold. improved monitoring and enforcement, more resources for the ministry of fisheries and their patrol ships (3) capacity building to sustainable management areas (SMAs) for inshore fisheries management. Channel: Factor usage and backward supply chain. Theory of Change: Forward factor usage and direct jobs: Improved spat hatchery capacity and direct provision of inputs to pearl producers to sell pearls will increase their (complementary) labor demand (indirect and directly in turn). Increased supply of pearls will increase demand for labor by pearl jewellery producers. Improved management of inshore fisheries and reduction in illegal fishing will eventually increase fish stocks and hence fish production; labor demand in the domestic fishing industry and related services will increase with the supply of fish. Method and data: Activity- specific input-output coefficients using data from Chile and Tonga are used to tie increased inputs to increased labor utilization. Findings: New or improved jobs number 20 direct, 14 indirect; 11 indirect from fisheries only. Funding Direct jobs (FTE) Indirect jobs Direct jobs per Indirect jobs per (FTE) US$ million US$ million Mabe pearls US$0.39mn 6 3 15.4 7.7 Inshore fisheries US$1.85mn 14 7 7.6 3.8 Outshore US$1.72mn 0 4 0 2.3 fisheries Total US$3.96mn 20 14 5.1 3.5 xxxiii Tonga: Pathways to Sustainable Oceans Project (AIMM Real Sector Assessment Model Estimation) Pros of approach: Uses empirically WBG project code: P164941 estimated elasticities of employment-to- Budget: $3.9mn. output and GTAP SAM data from similar Type: Ex Ante economy (Fiji). Cons of approach: Parameters are not Channels: Factor usage, backward supply chain, and specific to country due to lack of data. The number of working hours and labor consumption spillovers. income of the farms (used in Meneses) are not sufficient to use in the SAM multiplier Components Included in Estimation approach, which requires more detailed data from the companies affected. (2) Outshore (longline) fisheries: support for improved Furthermore, very specific jewelry monitoring and enforcement, more resources for the businesses might not be well represented in sectoral categories of IO/SAM multiplier ministry of fisheries and their patrol ships (3) capacity approach. building to sustainable management areas (SMAs) for inshore fisheries management. Noteworthy: AIMM Modeling was done after the first method as a means to compare for those channels of impact Theory of Change: Increased fisheries management captured in both. Estimates were very increases fish production, boosting demand along the close. value chain, as well as consumption demand. Methods and data: Estimation using IFC’s Anticipated Impact Measurement and Monitoring (AIMM) framework model for real sectors, using SAM-multiplier methods, broad (fisheries) sector coefficients and labor input elasticities from Bürgi, Hovhannisyan and Mondragon-Velez (IFC, 2020), who construct GDP-employment elasticities by decomposing annual GDP growth into job creation and improvements in labor productivity for individual countries and three sectors: agriculture, manufacturing, and services, over the period of 2000-2016. This GDP growth decomposition approach provides country and sector specific GDP-employment elasticities that account for varying trends and patterns across countries and sectors. Findings: 14 indirect jobs will be created / improved as follows IFC Note IFC Investment Meneses Estimate of IFC Estimate of Consumption Total (US$mn) Indirect (Factor Usage) Indirect (Factor spillovers Indirect Project jobs Usage) jobs jobs jobs Inshore fisheries 1.85 7 4 5 9 Outshore fisheries 1.72 4 2 3 5 xxxiv Uganda: Investing in Forests and Protected Areas for Climate-Smart Development Project WBG project code: P170466 Budget: (USD 85.2mn) Total project value US$ 148.2 Pros of approach: Direct mapping of value Type: Ex ante chains and use of varied sources of information. Components Included in Estimation: 1 and 2 (3 and 4 excluded63) Cons of approach: Estimates not based on 1. Investments in capacity to manage forests and survey data as intended, due to difficulty of reaching a representative sample. wildlife protected areas: Investment in staff housing and Counter factual is not well established. offices, opening of new tracks for visitors and park management, and procurement of equipment; Investments to reduce invasive species and forest fires; support for communities to restore degraded forest areas; and investment in forest protection near refugee settlements. 2. Targeted investments in tourism businesses and production forests (plantations): Infrastructure to leverage private tourism concessions; Conditional grants to encourage investment in high quality plantations by the private sector; Investments in community tourism activities; Matching grants to stimulate wood processing investment. Channels: Factor Usage, supply chain, and consumption spillover jobs. Theory of change: Matching grants to private sector (plantation, wood processing, tourism business) coupled with enhanced management of and investments in forests and protected areas increase supply of wood and improve tourism experience attracting more visitors. Expanded supply of wood increases demand for labor in wood processing and expanded demand for tourism services expands demand for labor in that sector. Method and data: Industry surveys for qualitative information and breakdown of types of jobs and workers64. Tourism: Social Accounting Matrix (with Tourism Activity Clusters) and Uganda’s projected GDP to forecast tourism employment. Wood sector: value chain analysis provides direct and indirect employment estimates based on projections of wood production and processing using existing and enhanced technology (through matching grants). Findings: Wood sector65: Additional jobs number 16,500 (in 2035 conservatively). Tourism: 107,000 (2026) to 143,300 (2036) 63 Component 3: Improved landscape management in refugee hosting areas; Component 4: Project management 64 The industry survey was not able to provide detailed quantitative information due to structural challenges with the underlying national enterprise registers. 65 In the wood sector, a large part of the jobs impacts in downstream processing materializes only after a long delay as the project-financed tree plantations reach their maturity and can be harvested only after 10 (eucalyptus) to 18 (pine) years. xxxv Uzbekistan: Livestock Sector Development Project WBG project code: P153613 Budget: US$150mn, US$243.8 total. Pros of approach: Utilizes information Type: Ex post from over half of project beneficiaries. Cons of approach: Counter-factual not Components Included in Estimation fully accounted for, more serious issue 1: Livestock Sector Public Investment Framework and with private for-profit activities; non- Public Services: (i) Strategy, Policy, and Public representative sample used. Investment Framework; and (ii) Strengthening Livestock Sector Public Services. Component 2: Livestock Value Chain Modernization: (i) Credit Line for Private Investments; and (ii) Value Chain Development and Smallholder Market Inclusion. Component 3: Project Coordination, Management, and Monitoring and Evaluation supports financial management (FM), procurement, environmental and social safeguard compliance, including Grievance Redress Mechanism (GRM) and impact evaluation. The project is being implemented in all the thirteen regions of Uzbekistan. Channel: Direct, supply chain, factor usage and consumption spillovers Theory of Change and Channels: Through the access to finance, farmers and agribusinesses will expand their production, raise productivity and income, and enhance commercialization as well as create new jobs. Methods and data: Before and after, quantitative and qualitative information from document reviews, telephone/telegram-based interviews using a semi structured questionnaire; and focused discussions with credit line beneficiaries, their employees, and project staff. Findings: 25,441 new jobs, including 7,093 direct, 1,710 women, 9,369 seasonal, and 8,979 indirect jobs. xxxvi ANNEX D: Spotlight on Structural Computable Model Versus Reduced Form Results: Chad-Cameroon Transport Corridor Projected outcomes varied by the approach taken – RF versus structural model – for both Cameroon and Chad in the pilots done for the transport corridor project. As shown in Error! R eference source not found., results differ by method and econometric model. It is possible that if the treatment varies substantially within a district, due to negative spillovers, impacts observed at the individual level still result in no net change at the district level; that is, gains by those positively impacted are offset by those negatively impacted. In fact, the district level impact estimate for Cameroon is not statistically significant, but it is of larger magnitude, confirming the sign and significance of the individual-level results: better transport infrastructure leads to a shift out of agricultural employment. In the case of Cameroon, the Figure 3: Estimated Shifts out of Agriculture as Sector of structural modelling approach Main Employment produced a predicted shift in sectors of employment of the opposite sign to the econometric models (Table 4). Rather than a shift out of agriculture, the model predicts a net shift into agriculture as a share of workers. In Chad, the district level econometric estimate is of the same sign as the structural model: there is a shift out of agriculture predicted for both. With respect to the conflicting results for Cameroon, of course, it is possible for a structural model to predict greater sector specialization at the country level, Source: Lebrand 2022. as would occur with the more “open” trade between the two countries resulting from improved transport co nnectivity. Nevertheless, the CGE result is not reliable when seen in comparison with the empirical evidence. The regression analysis is fully driven by the data and given that the instrumental variables approach was available for some closely related outcomes (LeBrand, 2022) to address simultaneity bias, and the results are consistent. Thus, this approach should be considered reasonably reliable. As a reduced form estimate, no (explicit) assumptions are needed about the economy’s or locality’s production or consumption patterns, shifts in firm or labor’s location, or the magnitude of any frictions to sectoral or spatial mobility. This approach in this case implicitly accounts for possible spillovers across units of observation, so can be used to compute a total xxxvii Table 4: Estimated Change in the Shares of Main Jobs in Agriculture with Respect to Without Project / Less Treated Scenario at National Level (percent) Policy Country Method type Method Estimate scope Cameroon Reduced Form Multinomial logit T -0.14* (individual level) OLS (individual level) T -0.31* OLS (district level) T -0.19 2SLS (district level) T -2.49 Structural model T 0.3 T+B 0.1 Chad Reduced Form Multinomial logit T 1.07* (individual level) OLS (individual level) T 0.89* OLS (district level) T -1.22* Structural model T -0.1 T+B -.02 Notes: Asterisks listed for reduced form estimates indicate estimate is statistically significant at the 15% level. There was no instrumental variable available for Chad. For policy scope T indicates only transport policies are considered in estimation, T+B that policies for transport as well as for more net impact at a higher level of geographic or population aggregation. Counting the number of expeditious international border crossings. jobs impacted, however, still requires one to impose a counter-factual assumption. Of course, for estimating future impacts, the approach assumes that one can extrapolate into the future, and that the coefficients of impact from the past remain valid. It also limited in being unable to capture unmodeled heterogeneity of impact over space or population sub-group. Understanding heterogeneity of impacts is desirable, in particular to address distributional effects, especially for a large investment of the sort made in a large transport corridor project. Large investments can have impacts on trade and commerce over a wide area, and in expanding trade, it can have winners and losers.66 This is where structural modeling can provide additional value.67 But the approach’s reliability depends crucially on the validity of the parameters used and economic features modeled (e.g., the change in trade costs from distance changes, 66 Even if a proxy such as distances to a paved highway (which could be in another district) was used, this would be in reference to single highway. 67 Assessing heterogeneity of this sort could be done with a regression model by including interaction terms in panel data. With a single survey round it might one might encounter collinearity problems between individual/district fixed effects and other characteristics. xxxviii elasticities of substitution, and so forth).68 In this case, the estimation relies upon a number of parameters from Ethiopia, as shown below. Moreover, in the case of the Cameroon-Chad corridor, there are likely to be important frictions and other unmodeled factors. Table 5: Parameter Assumptions for Chad-Cameroon Transport Corridor Structural Model To study heterogeneous effects and other outcomes not captured in an econometric estimation, the best approach would be to use a structural model in a way which selects parameters that provide a close match to the moments of the aggregate data, including, for example, the aggregate impact of the policy shock in question (such as in Moneke, 2020 and Osborne, 2004). The more empirically grounded the parameter values from the country’s data, the more accurate the model’s predictions would be. For Chad, another discrepancy arises for the RF estimations. Those done at the individual level predict a higher level of employment in agriculture; those at the district level and using the CGE model estimate the opposite shift. To check the consistency of these estimates requires a population-weighted approach. Differences could also be due to the inability to fully account for simultaneity bias, as the IV approach was not feasible for Chad due to data limitations. 68 Another approach to predicting results spatially would be to utilize parameters that give the closest fit of the model’s outcomes to the data – in this case the impacts estimated via reduced form. This “indirect inference” method would take the aggregate estimate as given and allow one to study spillovers, labor mobility, and other factors that may be of interest. xxxix Most Recent Jobs Working Papers: 76. Measuring Ex Ante Jobs Outcome of the Bangladesh Livestock and Dairy Development Project. (2023) Mansur Ahmed, FNU Jonaed and NazmulHoque. 75. Jobs, Food and Greening: Exploring Implications of the Green Transition for Jobs in The Agri-Food System. (2023) Gianluigi Nico and Luc Christiaensen. 74. Who Is Most Vulnerable to the Transition Away from Coal? Ruda Śląska Residents’ Preferences towards Jobs and Land Repurposing. (2023) 73. Does Agricultural Intensification Pay? (2023) Ghislain Aihounton and Luc Christiaensen. 72. Cost-Effectiveness of Jobs Projects in Conflict and Forced Displacement Contexts—Annexes. (2022) Virginia Barberis, Laura Brouwer, Jan Von Der Goltz, Timothy Hobden, Mira Saidi, Kirsten Schuettler and Karin Seyfert. 72. Cost-Effectiveness of Jobs Projects in Conflict and Forced Displacement Contexts. (2022) Virginia Barberis, Laura Brouwer, Jan Von Der Goltz, Timothy Hobden, Mira Saidi, Kirsten Schuettler and Karin Seyfert. 71. Towards a Just Coal Transition Labor Market Challenges and People’s Perspectives from Wielkopolska. (2022) Luc Christiaensen, Céline Ferré, Maddalena Honorati, Tomasz Janusz Gajderowicz and Sylwia Michalina Wrona. 70. Towards a Just Coal Transition Labor Market Challenges and People’s Perspectives from Silesia. (2022) Luc Christiaensen, Céline Ferré, Tomasz Gajderowicz and Sylwia Wrona. 69. Towards a Just Coal Transition Labor Market Challenges and People’s Perspectives from Lower Silesia. (2022) Luc Christiaensen, Céline Ferré, Tomasz Gajderowicz and Sylwia Wrona. 68. A Tale of Two Countries: Labor Market Profiles of Youth in Urban and Rural Cameroon. (2022) Ioana Botea and Mitja Del Bono Click here for full Jobs Paper Series Address: 1776 G St, NW, Washington, DC 20006 Website: http://www.worldbank.org/en/topic/jobsanddevelopment Twitter: @WBG_Jobs Blog: https://blogs.worldbank.org/jobs/