48597 THE WORLD BANK Using Knowledge to Improve Development Effectiveness An Evaluation of World Bank Economic and Sector Work and Technical Assistance, 2000–2006 THE WORLD BANK GROUP WORKING FOR A WORLD FREE OF POVERTY The World Bank Group consists of five institutions—the International Bank for Reconstruction and Development (IBRD), the International Finance Corporation (IFC), the International Development Association (IDA), the Multilateral Investment Guarantee Agency (MIGA), and the International Centre for the Settlement of Investment Disputes (ICSID). Its mission is to fight poverty for lasting results and to help people help themselves and their environment by providing resources, sharing knowledge, building capacity, and forging partnerships in the public and private sectors. THE INDEPENDENT EVALUATION GROUP ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION The Independent Evaluation Group (IEG) is an independent, three-part unit within the World Bank Group. IEG-World Bank is charged with evaluating the activities of the IBRD (the World Bank) and IDA, IEG-IFC focuses on assessment of IFC’s work toward private sector development, and IEG-MIGA evaluates the contributions of MIGA guarantee projects and services. IEG reports directly to the Bank’s Board of Directors through the Director-General, Evaluation. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank Group’s work, and to provide accountability in the achievement of its objectives. It also improves Bank Group work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. W O R L D B A N K I N D E P E N D E N T E VA L U AT I O N G R O U P Using Knowledge to Improve Development Effectiveness: An Evaluation of World Bank Economic and Sector Work and Technical Assistance, 2000–2006 2008 The World Bank http://www.worldbank.org/ieg Washington, D.C. ©2008 The International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org E-mail: feedback@worldbank.org All rights reserved 1 2 3 4 5 11 10 09 08 This volume is a product of the staff of the International Bank for Reconstruction and Development / The World Bank. The findings, interpretations, and conclusions expressed in this volume do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denomina- tions, and other information shown on any map in this work do not imply any judgement on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development / The World Bank encour- ages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: 978-750-8400; fax: 978-750-4470; Internet: www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW , Washington, DC 20433, USA; fax: 202-522-2422; e-mail: pubrights@worldbank.org. Cover: Image ©William Whitehurst/Corbis. Reproduced with permission. ISBN-13: 978-0-8213-7725-3 e-ISBN-13: 978-0-8213-7726-0 DOI: 1596/978-0-8213-7725-3 Library of Congress Cataloging-in-Publication Data The World Bank’s economic and sector work and technical assistance, FY00-06 : an independent evaluation. p. cm. 1. World Bank—Developing countries—Evaluation. 2. Economic assistance—Developing countries—Evaluation. I. World Bank. HG3881.5.W57W6947 2008 332.1’532090511—dc22 2008032228 World Bank InfoShop Independent Evaluation Group E-mail: pic@worldbank.org Knowledge Programs and Evaluation Capacity Telephone: 202-458-5454 Development (IEGKE) Facsimile: 202-522-1500 E-mail: eline@worldbank.org Telephone: 202-458-4497 Facsimile: 202-522-3125 Printed on Recycled Paper Contents vii Abbreviations ix Acknowledgments xi Foreword xiii Executive Summary xix Management Response xxiii Advisory Panel Statement 1 1 Introduction and Methodology 4 Objective, Framework, and Scope of the Evaluation 6 Evaluation Tools 9 2 Trends and Patterns of Bank ESW and TA, Fiscal 2000–06 11 Number and Cost of ESW and TA 11 Sectoral Distribution of ESW and TA 12 Regional Distribution of ESW and TA 12 Trust Fund Contribution to ESW and TA Costs 13 Origination of ESW and TA 14 ESW Report Types 16 Quality of ESW and TA 19 3 Effects of ESW and TA on World Bank Activities 21 Informing Bank Lending 25 Informing Bank Strategy 27 4 Effects of ESW and TA on Client Countries 29 The Extent to Which ESW and TA Met Their Objectives 31 Some Factors Affecting the Extent to Which ESW and TA Met Their Objectives 32 Influencing Government Policy 35 Building Capacity and Strengthening Institutions 38 Informing/Stimulating Public Debate 38 Facilitating Knowledge Exchange 42 Influencing the Development Community 44 Other Effects 47 5 Factors Influencing the Effectiveness of ESW and TA 49 Technical Quality 51 Partnership with Clients 55 Origination 57 Dissemination 62 Other Relevant Dimensions iii U S I N G K N OW L E D G E TO I M P R OV E D E V E L O P M E N T E F F E C T I V E N E S S 65 6 Main Findings and Recommendations 67 Effectiveness of ESW and TA 70 What Do Clients Want from the Bank? 73 What Would Have Happened Without Bank ESW and TA: Clients’ Views 74 The Bank’s Results Tracking Framework for ESW and TA 74 Conclusions and Recommendations for the Way Forward 77 Appendixes 79 A: ESW and TA Types 81 B: Methodology 107 C: Portfolio of World Bank ESW and TA, Fiscal 2000–06 121 D: Econometric Analysis 127 E: Specific Examples of Effects of ESW and TA from Country Reviews 137 Endnotes 141 Bibliography Boxes xv ES.1. Influential ESW: Examples from Vietnam and Mauritius xvi ES.2. When ESW and TA Were Not Effective 4 1.1 ESW and TA: What Are They? 14 2.1 IDA-Mandated ESW and Core Diagnostics 24 3.1 Loan TTLs’ Views on the Effects of ESW on Their Loans 26 3.2 The Role of ESW in Informing Bank Country Strategies 35 4.1 FSAP and PER Informing Government Policy and Legislation: Some Examples 36 4.2 Selected Examples of TA Assisting in Policy/Program Implementation 41 4.3 Some Country-Specific Factors Affecting the Extent to Which ESW and TA Stimulated Public Debate 45 4.4 Selected Examples of Bank ESW Influencing the Donor Community 54 5.1 Nature of Partnerships with Clients: Examples from Country Reviews 56 5.2 Views of In-Country Stakeholders on the Role of the Bank in Proposing ESW and TA Figures 12 2.1 Number and Cost of ESW and TA Delivered to Clients 15 2.2 Share of Client-Requested ESW in IBRD and IDA Countries, Fiscal 2000–06 15 2.3 Share of Client-Requested TA in IBRD and IDA Countries, Fiscal 2000–06 17 2.4 Quality and Average Cost of ESW 23 3.1 Ratings on the Extent to Which ESW Informed Lending, by Shares of Respondents 25 3.2 Ratings on the Extent to Which ESW Informed Strategy and Lending and Met Other ESW and TA Objectives, by Shares of Respondents 30 4.1 Ratings on the Extent to Which ESW and TA Met Various Objectives, by Shares of Respondents 32 4.2 Distribution of Ratings Given by In-Country Stakeholders in the Democratic Republic of Congo and Romania 33 4.3 Ratings on the Extent to Which ESW and TA Informed Policy, by Shares of Respondents from Different Sources 34 4.4 Ratings on the Extent to Which ESW and TA Informed Policy, from In-Country Stakeholders in Country Reviews 36 4.5 Ratings on the Extent to Which ESW and TA Built Capacity/Strengthened Institutions, by Shares of Respondents from Different Sources iv CONTENTS 37 4.6 Ratings on the Extent to Which ESW and TA Built Capacity/Strengthened Institutions, from In-Country Stakeholders in Country Reviews 40 4.7 Ratings on the Extent to Which ESW Stimulated Public Debate, by Shares of Respondents from Different Sources 41 4.8 Ratings on the Extent to Which ESW Stimulated Public Debate, from In-Country Stakeholders in Country Reviews 43 4.9 Ratings on the Extent to Which TA Facilitated Knowledge Exchange, by Shares of Respondents from Different Sources 43 4.10 Ratings on the Extent to Which TA Facilitated Knowledge Exchange, from In-Country Stakeholders in Country Reviews 44 4.11 Ratings on the Extent to Which ESW Influenced the Development Community, by Shares of Donors from Different Sources 44 4.12 Ratings on the Extent to Which ESW Influenced the Development Community, from Donors in Country Reviews 70 6.1 First-Ranked Bank Activity with Respect to Importance to Clients, by Shares of IBRD and IDA In-Country Stakeholders 71 6.2 First-Ranked Institutions with Respect to Usefulness of Reports and TA, by Shares of In-Country Stakeholders Tables 5 1.1 The Bank’s ESW and TA Objectives and Success Indicators 6 1.2 Evaluation Framework 12 2.1 Average Costs of Country, Regional, and Global ESW and TA, Fiscal 2000–06 13 2.2 Average Costs of ESW and TA by Region, IBRD versus IDA, Fiscal 2000–06 14 2.3 Origination of ESW, Fiscal 2000–06 15 2.4 Origination of TA, Fiscal 2000–06 16 2.5 Numbers and Average Costs of ESW, by Report Type, Fiscal 2000–06 17 2.6 Shares of In-Country Stakeholders Giving Ratings of 4 and Above on Different Dimensions of Bank ESW and TA, by Source 17 2.7 Shares of Tasks with QAG Ratings of Moderately Satisfactory and Above 24 3.1 Average Quality at Entry (Overall Assessment) Ratings for a Selected Sample of Loans, by Region, Fiscal 2003–05 30 4.1 Average Ratings by In-Country Stakeholders on the Extent to Which ESW and TA Met Various Objectives 31 4.2 Average Ratings by In-Country Stakeholders and TTLs on the Extent to Which ESW and TA Met Various Objectives 39 4.3 Selected Examples of ESW Building Capacity 40 4.4 Selected Examples of TA Building Capacity 42 4.5 Selected Examples of ESW and TA Stimulating Public Debate 50 5.1 Average Ratings of In-Country Stakeholders on Various Dimensions of ESW and TA 50 5.2 Average Rankings of In-Country Stakeholders on the Importance of Various Dimensions of ESW and TA for Achieving Objectives 51 5.3 Rankings of Efforts by TTLs 52 5.4 Ratings and Implied Rankings of Incentives Perceived by ESW TTLs 52 5.5 Ratings and Implied Rankings of Incentives Perceived by TA TTLs 60 5.6 Modes of Dissemination Selected as Most Useful, by Shares of Stakeholders 61 5.7 Modes of Dissemination Selected as Most Useful, by Shares of IBRD and IDA Stakeholders 72 6.1 Average Ratings of In-Country Stakeholders on the Usefulness of Country, Regional, and Global ESW and TA v In Bangladesh, this woman block prints fabric for sale in local markets and for export. World Bank trade reports informed the government’s trade policies. Photo by Shehzad Noorani, courtesy of the World Bank Photo Library. Abbreviations AAA Analytical and advisory activities AML/CFT Anti–money laundering and combating the financing of terrorism CAS Country Assistance Strategy CEM Country Economic Memorandum CFAA Country Financial Accountability Assessment CPAR Country Procurement Assessment Review CSO Civil society organization DPL Development policy lending/loan ESW Economic and sector work EU European Union FSAP Financial Sector Assessment Program IBRD International Bank for Reconstruction and Development ICA Investment Climate Assessment ICT Information, communications, and technology IDA International Development Association IEG Independent Evaluation Group IFC International Finance Corporation IMF International Monetary Fund MIC Middle-income country PER Public Expenditure Review PESW Programmatic ESW PSD Private sector development QAE Quality at entry QAG Quality Assurance Group ROSC Report on the Observance of Standards and Codes TA Technical assistance TTL Task team leader vii In Vietnam, researchers, undergraduates, and postgraduate students often use the World Bank’s Public Expenditure Reviews. Photo by Simone D. McCourtie, courtesy of the World Bank Photo Library. Acknowledgments This evaluation of the World Bank’s economic Banerji, Mona Haddad, Carlos Silva-Jauregui, and and sector work and nonlending technical assis- Steven Webb were much appreciated. tance was prepared by the Country Evaluation and Regional Relations Unit of the Independent Eval- The External Advisory Panel for the study con- uation Group. sisted of Bimal Jalan (Member of Parliament and former governor of Reserve Bank of India; former The evaluation was led by Helena Tang, the author executive director of the World Bank 1992–96) and of the report. Inputs to the report include coun- Roberto Junguito (former Finance Minister and try reviews, stakeholder and staff surveys, and sta- Agriculture Minister, Colombia). tistical and econometric analyses undertaken by Independent Evaluation Group staff and consult- The team received guidance and support from the ants. The 12 country reviews were prepared by Ju- Independent Evaluation Group management lian Berengaut (Romania, Serbia), Jeffrey Brown team: Arup Banerji, Ken Chomitz, Shahrokh Far- and Helena Tang (Guyana), Jorge Garcia-Garcia doust, Cheryl Gray, Ali Khadr, and Vinod Thomas. (Malaysia, Vietnam), Aylin Isik-Dikmelik and He- The peer reviewers were David Goldsbrough and lena Tang (Jordan), Robert Lacey (the Democratic Marcelo Selowsky. Republic of Congo, Mali), Chandrashekar Pant Editorial assistance was provided by William Hurl- (Bangladesh), and Violeta Rosenthal (Lesotho, but. Production and logistical support was pro- Mauritius, and Peru). Saubhik Deb and Aylin Isik- vided by Agnes Santos and Roderick de Asis. Dikmelik carried out the statistical and econo- Heather Dittbrenner copy edited the study for metric analyses for the report. Jeffrey Brown, publication. Saubhik Deb, and Aylin Isik-Dikmelik compiled the database for all the analysis in the report. The evaluation team greatly appreciates the time and insights of past and current country directors, In collaboration with Helena Tang, Jeffrey Brown, sector managers, task team leaders, government Aylin Isik-Dikmelik, and Mary McIntosh (Prince- officials, parliamentarians, researchers, and rep- ton Survey Research Associates International) de- resentatives of civil society, the donor community, signed the questionnaire for conducting interviews and the private sector who were interviewed and with in-country stakeholders. Also in collaboration surveyed. with Helena Tang, Jeffrey Brown, Aylin Isik- Dikmelik, and Matthew Petrie (Fusion Analytics Finally, the team acknowledges the generous fi- LLC) designed and conducted the electronic sur- nancial support of the government of Norway veys of Bank task team leaders and in-country (through the Ministry of Foreign Affairs and the stakeholders. Comments on the draft question- Norwegian Agency for Development Cooperation) naire received from Carlos Felipe Jaramillo and on for the stakeholder surveys and the government the draft survey instruments received from Hor- of Switzerland (through the Swiss Development moz Aghdaey, Ismail Arslan, Philippe Auffret, Arup Corporation) for the country reviews. Director-General, Evaluation: Vinod Thomas Director, Independent Evaluation Group–World Bank: Cheryl Gray Senior Manager, Country Evaluation and Regional Relations Unit, IEG: Ali M. Khadr Task Manager: Helena Tang ix Working group discussions in Mali. Knowledge exchange is one of the objectives of World Bank TA. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. Foreword Over the past year, the Bank has reinforced its tered for effectiveness, whether clients actually pro- focus on knowledge and learning, naming it duced part of the task or not. Sustained follow-up among its six strategic directions. In October after the completion of the tasks was important 2007, the president of the World Bank stated, for effectiveness. Whether clients requested the “The World Bank Group is a unique and special tasks did not matter for effectiveness, although all institution of knowledge and learning. Delivering, tasks needed to be tailored to client needs and in- expanding, and testing this learning—in tandem terests to be effective. ESW and TA were less ef- with financing or separately—is the most impor- fective in countries where government capacity tant part of our work.” was lower. Clients in middle-income countries prefer nonlending to lending services, and clients The Bank provides knowledge support to coun- in all countries prefer TA over ESW . tries through analytical and advisory activities. This evaluation focuses on two of these activities: The Independent Evaluation Group has five rec- economic and sector work (ESW) and nonlending ommendations for Bank action to obtain stronger technical assistance (TA). ESW is aimed at in- results from ESW and TA. First, reinvigorate the forming Bank lending and government policy, mandate for country teams to maintain a strong building client capacity, stimulating public debate, knowledge base for countries and sectors where and influencing the development community. TA the Bank is providing or planning to provide is aimed at assisting in policy implementation, funds. Second, ensure that ESW tasks in Interna- strengthening institutions, and facilitating knowl- tional Development Association countries are ad- edge exchange. ESW and TA are an essential part equately resourced (even if it means fewer ESW), of the Bank’s engagement with its clients—the because cost matters for quality, and quality mat- Bank spent $910 million (26 percent of its spend- ters for effectiveness. Third, enhance institutional ing on country services) on these products dur- arrangements for ESW and TA by ensuring sub- ing fiscal 2000–06. stantive task team presence in country offices— particularly in countries with low institutional This evaluation found that the majority of ESW and capacity—to facilitate closer client collaboration. TA met their objectives at least to an average ex- Fourth, recognize, receive, and build on client tent during fiscal 2000–06, although there were feedback to counterbalance current Bank incen- substantial differences across countries and tasks. tives for lending over nonlending and ESW over ESW and TA of higher technical quality were clearly TA. Fifth, take the results tracking framework for more effective in meeting their objectives. Close ESW and TA more seriously, including system- collaboration with clients during the process mat- atizing client feedback. Vinod Thomas Director-General, Evaluation xi Parliamentary discussions in the Republic of Yemen. Informing public debate is one of the objectives of World Bank ESW. Photo by Bill Lyons, courtesy of the World Bank Photo Library. Executive Summary I n 1996 the World Bank committed itself to becoming a “global knowledge bank,” using knowledge to improve the development effectiveness of its work. In fiscal 2008, the Bank reiterated its focus on knowledge and learn- ing, naming it as one of its six strategic directions. This evaluation focuses on two of the analytical and advisory activities through which the Bank provides knowledge to its client countries: economic and sector work (ESW) and non- lending technical assistance (TA). The objectives of ESW are to inform lending, in- effectiveness. Fourth, whether clients requested form government policy, build client capacity, the tasks or not did not matter for their effec- stimulate public debate, and influence the de- tiveness, although tasks not requested by clients velopment community. The objectives of TA are needed to be tailored to client needs and inter- to assist in policy implementation, strengthen in- ests to be effective. Fifth, there is some evidence stitutions, and facilitate knowledge exchange. that ESW and TA were less effective in countries ESW and TA are an essential part of the Bank’s en- where government capacity was lower. Finally, gagement with its clients—it spent $910 million an additional finding of the evaluation is that (26 percent of its spending on country services) clients in middle-income countries (MICs) prefer on these products during fiscal 2000–06. nonlending to lending services, and clients in all countries prefer TA to ESW . The majority of ESW and TA met their objectives at least to an average extent during fiscal 2000–06. In light of these findings and the Bank’s renewed Between 65 and 80 percent of users of ESW and focus on knowledge and learning, the Inde- TA in client countries and between 74 and 87 pendent Evaluation Group has five recommen- percent of users within the Bank gave ratings of dations for Bank action to obtain even better average and above when asked to what extent results from ESW and TA. First, reinvigorate the these products met their stated objectives. But mandate (which underpinned the ESW reforms there were substantial differences in ratings across in fiscal 1999) for country teams to maintain a countries and tasks, some of the reasons for which strong knowledge base on countries and sectors are as follows. where the Bank is providing or planning to pro- vide funds. Second, ensure ESW tasks in Inter- First, the technical quality of ESW and TA influ- national Development Association (IDA) countries enced their effectiveness. Second, close collabo- are adequately resourced (even if it means fewer ration with clients, from task initiation through the ESW). ESW tasks are less well resourced in the IDA formulation of recommendations, was important than in International Bank for Reconstruction for ESW and TA to be effective, whether clients ac- and Development (IBRD) countries. That can be tually produced part of the task or not. Third, sus- a problem, given the finding that cost matters tained follow-up after the completion of the tasks, for quality and quality matters for effectiveness. rather than just dissemination, was important for Greater selectivity will also reduce the burden xiii U S I N G K N OW L E D G E TO I M P R OV E D E V E L O P M E N T E F F E C T I V E N E S S on limited government capacity in some IDA countries, PERs, Financial Sector Assessment Pro- countries. Third, enhance institutional arrange- grams, and Investment Climate Assessments were ments for ESW and TA by ensuring substantive task among the more prominently cited ESW prod- team presence in country offices—particularly in ucts that had, variously, informed government countries with low institutional capacity—to fa- policy, built capacity, stimulated public debate, cilitate closer client collaboration, and including and influenced the development community. Box a clear strategy for ESW and TA dissemination ES.1 provides examples of particularly influential and follow-up at the concept paper stage. Fourth, ESW and TA. Examples of less effective ESW and recognize, receive, and build on client prefer- TA are provided in box ES.2. ences and feedback to counterbalance current Bank incentives for lending over nonlending and ESW and TA had effects beyond the stated ESW over TA. Fifth, take the results tracking frame- objectives in some cases. The more prominent work for ESW and TA more seriously, including by ones included effects on countries other than incorporating systematic client feedback. those for which they were intended, private in- vestment decisions by domestic and international The Extent to Which ESW and TA Met investors, support for local research, and the cre- Their Stated Objectives ation of informal networks between Bank staff and Most ESW and TA met their stated objectives policy makers. to at least an average extent, although their effectiveness was greater in shaping Bank Stakeholders in most but not all the coun- lending and strategy than in providing sup- tries reviewed indicated that ESW and TA port directly to client countries. The indirect had made a difference to the reforms in effects of ESW and TA on client countries— their countries. This view was expressed in through Bank lending—were greater than the countries with a wide range of income, govern- direct effects. Between 65 and 80 percent of ment capacity, and policy orientation. In some users of Bank ESW and TA in client countries countries, stakeholders indicated that the re- gave ratings of average and above on the extent forms in their countries would not have hap- to which ESW and TA met their stated objec- pened at all without Bank ESW and TA. Others tives; between 74 and 87 percent of such users indicated that without Bank ESW and TA, the re- in the Bank (task team leaders for loans and forms would have, variously, been of lower qual- strategies) gave such ratings. ESW had the largest ity, taken longer, and been more piecemeal and effects on informing Bank strategy, which is not less focused. Some indicated that Bank ESW and a stated objective. TA had helped them avoid or reduce the cost of policy mistakes. The presence of relevant ESW was statisti- cally associated with better loan design. Bank ESW and TA also provided a level of com- These results corroborate the relatively high rat- fort or confidence for some countries to move for- ings, on average, from loan task team leaders on ward. A few pointed to the credibility that Bank the extent to which ESW informed lending. The ESW and TA provided. The characteristics of Bank evaluation period was not long enough for as- ESW and TA that clients valued were high tech- certaining the effect of ESW on loan outcomes. nical quality, objectivity and neutrality, and pro- vision of international perspectives. In 2 of the 12 ESW in a range of sectors and report types, countries reviewed, however, a majority of the in- including tasks initiated by the Bank, had country stakeholders indicated that Bank ESW and positive effects both within the Bank and in TA made no difference to the reforms in their client countries. Country Economic Memo- countries. randa and Public Expenditure Reviews (PERs) were especially useful for informing Bank strate- The effectiveness of ESW and TA ranged gies and development policy loans. Within client from substantially above average to sub- xiv E X E C UT I V E S U M M A RY Box ES.1: Influential ESW: Examples from Vietnam and Mauritius The PER is one of the core diagnostic ESW products that was in- The Mauritius Aid for Trade TA (2006) exemplifies influential TA. fluential both within the Bank (in informing development policy The government requested the TA to help define its reform pro- loans and Country Assistance Strategies) and in client countries gram and to mobilize funds as part of the multidonor Aid for Trade (in informing government policy and building capacity). The Viet- Initiative. Within two months of the request, a Bank mission pro- nam PERs (2000, 2005) are among those that had important direct vided analysis of possible reforms, including estimates of adjust- effects. The two reports helped shape legislation to increase ment costs of opening up, tax reforms, and strategies to lower transparency in the state budget law (2002) and led to the adop- the cost of key services. At the end of the two-week mission, the tion of a medium-term expenditure framework and unification of Bank team presented the government with a summary of the the public accounts system. main elements of the reform options. Among the important fac- In addition to informing government policy in these ways, the tors underlying the effectiveness of the TA were the timely nature PERs also built government capacity through the collaborative of the support and the leveraging of existing ESW. After extensive way they had been done. This is reflected in the increasing in- internal debate, many of the mission’s recommendations were in- volvement of the government in the production of the PER, from pro- corporated into the government’s reform program, which was viding data (1996) to writing some sections of the report and supported by a follow-up Bank loan. providing data (2002) to writing the report on an equal footing with the Bank (2005). Source: IEG country reviews. stantially below average across countries the requisite skills for particular ESW and TA tasks, and tasks. The wide range of effectiveness is in which undermined the Bank’s credibility. part caused by country-specific factors, such as government capacity and government receptivity, ESW products of higher quality cost more. and in part by factors that are more amenable to Furthermore, the average cost of ESW tasks was being addressed by the Bank. lower in IDA than in IBRD countries. The Bank budget but not the trust fund component of cost Factors in ESW and TA Effectiveness was associated with the quality of ESW. When government capacity was lower, ESW and TA were less effective. ESW and TA were Close collaboration with clients, from the ini- less effective in postconflict and otherwise low- tiation of the task through the formulation capacity countries. Policy makers in such countries of recommendations, was important for had limited capacity to absorb all the ESW pro- ESW and TA to be influential. This may or may duced. High turnover of government officials also not have entailed a client actually producing part negatively affected capacity and, in turn, the ef- of the task, such as writing sections of a report. fectiveness of ESW and TA in some countries. Where partnership with clients made a dif- Government receptivity also influenced the ference, a variety of approaches was equally effectiveness of ESW. There were instances effective, depending on country conditions. where political constraints influenced gov- In countries with more limited government ca- ernment receptivity to certain ESW products, ei- pacity, partnering in the production of narrowly ther rendering such ESW less effective or delay- focused tasks was more effective than partnering ing its effectiveness until the political situation in broadly scoped tasks. changed. Whether the client specifically requested ESW and TA products of lower technical the ESW and TA did not matter for effec- quality were less effective. In some instances tiveness, although client buy-in was still the Bank did not have staff and consultants with important. In-country stakeholders in both xv U S I N G K N OW L E D G E TO I M P R OV E D E V E L O P M E N T E F F E C T I V E N E S S semination to the relevant counterparts resulted Box ES.2: When ESW and TA Were Not Effective in important changes; in other cases wide dis- semination stimulated the public debate that spurred changes. How wide dissemination Limited government capacity and insufficient collaboration/partnership needed to be for ESW and TA to be effective to garner government buy-in were the main reasons behind the lack of also varied within the same country for different success for several ESW in the Democratic Republic of Congo. In con- topics. trast, a few ESW there had very positive effects because of substantial partnering and collaboration. Translation is important to client countries. Even in countries with high levels of capacity, purely supply-driven Stakeholders in a majority of the countries re- tasks without government buy-in were not effective. An example is eco- viewed for this evaluation indicated that better and nomic monitoring ESW in MICs that had access to such analysis from the more rapid translation is needed. Having Bank staff private sector or international investors. The Bank’s economic monitor- or consultants that speak the local language clearly ing ESW had very little value added in such circumstances. enhanced the effectiveness of these tasks. Lack of government receptivity because of political constraints could also affect or delay the usefulness of ESW. In Peru, the Sierra Rural De- ESW and TA were not effective when some velopment Strategy became influential only four years after its comple- of these factors were not in place (see box tion, after a change in government. ES.2). Poor quality and slow translation were other reasons behind the poor effects of some ESW in some francophone West African and some Latin What Clients Want from the Bank American countries. In one of the countries, the translation took so long Stakeholders in IBRD countries indicate a that when the documents finally became available in the local language, strong preference for the Bank’s nonlend- they had lost much of their importance for policy making. ing services over its lending services. The Source: IEG country reviews. preference is less strong in IDA countries. TA is preferred to ESW in both groups of countries. MICs value the Bank’s advice, and some of them borrow from the Bank to obtain that low- and middle-income countries indicated that advice. Stakeholders in some MICs expressed a the Bank has a useful role in proposing topics for strong view that Bank ESW should be delinked ESW and TA, drawing on Bank knowledge and from its lending operations. IDA countries need international experience. Nevertheless, the Bank the Bank’s funds but also value the Bank’s advice. needs to ensure that there is genuine client in- TA is preferred to ESW in countries with high terest or needs to engender such interest. Tailoring government capacity as well as in those with tasks such as core diagnostics to clients’ needs and lower capacity. making efforts to collaborate from the conception stage were both effective in engendering client Stakeholders found the Bank’s reports and interest. TA more useful than those provided by other institutions. This finding was stronger for TA than Sustained follow-up beyond one-off dis- for ESW . semination was important for ESW and TA effectiveness. The ESW and TA with the great- The Bank’s clients have a strong desire for est effects were those for which there was sus- greater Bank presence in the country. Hav- tained follow-up on completion. The follow-up ing ESW and TA undertaken from country offices could be in the form of lending or nonlending ac- facilitates the close collaboration on these tasks tivities, with the latter encompassing formal or in- that is important for them to be effective. Follow- formal TA. up is also best sustained from country offices, given the long-term close interactions typically en- The scope of dissemination for effective tailed. Related to this is the need to further devolve ESW and TA varied. In some cases targeted dis- decision-making powers to the country offices, as xvi E X E C UT I V E S U M M A RY they are front-line recipients of government re- individual ESW tasks in IDA countries than in quests, especially for rapid-response TA. IBRD countries and is supported by the findings that cost matters for quality and quality matters The Bank’s Results Tracking Framework for effectiveness. Greater selectivity will also help for ESW and TA reduce the burden on limited government ca- The Bank needs to take the results tracking pacity in some IDA countries and will free re- framework for ESW and TA much more se- sources for more collaboration and follow-up riously. Although such a framework exists, its needed to enhance the effectiveness of ESW . Se- implementation is not enforced. This can be seen lectivity could be enhanced by giving priority to in the poor quality of information on ESW and TA ESW that informs Bank lending and strategy or that in the Bank’s administrative database, as well as in is clearly desired and needed by the client. the incomplete information in the results tracking framework itself. This in turn reflects the lack of Third, enhance the institutional arrange- serious monitoring or reporting of these tasks. Fi- ments for undertaking ESW and TA. To the ex- nally, there is no systematic follow- up with clients tent allowed by budget, ensure substantive task to evaluate the effects of these products. team presence in country offices, particularly in countries with low institutional capacity. This will Recommendations facilitate closer collaboration with clients from During fiscal 2000–06, the Bank spent nearly task initiation through follow-up. In addition, for- $1 billion on ESW and TA, slightly more than one- mulate a dissemination and implementation strat- quarter of its spending on country services. Al- egy for ESW and TA at the concept paper stage. though it is not possible to quantify the impact of Such a strategy should identify the target audience, this spending, this evaluation found that in the ag- the mode of dissemination, and the follow-up gregate, Bank ESW and TA met their objectives to arrangements after dissemination. All of these at least an average extent. Furthermore, Bank should be explicitly budgeted for as integral to clients find these products useful, with clients in the task. MICs finding them more useful than lending. Fourth, recognize and build on client pref- In light of these findings, the Independent Eval- erences, whether for nonlending versus uation Group has several recommendations to lending services or for TA versus ESW. Insti- help the Bank obtain better results for ESW and tute a mechanism to obtain client feedback on a TA. These recommendations will also help support periodic basis on delivered ESW and TA prod- the Bank’s six strategic directions, as knowledge ucts. Such feedback should include clients’ views is important for all of them. on collaboration, follow-up, and usefulness of the tasks (including specifics of how the tasks First, reinvigorate the mandate—which un- were used). The client feedback should be re- derpinned the fiscal 1999 ESW reforms—for quested at a set period (for instance, around one country teams to maintain a strong knowl- year) after the delivery of the task to the client to edge base on countries and sectors where the allow time for follow-up, and it should be the last Bank is providing or planning to provide milestone for ESW and TA. Obtaining client feed- funds. Bank country strategies and lending activ- back would encourage a stronger results focus for ities need to continue to be supported by requi- ESW and TA and would help counterbalance cur- site analysis, although a return to strictly defined rent Bank incentives for lending over nonlending “core diagnostic” ESW is unnecessary. and for ESW over TA. Second, ensure that ESW tasks in IDA coun- Fifth, take the results tracking framework tries are adequately resourced, even if it seriously, including by incorporating syste- means fewer ESW in some countries. This will matic client feedback, as noted in the above help to address the lower level of resources for recommendation. xvii At work on computers in South Africa. Building capacity is an objective of World Bank TA. Photo by Francis Dobbs, courtesy of the World Bank Photo Library. Management Response M anagement welcomes this Independent Evaluation Group (IEG) re- port on the World Bank’s economic and sector work (ESW) and tech- nical assistance (TA). In the findings of the review, management sees much on which it can build, and it regards the review as providing use- ful inputs to the ongoing work on analytic and advisory activities (AAA). Man- agement broadly concurs with the recommendations emerging from the evaluation. Introduction different Bank instruments and their delivery modes provide useful and welcome insights. Quality The IEG evaluation underscores the importance Recommendations of ESW and TA and provides a broad endorsement The recommendations include a call to reinvigo- of the quality and relevance of these Bank prod- rate the Bank’s mandate for a strong knowledge ucts. Overall, the evaluation notes, the ESW re- base; to focus ESW resources on fewer but better- forms of fiscal 1999 were accompanied by an funded studies; to enhance the institutional increase in the quality of ESW . It also notes that arrangements for undertaking ESW and TA by en- in-country stakeholders validated the Quality As- suring substantive task team presence in the field, surance Group’s high ratings on internal quality, particularly in countries with low institutional ca- although stakeholders rated dissemination activ- pacity, and by formulating at the concept stage a ities lower than technical quality. dissemination and follow-up approach; and to build on client preferences, including feedback Effectiveness after completing the task. Last, the report under- The evaluation concludes that most ESW and TA scores the need to strengthen the results tracking tasks met their stated objectives and were effec- framework for ESW and TA, specifically the re- tive in shaping lending. It also corroborates the porting mechanisms, and to improve its imple- link between the existence of relevant ESW and mentation. Management broadly agrees with these the quality of loan design. recommendations. From a client perspective, stakeholders in most countries indicated that ESW and TA tasks had General Comments made a difference to the reforms in their coun- The IEG evaluation underscores the need to make tries, although the effectiveness of these activities ESW more responsive to clients (during plan- differed across countries. Management finds that ning, preparation, and feedback stages). Man- IEG’s discussion of the several factors that influ- agement considers this approach to be good ence ESW and TA effectiveness and the summary practice. But there is an implicit assumption that of the preferences expressed by clients regarding the client is a specific entity that can express its xix U S I N G K N OW L E D G E TO I M P R OV E D E V E L O P M E N T E F F E C T I V E N E S S needs and preferences at the time these activities ative usefulness of regional and global studies in are planned. This is not always necessarily the case: this regard. Management notes that these latter sometimes the intended client is a broader and studies do have a wider reach that fell outside the more diffuse set of stakeholders. In some cases, scope of the IEG evaluation. Country users can be too, ESW is used upstream to help inform the expected to rate country ESW higher than re- country’s overall process of developing a poverty gional or global work, because country ESW is cus- strategy (poverty assessments, for example), a tomized to their needs. But global and regional role the importance of which is recognized in products often deepen methodological tools and the IEG evaluation. make them more accessible to teams, while pro- viding a cross-country analytical context that can Feedback and instrument choice serve as a very valuable background to country- Management agrees with the importance of ob- specific ESW . taining client feedback, but it must be done in a way that does not overburden the Bank’s coun- Caveats terparts. In preparing its review, management will Although agreeing with the broad thrust of the analyze options for incorporating client feedback. findings, management would like to point out Organizing such feedback needs to take into that the evaluation contains a few examples of account the Country Assistance Strategy (CAS) strong statements related to a single individual or context, particularly the CAS pillar(s) or themes a limited sample. These statements sometimes ap- to which they contribute. The report also men- pear to be going beyond the main points being tions incentive issues regarding instrument choice. made. The choice between different knowledge and learning instruments and lending is a fundamental Overall, management agrees with most of IEG’s part of the CAS process and of the CAS country findings and accepts its recommendations. Man- consultations that help address incentive biases. agement’s specific responses to IEG recom- mendations are given in the Management Action Regional and global studies Record. The IEG evaluation, by design, focused on ESW aimed at specific countries and reviewed the rel- xx M A N AG E M E N T R E S P O N S E Management Action Record Recommendation Management response Reinvigorate the mandate—which underpinned Agreed/Ongoing. Management agrees with the recommendation of main- the fiscal 1999 ESW reforms—for country teams taining a strong knowledge base in countries where the Bank is providing or to maintain a strong knowledge base on coun- planning to provide funds. This recommendation, which is supported by the em- tries and sectors where the Bank is providing or pirical association between the existence of upstream ESW and the quality of planning to provide funds. Bank country strate- loan design, is particularly relevant as the Bank moves into thematic lending gies and lending activities need to continue to be areas such as governance and climate change mitigation. In the context of the supported by requisite analysis, although a return review of AAA starting this fiscal year, and particularly for International De- to strictly defined “core diagnostic” ESW is velopment Association (IDA) countries, management will take stock of the ex- unnecessary. isting body of ESW and outline steps to address identified issues. Ensure ESW tasks in IDA countries are ade- Agreed/Ongoing. Management agrees with the importance of selectivity in al- quately resourced, even if it means fewer ESW locating ESW resources. In the context of ESW and TA, greater selectivity may in some countries. This will help to address the lead to reducing the number of tasks and to increasing the budget assigned lower level of resources for individual ESW tasks to each of the tasks. But the issue should not be seen just in terms of numbers in IDA countries than those in International Bank of tasks, but also in terms of consolidating tasks within a programmatic, mul- for Reconstruction and Development countries tiyear, results-based approach so as to enhance their overall impact. The and is supported by the finding that cost matters planning of the AAA program, including ESW and TA activities, takes place in for quality and quality matters for effectiveness. the context of preparation of the CAS, and selectivity is a major consideration Greater selectivity will also help to reduce the in preparing a CAS program that is tailored to country needs. As part of the con- burden on limited government capacity in some sultation process, as well as ongoing country policy dialogue, specific knowl- IDA countries and will free resources for more edge gaps are identified. How and when to address them is planned, taking collaboration and follow-up needed to enhance into account country needs, the work programs of other partners, and coun- the effectiveness of ESW. Selectivity could be en- try circumstances. For IDA countries, issues with data quality and availability hanced by giving priority to ESW that informs of analytic work prepared by the government or other sources may entail a larger Bank lending and strategy or that is clearly de- effort to fill gaps over time. In countries where what is needed is knowledge sired and needed by the client. on implementation (the engineering of reform) rather than traditional ESW, it is possible that, given weaker country capacity, smaller and more narrowly fo- cused ESW or TA is more effective than larger efforts. In its AAA review, man- agement will analyze resource allocation issues surrounding ESW and TA tasks and will propose steps to strengthen collaboration with in-country in- stitutions and other stakeholders to ensure an effective ESW/TA program. Enhance the institutional arrangements for un- Agreed/Ongoing. Management agrees that task team presence in field offices, dertaking ESW and TA. To the extent allowed by particularly in countries with low institutional capacity, will facilitate close col- budget, ensure substantive task team presence laboration with clients. The delegation of task management responsibilities to in country offices, particularly in countries with the field has been an ongoing process—for example, in the Sub-Saharan Africa low institutional capacity. This will facilitate Region and fragile states—and further steps are being considered. There are closer collaboration with clients from task initi- associated trade-offs that transcend budget constraints. Ensuring that ESW/TA ation through follow-up. In addition, formulate a is of top quality often requires that task management be carried out by thematic dissemination and implementation strategy for experts who cannot be decentralized, given their global role. In many countries, ESW and TA at the concept paper stage. Such a the size of the ESW program and budgetary constraints limit decentralization strategy should identify the target audience, the of specialized staff. In any country, the balance depends on the country con- mode of dissemination, and the follow-up text and the specific issues. Management is reviewing these considerations in arrangements after dissemination, all of which the context of the ongoing review of decentralization. Management fully agrees should be explicitly budgeted for as integral to the with the recommendation to clearly formulate and adequately fund the dis- task. semination and follow-up strategy for ESW and TA at the concept stage of the task. The AAA review will provide further guidance in this area. (continues on next page) xxi U S I N G K N OW L E D G E TO I M P R OV E D E V E L O P M E N T E F F E C T I V E N E S S Management Action Record (continued) Recommendation Management response Recognize and build on client preferences, Agreed. Management agrees that client feedback is a key input to assess how whether for nonlending versus lending services clients view the different dimensions of ESW/TA in particular and the design or for TA versus ESW. Institute a mechanism to of the Bank’s program in general. Feedback is already sought and used in the obtain client feedback on a periodic basis on context of client surveys, CAS preparation, policy dialogue, lending prepara- delivered ESW and TA products. Such feedback tion, and project implementation. Feedback focused on ESW and TA should pro- should include clients’ views on collaboration, vide further insights on client views as well as lessons for the design and follow-up, and usefulness of the tasks (including implementation of the ESW and TA program. Overall, the focus should be specifics of how the tasks were used). The client more on the development results that might follow from changes in client ap- feedback should be requested at a set period (for proach and policies, rather than short-term client satisfaction with the pro- instance, around 1 year) after the delivery of the duction process. In the context of the AAA review, management will review task to the client to allow time for follow-up, and options regarding the periodicity of requests for feedback to minimize the it should be the last milestone for ESW and TA. burden on country counterparts. Management will also analyze whether the Obtaining client feedback would encourage a feedback should focus on independent pieces of work rather than on a body stronger results focus for ESW and TA and would of work with a common thematic focus. help counterbalance current Bank incentives for lending over nonlending and for ESW over TA. Take the results tracking framework seriously, Agreed. Management agrees with the recommendation to ensure greater at- including by incorporating systematic client feed- tention to results, for example, by incorporating client feedback. Although use back, as noted in the above recommendation. of feedback is an important component of a results framework, other ele- ments could also be quite relevant. Attribution is certainly an issue: it is diffi- cult to ascribe results to a single piece of ESW. In some cases a credible results framework can be established around a thematic program of activities that will help monitor results within an adequate time frame. In the context of the AAA review that is now starting, management will examine the results framework for ESW and TA and outline steps to strengthen it. Management will consider all these agreed actions completed with the con- clusion of the AAA review and management decision based on the results of the review. Management will report to the Board on the outcome of that process in fiscal 2010. xxii Advisory Panel Statement Bimal Jalan Member of Parliament and former Governor of Reserve Bank of India; former Executive Director of the World Bank (1992–96) The evaluation report on the Bank’s economic and cellent World Development Report, World Bank sector work (ESW) and technical assistance (TA) 2004d). Although most developing countries, in- is well documented and covers all relevant issues cluding my own, have made progress in this area, including cost, utility, rating, scope, and client there is a still long way to go. In fact, it can be ar- reactions. The overall findings are, on balance, fa- gued that although growth rates in many poor re- vorable and make a case for the Bank to “rein- gions have increased with the opening and vigorate” its work in these areas—both to inform liberalization of economies, the position with re- lending as well as to help clients meet their de- spect to public sector governance and delivery of velopment goals. public services to the poorest has actually wors- ened in terms of quality and effectiveness. This is I found the independent evaluation highly in- particularly true in the rural sector (which is one formative, candid, and objective. In addition to the of the five dominant sectors covered by Bank views of Bank staff, country experts, and econo- ESW). mists, it also reflects the views of government officials of member countries and other benefi- My suggestion is that in the next five to six years ciaries. Recommendations for further action by the (say, 2009–15), the Bank’s ESW in IDA countries Bank are reasonable and deserve support. should concentrate on identifying and imple- menting global “best practices” in public delivery I have just one further suggestion for considera- of specified public services (such as shelter, lit- tion by the Independent Evaluation Group and eracy, nutrition, infant mortality, drinking water, Bank management. This relates to the need for a sanitation, and electricity) to the poorest, that is, global ESW on delivery of public services to the the bottom 20 percent of the population. Stud- poorest of the poor. As far as TA is concerned, ies show that delivery outcomes in providing present arrangements and orientation do not re- basic life services are worst for the poorest fifth quire much change. in most countries and across countries (see World Bank 2004d). Yet we now have sufficient infor- On ESW , an important recommendation made in mation available on what kind of institutional the evaluation is to focus primarily on International arrangements and practices work best for deliv- Development Association (IDA) countries and to ering services to the poorest of the poor. make them more cost-effective through greater selectivity. This is a welcome suggestion, but I An urgent task is to design a practical and work- would go a step further. By now the Bank has been able program of ESW in IDA countries that can be involved in efforts to alleviate the worst forms of replicated across countries with minimum re- poverty for more than 60 years and to “make ser- gional and country-specific variations. I should clar- vices work for poor people” (please see the ex- ify that such an exercise is not meant to be a xxiii U S I N G K N OW L E D G E TO I M P R OV E D E V E L O P M E N T E F F E C T I V E N E S S “universal rule book” that is mandatory and rigid, ation Group researchers involved in this study, but only a guide to “best practices” that have and (3) the editor or a staff member involved in the achieved the best results in countries with large preparation of the 2004 World Development Report percentages of poor people. (World Bank 2004d) on public services. To prepare such a global ESW for the poorest quin- Based on the group’s recommendations, regional tile in IDA countries, it may be desirable for the and/or country-specific ESW could be prepared by Bank to constitute a steering group of senior staff the Bank’s offices in IDA countries for imple- with relevant expertise from different parts of the mentation during 2009–15. Reports on actual re- Bank. The group could have (1) country special- sults in improving public services for the poor ists from 12 countries in the sample used for the should then be presented for consideration by the present study, (2) one or two Independent Evalu- Board every year. Roberto Junguito Former Finance Minister and Agriculture Minister, Colombia The report on the ESW and TA prepared by the Regarding the user ratings, I agree that they are Independent Evaluation Group is excellent. On mostly very positive both for ESW and TA. How- reading the report, I would fully share the view ever, I note that ratings especially for ESW are bet- that “the Bank needs to take the results tracking ter in the higher-income countries in the sample. frameworks for ESW and TA more seriously.” No comments are made regarding ratings ac- cording to the per capita income levels of the On the methodological aspects of the paper I client countries. only have a couple of comments. The first is that the report is based only on a sample of 12 coun- I would also have liked to see some measures of tries. One would have wished to see a larger sam- dispersion of ratings among countries. It is sim- ple. For example, in the case of the Latin America ply highlighted that there were opinions that went and the Caribbean Region, it only includes two from significant impacts on countries’ policies to countries—Peru and Guyana. no differences in reforms to the countries. My second comment regards the time frame. The The relative preference of clients for TA over ESW report includes experiences from 2000 to 2006. depends on World Bank requirements (the Fi- This was a period when emerging countries tran- nancial Sector Assessment Program, Public Ex- sited from crisis in 2000–02, when lending pro- penditure Reviews, and Policy Notes) over country grams from multilaterals were badly needed, to preferences revealed in TA service demands. This a period of large market access. Nevertheless, no does not mean that ESW are not important, but differences of opinions among clients or task that ESW are simply seen as part of potential con- team leaders are made regarding ESW and TA ditionality for future loans. undertaken in the first and second periods. Regarding the recommendations made as a result The fact that most opinions would have been of the independent evaluation, I would agree on made recently would also explain the clients’ in- reinvigorating the mandate but not on devolving terests in the Bank’s nonlending services more management to country offices. Nowadays, they than lending services. are usually of very poor technical quality and xxiv A D V I S O R Y PA N E L S TAT E M E N T mainly serve social purposes of the World Bank. Bank client countries, it is especially important to Rather, I would stress the importance of using local issue regional and global ESW . economists and research centers as partners in ESW and TA efforts. Finally, I would like to support the future quality issues stressed in the report. Technical quality, part- Neither do I share the view that World Bank ser- nerships, policy options, working with different vices on ESW should focus on IDA countries. I ministries and levels of government, and timeli- know this is the “fashion” at the World Bank, ness are all very important. Dissemination issues given pressures made by the Development Com- are extremely important, because in my experi- mittee. Nevertheless, I do not see that this con- ence, policy-maker technical papers are restricted clusion follows from the evaluation. In World to the authorities. xxv Chapter 1 Evaluation Highlights • This evaluation focuses on two of the Bank’s analytic and advisory ac- tivities, economic and sector work (ESW) and technical assistance (TA), over the period fiscal 2000–06. • ESW seeks to influence client poli- cies and programs, TA to enable clients to implement reforms and strengthen institutions. • Reforms have introduced frame- works for tracking results on spec- ified objectives for ESW and TA. • The evaluation assesses the extent to which the stated objectives of ESW and TA have been met. • The evaluation also assesses how certain factors influence the extent to which the stated objectives are met—the way ESW and TA are orig- inated, partnership with clients in production, technical quality, and dissemination of these products. In Mali, a World Bank transport sector report informed donor funding in the sector. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. Introduction and Methodology O ver the past year the Bank has reiterated its focus on knowledge and learning, naming it among its six strategic directions. In October 2007 the president of the World Bank stated, “The World Bank Group is a unique and special institution of knowledge and learning. Delivering, ex- panding, and testing this learning—in tandem with financing or separately— is the most important part of our work” (World Bank 2007a). The Bank provides knowledge to its client coun- intent of influencing an external client’s ESW specifically aims to tries through analytical and advisory activities policies and/or programs, and (3) repre- influence client policies (AAA). This AAA consists of economic and sector sents the views of the Bank (that is, is not and programs. work (ESW), nonlending technical assistance (TA), attributed to individual authors). ESW donor and aid coordination, research services, underpins the Bank’s policy dialogue impact evaluation, and the World Development with clients, its development of country strate- Report. gies, and its formulation and implementation of effective lending programs. ESW is also used to This evaluation focuses on two of these prod- build institutional capacity, especially when it is un- ucts, ESW and TA, which are intended to sup- dertaken in partnership with local institutions. port client countries’ efforts in strengthening their policies and institutions. The evaluation The Bank defines TA as an activity that (1) TA primarily aims to does not cover research or the other AAA. has the primary intent of enabling an ex- enable clients to ternal client to implement reforms and implement reforms and How much importance does the Bank afford ESW strengthen institutions, (2) is freestand- strengthen institutions. and TA? In 1996 the Bank committed itself to be- ing (that is, is not an essential part of an- coming a “global knowledge bank,” using knowl- other activity, such as lending or ESW), and (3) is edge to improve the quality of its operations linked to a Bank unit with clear accountability and to obtain development results, making such for the service provided. knowledge products a high priority (IEG 2003, p. ix). At the end of fiscal 1999, the Bank launched The Bank’s ESW activities have been evolving over a comprehensive program of ESW reforms to im- time. Since July 2004, ESW has been classified into prove the effectiveness with which resources are two output types: reports (studies outlining pol- used. In 2004, it launched reforms to improve the icy recommendations) or policy notes (“just- focus and quality of TA and to eliminate the over- in-time” advice to a client). Since September lap between TA and ESW product lines. 2004, TA has been classified into five categories: (1) institutional development plans, (2) “how to” The Bank defines ESW as an activity that (1) in- guidance, (3) models/surveys, (4) client docu- volves analytic effort, (2) is undertaken with the ment reviews, and (5) knowledge-sharing forums. 3 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Since fiscal 2005, it has been mandatory for ESW Box 1.1: ESW and TA: What Are They? to have objectives specified at task initiation, al- though some tasks already had objectives speci- fied even prior to fiscal 2005.1 Of the 1,817 ESW An ESW is what is typically referred to as a Bank report. It could be broad delivered during fiscal 2002–06 that had objec- based, such as a Country Economic Memorandum (CEM) covering sev- tives specified, the majority (69 percent) had eral sectors of the economy, or narrow, such as a sector report covering informing government policy as one of the ob- a specific sector, for example, transport. jectives, followed by stimulating public debate The Bank produces more than 50 types of such reports. An example (42 percent), informing lending (41 percent), is the Public Expenditure Review (PER), which aims to assist the client building capacity (32 percent), and informing the country reform public expenditures. A PER can include analysis of the development community (32 percent). More than appropriate public-private mix of goods and services provision, evalua- half of the tasks had multiple objectives. tion of public expenditure priorities, analysis of the sustainability of the fiscal deficit, review of revenue generation, and an examination of pub- As with ESW , since fiscal 2005, all TA have been re- lic sector institutional arrangements. quired to have objectives specified at task initia- TA provides technical advice to clients. It can be advice on formulat- tion, although some tasks already had objectives ing and implementing policies and programs, such as on public-private specified prior to fiscal 2005. Of the 908 TA de- partnerships in infrastructure, or on drafting legislation. It can also be the livered during fiscal 2000–06 that had objectives provision of training on data analysis (such as on household survey data) specified, a large majority (three-quarters) had or on model simulation (such as simulation of the effects of tariff changes strengthening institutions as one of the objec- on revenues and on production). tives, followed by policy/program implementa- Chapter 4 provides specific examples of the effects of PERs and other tion (52 percent) and knowledge exchange (46 ESW, as well as effects of various TA undertaken in the countries reviewed. percent). As in the case of ESW , more than half of Source: World Bank. the tasks had multiple objectives. Objective, Framework, and Scope Reforms of ESW and TA Appendix A lists ESW report types and TA of the Evaluation introduced frameworks output types. Box 1.1 provides examples This evaluation assesses the extent to which ESW for tracking their results. of ESW and TA. and TA have met their stated objectives2 as listed in table 1.1 and derives recommendations on As part of the reforms of ESW and TA, in 2004 the how to improve the effectiveness with which Bank introduced results tracking frameworks for these products meet their objectives. The con- enhancing the reporting and monitoring of these ceptual framework underpinning the evaluation activities. The tracking frameworks specify the is presented in table 1.2. The focus is on evaluat- objectives—and success indicators for ing ESW and TA aimed at specific countries (called Nearly all the objectives achieving these objectives—for ESW and country-specific ESW and TA in this evaluation). of ESW and TA focus TA (table 1.1). In so doing, the Bank directly on client made explicit the implicit objectives for The evaluation also reviews the relative usefulness countries. these tasks. of regional and global—compared with country- specific—ESW and TA in addressing specific coun- Of the five ESW objectives, one is aimed at in- try concerns. But the evaluation does not—nor forming Bank activities (inform lending), and the is it intended to—provide a thorough evaluation other four are aimed directly at client of the effectiveness of these other types of ESW The evaluation assesses countries. Of course, informing lending and TA, which may have objectives other than the extent to which the is a channel through which the Bank those directed at country-specific concerns. two product lines have provides support to client countries. All met their stated three TA objectives are aimed directly at Specifically, the evaluation derives findings re- objectives. client countries. garding inputs (such as Bank rules and processes, 4 INTRODUCTION AND METHODOLOGY Table 1.1: The Bank’s ESW and TA Objectives and Success Indicators ESW TA Objectives and success indicators Objectives and success indicators Inform lending Assist in client’s policy/program implementation • Agreed lending program for sector/country • Policy/program implemented • New loan under preparation/implementation • Compliance with law/regulation achieved • Other • Other Inform government policy Develop/strengthen institutions • Government decree issued • Increased capacity to design strategies/policies • New legislation adopted • Increased effectiveness in managing operations • New government strategy adopted • Increased ability to monitor/evaluate operations • Other • Other Build client analytical capacity • Client makes major analytical contribution Facilitate knowledge exchange • Client learns to produce output independently • New conceptual frameworks learned • Other • Learned knowledge applied to local circumstances Inform/stimulate public debate • Other • Media widely report Bank analysis • Major stakeholders reflect on Bank views • Other Influence the development community • Additional resources mobilized • Shift in donor policies/priorities • Other Source: World Bank. Note: ESW = economic and sector work; TA = technical assistance. budgets, corporate incentives, and so on) and The evaluation examines the effects of The findings regarding outputs (for example, the number and types of ESW and TA completed since the ESW re- inputs and outputs are product, technical quality, dissemination strategy, form initiative, that is, those products de- intended to enhance the and so on) that can enhance the effect of ESW and livered during fiscal 2000–06. During this effects of ESW and TA. TA on outcomes (such as the quality of Bank lend- period, the Bank delivered 3,930 ESW ing or country policy or institutional changes). and 1,856 TA tasks to clients (see appendix C). The Bank’s work—with ESW and TA being a com- ponent of this work—focuses ultimately on In addition to evaluating the extent to which ESW growth, poverty reduction, and other develop- and TA met their objectives, for each objective the ment objectives, but given the enormous diffi- evaluation also addresses the following four eval- culty in attribution, this study does not evaluate uative questions: the effect of ESW and TA on these development The evaluation covers outcomes. In other words, it does not evaluate the • To what extent does the way ESW and the period since reforms “impact” part of the evaluation framework laid TA originate affect achievement of began (that is, fiscal out in table 1.2. objectives? 2000 through 2006). 5 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Table 1.2: Evaluation Framework Initiation of ESW/TA Originated by Bank rules (IDA requirements, corporate mandates; core diagnostics) Inputs Other origination (country units; sector units; clients) Partnership (clients/local researchers/donors) Budget Delivery of ESW/TA Number, cost, and type (reports vs. policy notes; core diagnostic versus others; Outputs and so on) Technical quality Dissemination Objectives of ESW Objectives of TA • Informed lending • Assisted in policy/program implementation • Informed policy • Developed/strengthened institutions Outcomes • Built client analytical capacity • Facilitated knowledge exchange • Informed/stimulated public debate • Influenced development community Development outcomes Impact Growth and poverty reduction and other development objectives Source: IEG. Note: ESW = economic and sector work; IDA = International Development Association; TA = technical assistance. • How does partnership with local institutions Evaluation Tools in the production of ESW and TA affect the The findings of this evaluation rest on the fol- achievement of objectives? lowing five sets of evidence. Details on each are • How has the technical quality of ESW and TA in appendix B. affected achievement of objectives? • How does dissemination affect the achieve- 1. Reviews of the effects of ESW and TA and ment of ESW and TA objectives? the factors affecting these effects in 12 countries. The reviews entailed desk reviews The evaluation also reports the other effects of and structured interviews (including in the ESW and TA that emerged from the findings. field) with in-country stakeholders, Bank staff, 6 INTRODUCTION AND METHODOLOGY and management working on those countries. The tasks than those who did not re- The findings derive from 12 countries are Bangladesh, the Democratic Re- spond. five sets of evidence. public of Congo, Guyana, Jordan, Lesotho, Malaysia, Mali, Mauritius, Peru, Romania, Serbia, and Vietnam. 3. An electronic survey of Bank loan TTLs The countries were selected to be broadly repre- on their views on (and quantitative ratings for) sentative of Bank clients with respect to Region, in- specific ESW delivered during fiscal 2000–06 come, population, Country Policy and Institutional that may have informed their loans. Those Assessment, size of lending (per capita), and share ESW TTLs who were also loan TTLs were ex- of ESW cost in the cost of country services. cluded from the survey. The country reviews sought the views (with There were 91 responses out of a net ad- quantitative ratings) of the stakeholders on var- dressable sample of 186 loan TTLs, a 49 per- ious dimensions of specific ESW and TA tasks cent response rate. Bias checks indicated no (quality, partnership, origination, dissemina- Regional or sectoral biases in the responses. tion, timeliness, and so on) and the extent to which these tasks met the various ESW and TA 4. Electronic surveys of in-country stake- objectives listed in table 1.1. holders for the purpose of validating findings from the country reviews. These are a survey A total of 353 in-country stakeholders were in- on specific ESW tasks, a survey on specific terviewed for the country reviews, of which TA tasks, and a survey on general views on 191 were public officials, 9 were members of ESW and TA tasks. legislatures, 63 were donor representatives, 39 were civil society representatives, 33 were For the survey on specific ESW tasks, out of from the private sector, and 18 were from ac- a net addressable sample of 297 stakeholders, ademia or research institutions. A total of 129 70 responded on ESW for 41 countries. The ESW , 64 TA, and 88 loans were reviewed. The response rate was 24 percent. For the survey sectoral distributions of the ESW and TA re- on specific TA tasks, of a net addressable sam- viewed were in line with the sectoral distri- ple of 144, 33 responded on TA for 17 coun- butions of all such tasks delivered Bank-wide tries, a 23 percent response rate. For the during fiscal 2000–06. general ESW/TA survey, of a net addressable sample of 2,418 stakeholders, 550 responded 2. Electronic surveys of all Bank task team from 114 countries. The response rate was leaders (TTLs) on specific ESW and TA again 23 percent. Bias checks did not reveal they had led during fiscal 2000–06. The surveys significant biases in the sectoral or Regional sought the views of TTLs on various dimensions distribution of respondents with respect to of ESW and TA and their perceptions (with those who were sent the survey or with re- quantitative ratings) on the extent to which spect to the sectoral or Regional distribution the various ESW and TA objectives were met. of all Bank ESW and TA. The net addressable sample (see appendix B 5. Statistical analysis of associations between for definition) of TTLs was 1,143 for ESW and different aspects of ESW and TA (numbers, 419 for TA; of these, 696 ESW TTLs and 220 TA average costs, quality, types, partnerships, TTLs responded, representing response rates origination, and so on) and indicators of of 61 and 53 percent, respectively. Bias checks achievement of objectives (loan quality, gov- revealed no significant biases with respect to ernment policies, institutions, and so on). the respondents: specifically, those TTLs who This component of the work was based on responded did not generally have higher Qual- Bank administrative data, data collected from ity Assurance Group (QAG) ratings for their 7 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS In most cases, only surveys, and other pertinent data on all the surveys are on a scale of 1 to 6, with 1 being findings supported by all country conditions. low and 6 being high, 4 being slightly above av- five sources of erage, and 3 being slightly below average. information are reported. Only those findings that are supported by all five sources of information are re- Finally, the evaluation also considered the findings ported, unless otherwise noted. All the quantita- of other Independent Evaluation Group (IEG) tive ratings provided in the country reviews and evaluations where applicable. 8 Chapter 2 Evaluation Highlights • The number and cost of ESW and TA rose over the evaluation period. • About one-third of ESW and 60 per- cent of TA were requested by clients. • Larger shares of both ESW and TA were requested by International Bank for Reconstruction and Devel- opment countries than by Inter- national Development Association countries. • Core diagnostic reports are more costly than other types of reports. • In-country stakeholders rated tech- nical quality of ESW and TA signifi- cantly higher than either dissemi- nation or overall impact. • It can be inferred that the ESW re- forms resulted in increased ESW quality. Newsroom in Maputo, Mozambique, that provides copy for its own and other publications. Stimulating public debate is one of the objectives of ESW. Photo by Eric Miller, courtesy of the World Bank Photo Library. Trends and Patterns of Bank ESW and TA, Fiscal 2000–06 Number and Cost of ESW and TA D uring fiscal 2000–06, the Bank delivered 3,930 ESW and 1,856 TA to clients, at a total cost of around $910 million ($630 million for ESW and $280 million for TA).1 The annual outlays on ESW more than dou- bled—from $62 million in fiscal 2000 to $133 million in fiscal 2005—concur- rent with the ESW reforms described in chapter 1. The increase in annual outlays on TA was even cost of ESW was 13 percent higher for In- The Bank increased greater, more than tripling, from $16 million to $52 ternational Bank for Reconstruction and spending on ESW and TA million over the same period (figure 2.1).2 The in- Development (IBRD) countries than In- during fiscal 2000–06 in crease in outlays was due to both an increase in ternational Development Association dollar terms and as a the number of ESW and TA products delivered to (IDA) countries, while the average cost share of spending on clients (figure 2.1) and an increase (albeit more of TA was 7 percent higher. The average country services. modest) in the average cost of these products (see cost of ESW was higher for 15 of the 20 appendix C). sectors for IBRD compared with IDA The average cost of ESW countries, and the average cost of TA was slightly higher than In parallel with the increase in outlays on ESW and was higher for IBRD countries for only for TA, and the average TA, Bank spending on these products as a share of half of the sectors (see appendix C). The costs of both ESW and TA its spending on country services rose from 22 per- average costs of ESW and TA tasks were were higher in IBRD than cent in fiscal 2001 to 27 percent in fiscal 2006. higher in IBRD than in IDA countries in IDA countries. Over the same period, Bank spending on lending all the Regions (table 2.2).3 operations (lending preparation and development and project supervision) fell from 59 to 56 percent. Sectoral Distribution of ESW and TA During fiscal 2000–06, around 78 percent of the During fiscal 2000–06, the largest number of ESW ESW delivered was country specific, 16 percent re- delivered by the Bank (including country-specific, gional, and 6 percent global. Over the same pe- regional, and global ESW) was in economic pol- riod, around 80 percent of the TA delivered was icy, followed by the financial sector, public sector country specific, 18 percent regional, and 2 per- governance, private sector development (PSD), cent global (see appendix C). and the rural sector. ESW in these five sectors made up more than half of the number of ESW de- The average cost of ESW was somewhat higher livered during that period.4 The same five sectors than that of TA (table 2.1). Overall, the average are also at the top with respect to the cost of ESW. 11 U S I N G K N OW L E D G E TO I M P R OV E D E V E L O P M E N T E F F E C T I V E N E S S TA) delivered by the Bank were in the financial sec- Figure 2.1: Number and Cost of ESW and TA tor, followed by PSD, public sector governance, Delivered to Clients and almost equal numbers in environment, social 800 140,000 development, energy and mining, and economic policy. TA in these seven sectors made up 62 per- 700 120,000 cent of all Bank TA delivered during that period. 600 TA in these sectors dominated in most of the Re- 100,000 gions (see appendix C) and in IBRD and IDA Cost (US$ 000) 500 countries. For fragile states, water supply and 80,000 Number 400 sanitation and the rural sector replaced financial 60,000 sector and PSD among the top sectors. 300 40,000 Regional Distribution of ESW and TA 200 During fiscal 2000–06, the Sub-Saharan Africa 100 20,000 Region delivered the largest number of ESW 0 0 (country-specific and Regional), accounting for 23 2000 2001 2002 2003 2004 2005 2006 percent of ESW delivered by the Bank. However, Fiscal year per country, the Region delivered the fewest ESW deliveries TA deliveries ESW , and the South Asia Region delivered the ESW cost TA cost most. Per task, country-specific ESW costs the least, but Regional ESW costs the most in Sub- Source: World Bank administrative data. Saharan Africa, compared with other Regions. Note: ESW = economic and sector work; TA = technical assistance. Hence, Sub-Saharan Africa had the highest ratio of average cost of Regional versus country-specific ESW (see appendix C). These five sectors overlap with many of the sec- tors of core diagnostic ESW (see box 2.1). ESW in With respect to TA, East Asia and Pacific deliv- these sectors dominated in most of the Regions ered the largest absolute number (accounting (see appendix C) and in IBRD and IDA countries. for 24 percent of TA delivered Bank-wide), as well In fragile states, social development replaced the as the largest number per country. Per country, the financial sector as one of the top sectors. Middle East and North Africa Region accompanied East Asia and Pacific in delivering the most; Latin During the same period, the largest numbers5 of America and the Caribbean delivered the fewest. TA (including country-specific, regional, and global As in the case of ESW , Sub-Saharan Africa also had the most expensive Regional TA, as well as the highest ratio of the average cost of Regional to Table 2.1: Average Costs of Country, country-specific TA (see appendix C). Regional, and Global ESW and TA, Fiscal 2000–06 Trust Fund Contribution to ESW Average cost Average cost and TA Costs of ESW of TA Trust fund contributions rose in parallel with the ($US) ($US) increase in Bank budget for ESW and TA during Country 159,846 129,723 fiscal 2000–06, in line with the Bank’s focus on Regional 175,609 239,807 these activities during the period.6 During that pe- Global 127,597 194,471 riod, roughly equal amounts of trust funds went All 160,384 150,608 to ESW and TA ($115 million and $118 million, re- Source: IEG staff calculations based on World Bank administrative data. spectively), contributing to around 18 percent of Note: ESW = economic and sector work; TA = technical assistance. expenditures on ESW and a much higher 42 per- 12 T R E N D S A N D PAT T E R N S O F B A N K E S W A N D TA Table 2.2: Average Costs of ESW and TA by Region, IBRD versus IDA, Fiscal 2000–06 ($US 000) Europe Latin America Middle East East Asia and and the and South Africa and Pacific Central Asia Caribbean North Africa Asia Total Average cost of ESW tasks IBRD 153.5 192.9 167.7 174.4 160.0 n.a. 172.1 IDA 139.9 161.9 158.0 116.5 110.3 170.7 152.1 Average cost of TA tasks IBRD 154.4 168.0 123.7 151.5 90.4 n.a. 136.4 IDA 116.1 154.1 94.1 124.6 84.5 133.6 127.7 Source: IEG staff calculations based on World Bank administrative data. Note: n.a. = not applicable; ESW = economic and sector work; IBRD = International Bank for Reconstruction and Development; IDA = International Development Bank; TA = technical assistance. cent of expenditures on TA. The share of trust governments and the Bank; 1 percent The Sub-Saharan Africa funds in TA expenditures rose from 35 to 43 per- requested by the Board; and 1 percent by Region delivered the most cent during the period but remained more or donors (table 2.3). ESW but the fewest per less the same for ESW , at around 20 percent (with country, and East Asia lower shares in fiscal 2002 and fiscal 2003 of 12 and Compared with ESW , client governments and Pacific delivered 7 percent, respectively). Certain sectors were fa- requested a much larger share (around the most TA and the most vored by trust funds for ESW and TA, not all of 60 percent) of TA tasks delivered during per country. which overlapped with the top sectors of focus by fiscal 2000–06 (table 2.4). This informa- the Bank for these products (see appendix C). tion came from TA TTLs in two sources of information: the Bank’s administra- Trust fund contributions Origination of ESW and TA tive database and responses to the TA increased, along with the According to ESW TTLs, client governments re- TTL survey. Unlike for ESW , there is a Bank budget for ESW quested about one-third of the ESW delivered dur- higher degree of congruence between and TA. ing fiscal 2000–06. This came from two sources of the two sources of information: of the information, both provided by ESW TTLs: the 134 TA tasks classified as client requested Bank’s administrative database and the responses in the Bank’s administrative database,Client governments to the ESW TTL survey.7 Although the aggregate 95 were reported as requested by the requested about one-third shares from the two sources are similar, there are government in the survey responses. of the ESW tasks and significant differences for individual tasks. Specif- about 60 percent of the ically, of the 218 ESW tasks classified in the Bank’s Twenty-seven percent of TA tasks were TA tasks. administrative database as requested by the client, originated by the Bank either by sector only about half were reported that way in the sur- or country units or a combination thereof, or by vey responses. In other words, in only half the cases Regional management, by the World Bank Insti- did TTLs give the same information in the survey tute, or by team leaders themselves. Eight percent as they did in the Bank’s administrative database. were requested by senior management,8 1 per- cent by donors, and 1 percent by a combination Fifty-four percent of ESW tasks were originated by of client governments and the Bank. the Bank, either by sector or country units or a combination thereof, by Regional management, or According to the survey responses of IBRD countries requested as IDA mandates (box 2.1); 12 percent by senior Bank TTLs, larger shares of both ESW and more ESW and TA than management; 2 percent by a combination of client TA were requested by clients in IBRD IDA countries. 13 U S I N G K N OW L E D G E TO I M P R OV E D E V E L O P M E N T E F F E C T I V E N E S S Box 2.1: IDA-Mandated ESW and Core Diagnostics Some ESW products have been required for IDA countries in the In fiscal 2002, Bank management made a commitment to the context of IDA replenishment exercises. The first ESW products Board that all active countries have in place by the end of fiscal mandated by IDA were Poverty Assessments; specifically, IDA 11 2004 current (that is, delivered in the past five years) core diag- (fiscal 1997–99) provided that Poverty Assessments should be nostics. The core diagnostics are Poverty Assessments, CEMs or completed for all IDA borrowers. IDA 12 (fiscal 2000–02) provided a Development Policy Review, PERs, CFAAs, and CPARs, or their that Poverty Assessments and PERs should be undertaken in each functional equivalents. IDA country. IDA 13 (fiscal 2003–05) set out targets for the com- In August 2005, the Bank stated that the experience since 2003 pletion of Country Financial Accountability Assessments (CFAAs), of required coverage and mix of diagnostic work sometimes resulted Country Procurement Assessment Reviews (CPARs), PERs, and In- in inefficient use of Bank resources and uneven benefits to clients. vestment Climate Assessments (ICAs). These targets were met in Management decided that diagnostic work from then on would be the spring of 2004, as scheduled. The IDA 14 replenishment (fiscal programmed country by country, depending on the type and level 2006–08) did not specify targets for ESW. of Bank engagement, client priorities and circumstances, and the availability of relevant knowledge from partners. Management would continue to monitor the status of diagnostic work. Source: World Bank. than in IDA countries (figures 2.2 and 2.3). Ac- sources, more than half of the TA tasks were cording to both TTL survey responses and Bank client requested for all sectors except social administrative data, during fiscal 2000–06 the development.10 sectors with the highest shares of client-requested ESW were PSD; global information, communica- ESW Report Types tions, and technology (ICT); energy and mining; According to the Bank’s administrative database, and urban development.9 Also according to both during fiscal 2000–06, 16 percent of the ESW de- livered to clients were core diagnostic reports, 14 percent were other diagnostic reports, 37 percent Table 2.3: Origination of ESW, Fiscal 2000–06 were advisory reports, and 32 percent were not (percent) assigned any report types provided by the Bank’s According to According ESW report type classification system (see table survey of to Bank 2.5; also see appendix A for the report type clas- Requestor/Originator Bank TTLs administrative data sification system and appendix C for the numbers Client 31 32 of ESW by report type). Board 1 1 Senior management 12 9 Given the already large number of report types Strategic task — 57 (53) provided by the Bank’s classification system, the fact that nearly one-third of the ESW delivered Government and Bank 2 — to clients (1,240 tasks) was not assigned any of Donor 1 — these types raises questions about the current Sector unit 23 — system in the Bank for reporting and monitor- Country unit 19 — ing ESW. It is possible that the 53 types of reports IDA mandate 2 — provided in the system are not exhaustive, al- Other Banka 10 — though an “other types” category is provided and Source: IEG ESW TTL survey and World Bank administrative data. 1.4 percent of the ESW was assigned that classi- Note: ESW = economic and sector work; TTLs = task team leaders. Numbers do not total 100 because of rounding. fication. The other possibility is that Bank staff a. See text description. do not take reporting on ESW seriously, which in 14 T R E N D S A N D PAT T E R N S O F B A N K E S W A N D TA turn reflects the lack of attention of the Bank to Table 2.4: Origination of TA, Fiscal 2000–06 such data in the system. This point will be ad- (percent) dressed in more detail in chapter 6. According to According Of the tasks assigned report types in the Bank’s survey of to Bank administrative data, among the most numerous Requestor/Originator Bank TTLs administrative data were the core diagnostic reports. This reflects Client 61 62 the Bank’s emphasis on such report types during Senior management 8 2 the first part of the decade (box 2.1). Additionally, Strategic task — 30 the Financial Sector Assessment Program (FSAP), Government and Bank 1 — introduced by the governors of the Bank and the Donor 1 — International Monetary Fund (IMF) in 1999 in Sector unit 14 — the wake of the financial crises of the late 1990s, Country unit 10 — and the ICA, which was required for IDA countries Other Banka 3 — (box 2.1), were also numerous. Finally, the types Source: IEG TA TTL survey and administrative data. of Other Financial Sector Studies, Other Infra- Note: TA = technical assistance; TTL = task team leader. Numbers do not total 100 because of rounding structure Studies, and Other Public Sector Stud- or “not assigned” category in the Bank database. a. See text description. ies were also numerous, although none of these was mandated. one-third. Of the core diagnostic reports, the In- During fiscal 2000–06, the average cost of a report tegrative Fiduciary Assessments were the most ex- was $160,384. On average, core diagnostic re- pensive, while the CFAAs and CPARs were the ports were the most costly (see table 2.5), ex- least costly, on average around half the cost of an ceeding the cost of an average report by more than average report. With respect to other mandated Figure 2.2: Share of Client-Requested ESW Figure 2.3: Share of Client-Requested TA in in IBRD and IDA Countries, Fiscal 2000–06 IBRD and IDA Countries, Fiscal 2000–06 60 100 50 80 40 60 Percent Percent 30 40 20 20 10 0 0 2000 2001 2002 2003 2004 2005 2006 2000 2001 2002 2003 2004 2005 2006 Fiscal year Fiscal year IBRD IDA IBRD IDA Source: IEG ESW TTL survey. Source: IEG TA TTL survey. Note: ESW = economic and sector work; IBRD = International Bank for Reconstruction and Note: IBRD = International Bank for Reconstruction and Development; IDA = International Development; IDA = International Development Association. Development Association; TA = technical assistance. 15 U S I N G K N OW L E D G E TO I M P R OV E D E V E L O P M E N T E F F E C T I V E N E S S Similar dimensions of Bank ESW and TA are rated Table 2.5: Numbers and Average Costs of ESW, by by the Bank’s internal QAG. QAG assesses the Report Type, Fiscal 2000–06 quality of these tasks on relevance, quality of the Average cost analysis, dialogue and dissemination, and likely im- Report types Numbers ($US) pact. In addition, there is an overall quality rating Core diagnostic 622 214,151 that synthesizes these ratings. The QAG ratings are Public Expenditure Review 148 242,792 on a six-point scale:12 highly satisfactory, satisfac- Country Procurement Assessment Review 109 76,360 tory, moderately satisfactory, moderately unsat- isfactory, unsatisfactory, and highly unsatisfactory. Poverty Assessment 107 308,489 Country Economic Memorandum 105 332,955 Tables 2.6 and 2.7 present the ratings provided by Country Financial Accountability the in-country stakeholders from the different Assessment 104 89,367 sources and by QAG on the various dimensions. Development Policy Review 35 191,388 Several observations emerge. Integrative Fiduciary Assessment 14 356,030 Other diagnostica 550 165,593 First, for each IEG source of information, the Financial Sector Assessment Program 100 240,946 ratings of most of the dimensions are positively Investment Climate Assessment 80 184,549 and significantly13 correlated with each other. The Advisory reportsa 1,455 175,134 exception is timeliness and relevance, which were Other financial sector study 141 154,804 not correlated consistently across the different Other infrastructure study 131 140,332 sources of information. The absence of a signifi- Other public sector study 109 162,269 cant correlation between these two dimensions Not assigned reports 1,240 115,429 from the responses to the specific ESW and spe- Total number of reports 3,930 160,384 cific TA surveys is noteworthy in light of the com- Source: World Bank administrative data. bination of these two dimensions in the QAG Note: ESW = economic and sector work. ratings (that is, QAG considers them to be similar). a. For a complete list of other diagnostic reports, see appendix A and appendix table C.7. Second, technical quality received high ratings from all sources. More than 95 percent of the in- Although there are reports, FSAPs cost more than the aver- country stakeholders interviewed in the country already 53 report types in age core diagnostic (hence much more reviews gave ratings of 4 and above for the tech- the Bank classification than the average report), and ICAs cost nical quality of Bank ESW and TA. Slightly smaller system, nearly a third of somewhat more (about 15 percent) than shares of in-country stakeholders from the other the ESW was not assigned the average report. three sources gave ratings of 4 and above for to any of those categories. technical quality (table 2.6), although the average Quality of ESW and TA ratings from these other sources are not signifi- This evaluation sought the ratings from cantly different from those obtained in the coun- in-country stakeholders on various di- try reviews. Ratings of 4 and above provided by Core diagnostic reports mensions of ESW and TA through coun- in-country stakeholders are comparable to QAG were among the most try reviews and three electronic surveys ratings of moderately satisfactory and above (table costly products. (one on specific ESW , one on specific TA, 2.7), which are given to 91 percent of Bank ESW and one on general views on Bank ESW and TA for internal quality. Hence, this evaluation and TA; see appendix B for details). The validates the internal quality ratings by QAG. Technical quality dimensions are technical quality, rele- received high ratings vance, timeliness, dissemination, and Third, dissemination received significantly from in-country overall impact. The ratings are on a six- lower ratings than technical quality from the in- stakeholders, validating point scale (where 1 is low and 6 is high, country stakeholders from almost all sources the high QAG ratings on 4 is slightly above average, and 3 slightly (the exception is the respondents to the specific internal quality. below average).11 TA survey, who provided the same ratings on 16 T R E N D S A N D PAT T E R N S O F B A N K E S W A N D TA Table 2.6: Shares of In-Country Stakeholders Giving Ratings of 4 and Above on Different Dimensions of Bank ESW and TA, by Source (percent) Country Specific ESW Specific TA User ESW/TA reviews survey survey survey Relevance 83 99 94 85 Timeliness 60 81 88 75 Technical quality 95 90 88 88 Dissemination 42 79 85 69 Overall impact 66 81 85 78 Source: IEG surveys and country reviews. Note: ESW = economic and sector work; TA = technical assistance. Table 2.7: Shares of Tasks with QAG Ratings of Moderately Satisfactory and Above (percent) ESW and TA ESW TA Strategic relevance and timeliness 98 97 99 Internal quality 91 90 95 Dialogue and dissemination 90 87 98 Likely impact 88 87 93 Source: QAG. Note: ESW = economic and sector work; QAG = Quality Assurance Group; TA = technical assistance. average for the two dimensions). In contrast, sim- ilar shares of ESW and TA received QAG ratings of Figure 2.4: Quality and Average Cost of ESW moderately satisfactory and above for internal qual- ity (91 percent) as for dialogue and dissemination 3.5 250,000 (90 percent). In other words, QAG gave much higher ratings than in-country stakeholders for dis- 3.0 200,000 semination relative to technical quality. Fourth, overall impact received significantly 2.5 150,000 QAG rating US$ lower ratings than technical quality from the in-country stakeholders from almost all sources 2.0 100,000 (the exception again is the respondents to the specific TA survey). In contrast, only a slightly 1.5 50,000 lower share of ESW and TA received QAG ratings of moderately satisfactory and above for likely impact (88 percent) than for internal quality (91 1.0 0 2000 2001 2002 2003 2004 2005 2006 percent). This could in part be because QAG pro- vides ratings on “likely impact,” compared with the Fiscal year ratings provided by in-country stakeholders on Average cost of ESW QAG internal quality “impact.” Source: Quality Assurance Group and Bank administrative data. Note: The QAG internal quality rating scale is from 1 to 4, with 1 being high. ESW = economic and sector Further, QAG internal quality ratings for ESW are work. significantly and positively correlated with the 17 U S I N G K N OW L E D G E TO I M P R OV E D E V E L O P M E N T E F F E C T I V E N E S S Dissemination and cost of the ESW, but the same is not true (the average cost of ESW rose for 16 of 20 sec- overall impact received for TA.14 Also, the positive and significant tors).15 At the same time, there was also an in- significantly lower ratings association only holds for the Bank crease in the quality of ESW (as seen in figure 2.4), than technical quality budget component of cost; no associa- consistent with the finding of a positive correla- from in-country tion was found for the trust fund com- tion between cost and quality. In other words, the stakeholders. ponent of cost (see appendix D). ESW reforms of fiscal 1999 were accompanied by an increase in the quality of ESW . An increase in ESW As mentioned on page 11, the increase quality accompanied in annual outlays on ESW during fiscal the ESW reforms of 2000–06 was accompanied by a moder- fiscal 1999. ate increase in the average cost of ESW 18 Chapter 3 Evaluation Highlights • For a sample of loans examined, two- thirds were preceded by ESW that could have informed the loans, with development policy loans more likely to have been informed by ESW than investment loans. • The Sub-Saharan Africa Region had the lowest share of investment loans preceded by relevant ESW, and the East Asia and Pacific Region had the highest. • Loans preceded by relevant ESW had better loan design. • The effects of ESW on informing Bank strategy—not a stated objec- tive—were larger than its effects on meeting stated Bank objectives. • Core diagnostics, in particular CEMs and PERs, informed development policy loans and country assistance strategies. In Jordan, a World Bank report on urban management led to changes in the intergovernmental transfer formula between the central government and municipalities. Photo by Francis Dobbs, courtesy of the World Bank Photo Library. Effects of ESW and TA on World Bank Activities T his chapter presents findings on the extent to which Bank ESW informed lending—the only objective aimed directly at Bank activities. It also pre- sents findings of TA informing Bank lending, although this was not a stated objective of TA. Finally, the chapter presents findings on another effect of ESW and TA on Bank activities that emerged from the evaluation, inform- ing Bank strategy, although this was not a stated objective of Bank ESW or TA. The findings in this chapter draw on the follow- a detailed review of the portfolio of ESW Of ESW delivered ing sources of information: delivered up to three years prior to the during fiscal 2002–06, 41 approval date of the loans to identify percent of those that had • Structured interviews with TTLs of ESW , loans, those ESW that could actually have in- objectives specified were and strategies from the 12 country reviews formed the loans (see appendix B for aimed at informing Bank on their perceptions of the effects (with quan- the methodology on loan selection and lending. titative ratings) of ESW on loans and strategies ESW identification).2 • An electronic survey of loan TTLs about their perceptions of the effects (with quanti- For the selected sample of 119 loans, about two- tative ratings) of specific ESW on their loans thirds were preceded by ESW that could have in- • An electronic survey of ESW TTLs regarding formed the loans, and one-third was not. The their perceptions of the effects (with quantita- selected sample of loans is representative of the tive ratings) of specific ESW on lending Regional and sectoral distributions of all Bank • Econometric analysis to determine the as- loans, as well as the distribution between invest- sociation between ESW and loan design with ment loans and development policy loans (DPLs) ratings on loan design coming from QAG. approved during fiscal 2003–05 (see appendix B). Informing Bank Lending DPLs were far more likely to have been About two-thirds of the ESW has long been an important part of the informed by ESW than investment loans. selected sample of 119 Bank’s work and has underpinned the Bank’s Of the sample of 119 loans, 91 percent of loans was preceded by country dialogue and operations. Of the ESW de- DPLs (20 of 22 loans) were preceded by relevant ESW, including livered during fiscal 2002–06 that had objectives ESW that could have informed the loans, nearly all the DPLs. specified, 41 percent had informing lending as whereas only 61 percent of investment one objective. loans (59 of 97 loans) were preceded by such ESW . Although informing lending is one of the main objectives of Bank ESW , not all Bank loans were On a Regional basis, Sub-Saharan Africa had the informed by ESW . For a selected sample of loans lowest share of investment loans that were pre- approved during fiscal 2003–05,1 IEG undertook ceded by relevant ESW (that is, those ESW that 21 U S I N G K N OW L E D G E TO I M P R OV E D E V E L O P M E N T E F F E C T I V E N E S S Sub-Saharan Africa could have informed the loans), and East The importance of ESW in informing lending is had the lowest share Asia and Pacific had the highest. (The evident in the responses of loan TTLs to the sur- of investment loans shares, in ascending order, were 36 per- vey question of how their loans would have been preceded by relevant cent for Sub-Saharan Africa, 50 percent affected if the Bank had not produced the specific ESW and East Asia and for Latin America and the Caribbean, 56 ESW (box 3.1). Loan TTLs interviewed for the Pacific the highest. percent for the Middle East and North country reviews also provided evidence of the Africa, 59 percent for South Asia, 70 per- importance of ESW for their loans (box 3.1). cent for Europe and Central Asia, and 83 percent for East Asia and Pacific). What kinds of ESW informed lending? A range of ESW informed lending, with country- Did ESW inform Bank lending? TTL views specific ESW having more effects than regional ESW informed lending to an above average extent. or global ESW and core diagnostics having more From the 12 country reviews conducted for this effects on DPLs than on investment loans. The for- evaluation, loan TTLs gave, on the whole, an above mer finding from the country reviews is not sur- average rating of 4.33 to ESW for inform- prising because country-specific ESW is clearly Between 74 and 87 ing their loans. This finding was validated geared more toward lending (which is done at the percent of loan TTLs gave by the overall rating of 4.64 given by loan country level) than regional or global ESW . What above average ratings TTLs in a survey of their views on specific is perhaps surprising is that regional and global to ESW for informing ESW informing their loans.5 Specifically, ESW did inform lending in some instances. their loans. 74 percent of loan TTLs in the country re- views and 87 percent of loan TTLs in the Regional and global ESW received above average survey gave ratings of 4 and higher for the extent ratings for informing lending, although their ef- to which ESW had informed their work (figure 3.1). fects were generally rated lower than country-spe- cific ESW. When asked in the survey to provide a ESW not intended ESW that was not intended to inform rating on the extent to which the different types to inform lending lending had similar effects on lending as of ESW informed their lending, loan TTLs on av- had effects similar to ESW that was intended to inform lend- erage gave a rating of 4.8 to country-specific ESW , ESW that did. ing. The average rating given by loan 3.7 to regional ESW , and 3.8 to global ESW . 6 The TTLs for ESW that had informing lending country reviews provided some specific examples as an objective was not significantly different from of regional ESW informing lending. For instance, the average rating given to ESW that did not have in Guyana, the HIV/AIDS in the Caribbean ESW that objective. (World Bank 2001) informed the Guyana HIV/ AIDS Prevention and Control Project. ESW TTLs had a realistic perception of the effects of their tasks on informing lending. The ratings Among the core diagnostic ESW , PERs and CEMs given by ESW TTLs from the country reviews and were particularly useful for informing DPLs. Core from the surveys of specific tasks were not signif- diagnostic reports were cited as having informed icantly different from the ratings given by the loan DPLs in virtually all 12 countries reviewed. PERs, TTLs from the same two sources. In the country followed by CEMs, were the most commonly cited reviews, 77 percent of ESW TTLs and 74 percent ESW to have informed DPLs. The DPLs informed of loan TTLs gave ratings of 4 and higher on the by PERs included Poverty Reduction Structural extent to which ESW informed lending. Loan TTLs Credits and Public Expenditure Reform Loans. were even more positive than ESW TTLs in the sur- Other core diagnostic reports that had also in- veys: 87 percent of loan TTLs, compared with 74 formed DPLs were the CFAA, the Development percent of ESW TTLs, gave ratings of 4 and higher Policy Review, and the CPAR. Core diagnostics on the extent to which the same ESW they were also informed investment loans, although not as being surveyed on informed lending (figure 3.1). commonly as they did DPLs. 22 E F F E C T S O F E S W A N D TA O N W O R L D B A N K A C T I V I T I E S Other mandated reports (either because of IDA Figure 3.1: Ratings on the Extent to Which ESW or other mandates) also informed DPLs more Informed Lending, by Shares of Respondents than they did investment loans. These included the ICA and the Diagnostic Trade Integration 100 Study. Did ESW improve the quality of loans? 80 1 Loans that were preceded by relevant ESW had 2 better design than those that were not. Specifically, 60 Percent 3 for the same selected sample of 119 loans men- 4 tioned above, the evaluation finds that those pre- 40 ceded by relevant ESW had better ratings for loan 5 design,7 on average, than those that were not 20 6 (table 3.1).8 This was the case overall and for every Region except the Middle East and North 0 Africa (where there was no difference) and East ESW TTL Loan TTL ESW TTL Loan TTL Asia and Pacific (where it was the reverse). Country reviews TTL surveys Response source More detailed analysis found that loans preceded by relevant ESW were associated with better qual- Source: IEG surveys and country reviews. ity of certain aspects of loan design. These find- Note: Rating scale is from 1 to 6, with 1 denoting “no extent” and 6 denoting “great extent.” ESW = eco- nomic and sector work; TTL = task team leader. ings were based on econometric analysis (see appendix D for details) using QAG quality at entry (QAE) ratings for loan design. • There is an association (significant at Core diagnostic reports, This evaluation finds that loans preceded by rele- the 10 percent level) between the num- particularly PERs and vant ESW were likely to have better QAE ratings for ber of ESW and one particular aspect CEMs, were especially strategic relevance and approach, policy and of the strategic relevance and approach useful in informing DPLs. institutional aspects, and the financial manage- dimension, that is, the adequacy of ment capacity aspect of the risk assessment di- country and sector knowledge under- mension. They were also likely to have better QAE pinning the project. Specifically, loans Loans preceded by overall assessment ratings but were not associ- preceded by a greater number of rel- relevant ESW were ated with better ratings for technical, financial, evant ESW were more likely to have associated with better and economic aspects, or for fiduciary aspects.9 better QAE ratings for this aspect. The quality in some aspects Specifically, the regression results are as follows. average cost of the relevant ESW did of design. not matter. • Loans preceded by at least one relevant ESW • Loans preceded by at least one relevant ESW were more likely to have better overall QAE were more likely to have better QAE ratings assessment ratings.10 The actual number and for the policy and institutional aspects of the average cost of relevant ESW (that is, any num- loan. ber greater than one) did not matter for the • Loans that had actually cited CFAA (or similar QAE rating. financial management ESW) in the project ap- • Loans preceded by at least one relevant ESW praisal documents were more likely to have bet- were more likely to have better QAE ratings ter QAE ratings for the financial management for the strategic relevance and approach di- capacity aspect of the risk assessment dimen- mension. The actual number and average cost sion. However, the association was not present of relevant ESW had no effect. if CFAAs existed but were not mentioned in the 23 U S I N G K N OW L E D G E TO I M P R OV E D E V E L O P M E N T E F F E C T I V E N E S S The evaluation period was not long enough to as- Box 3.1: Loan TTLs’ Views on the Effects of ESW on certain the effect of ESW on loan outcomes. Their Loans Lending: A channel through which ESW have From the loan TTL survey, of the 65 respondents (of 91 loan TTLs surveyed, effects in client countries representing a 71 percent response rate), the majority (55 of 65) highlighted Informing Bank lending is an important chan- the usefulness of ESW to their loans; 9 percent (6 of 65) indicated that it nel—though not the only one—through which would have made no difference. Among the 55 who indicated that the spe- Bank ESW informed country policies and strength- cific ESW was useful, 4 indicated that the loan would not have happened ened institutions. In some of the countries re- without the ESW. Others indicated, variously, that more analysis (or ESW) viewed, such as Bangladesh, ESW informed policy would have been needed; more time would have been needed to prepare mainly through lending.11 Similarly in Lesotho, the loan, the quality of the loan would have been compromised, and some ESW needed to be followed by lending to inform of the reforms supported by the loan would not have taken place without policy. For example, much of the effects of the the ESW. Lesotho Private Sector Development Strategy ESW From the country reviews, a loan TTL for Guyana said that if there had were through a follow-up loan (Private Sector been no ESW, there would have been no loans in Guyana. In Peru, loan Competitiveness and Economic Diversifica- TTLs of two different loans indicated that without ESW, the respective loans tion Project), whereas the absence of follow-up to either would not have happened at all or would have taken longer to pre- the Lesotho Public Expenditure Management and pare and would not have been delivered in a timely manner at the transi- Financial Accountability Review (World Bank tion government’s request. In postconflict Democratic Republic of Congo, 2007b) limited its effects. In Guyana, ESW in- ESW was conducted mainly to support Bank operations in the posten- formed government policy mostly through lend- gagement period. The country status reports on education and on health ing, not only by the Bank but also by other and the ESW on public sector governance were all carried out to facili- institutions, such as the Inter-American Develop- tate loan preparation. ment Bank and the IMF (see chapter 4 for details). Source: IEG surveys and country reviews. Other findings Lending informed ESW in some countries, rather project appraisal document. This clearly implies than the reverse. In Lesotho and Mauritius, for ex- that CFAAs need to be actually used to have an ef- ample, the analytical work done for loan prepara- fect on project quality; it also means that not all tion was packaged afterwards as ESW . In Lesotho, CFAAs that had been produced had been used to a study on the education sector was informed by inform lending. a loan (the Second Education Sector Development Table 3.1: Average Quality at Entry (Overall Assessment) Ratings for a Selected Sample of Loans, by Region, Fiscal 2003–05 Region Preceded by relevant ESW Not preceded by relevant ESW Sub-Saharan Africa 2.0 2.3 East Asia and Pacific 2.1 1.5 Europe and Central Asia 1.7 2.3 Latin America and the Caribbean 1.7 2.1 Middle East and North Africa 2.0 2.0 South Asia 1.9 2.0 Average 1.9 2.2 Source: IEG, based on quality at entry (overall assessment) ratings from QAG. Note: The rating scale is from 1 to 4, with 1 being the highest. In fiscal 2004 the QAE rating scale was switched from a four-point scale to a six-point scale (with 1 being the highest on both scales). For the purposes of this analysis, the six-point scale was transformed into a four-point scale. ESW = economic and sector work; QAG = Quality Assurance Group; QAE = quality at entry. 24 E F F E C T S O F E S W A N D TA O N W O R L D B A N K A C T I V I T I E S Project). Three ESW products in Mauritius were the Figure 3.2: Ratings on the Extent to Which ESW result of repackaging done through project prepa- Informed Strategy and Lending and Met Other ESW ration funded by grants (Institutional Develop- and TA Objectives, by Shares of Respondents ment Fund and the Japan Policy and Human Resources Development Fund): pension mod- 100 ernization (World Bank 2004a), fiscal sustainability (World Bank 2004b), and a study on the transport 80 sector. 1 2 ESW was found to inform lending not only in the 60 Percent 3 intended country but also in other countries. For example, the Pakistan and the Indonesia CPARs, 4 40 together with the Bangladesh CPAR, informed the 5 Public Procurement Reform Credit in Bangladesh. 6 20 TA also informed lending, although that was not an intended objective. Among the countries reviewed, 0 the Aid for Trade TA in Mauritius informed the Strategy Lending Other ESW and TA objectives 2007 Trade and Competitiveness DPL that followed, which was the first DPL for Mauritius in five years. Source: IEG country reviews. Note: Rating scale is from 1 to 6, with 1 denoting “no extent” and 6 denoting “great extent.” ESW = eco- Informing Bank Strategy nomic and sector work; TA = technical assistance. The effect of ESW on informing Bank strategy— which is not a stated objective of ESW—was larger than ESW’s effects on meeting any of the Bank ob- jectives that are stated. This finding is based on was ESW not used to inform CASs. The ESW had larger effects ratings provided by various ESW users (both in- CASs in the 10 remaining countries were on informing Bank staff side and outside the Bank) in the context of coun- informed by ESW , and in 5 of these 10 than on meeting the try reviews. countries ESW had a particularly impor- stated Bank objectives. tant role. Box 3.2 provides some details Specifically, TTLs of Bank Country Assistance on the role of ESW in the Bank’s assis- Strategies (CASs) gave an average rating of 5.4 to tance strategies in all these countries. ESW for informing their work. This rating is higher than the average rating of 4.3 given by loan TTLs A range of ESW informed CASs, with core diag- to ESW for informing lending. It is also higher than nostics cited most frequently. For the 10 countries the average rating of 3.9 that in-country stake- for which the CASs had been informed by ESW , the holders gave to ESW and TA for the various ob- ESW most mentioned as having informed those jectives achieved in their countries (see chapter strategies was the CEM (in six countries, seven if De- 4). Basically, 100 percent of the strategy TTLs velopment Policy Reviews are included), the PER gave ratings of 4 and higher to ESW for inform- (in six countries), education sector reports (four ing their work, compared with 74 percent of loan countries), and reports on the rural sector (four TTLs and 64 percent of in-country stakeholders countries). In other words, these CASs had been for informing their work (figure 3.2). informed by at least one core diagnostic, and in sev- eral instances by more than one core diagnostic. ESW , and in some instances TA, was used to inform CASs. Of the 12 countries reviewed, only in the Other ESW and even TA also informed CASs, if Democratic Republic of Congo, a postconflict not as prominently as core diagnostics. Corporate country, and in Malaysia, which is not borrowing ESW informed the CAS in two countries—the from the Bank and for which there was no CAS, Diagnostic Trade Integration Study (Integrated 25 U S I N G K N OW L E D G E TO I M P R OV E D E V E L O P M E N T E F F E C T I V E N E S S Box 3.2: The Role of ESW in Informing Bank Country Strategies For the Democratic Republic of Congo, instead of a CAS TA was particularly important for providing quantifiable there was a Transitional Support Strategy, which was results indicators for the strategy (the impact of HIV/AIDS informed by analytical work done in other parts of the on household poverty). In Peru, ESW was the basis of the Banka rather than by ESW. No ESW had been produced Bank’s country strategy, with Policy Notes being the prior to the Transitional Support Strategy to have in- main input for the 2006 CAS. In Bangladesh, the CAS team formed the latter. leaders cited many ESW products as inputs for their In 5 of the 12 countries reviewed, CAS TTLs indi- strategy documents. In Guyana, the CAS would have cated that ESW played a particularly important role in been very different if the assessment on financial ac- informing their work. In Mauritius, the CEM provided countability and the HIV/AIDS in the Caribbean: Issues the country knowledge base to underpin country dialogue and Options report (World Bank 2001) had not been in the earlier absence of a country office. In Lesotho, a done. Source: IEG country reviews. a. In particular, analytical work done in Development Economics and Chief Economist, “Breaking the Conflict Trap: Civil War and Development Policy” (World Bank 2003), and in the Sub-Saharan Africa Region, “Post-Conflict Recovery in Africa: An Agenda for the Africa Region” (World Bank 2002). ESW (and sometimes TA) Framework 2003) for Lesotho and the as- egy was not an objective (explicit or implicit) of was generally used to sessment of the financial sector for Peru. TA. Specifically, the Aid for Trade TA had an im- inform country strategies. Regional ESW were mentioned in two portant role in informing the CAS for Mauritius, countries as having informed CASs: the and the Private Participation in Infrastructure TA HIV/AIDS in the Caribbean ESW (World informed the CAS for Jordan. Bank 2001) in Guyana, and the Lessons from the North America Free Trade Agree- Finally, effects of ESW on strategy were facilitated ment for Latin America and the Caribbean by either having the ESW team leaders be on the The most frequently cited Regional ESW (World Bank 2005c) in Peru. team of the strategy or lead the strategy itself, but influences on country these factors were not always essential for ESW to strategies were core TA also informed CASs in two countries, play a pivotal role. diagnostic reports. although informing Bank country strat- 26 Chapter 4 Evaluation Highlights • ESW and TA delivered to clients dur- ing fiscal 2000–06 generally met their stated objectives of influencing pol- icy, building capacity, stimulating public debate, facilitating knowledge exchange, and influencing the de- velopment community. • The effects of ESW and TA ranged from substantially above average in some countries to substantially below average in others. • TTLs generally had realistic percep- tions of the extent to which their tasks had effects. • Differences in government capacity account for some of the differences in effects across countries. • ESW and TA in a range of sectors and of various types had effects, in- cluding several corporate or IDA- mandated or core diagnostic tasks. In Serbia, World Bank reports on the financial sector were associated with major reforms in the banking sector. Photo by Zoran Mrdja, courtesy of the World Bank Photo Library. Effects of ESW and TA on Client Countries lthough ESW has its genesis in informing Bank country dialogue and op- A erations, it also has direct effects on the Bank’s client countries. Indeed, in addition to the objective of informing lending, the Bank has three ob- jectives for ESW that are aimed directly toward its client countries: (1) inform- ing government policy, (2) building client analytical ability, and (3) informing or stimulating public debate. Another objective of ESW is directed toward its client countries through the effect of ESW on the development community. Unlike ESW , all the Bank objectives for TA are This chapter’s findings present evidence (in- aimed directly toward its client countries. These cluding ratings) gathered from the country re- are assisting in client’s policy/program imple- views. Unless otherwise specified, only those mentation, developing/strengthening institutions, findings validated by the different sources of in- and facilitating knowledge exchange. formation are reported. This chapter presents findings on stakeholders’ The Extent to Which ESW and TA Met perceptions of the extent to which these four Their Objectives ESW and three TA objectives were met in the The discussion in this section is based on the Bank’s client countries. The evidence and ratings stakeholder ratings (table 4.1).1 In general, the rat- come from six sources of information (see ap- ings provided by in-country stakeholders in the pendix B for details): country reviews are lower than those provided in the electronic surveys, although not all the dif- • Structured interviews of stakeholders from the ferences are statistically significant.2 Specifically, 12 country reviews the various sources of information indicated that • An electronic survey of in-country stakehold- between 65 percent (in the country reviews) and ers on specific ESW (the “specific ESW user 79 percent (in the specific TA user survey) of survey”) stakeholders gave ratings of 4 and higher to the • An electronic survey of in-country stakehold- extent to which all ESW and TA objectives, com- ers on specific TA (the “specific TA user survey”) bined, were met (figure 4.1). The findings are as • An electronic survey of in-country stakehold- follows. ers on their general views on Bank ESW and TA (the “ESW/TA user survey”) On the whole, ESW and TA delivered to • An electronic survey of ESW TTLs on specific clients during fiscal 2000–06 generally ESW and TA delivered ESW tasks (the “ESW TTL survey”) met their stated objectives. All objec- during fiscal 2000–06 • An electronic survey of TA TTLs on specific TA tives, on average, received average (3.5)3 generally met their tasks (“TA TTL survey”). and above ratings from all the sources of objectives. 29 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Table 4.1: Average Ratings by In-Country Stakeholders on the Extent to Which ESW and TA Met Various Objectives From specific From From ESW specific ESW/TA country user TA user user Objective reviews survey survey survey Influencing policy Informing policy (ESW) 3.9 4.3 — 4.0 Implementing policy (TA) 4.6 — 4.4 Building capacity/institutions Building capacity (ESW) 3.5 3.8 — 4.0 Strengthening institutions (TA) 4.5 — 4.1 Informing/stimulating public debate (ESW) 3.4 4.5 — 4.0 Influencing development community (ESW) 3.7 4.3 — 4.3 Facilitating knowledge exchange (TA) 4.3 — 4.9 — Source: IEG surveys and country reviews. Note: The ratings are for all ESW and TA (that is, country, regional, and global). The ratings are on a six-point scale, where 1 denotes “no extent,” 6 denotes “great extent,” 4 is slightly above average, and 3 is slightly below average. The same rating scale is used for all the surveys and interviews undertaken for this evalua- tion. ESW = economic and sector work; TA = technical assistance. information, with the exception of stimulating (4.9) for knowledge exchange (from the specific public debate from country reviews. The ratings TA user survey).4 for the other ESW and TA objectives ranged from just average (3.5) for capacity building (from the Although not all ESW and TA were aimed at meet- country reviews) to significantly above average ing all the objectives, many of them met objectives not specified at the outset. For all but one ob- Figure 4.1: Ratings on the Extent to Which ESW jective, there was no difference between the rat- and TA Met Various Objectives, by Shares of ings for tasks with a certain objective and those Respondents without that objective. The exception pertained to the ESW objective to inform policy: the ratings 100 received by tasks with this objective were signif- 90 icantly higher than those without this objective. 80 1 2 TTLs generally had realistic perceptions of the ex- 70 tent to which their tasks had effects. Responses 60 3 Percent to the surveys on specific tasks indicate no dif- 50 4 ference between the ratings given by TTLs and 40 5 those given by in-country stakeholders for the 30 same tasks for each of the objectives (table 4.2). 6 20 There are some differences in ratings between 10 these two groups in the country reviews (where 0 ratings being compared are for the entire portfolio Country reviews Specific ESW Specific TA ESW/TA user of ESW and TA, rather than for specific tasks). survey survey survey Response source Specifically, for informing policy and stimulating Source: IEG surveys and country reviews. Note: Rating scale is from 1 to 6, with 1 denoting “no extent” and 6 denoting “great extent.” public debate, TTLs gave significantly higher ESW = economic and sector work; TA = technical assistance. ratings than in-country stakeholders; for building capacity and informing the development com- 30 E F F E C T S O F E S W A N D TA O N C L I E N T C O U N T R I E S Table 4.2: Average Ratings by In-Country Stakeholders and TTLs on the Extent to Which ESW and TA Met Various Objectives From specific ESW and From country reviews specific TA surveys In-country In-country Objectives stakeholders TTLs stakeholders TTLs ESW Informing policy 3.9 4.4 4.4 4.4 Building capacity 3.5 3.3 3.8 3.9 Informing/stimulating public debate 3.4 3.8 4.5 4.4 Informing development community 3.7 3.7 — — TA Implementing policy/program 4.6 4.7 4.3 4.2 Strengthening institutions 4.5 3.9 4.2 3.8 Knowledge exchange 4.3 4.9 4.9 5.0 Source: IEG surveys and country reviews. Note: Bold figures denote ratings that are statistically significantly different at the 5 percent level between the in-country stakeholders and the TTLs. The rating scale is from 1 to 6, with 1 denoting “no extent” and 6 denoting “great extent.” ESW = economic and sector work; TA = technical assistance; TTL = task team leader. munity, there was no statistically significant dif- so on). These are addressed in chapter 5. TTLs generally had ference in ratings between the two groups. For TA, Other factors are specific to the country, realistic perceptions of the TTLs gave, on average, a significantly higher rat- not all of which can be addressed by the extent to which their tasks ing to knowledge exchange, but they gave a sig- Bank. had effects. nificantly lower one to strengthening institutions than in-country stakeholders did. They gave vir- For the 12 countries reviewed, the average ratings tually the same rating as in-country stakeholders provided by in-country stakeholders on all ESW to the implementing policy/program objective. and TA objectives ranged from 2.7 for the Dem- ocratic Republic of Congo to 4.7 in Ro- TA TTLs gave significantly higher ratings on the mania. Most stakeholders in Romania TA TTLs rated the extent to which TA objectives were met than ESW gave ratings of 4 and higher, whereas achievement of the TTLs gave on the extent to which ESW objectives most stakeholders in the Democratic Re- objectives of their tasks were met. This was the case in the country reviews public of Congo gave ratings of 3 and higher than ESW TTLs and was confirmed by survey responses of TTLs. lower (with 40 percent of the stake- rated theirs. This could reflect the nature of TA tasks, whereby holders giving a rating of 2 and some TTLs work more closely with clients and see more giving a rating of 1, which denotes that the ESW immediate results of their tasks. and TA objectives were not met at all; see figure 4.2). However, even in the Democratic Republic Some Factors Affecting the Extent to of Congo, some ESW and TA had great effects, with Which ESW and TA Met Their Objectives high ratings of 5 given by some in-country stake- The country reviews indicate that ESW and TA holders to some tasks. In these instances, specific met their objectives to different extents in differ- efforts by Bank staff, in particular on the collab- ent countries. Some of the factors accounting for oration or partnership front, made the difference these differences could be addressed by the Bank, (see discussion in chapter 5 for details). including those related to Bank processes and corporate incentives (technical quality of the prod- The difference in ratings between the two coun- ucts, the extent of collaboration or partnership, is- tries could be explained, at least in part, by the sues related to dissemination and follow-up, and different country conditions. On one hand, the 31 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Figure 4.2: Distribution of Ratings Given by In-Country Stakeholders in the Democratic Republic of Congo and Romania Democratic Republic of Congo Romania 50 50 45 45 45 40 40 40 35 35 29 30 30 27 Percent Percent 25 25 19 20 20 15 13 15 12 10 8 10 7 5 5 0 0 1 2 3 4 5 6 1 2 3 4 5 6 Ratings Ratings Source: IEG country reviews. Note: Rating scale is from 1 to 6, with 1 denoting “no extent” and 6 denoting “great extent.” Differences in ratings postconflict status and low government building effects of ESW and TA in these coun- between countries are in capacity in the Democratic Republic of tries. Finally, factors such as government recep- part explained by country Congo could have contributed to the low tivity arising from political and other constraints conditions such as effects of ESW and TA in that country. affected the extent to which ESW and TA met government capacity. Policy makers in the Democratic Repub- their objectives. lic of Congo did not have time to read and absorb much of the ESW produced by the Bank. Influencing Government Policy On the other hand, higher government capacity The overall ratings given by in-country stake- in Romania, coupled with the appreciation of holders on the extent to which ESW and TA in- Bank involvement after its opening up—and in formed and helped them implement policies particular, Bank assistance with respect to the from the different sources were all above average European Union (EU) accession process—could (table 4.1). At least 70 percent of in-country stake- have all contributed to the higher ratings for holders and TTLs gave ratings of 4 and higher Romania. (around 80 percent in the case of respondents to the specific ESW and specific TA surveys) on In-country stakeholders Government capacity also affected the ef- the extent to which the objective of informing/ rated ESW and TA above fects of ESW and TA in the other coun- implementing policy was met (figure 4.3).5 average for informing tries reviewed. In Jordan and Serbia, Bank and assisting in the staff pointed to the high turnover of min- For the 12 countries reviewed, the ratings for ESW implementation isters as having worked against the ef- and TA’s influence on policy ranged from a low of policies. fectiveness of ESW and TA. In Serbia, the (and below average rating) of 3.1 for the Demo- rapid turnover of ministers undermined cratic Republic of Congo and Mali to highs of 4.9 the effects of ESW and TA (which were and 4.7 for Romania and Serbia, respectively (fig- nevertheless quite significant, although they could ure 4.4). This dispersion of ratings is also reflected have been greater). The high turnover of gov- in the fact that around 30 percent of respondents ernment officials also undermined the capacity- from the different sources (20 percent in the case 32 E F F E C T S O F E S W A N D TA O N C L I E N T C O U N T R I E S Figure 4.3: Ratings on the Extent to Which ESW and TA Informed Policy, by Shares of Respondents from Different Sources 100 90 80 1 70 2 60 3 Percent 50 4 40 5 30 6 20 10 0 Country Specific ESW Specific TA ESW/TA user ESW TTL TA TTL reviews user survey user survey survey survey survey Response source Source: IEG surveys and country reviews. Note: Responses presented for country reviews are those of in-country stakeholders and exclude those of TTLs. Rating scale is from 1 to 6, with 1 denoting “no extent” and 6 denoting “great extent.” ESW = economic and sector work; TA = technical assistance; TTL = task team leader. of the specific ESW and specific TA surveys) gave indicated that it was used for ESW and TA affected ratings of 3 and lower (figure 4.3). government policy, and nearly policy either directly or by one-third indicated that it was informing government More ESW specified informing policy as an ob- used in government legislation. strategy and legislation. jective than other objectives. Assisting in the im- plementation of policy/program is the second ESW informing government policies: most important TA objective with respect to the Evidence from country reviews number of TA specifying this as an objective (see In-country stakeholders cited ESW in a wide range chapter 1). Correspondingly, the country reviews of sectors as having directly informed policies in provided many examples of ESW and TA affecting the 12 country reviews. In line with the fact that government policy. ESW in economic policy, financial sector, public sector governance, and PSD were among the ESW and TA affected government policy either di- most numerous undertaken by the Bank, ESW in rectly or by informing government strategy and these sectors was among the most mentioned in legislation. More than half of the respondents to the country reviews for informing government pol- the survey on specific ESW indicated that the icy, including through informing government ESW on which they were being surveyed was strategy or legislation. used in government strategy and government policy, and 18 percent indicated it was used in gov- Of the ESW undertaken in these sectors, the FSAP ernment legislation. For the survey on specific TA, and the PER had the highest incidences of being nearly two-thirds of the respondents indicated cited as having informed policies and legislation. that the TA on which they were being surveyed The FSAP was mentioned in six of the seven coun- was used for government strategy, more than half tries for which an FSAP had been undertaken. 33 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Figure 4.4: Ratings on the Extent to Which ESW and TA Informed Policy, from In-Country Stakeholders in Country Reviews 6 5 4 Rating 3 2 1 p. . ali h na ia o ru an ius nia ia m Re Dem es th ys rb na Pe M ya rd rit ma so of lad Se ala t Jo Gu au Vie Le o, Ro ng M M ng Ba Co Country Source: IEG country reviews. Note: Rating scale is from 1 to 6, with 1 denoting “no extent” and 6 denoting “great extent.” ESW = economic and sector work; TA = technical assistance. For the financial and The PER (and other ESW on public ex- Aside from the FSAP , PER, and ICA, other core di- public governance sectors, penditure management ) was cited in 6 agnostics—such as the CFAA and the CPAR—and the FSAP and the PER of the 10 countries for which such ESW corporate ESW , such as the various Reports on the were most often cited as had been undertaken. Box 4.1 highlights Observance of Codes and Standards (ROSCs) having informed some of the specific effects of the FSAP (which include Anti-Money Laundering and Com- government policy. and the PER in the countries reviewed. bating the Financing of Terrorism [AML/CFT], the Corporate Governance ROSC, and the Account- For ESW in the PSD sector, the ICA had ing and Auditing ROSC), were also cited in a num- the highest incidence of being mentioned, hav- ber of countries as having informed legislation. ing been cited in four of the seven countries where it had been undertaken as having informed At the same time, many noncore diagnostic and policy, strategy, and legislation. In Malaysia, noncorporate ESW also had effects, including changes in the labor law and in the reg- those ESW that do not belong to the top ESW sec- In PSD, the ICA was most istration of property were attributed to tors Bank-wide. For instance, ESW in social pro- often named as having the ICA; it has also led the government tection had significant effects in Jordan and Peru, informed policies. to establish a committee to ensure that although this was not a sector that was selected deregulation and improvements in pub- for review for those countries. lic service delivery were carried out smoothly. In Serbia, the ICA was credited with the TA assisting in policy/program implementation: country’s regaining momentum in the privatiza- Evidence from country reviews tion process and in attracting foreign investment, TA in a wide range of sectors was rated highly in among other changes. In Guyana, the ICA was the country reviews for assisting in the imple- cited as having informed the country’s National mentation of government policies and programs, Competitiveness Strategy. including the financial sector, public sector gov- 34 E F F E C T S O F E S W A N D TA O N C L I E N T C O U N T R I E S Box 4.1: FSAP and PER Informing Government Policy and Legislation: Some Examples This box highlights examples from the country reviews of the FSAP the Fiscal Management Accountability Act and led to more timely and PER informing government policy, government strategy, and budgets and greater transparency through the removal of discre- legislation. More examples of other ESW informing government pol- tion. In Jordan, it resulted in the consolidation of expenditures by icy, strategy, and legislation are presented in appendix E. sectors rather than by ministries and the development of a Medium- Term Expenditure Framework to replace annual budgeting. FSAP In Peru, the PER focused attention on the quality of expendi- The extent of the effects of the FSAP ranged widely across the six tures, especially in the social sectors, and moved the government countries where it was cited as having effects. It had the largest toward results budgeting. It also led to the Law of Fiscal Decen- effect in Serbia, where it was associated with changes to and im- tralization. In Romania, the Public Expenditure and Institutional Re- plementation of the Banking Law, major improvements in banking view and the Public Expenditure Management ESW became a supervisions, bank privatization, and bank closures. basis for the macroeconomic stabilization policies of the Natase The FSAP also had important effects in Peru, where all its rec- government and contributed to the modification of the financial ommendations were implemented, including improvements in the structure of the public sector. payments and exchange systems and issuance of legislation on In Serbia, the PER update led to new/revised laws on wages the legal protection for supervisors. Among other changes, the FSAP in state administration and on civil servants and amendments to helped strengthen supervision in Mauritius. It led to the adoption the social security law, as well as improvements in public sector of a supervisory ladder in Jordan, strengthened the oversight of governance. In Vietnam, the PER helped shape legislation to in- the insurance industry in Guyana, and defined the agenda for fi- crease the transparency in the state budget law and led to the adop- nancial sector reforms in Bangladesh. tion of a Medium-Term Expenditure Framework and to the PER unification of the public accounts system. The extent of the effects of the PER was similar across the six coun- tries where it was cited as having effects. In Guyana, it informed Source: IEG country reviews. ernance, PSD, rural sector, education, and infra- were still average or above (table 4.1). TA was rated highly structure. The first three were among the top This is also reflected in figure 4.5, which for assisting in the sectors for TA Bank-wide, although infrastruc- shows that the shares of in-country implementation of ture (and transport) ranked next to the bottom stakeholders giving ratings of 4 and government policies in the number (and cost) of TA undertaken dur- higher ranged between 60 and 70 per- and programs. ing the evaluation period (see appendix C). TA also cent (compared with between 70 and helped inform government legislation, although 80 percent for influencing policy). not quite as much as ESW , with far fewer examples cited in the country reviews. Box 4.2 presents TA received higher ratings for capacity building some examples of TA assisting in the implemen- than ESW (although ratings for this TA objective tation of government policies and programs. were lower than for other TA objectives) from both in-country stakeholders and TTLs, as shown in figure 4.5. The figure shows that around 70 per- Building Capacity and Strengthening cent of TTLs and in-country stake- Institutions holders gave ratings of 4 and higher ESW received among The objective of building capacity by ESW received when surveyed on specific TA tasks, the lowest ratings for among the lowest ratings from in-country stake- compared with between 50 and 60 per- building capacity holders compared with the other ESW objectives, cent of those surveyed on specific ESW compared with other although the ratings from all the different sources tasks. objectives. 35 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Box 4.2: Selected Examples of TA Assisting in Policy/ Program Implementation This box highlights some examples of TA assisting in policy/ kets (in conjunction with the European Commission and the IMF). program implementation. More examples of TA in these and other The Challenges of EU Accession in Romanian Agriculture TA was countries having such effects are in appendix E. very helpful for Romania in closing the chapter on agriculture in Romania provides several examples of influential TA. One of the the pre-accession discussions with the EU. most prominent was the Anticorruption Diagnostic TA, which was In Peru, the Public-Private Infrastructure TA was influential— accompanied by the Anticorruption ESW. The TA formed the basis it helped jump start the concessions program, revised tariffs in the for the country’s anticorruption efforts through initiation of a pub- port system, and generally propelled reforms in transport. The Pri- lic debate and educational efforts. vate Participation in Infrastructure TA in Jordan led to the launch Another influential TA in Romania was the Capital Market De- of the public-private partnership in the country, and the Forest Sec- velopment Program TA, which was instrumental in establishing a tor Support TA in Vietnam helped establish a system of forest cer- regulatory regime for the pension, insurance, and mortgage mar- tification for plantations. Source: IEG country reviews. Figure 4.5: Ratings on the Extent to Which ESW and TA Built Capacity/ Strengthened Institutions, by Shares of Respondents from Different Sources 100 90 80 1 70 2 60 3 Percent 50 4 40 5 30 6 20 10 0 Country Specific ESW Specific TA ESW/TA user ESW TTL TA TTL reviews user survey user survey survey survey survey Response source Source: IEG surveys and country reviews. Note: Responses presented for country reviews are those of in-country stakeholders and exclude those of TTLs. Rating scale is from 1 to 6, with 1 denoting “no extent” and 6 denoting “great extent.” ESW = economic and sector work; TA = technical assistance; TTL = task team leader. Not surprisingly, TA was more focused on this As with informing policy, there is a range of effects objective than ESW . More TA aimed at this ob- across the countries reviewed (figure 4.6). Coun- jective than at other TA objectives, whereas this tries that had higher ratings for influencing pol- was the least important objective for ESW (see icy generally also had higher ratings for capacity chapter 1). building. 36 E F F E C T S O F E S W A N D TA O N C L I E N T C O U N T R I E S Figure 4.6: Ratings on the Extent to Which ESW and TA Built Capacity/ Strengthened Institutions, from In-Country Stakeholders in Country Reviews 6 5 4 Rating 3 2 1 h ali p. . o ru ius na an ia ia nia m Re Dem es th ys rb na Pe M ya rd rit ma so of lad Se ala t Jo Gu au Vie Le o, Ro ng M M ng Ba Co Country Source: IEG country reviews. Note: Rating scale is from 1 to 6, with 1 denoting “no extent” and 6 denoting “great extent.” ESW = economic and sector work; TA = technical assistance. The existing level of government capacity in the velopment community, and the author- TA, which was more country itself appeared to matter for the extent to ities in the context of the country re- focused on capacity which ESW and TA built capacity in the country. In views. In Mali, for example, ESW TTLs building than ESW, was three of the countries reviewed (Bangladesh, believed that capacity building was not rated higher than ESW. Guyana, and Lesotho), in-country stakeholders considered an important objective of indicated that government capacity was too low for their tasks and that they were not given the country to have benefited from the potential sufficient time or resources for it. capacity-building effects of ESW and TA. High turnover of government officials in the sector min- In Peru, a representative of the devel- The existing level of istries exacerbated the capacity problem in Peru, opment community perceived that the government capacity which received low ratings for capacity building. Bank’s incentive system was not quite affected the extent to right for meeting this objective, as it which ESW and TA built Aside from the country-specific conditions dis- could not be measured and was not cen- capacity. cussed in the preceding paragraph, the Bank’s in- tral to the process of ESW. Local re- centive system has also not been conducive to searchers were used as data and information building capacity through ESW and TA. This was gatherers but were not true partners in produc- clearly the perception of the TTLs in their re- tion. In the Democratic Republic of Congo, the sponses to the survey question of the incentives authorities saw little emphasis on part- they faced in the Bank for partnership in the pro- nership on the Bank’s side, with the Bank The Bank’s incentive duction of their tasks, the channel through which team and its consultants often working system has also been a the Bank envisioned that these tasks would help in isolation, limiting their interactions constraint on the extent build capacity. This view was corroborated by the with government to data collection and to which ESW and TA built views of some TTLs, representatives of the de- presentation of results. capacity. 37 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Notwithstanding the generally low ratings given Bank products showed the basic instruments to this objective, capacity building has taken place through which the problem (which was widely ac- through ESW and TA. The main channel through cepted in the country to be very serious) could which this happened was the interaction between be approached, including for issues such as trans- Bank staff and consultants and government offi- parency, disclosure, the legal framework, and cials during the preparation of the reports or pro- dedicated agencies. duction of TA. TA building capacity: Evidence ESW building capacity: Evidence from from country reviews country reviews Malaysia and Vietnam stood out among the coun- Although ESW did not generally have strong tries reviewed in terms of examples of TA building effects on building capacity, some corporate or capacity. These countries also received among the IDA-mandated ESW or core diagnostics were men- highest ratings for this objective. Malaysia re- tioned by in-country stakeholders for quested many of the TA tasks provided by the The ICA, FSAP, PER, CEM, building capacity. These were the ICA, the Bank, and for several of these the authorities and Poverty Assessment FSAP , the PER, the CEM, and the Poverty shared the cost, either through financing or in-kind contributed to Assessment. The ICA was mentioned as contribution. Specific examples of TA building ca- building capacity. having built capacity in 3 of 7 countries pacity for these countries are presented in table 4.4. for which it was undertaken, the PER in In some cases, 3 of 10 countries, and the FSAP in 2 of 7 Informing/Stimulating Public Debate ESW and TA worked in countries. Even in countries where rat- This is the second most frequently cited objective combination to ings for capacity building were low, such for ESW (see chapter 1). Unlike the objectives dis- build capacity. as Guyana, some of these products were cussed so far, there was a much greater disparity cited as having built capacity, as can be in views between the different groups of stake- seen in table 4.3. holders, with those interviewed in the country reviews giving generally lower ratings than those Similarly in Bangladesh, which received very low who were surveyed (figure 4.7). Only 50 percent ratings for capacity building, some in-country of the former gave ratings of 4 and higher, com- stakeholders acknowledged that over the longer pared with between 70 to 80 percent of the latter term, there have been gains, notably in the Ministry (the views of ESW TTLs were closer to those of in- of Finance on budgeting, accounting, auditing, country stakeholders who responded to surveys). public expenditure management, medium-term macroframework, and in the Central Bank and As with influencing policy and building capacity, commercial banks. Such capacity building has re- there is a range of effects across the countries re- sulted from a long period of interaction and col- viewed (figure 4.8). Countries that had higher laboration with the Bank and IMF in the context ratings for influencing policy and building ca- of specific tasks (such as PER, CPAR, FSAP , and the pacity generally also had higher ratings for this ob- Financial Accountability Assessment) among other jective. Various country-specific factors were interactions with the Bank. associated with the extent to which this objective was met in the countries reviewed (box 4.3). Spe- In some instances, a combination of cific examples of ESW and TA stimulating public ESW and TA helped build capacity. For debate are presented in table 4.5. Countries that had higher instance, ESW and TA in the financial ratings for influencing sector resulted in Romanian agencies Facilitating Knowledge Exchange policy and building adopting the “best practice” guidance The fewest TA had this as an objective compared capacity generally also provided by the Bank. Also in Romania, with the other TA objectives. Yet, along with im- had higher ratings the combination of ESW and TA in anti- plementing policies/programs, this objective re- for informing and corruption built capacity, with in-country ceived the highest average rating among all the stimulating public debate. stakeholders repeatedly noting that these ESW and TA objectives from in-country stake- 38 E F F E C T S O F E S W A N D TA O N C L I E N T C O U N T R I E S Table 4.3: Selected Examples of ESW Building Capacity ESW task Country Effects ICA Malaysia Government officials gained knowledge on designing surveys and analyzing survey results, and they feel they can now assess the investment climate alone, without the Bank’s help. Mauritius Built capacity of a local research company that carried out the firm survey and collected data from 200–300 companies. Guyana Those who were interviewed or surveyed for the ICA were later involved in development of the country’s National Competitiveness Strategy. PER Jordan Built capacity in the line ministries through their involvement in its production. Peru Government officials indicated that capacity building occurred during the preparation of the PER. Vietnam Government officials increased their participation in the production of the PER over time, from providing data (1996), to producing some sections (2002), to writing the report on equal footing as the Bank (2005). The work has helped the government design strategies and policies, such as the Comprehensive Poverty Reduction and Growth Strategy. FSAP Guyana Bank of Guyana staff learned about stress testing, banking supervision, and insurance from foreign experts during the FSAP mission and through the process of reviewing and providing comments on the FSAP. Jordan Undergoing the FSAP process was a learning experience for the insurance sector. CEM Romania Showed government agencies how to do policy analysis (developing proper databases, identifying intermediate steps, and doing policy scenarios) and how to prioritize. This improved the internal policy-making process in Romania and was arguably the greatest effect of the CEM, even though it was not undertaken for this purpose. Poverty Vietnam Enhanced the capacity of local researchers involved in the preparation of the report, as Assessment they are now doing the poverty analysis that the Bank used to do, and they have also applied the skills of doing poverty maps to agricultural mapping. Assessing Malaysia Helped government officials build a model and trained them in using it for simulations of Fiscal impacts of big changes in tax policy. Now the Ministry of Finance runs the basic aspects Incentives of the model, although it still needs some TA when simulating big changes in the tax system. Source: IEG country reviews. Note: CEM = Country Economic Memorandum; ESW = economic and sector work; FSAP = Financial Sector Assessment Program; ICA = Investment Climate Assessment; PER = Public Expenditure Review. holders. The high rating was validated by responses (figure 4.10).6 Both ESW and TA facilitated In-country stakeholders to the specific TA user survey and was congruent knowledge exchange in the countries gave facilitating with those by TA TTLs. The high ratings for this ob- reviewed, although this is a stated ob- knowledge exchange the jective can be seen in figure 4.9, which shows that jective only for TA. highest average rating of more than 80 percent of in-country stakeholders any objective for both interviewed in the country reviews and surveyed, One aspect of this objective highlighted ESW and TA. as well as more than 80 percent of TTLs, gave rat- by in-country stakeholders was the learn- ings of 4 and higher to this objective. ing of new concepts through Bank ESW . Stake- holders in Mauritius and Vietnam indicated that the As with the other objectives, this one received a Corporate Governance ROSC mainstreamed and range of ratings across the countries reviewed introduced the concept of corporate governance 39 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Table 4.4: Selected Examples of TA Building Capacity Country Tasks Effects Malaysia Financial Sector Monitoring Contributed to the Central Bank using better statistical techniques for projecting nonperforming loans and refining the techniques to do stress tests for the financial sector. Financial institutions now do their own stress tests under the supervision of the Central Bank, which is developing aggregate stress tests. Measuring Contribution of Government officials learned to gather information on services in the Services to GDP economy and to better measure their contribution to GDP. Mauritius Real Time Gross Helped modernize the payments system by making transactions more Settlement Initiative secure, facilitating financial sector deepening as a result. Vietnam Banking Sector Reforms (2002) Helped the State Bank of Vietnam (the Central Bank) to design and and Banking Reform implement a program for bank restructuring; audit commercial banks Program (2004) following international standards and supervise commercial banks; and design projects to monitor information in the financial sector. Source: IEG country reviews. Note: GDP = gross domestic product; TA = technical assistance. into the respective countries. A civil society rep- Figure 4.7: Ratings on the Extent to resentative indicated that people listen to the Which ESW Stimulated Public Debate, Bank regarding new conceptual frameworks (the by Shares of Respondents from example was education) in Jordan. Different Sources Bank ESW was also credited with introducing 100 new thinking, ideas, and terminology into coun- 90 tries. In Peru, the Bank is seen as the premier in- 80 stitution for guiding development thinking and 70 1 bringing new ideas to the country. In Romania, the 2 translation of the Challenges of EU Accession in 60 Romanian Agriculture TA into Romanian pro- Percent 50 3 vided the Ministry of Agriculture with the Ro- 40 4 manian equivalents of EU technical agricultural 30 5 terminology. In Vietnam, a private sector repre- 6 sentative claimed that a significant portion (70 per- 20 cent) of the country’s knowledge of the key terms 10 of market economy comes from the World Bank. 0 Country Specific ESW ESW/TA ESW TTL In some countries, although there were clear in- reviews survey user survey survey stances of knowledge transfer, sometimes the ef- Response source fects were not spread across sectors or did not go Source: IEG surveys and country reviews. far enough. In the Democratic Republic of Congo, Note: Responses presented for country reviews are those of in-country stakeholders the Public Financial Management TA transferred and exclude those of TTLs. Ratings scale is from 1 to 6, with 1 denoting “no ex- tent” and 6 denoting “great extent.” ESW = economic and sector work; TA = tech- a substantial amount of knowledge concerning nical assistance. budget processes, techniques, nomenclatures, and improved accounting procedures. However, 40 E F F E C T S O F E S W A N D TA O N C L I E N T C O U N T R I E S Figure 4.8: Ratings on the Extent to Which ESW Stimulated Public Debate, from In-Country Stakeholders in Country Reviews 6 5 4 Rating 3 2 1 p. . ali h ia an o ius ru ia na nia m Re Dem es th rb ys na Pe M rd ya rit ma so of lad Se ala t Jo Gu au Vie Le o, Ro ng M M ng Ba Co Country Source: IEG country reviews. Note: Rating scale is from 1 to 6, with 1 denoting “no extent” and 6 denoting “great extent.” ESW = economic and sector work. Box 4.3: Some Country-Specific Factors Affecting the Extent to Which ESW and TA Stimulated Public Debate In Vietnam, within-country differences in capacity af- not believe that the media reporting on ESW—or the fected the extent to which ESW and TA stimulated pub- public debates that were sometimes stimulated—helped lic debate. A private sector representative noted that change the perceptions or influence the views of key although greater government capacity over the past few stakeholders. Often the effect was not constructive, and years has made it much easier for the private sector to civil society continues to carry a negative image of the argue with the government about its role in the economy, Bank’s work. In fact, given the generally negative per- many in the media still do not understand microeco- ception of the Bank in civil society and among think tanks nomic and macroeconomic issues. (not always for reasons of merit but ideological), such de- For Romania, its transition from a formerly closed so- bate may actually scare the government from pursuing ciety was an important factor, as in-country stakehold- reforms.a ers are particularly appreciative of the opportunity to In Jordan, the Bank is generally not very popular with debate issues publicly. the media, Parliament, or the political parties, as it is For Serbia, the transition factor did not apply. Gov- perceived to have imposed conditions on the country. One ernment officials often agreed to recommendations of example is with respect to privatization, which is per- ESW, which obviated the need for public debate. ceived to have led to higher unemployment. However, In Bangladesh and Jordan, the Bank has a relatively even in Jordan, there were some positive examples of negative image. Most respondents in Bangladesh did ESW stimulating public debate (see table 4.5). Source: IEG country reviews. a. From an interview with a former senior public official. 41 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Table 4.5: Selected Examples of ESW and TA Stimulating Public Debate Country Tasks Effects Jordan AML/CFT Stimulated debate in the media on AML regulations. FSAP Stimulated debate within the insurance sector. Corporate Governance ROSC Led to the posting of the draft corporate governance code on the Internet, which stimulated debate. Doing Business Stimulated discussion/argument/debate and resulted in the Ministry of Industry and Trade taking into account the recommendations and reducing the number of days for doing business. Romania Anticorruption ESW, In-country stakeholders valued the approach that the Bank took, which Anticorruption Diagnostic TA was to present objective information and based on it, initiate a public debate. In turn, the opening of the issue of corruption to public debate led to the institutionalization of anticorruption efforts within the broadly defined government (for example, asset declaration for politicians). CEM, Public Expenditure Through public outreach, helped spread awareness of policy and Management (ESW), and PEIR implementation gaps that were slowing down EU accession. Vietnam Doing Business Widely reported in the media, which helped the private sector raise its voice. PER Disagreements over its findings on the irrigation sector led to a lot of discussion inside the government and among researchers. Source: IEG country reviews. Note: AML/CFT = Anti-money laundering and combatting the financing of terrorism; CEM = Country Economic Memorandum; ESW = economic and sector work; EU = European Union; FSAP = Financial Sector Assessment Program; PEIR = Public Expenditure and Institutional Review; PER = Public Expenditure Review; ROSC = Report on the Observance of Standards and Codes; TA = technical assistance. Stakeholders particularly the TA focused on a small group of tech- the knowledge transfer has occurred widely within appreciated learning new nical staff in the Ministry of Budget. There the Ministry of Finance, the perception of repre- concepts, new thinking, has been no political buy-in and little sentatives from this ministry as well as of donors and new terminology dissemination beyond the target group. was that more needed to be done for the sectoral through ESW. As a result, although the budget processes ministries. have improved, an in-country stake- 7 holder indicated that the Democratic Republic of Influencing the Development Community Congo is “still a long way from achieving budgetary There was a disparity in views on this objective discipline.” from the different sources, with donors that were interviewed giving lower ratings than donors that In Mali, a substantial amount of knowledge con- were surveyed (figure 4.11). Specifically, less than cerning the preparation of annual PERs, budget 60 percent of the donors that provided ratings processes and techniques, improved accounting from the country reviews gave ratings of 4 and procedures, multiyear public expenditure frame- higher, compared with between 70 and 90 percent works, and improved transparency in of the donors from the two surveys (specific ESW In some cases, knowledge public financial management was trans- survey and the ESW/TA user survey). ESW TTLs transfer did not spread ferred as a result of formal TA (PER TA)— who were surveyed also gave higher ratings for this across sectors or go but also especially informal TA provided objective, with nearly 70 percent giving ratings of far enough. by the Bank and other donors. Although 4 and higher. 42 E F F E C T S O F E S W A N D TA O N C L I E N T C O U N T R I E S Figure 4.9: Ratings on the Extent to Which Figure 4.10: Ratings on the Extent to TA Facilitated Knowledge Exchange, by Which TA Facilitated Knowledge Exchange, Shares of Respondents from Different from In-Country Stakeholders in Country Sources Reviews 100 6 90 80 5 1 70 2 60 4 Percent 3 Rating 50 4 40 3 5 30 6 20 2 10 0 1 Country Specific TA TA TTL ia ius m ali an ru nia ia p. . Re Dem reviews survey survey ys na rb Pe M rd rit ma of Se ala t Jo au Vie o, Ro M M ng Response source Co Source: IEG surveys and country reviews. Country Note: Responses presented for country reviews are those of in-country stakeholders and ex- clude those of TTLs. Rating scale is from 1 to 6, with 1 denoting “no extent” and 6 denoting Source: IEG country reviews. “great extent.” ESW = economic and sector work; TA = technical assistance; TTL = task team Note: Rating scale is from 1 to 6, with 1 denoting “no extent” and 6 denoting “great extent.” leader. TA = technical assistance. Although the fewest ESW specified this as an ob- semination of certain Bank ESW (Viet- More than half of all jective (see chapter 1), this belies the effects of nam), and Bank ESW not being done in survey respondents ESW on this objective. In particular, in response areas of donor interest (Mauritius). indicated that ESW was to the question on the specifics of how the ESW used by donors. was used, more than half of all respondents (not Even in countries where donors gave just the donors) to the specific ESW survey indi- below average ratings on this objective, there cated that the ESW on which they were being were divergent views among donors, as well as surveyed was used by donors. The same share was specific examples of ESW having effects. In Mau- found from the specific TA survey, when influ- ritius, for example, where several donors gave encing the development community was not even below average ratings of 3, one gave a relatively a stated objective of TA. high rating of 5 (the European Commission). Even the donor that gave below average rat- In 4 of the 10 countries for which donors provided ings—the French Development Agency—pro- ratings, donors gave ratings of below average vided specific examples of ESW informing its (that is, below 3.5): the Democratic Republic of activities. The same applies to Mali, where, despite Congo, Mali, Mauritius, and Vietnam (figure 4.12). generally low ratings for this objective, one donor The low ratings stem from a variety of country- (European Commission) cited a specific specific reasons, including donor fatigue (in the example of Bank ESW having an effect. ESW had effects even in Democratic Republic of Congo), donors already There were also examples of TA influ- countries where donors having large programs aligned with government encing the development community, al- gave below average priorities (Mali and Vietnam), the lack of dis- though this is not a stated objective for ratings on this objective. 43 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS TA. Some of the specific effects of ESW influ- Figure 4.11: Ratings on the Extent to Which ESW encing the development community are pre- Influenced the Development Community, by Shares sented in box 4.4. of Donors from Different Sources 100 In addition to specific effects, donors also high- lighted some general effects of ESW on their ac- 90 tivities. First, Bank ESW has created significant 80 1 awareness of issues that did not exist before, and 70 donors have subsequently gotten involved in 2 60 areas they had not previously considered. Second, 3 Percent 50 Bank ESW provided intellectual leadership for 4 reform priorities in countries. Third, Bank ESW 40 5 sometimes provided the empirical and intellectual 30 cover for activities that donors wished to engage 6 20 in anyway. 10 0 Other Effects Country Specific ESW ESW/TA user ESW TTL In addition to examining the various stated ob- reviews survey survey survey jectives, the country reviews also found that ESW Response source and TA had other effects. These included (1) ef- Source: IEG surveys and country reviews. fects that went beyond the specific country for Note: The rating scale is from 1 to 6, with 1 denoting “no extent” and 6 denoting “great extent.” ESW = which they were intended (regional effects); economic and sector work; TA = technical assistance; TTL = task team leader. (2) effects that went beyond the client govern- ments (toward which the ESW and TA were mainly directed) to the private sector, civil society, and local researchers and academia; (3) creation of in- Figure 4.12: Ratings on the Extent to Which ESW formal networks between Bank staff and gov- Influenced the Development Community, from ernment officials; and (4) monitoring of country Donors in Country Reviews economic conditions. 6 Regional effects 5 The Real Time Gross Settlement System Initiative TA for Mauritius is now being replicated all over Africa, and the Mauritians are sharing their expe- 4 rience with respect to the Aid for Trade TA with oth- Rating ers in the Region. The Jordan Mortgage Refinance 3 Company, established with the assistance of the Mortgage Market TA, is providing a model for other 2 countries in the region (the Arab Republic of Egypt and Pakistan). The Jordan AML/CFT Assessment, which was released in an Organisation for Eco- 1 nomic Co-operation and Development working p. . ali ius ho h u ia a an am Re Dem es r an group hosted by Jordan, has affected other coun- rb Pe M ot rd rit tn of lad y Se s Jo Gu au Vie Le o, ng M ng tries in the Region that attended the meeting. Ba Co Country The Regional Conference on Investment Climate Source: IEG country reviews. Note: Rating scale is from 1 to 6, with 1 denoting “no extent” and 6 denoting “great extent.” ESW = eco- and Competitiveness ESW in Malaysia seemed to nomic and sector work. The rating for Jordan is from only one donor. have created a sense of urgency in other South- east Asian countries about the need for policy 44 E F F E C T S O F E S W A N D TA O N C L I E N T C O U N T R I E S Box 4.4: Selected Examples of Bank ESW Influencing the Donor Community This box highlights some examples of Bank ESW influencing donor velopment provided funding for improving budget management. The activities. More examples of Bank ESW and TA having such effects latter agency used the FSAP as a blueprint for supporting the im- are in appendix E. plementation of the Prompt Corrective Action Framework and the Guyana provides many examples of Bank ESW influence on Corporate Governance of Banks. donor activities, the CFAA prominent among them. The U.K.’s De- In Mauritius, following the Transport Action Plan and the PER, partment for International Development indicated that it started the French Development Agency mobilized additional resources working on procurement issues because of the CFAA. The CFAA for and shifted its priorities toward the transport sector. The Eu- was one of the three conditions required for eligibility for the ropean Commission provided additional budget support through European Commission’s poverty-reduction budget support. The cofinancing the Trade and Competitiveness DPL because of the IMF used the CFAA to track debt relief conditions for heavily in- CEM. The recommendations in the CPAR provided conditions for debted poor countries. The report also informed the Inter-Ameri- European Commission financial support. can Development Bank’s Technical Cooperation Project on In Mali, the Transport Support to Sustainable Economic Growth Strengthening the Auditor General’s Office. ESW led to increased funding by the European Commission for the In Jordan, following the PER, the German Agency for Techni- road subsector; the Commission agreed to directly finance routine cal Cooperation and the United States Agency for International De- road maintenance for the first time. Source: IEG country reviews. changes to maintain competitiveness. Thailand, Peru, the country’s ranking in Doing Some effects went beyond in particular, has asked for a second round of Business is seen as having an impact on the intended country to competitiveness analysis following this work. investors’ decisions. The Sierra Rural influence the wider Development Strategy is viewed as hav- Region. Influencing private sector investment ing spurred private investment in Sierra. In Lesotho, the value chain studies that were part of the PSD Strategy have influenced private sec- Informing and influencing civil society tor decisions. In Malaysia, the Corporate Gover- The Accountability in Social Sectors ESW in Peru nance ROSC gave foreign fund managers an appears to have generated interest in educa- independent assessment of the level of corporate tion issues among civil society organizations governance in Malaysia and reiterated Malaysia’s (CSOs), academia, and the private sector. In readiness to receive foreign funds. Had this same Mauritius, the Mauritius Employers Federation information come from the government, the in- uses ESW to inform its own research and to ad- vestment community would have believed it less. vocate for policy changes in key areas such as The report has also helped validate the assessment pension reform. of the Malaysia Stock Exchange. In addition, the California Public Employees’ Retirement System Bank ESW has also helped improve relationships (one of the largest pension funds in the world) between in-country stakeholders. For instance, the uses it as a secondary source to confirm local Corporate Governance ROSC in Malaysia has information. helped improve relations among CSOs, shareholders, activist groups, and the In Vietnam, the Corporate Governance ROSC government. In Peru, ESW has facilitated Some effects went beyond has helped enhance the knowledge of the foreign the bringing together of stakeholders the client government to investment community on the situation in Viet- from different groups with different po- influence private sector nam. The Vietnam Development Reports have sitions by presenting a neutral piece of investment, civil society, had substantial effects on foreign investors. In analysis that groups can react to. and research. 45 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Supporting local research and academia Informal networks In Mauritius, the Labor Market Review generated The process of undertaking ESW and TA has also debate on minimum wages and provided ideas for led to the creation of informal networks through future research among local researchers. The interactions between Bank staff and government Mauritius CEM is used as a reference for analysis officials. In both Peru and Serbia, government of- by academics. A professor at the University of ficials indicated that when relationships are de- Guyana uses that country’s PER for re- veloped with Bank experts, they can call on them Some ESW and TA helped search purposes. The Regional ESW A for advice and information or for help in sup- create informal networks Time to Choose: Caribbean Develop- porting a position in a policy discussion. and contributed to better ment in the 21st Century (World Bank economic monitoring of 2005d) is almost a textbook at the same Monitoring the country. university. Some countries and organizations appreciate the monitoring function that Bank ESW provides. For In Lesotho, Bank ESW is generally important for instance, it was mentioned in Mauritius that ESW research. In Vietnam, researchers and under- provides an assessment of how well Mauritius is graduate and postgraduate students use the PERs, doing, which helps keep the country on track. The and universities have included many PER topics in coffee sector review for Vietnam provided the their curricula. In Peru, Bank reports are used for International Coffee Organization with informa- degree programs at universities. Universities and tion it had been unable to obtain. In contrast, local think tanks (in addition to the government) countries that have more capacity and resources, depend on Bank studies to develop their own such as Malaysia, would not benefit as much from analyses; studies that generate hard data such as this function, because there are many sources the ICA and the Municipal Debt ESW were found (local and international) that provide this moni- to be particularly useful in this regard. Furthermore, toring service. in Peru, when local researchers participate in World Bank research, they are taken more seriously. 46 Chapter 5 Evaluation Highlights • In-country stakeholders generally rated ESW and TA high for technical quality, the dimension on which task team leaders focused their efforts. • Partnership with clients in the pro- duction of ESW and TA received some of the lowest ratings. • Whether tasks are requested by clients or not did not matter for the achievement of objectives, although clients still needed to be genuinely interested in the tasks for them to be effective. • Dissemination was rated lowest by stakeholders and task team lead- ers—sustained follow-up through lending or nonlending TA had greater effects than dissemination. • Several additional dimensions were found to be important—provision of a menu of policy options, adequate resources, timeliness, and engaging ministries other than finance. In Lesotho, a World Bank health sector report informed the planning and implementation of a health project. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. Factors Influencing the Effectiveness of ESW and TA T his chapter assesses how the different dimensions of ESW and TA af- fected the extent to which objectives were met. These dimensions are technical quality, partnership with clients, origination (relevance), and dissemination (see chapter 1). The findings are based on the country reviews and the various surveys. The chapter also reports on the other dimensions that the country reviews and surveys found important for enhancing the ef- fects of ESW and TA. Technical Quality In-country stakeholders widely ac- Stakeholders generally Of the five dimensions of ESW and TA this evalu- knowledged the high technical quality of gave high ratings to ESW ation asked in-country stakeholders to rate, tech- Bank ESW . Even those who were critical and TA technical quality, nical quality was among the most highly rated of the role and influence of the Bank ad- which is correlated with (table 5.1). Those relatively high ratings validate mitted that the technical quality of the re- the extent to which the QAG ratings on internal quality (see chapter 2).1 ports was good. Donors in Bangladesh tasks met all their Overall, in-country stakeholders’ ratings on the were especially appreciative and believed objectives. technical quality of ESW and TA significantly and the Bank’s ESW was professional and positively correlate with their ratings on the extent trustworthy. Stakeholders in Romania noted that to which every ESW and TA objective has been the Bank’s comparative advantage is in accumu- met.2 lating practical knowledge and experience from many different countries. The relatively high ratings for technical quality— as well as the positive and significant correlation Hence, the Bank’s effectiveness in providing ESW between these ratings and ratings of the effects and TA is heavily dependent on its reputation of ESW and TA—indicate that the perceived high for technical expertise, based on its involvement quality of those tasks has been important for in a variety of countries. Without this reputation, achieving effects in the Bank’s client countries.3 the Bank’s effectiveness would be greatly re- This is confirmed by in-country stakeholders, duced. In Serbia, demand for ESW and TA from who, when asked to rank the importance of the in-country stakeholders was primarily based on five dimensions of ESW and TA for enhancing the reputation of Bank staff for experience, tech- their effects, on average gave rankings of 2 (sec- nical competence, and willingness to work on ond to the top) to technical quality (table 5.2). local priorities. There does not seem to be a large difference be- tween in-country stakeholders in IBRD versus IDA There were a few dissenting comments, however. countries with respect to the importance they In particular, some stakeholders in Bangladesh give to technical quality. criticized the Bank’s work as dogmatic, ideological, 49 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Table 5.1: Average Ratings of In-Country Stakeholders on Various Dimensions of ESW and TA Country Specific ESW Specific TA Dimension reviews user survey user survey ESW/TA user survey Technical quality 4.8 4.8 4.9 4.8 Relevance 4.5 5.4 5.2 4.6 Timeliness 3.9 4.4 4.5 4.2 Partnership with clients 3.6 4.5 4.5 3.7 Dissemination 3.4 4.6 4.9 4.3 Source: IEG surveys and country reviews. Note: Ratings are on a six-point scale, with 1 being low and 6 being high; 4 is slightly above average, and 3 is slightly below average. The same rating scale is used throughout this chapter. ESW = economic and sector work; TA = technical assistance. Table 5.2: Average Rankings of In-Country Stakeholders on the Importance of Various Dimensions of ESW and TA for Achieving Objectives Specific ESW Specific TA ESW/TA Country reviews user survey user survey user survey Dimension All IBRD IDA All IBRD IDA All IBRD IDA All IBRD IDA Technical quality 2 1 2 2 3 2 2 2 2 2 2 2 Relevance 1 2 1 3 1 3 4 3 3 3 3 3 Partnership 3 3 3 1 2 1 1 1 1 1 1 1 Timeliness 4 4 4 4 4 4 3 3 5 5 4 Dissemination 5 5 5 5 5 5 5 5 4 4 4 5 Source: IEG surveys and country reviews. Note: 1 is the top rank. ESW = economic and sector work; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; TA = technical assistance. and predictable, consistent with its “neoliberal” Some Bank staff interviewed during the country re- paradigm. Others said that Bank reports were views indicated that the technical quality of Bank more useful in analyzing the state of a particular sec- ESW and TA could be higher. Many TTLs working tor than in providing innovative solutions to known on the Democratic Republic of Congo commented problems, that the reports were too long, and that on the acute budget constraints in the Sub- sometimes analysis was too sophisticated and dif- Saharan Africa Region, which contradict the insti- ficult for the government to absorb. Some in-coun- tutional imperative to produce high-quality work. try stakeholders in Peru criticized the one-size-fits- This evaluation also found that Sub-Saharan Africa all approach, which they indicated the Bank could had the lowest average cost for country-specific ESW address by partnering with local counterparts. of all Regions. It also found a positive correlation In-country stakeholders in Lesotho and Peru men- between the cost and quality of tasks in the Region. tioned the need for higher qual- This Region also had the lowest QAG internal qual- Some stakeholders ity consultants, as some Bank ity ratings for country-specific ESW of all Regions.4 criticized the Bank’s ESW consultants did not have the req- and TA, and some Bank uisite knowledge for particular TTLs working on Romania indicated that the Bank staff indicated that tasks, which hurt the Bank’s should require higher technical standards, which quality could be better. credibility. had sometimes taken a back seat to timeliness. 50 FA C T O R S I N F L U E N C I N G T H E E F F E C T I V E N E S S O F E S W A N D TA TTLs working on Serbia echoed this, indicating a The focus on technical quality by TTLs re- The generally high ratings need to improve the review process so it focused flects their perceptions of the incentives on technical quality are more on quality and less on deadlines and a need they face in the Bank. ESW and TA TTLs consistent with the efforts to improve the budgeting of time for the work. interviewed for the country reviews—as exerted by TTLs. A TA TTL working on Peru suggested that TA is not well as those who responded to the sur- subject to the same quality control as ESW . veys—gave, on average, the highest rat- ing for incentives they face within the Bank to The generally high ratings on technical quality delivering a product with high technical quality from all sources (inside and outside the Bank) are (tables 5.4 and 5.5). consistent with efforts exerted by TTLs. In the con- text of the country reviews, when asked to rank Partnership with Clients the various dimensions on which they had exerted The Bank encourages partnership with national efforts, ESW TTLs gave the top rank to technical counterparts in the production of ESW as a means quality (table 5.3). This finding is validated by the of building national capacity. Among the various findings of the survey of ESW TTLs, who also, on dimensions of ESW and TA that in-country stake- average, assigned the first rank to technical qual- holders were asked to rate, partnership in ity when asked the same question. Specifically, production with clients received among Partnership with clients in more ESW TTLs (44 percent) gave the first rank- the lowest ratings in all the different the production of ESW ing to technical quality than to other dimensions. sources of information (country reviews and TA received among and electronic surveys; see table 5.1). the lowest ratings. The situation was slightly different with TA TTLs. When asked to rank the various dimensions on The in-country stakeholders’ ratings on partner- which they had exerted efforts in the context of ship in production with clients are significantly and the country reviews, they gave equal rankings to positively correlated with their ratings on the technical quality and relevance, on average. For extent to which the various ESW and TA objectives the same question in the survey of TA TTLs, how- were met.5 This implies that the perception of ever, relevance was ranked over technical quality partnership in production is correlated with the (which was ranked second). perception of effects of ESW and TA. Table 5.3: Rankings of Efforts by TTLs Ranking of efforts Ranking of efforts by ESW TTLs by TA TTLs From From From From country ESW TTL country TA TTL Dimension reviews survey reviews survey Technical quality 1 1 1 2 Relevance 2 2 1 1 Partnership with clients 4 3 4 3 Politically supportable recommendations 3 5 3 5 Menu of options 5 4 6 4 Timeliness 6 7 5 7 Dissemination 7 8 7 6 Shorter and more narrowly focused 8 6 na na Translation 9 9 na na Source: IEG surveys and country reviews. Note: 1 is the top ranking. ESW = economic and sector work; na = not applicable; TA = technical assistance; TTL = task team leader. 51 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Table 5.4: Ratings and Implied Rankings of Incentives Perceived by ESW TTLs Ratings of Implied rankings incentives of incentivesa From From From From country ESW TTL country ESW TTL Dimension reviews survey reviews survey Technical quality 4.6 4.6 1 1 Relevance to Bank operational teams 4.1 4.5 2 2 Relevance to clients 4.1 4.3 2 3 Menu of options 4.0 4.0 4 4 Timeliness 3.8 4.0 5 4 Politically supportable recommendations 3.6 3.7 6 6 Dissemination 3.2 3.3 7 7 Relevance to other donors 3.0 3.1 8 8 Relevance to other stakeholders 3.0 3.1 8 8 Partnership with clients 2.9 2.9 10 10 Source: IEG surveys and country reviews. Note: The rating scale is from 1 to 6, with 1 denoting “no extent” and 6 denoting “great extent.” ESW = economic and sector work; TTL = task team leader. a. 1 is the top rank. Table 5.5: Ratings and Implied Rankings of Incentives Perceived by TA TTLs Ratings of Implied rankings incentives of incentivesa From From From From country TA TTL country TA TTL Dimension reviews survey reviews survey Technical quality 4.3 4.6 1 1 Politically supportable recommendations 3.8 3.7 2 3 Menu of options 3.7 3.7 3 3 Timeliness 3.4 3.9 4 2 Dissemination 3.3 3.5 5 5 Partnership with clients 3.2 3.1 6 6 Source: IEG surveys and country reviews. Note: The rating scale is from 1 to 6, with 1 denoting “no extent” and 6 denoting “great extent.” TA = technical assistance; TTL = task team leader. a. 1 is the top rank. However, findings from the specific ESW survey not. The information on whether the ESW had a indicate that actual partnership in production partner came from the responses of TTLs to the with clients did not matter for the effects. Specif- ESW TTL survey and was applied, for the same spe- ically, in-country stakeholders responding to the cific task, to the responses to the specific ESW sur- survey did not rate the effects of ESW that had a vey. In the TTL survey, partnership in production partner significantly higher than ESW that did was specified to entail substantial comments on 52 FA C T O R S I N F L U E N C I N G T H E E F F E C T I V E N E S S O F E S W A N D TA the concept paper, data collection and analysis, In Bangladesh, in-country respondents Partnership with clients in writing background papers, writing sections of the indicated that involvement of local part- the production of ESW ESW , and providing substantial comments on the ners would not only help build local re- and TA is correlated with draft report. search and analytical capacity, but would the perceived effects of also help develop a sense of ownership ESW and TA. These seemingly contradictory findings indicate that was important for implementing that what partnership means to the in-country changes. Several respondents were pre- stakeholders may not necessarily be actual pro- pared to trade some decline in technical duction of (part of) the tasks in the ways specified quality for increased local partnership. in the previous paragraph. This is corroborated by findings from the country reviews. Those Lesotho was the only country of the 12 The findings suggest findings indicate that a variety of partnership reviewed that is not seeking more part- that stakeholders’ approaches were equally effective and did not nership in production because of ca- understanding of always entail actual production of parts of the pacity constraints. One high-level partnership does not task (see next section). What is important is work- government official indicated that the necessarily mean actual ing closely with clients to ensure that they are quality of the product could be com- involvement in the part of the process and that they assume owner- promised if they do not have the right production of the tasks. ship of the task, which entails close collaboration people to partner with the Bank. from task initiation to the formulation of conclu- sions and recommendations. These findings from the country reviews are cor- roborated by the high ranking that in-country The country reviews found that the capacity level stakeholders gave to partnership (relative to other of the country did not affect the desire for part- dimensions) for enhancing the effects of ESW nership or for partnership to have positive ef- and TA. Stakeholders in both IBRD and IDA coun- fects. In countries that have the capacity, such as tries across the different sources of information Malaysia and Serbia, partnership is a given. The value partnership similarly (table 5.2). Malaysian government was involved in the pro- duction of all four (of eight) ESW tasks that it The degree of partnership with clients that is im- had either requested or wanted. portant for enhancing the effects of ESW and TA differs. It could involve writing parts of the report In countries with lower levels of capacity, such as or just close collaboration from task initiation Vietnam, the government is interested in part- through formulation of conclusions and recom- nering with the Bank, as it wants to learn to do mendations. Box 5.1 presents examples from the things itself. The Vietnam PERs are now done on country reviews of the different approaches to a fully joint basis; Poverty Assessments are done partnership with clients that have worked. completely by the Vietnamese themselves; and background papers and chapters for the Vietnam Although most of the examples of part- Countries of a wide development reports are subcontracted to local nership in the country reviews were part- range of capacity desire researchers. nerships with the government, other partnership with the Bank partners also contributed to enhanc- in ESW and TA. Several In the Democratic Republic of Congo, where ing the effects of ESW and TA. In Roma- approaches to partnership many ESW tasks received low ratings for inform- nia, partnership with donor agencies im- were equally effective, ing policy (ratings of 2 and 3), partnership ap- proved the technical quality of ESW. depending on country peared to have been essential for those ESW that Partnerships with other international in- conditions. had effects. The three ESW that were given high stitutions have also enhanced the effects ratings (ratings of 5 and 6 for ESW in the forestry, of the ESW by giving the product stronger back- education, and health sectors) all fully involved the ing and hence commanding greater attention from government in various ways. the government. This was the case in Peru, where 53 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Box 5.1: Nature of Partnerships with Clients: Examples from Country Reviews In the Democratic Republic of Congo, in a review of the forest sec- meant having the Bank team sit with the government team to de- tor (perhaps the most successful ESW in the Democratic Repub- fine the competitiveness strategy—that there was “engagement.” lic of Congo in terms of informing policy during the evaluation For the ICA, partnership meant close collaboration in defining the period), the government collaborated with Bank staff from begin- study and the derivation of recommendations through a wide con- ning to end, including participation in substantive drafting of the sultative process. report. For the Education in the Democratic Republic of Congo: Pri- For Malaysia, partnership entailed collecting and providing orities and Options for Regeneration (World Bank 2005b), which the data and making substantive comments on the concept papers also received high ratings for informing policy, the counterpart in (as well as on the draft report in two cases). Partnership also en- the Ministry of Education was kept informed of the progress of the tailed the government contributing financially to the tasks: it paid study at all stages, with international consultants regularly dis- for the firm-level survey for the ICA, paid the cost of consultants cussing the substance of the report with the counterparts. For the from foreign universities for the Assessing Fiscal Incentives ESW, country status report on health, nutrition, and population, another and funded the Regional Conference on Investment Climate and highly rated ESW for informing policy, the counterpart in the Min- Competitiveness (an ESW). istry of Health indicated that he coordinated the study himself and Even Lesotho, which was not eager to seek more partnership directly supervised the work of the international consultants. because of its capacity constraints, partnered with the Bank for In Mauritius, partnership was generally understood to be part the PSD strategy, for which the government took ownership and of the production team, from the conceptualization of the reports organized a three-day PSD forum. In addition, the government to the formulation of recommendations. Partnership does not nec- created working groups composed of government, civil society, and essarily entail writing a section of a report or producing the TA. For private sector representatives to work on follow-up implementa- the Aid for Trade TA, partnership for a senior government official tion and project preparation. Source: IEG country reviews. In some cases, joining forces with the Inter-American Where champions do not already exist, the Bank partnerships with Development Bank (on the PER and the could try to cultivate them, as it did for the for- institutions other than the CPAR) and the IMF (on the FSAP) helped. est sector review in the Democratic Republic of government contributed It was also the case in Guyana, where a Congo. In this case, the Bank deliberately took a to enhancing the effects of government official indicated that the very partnership-oriented approach from the be- ESW and TA. consistency of recommendations across ginning, not only with the government but also donors enhances the effects of ESW . with other stakeholders (local and international CSOs, the logging companies, and donors). This Engendering partnership with clients: was necessary, as the objective of the ESW was not Examples from country reviews just to fill in knowledge gaps and provide the an- Partnership with the government is obviously fa- alytical underpinning for future lending, but also cilitated if there is a champion in the government to gain consensus on an extremely controversial for the task. This was the case for the FSAP and as- topic. This was very costly in both time and sociated reports in Mauritius, the Gender Assess- money—the task was completed with a delay of ment and the ESW and TA in the education nearly two years. Partnerships with the sector in Jordan, a review of the forest government are sector in the Democratic Republic of Efforts and incentives facilitated where there is Congo, and the PSD Strategy in Lesotho. The lower ratings on partnership in production a government champion In all these cases, a high-level govern- by in-country stakeholders are commensurate for the task. ment official championed the reports. with the efforts exerted by TTLs on this dimen- 54 FA C T O R S I N F L U E N C I N G T H E E F F E C T I V E N E S S O F E S W A N D TA sion, which in turn reflect the low incentives they The specific TA survey also showed that Where champions did not perceive for partnership in production with clients whether the task was client requested already exist, they could in the Bank. In terms of exertion of efforts, TTLs or not was not significantly associated be cultivated through ranked partnership after technical quality and with in-country respondents’ ratings on substantial efforts in relevance in the country reviews and the surveys the extent to which TA strengthened in- collaborating with clients. (table 5.3). They also gave the lowest (and a below stitutions or facilitated knowledge ex- average) rating to the incentives they face within change. The TA objective for which the the Bank with respect to exerting efforts on part- origination of the task seemed to matter nership (tables 5.4 and 5.5). Given that TTLs did was implementing policies/programs. The lower ratings on not rank partnership the lowest in terms of where For this objective, in-country respon- partnership in production they exerted efforts, it appears that TTLs actually dents gave, on average, a higher8 rating with clients reflect the put more effort into partnership compared to the of 4.8 to tasks that were requested by efforts exerted by TTLs incentives for such efforts in the Bank. the government than to tasks that were and the incentives for not (4.0 rating). such efforts in the Bank. This suggests that TTLs consider partnership im- portant and would invest more effort in this area In 3 of the 12 countries reviewed, there were in- if they were given the resources and recognition dications that the effects of ESW and TA were en- for it. In Lesotho, six of nine TTLs mentioned hanced when the clients had requested the work. partnership as one of their three top concerns, For the other nine countries, whether the tasks even though the government is not seeking more were client requested did not appear to have in- partnership. One TTL working on Lesotho said fluenced the effects of the tasks. In fact, there are that Bank staff in general pay more attention to many examples of tasks not requested by clients comments by other Bank staff than to comments that had significant effects (as highlighted in chap- from the client government. ter 4), such as the IDA-mandated CPAR, ICA, and PER. The FSAP also had significant effects (as in- Origination dicated in chapter 4), and although the FSAP The main indicator of relevance for ESW and TA needs to be officially requested by the authorities, lies in the circumstances under which such work the country reviews indicated several instances is initiated, that is, whether it originates with the where FSAPs were undertaken at the urging of the client (demand driven) or with the Bank (supply Bank or the IMF. driven). Whether the ESW and TA were requested by the client generally did not make a significant In Romania, the extent to which ESW achieved its difference to the extent to which the various ESW objectives had little relation to whether the ESW and TA objectives had been met. This was the were demand or supply driven. Instead, finding from the specific ESW and TA surveys,6 as the key determinants were the political Whether the client well as from the country reviews. conditions in the country and the coun- requested the ESW and TA try’s receptivity to implementing re- did not significantly The specific ESW survey revealed that whether the form, which was driven by the EU affect the extent to which task was client requested or not was not signifi- accession process. objectives were met. cantly associated with in-country respondents’ ratings on the extent to which the ESW informed Similarly in Serbia, the main determinant of the policy, built capacity, or influenced donor activi- extent to which ESW and TA achieved their ob- ties. The only ESW objective for which the origi- jectives was receptivity on the Serbian side, which nation of the task seemed to matter was very much depended on the political situation in stimulating public debate. On this objective, in- the country. country respondents gave, on average, a some- what higher7 rating of 4.8 to tasks that were These findings from the country reviews indicate requested by the government than to tasks that that ESW and TA did not always have to be demand were not (4.4 rating). driven to have effects, but where they were not 55 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Where ESW and TA were there was still genuine client interest in ocratic Republic of Congo, client interest in the for- not demand driven, the task. For example, although the est sector review was engendered by substantial genuine client interest in Bank proposed the ICA for Mauritius, Bank efforts in collaborating with the clients. the tasks was important the government took interest in it and for them to have effects. contributed funds to cover the costs In-country stakeholders in both low- and middle- of hiring a local research company to income countries indicated that the Bank has a collect firm data. In Malaysia, where the Bank useful role in proposing topics for ESW and TA, proposed the Corporate Governance ROSC, the as the Bank can draw on its own knowledge and government was genuinely interested because it international experience to analyze and propose wanted an independent assessment of a vision for what is needed for the country. These Bank efforts can also what its Securities and Exchange Com- views are presented in box 5.2. engender client interest. mission had done. Although government receptivity is important Client interest can also be generated by Bank ef- for tasks proposed by the Bank, government com- forts. In Mauritius, for example, the Bank engen- mitment to implementing tasks would likewise be dered client interest by tailoring core diagnostics important before the Bank undertakes tasks the to meet government needs. For example, the CEM government requests. In Jordan, for instance, focused on competitiveness and the PER on ways some respondents indicated that the Bank should to control the fiscal deficit—both issues of inter- verify the need for the task (TA), including est to the Mauritian government. Another exam- whether TA previously provided on the same sub- ple of tailoring mandated products that ject had worked and whether other donors are The Bank has a useful role helped with impact was the Guyana doing the same thing. In the same vein, a donor in proposing topics for CFAA, which had important effects be- in Bangladesh suggested that the government ESW and TA, drawing on cause it was done as a strategic product should first demonstrate that it recognizes the its own knowledge and rather than the descriptive product need for capacity building—such as spending its international experience. called for by the guidelines. In the Dem- own resources on capacity building—before the Box 5.2: Views of In-Country Stakeholders on the Role of the Bank in Proposing ESW and TA • In Guyana, some government officials indicated that the Bank can • In Peru, some respondents indicated that ESW and TA should identify the driving forces of the economy, including those stem- not be too demand driven, as politicians have a short-term hori- ming from international trends (such as the international trading zon, and the Bank with its international perspective can see prob- regime for sugar). This is because the Bank is more detached and lems that the Peruvians may fail to recognize. The Bank can put can identify areas that need to be studied that the country may big issues on the agenda, then provide specific help to address not be able to. the issue. • In Jordan, some respondents indicated that the Bank could • In Romania and Serbia, respondents were interested in the Bank come up with its own proposals of topics to study based on looking ahead and identifying what policies and issues would be analysis of social and economic indicators in the country or de- important in the future (such as the prospective 2013 reform of velopments in the region. the EU Common Agricultural Policy for Romania). Some re- • In Lesotho, some respondents indicated that the Bank has a spondents in Serbia were interested in more systematic identi- comparative advantage in identifying what needs to be done. fication of areas in which the Bank could have the strongest • In Mauritius, some respondents indicated that the Bank can impact on growth, as well as areas related to EU accession. provide a long-term perspective. Source: IEG country reviews. 56 FA C T O R S I N F L U E N C I N G T H E E F F E C T I V E N E S S O F E S W A N D TA Bank supplies it. This would provide some as- amples of Bank ESW and TA informing In-country stakeholders surance that the government would use such ca- donor activities. gave among the lowest pacities better. ratings to dissemination Sustained follow-up can also take the of ESW and TA. Dissemination form of nonlending TA, or just continu- Dissemination efforts for ESW and TA are among ous dialogue (informal TA) for a period the dimensions rated lowest by in-country stake- after the completion of the ESW . A very good ex- holders (table 5.1). As with technical quality and ample of nonlending follow-up leading to achieve- partnership, ratings for dissemination efforts are ment of objectives was the AML/CFT for Jordan. significantly and positively correlated with rat- The Bank TTL for the ESW followed up with con- ings on the extent to which the various ESW and tinuous support, starting with a workshop, then TA objectives were met.9,10 with TA to assist in the drafting of new AML leg- islation. This led to new, improved draft AML/CFT Despite the relatively low rating for dissemination legislation replacing older draft legislation in the and the perception that dissemination efforts en- Parliament. hance the effects of ESW and TA, in-country stake- holders ranked dissemination near the bottom in Training workshops have also been important for terms of priorities for ESW and TA (table 5.2). This achieving objectives in other instances, such as could be because dissemination (which conjures those that followed the poverty report in Jordan. up a one-time event) by itself was not perceived A number of stakeholders in Serbia also proposed to be important for enhancing the effects of ESW linking dissemination to training opportunities. and TA. For example, a donor in Mali indicated that In addition, respondents in Serbia indicated he found workshops that disseminated ESW a that to enhance impact, the Bank needs to commit waste of time and money. to engage for the long haul. This sentiment was echoed in Bangladesh, where respondents in- What has emerged from the country reviews as dicated that short-term interventions important for enhancing the effects of ESW and (such as helping the government draft Sustained follow-up TA is sustained follow-up beyond dissemination. an Insurance Regulation Law) do not beyond dissemination is Sustained follow-up can take various forms. One lead to sustained capacity improvement important for enhancing of the most common forms is lending, which, as and that the Bank needs to provide con- the effects of ESW and TA. discussed in chapter 3, is one of the main chan- stant support. nels through which ESW products have achieved their nonlending objectives.11 Lending also helped Another aspect of follow-up that has emerged with the effects of TA (not just of ESW). For in- from the country reviews as important is ade- stance, the Mauritius Aid for Trade TA was followed quate capacity in the country office, in particular, by the Trade and Competitiveness DPL to help im- relevant expertise that can provide TA to local plement the recommendations of the TA. counterparts. This issue was raised by several counterparts in Mali, who very much appreciated Follow-up funding has included not just lending the support of the resident economist for from the Bank but also funding from other the public financial management reform Follow-up can include donors. In Jordan and Mauritius, the FSAP was fol- program, but indicated that they would lending as well as lowed by a multidonor grant from the Financial like to see greater capacity in the coun- nonlending assistance or Sector Reform and Strengthening Initiative, which try office to support other sectors also. just continuing dialogue. helped implement the recommendations. In Lesotho, many of the recommendations of the ed- The issue is more acute in countries where the ucation sector study were implemented with the Bank has no country office. In Mauritius, for in- support of a grant from another multidonor ini- stance, the earlier absence of a country office13 tiative, the Education for All–Fast-Track Initiative limited Bank interactions mostly to the Ministry Catalytic Fund.12 Chapter 4 provides other ex- of Finance. This limited the knowledge of Bank 57 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS products in the sectoral ministries and among semination strategies. In Jordan, for instance, CSOs, a concern that was voiced not only by rep- both the Corporate Governance ROSC and the resentatives of the sectoral ministries but also PER were perceived to have great effects, although by those of the Ministry of Finance. A donor the PER was distributed only to the government in Lesotho (where the Bank did not have a coun- and the ROSC was distributed much more widely. try office but has now stationed a senior coun- Also in Jordan, the Poverty Institutional Framework try officer in the field) indicated that the influence TA was perceived to have more significant effects of the Bank without having a physical presence than the Private Sector Participation and Regula- is limited. tory Reform in the Jordanian Water and Sanitation Sector TA, although the latter was disseminated Finally, both in-country stakeholders and TTLs much more widely than the former. remarked on the need for more resources (staff time) for sustained follow-up. Stakeholders in In several instances, government interest had a Serbia also noted the lack of adequate resources much stronger effect than the Bank’s dissemina- for follow-up. A TTL working on Viet- tion efforts for ESW and TA. In Malaysia, for ex- Adequate country nam indicated that it is necessary to visit ample, a typical sequence of events for influential office capacity is the country many times to create support ESW and TA was as follows: first, government in- also important for for change, and this requires sufficient terest led to the request for the ESW or TA, then follow-up. budget resources. intense internal debate within the government, then wide dissemination by the government itself. Scope of dissemination Government interest in the Regional Conference The Bank generally encourages broad dissemi- on Investment Climate and Competitiveness nation of ESW and TA, which is often important ESW was such that it actually provided funding for for enhancing their effects. However, the coun- dissemination. try review findings indicate that the scope of dis- semination associated with ESW and TA having Similarly in Mauritius, the government was in- effects ranged greatly. In some cases, dissemina- terested in the ICA, FSAP, and Aid for Trade TA and tion narrowly targeted to the relevant counterparts undertook their dissemination to help create resulted in important changes; in other cases, broader support for reforms to which it was al- broad dissemination stimulated the public de- ready committed. Another example is Lesotho, bate that spurred changes. In a few review coun- where the government requested the influential tries, in-country stakeholders strongly believed PSD Strategy, then took ownership of it after its that their government is too influenced by the completion and promoted it through a three-day Bank, such that broad dissemination could be private sector forum. In all these examples, gov- counterproductive. It appears, therefore, that the ernment interest was key to the effects and dis- scope of dissemination associated with semination, which the governments actually took The scope effective ESW and TA depends on the charge of or funded. of dissemination country and the issue; it could even be associated with effective different for different issues within the The Malaysia example also indicates that there ESW and TA varied. same country. could be a difference between stimulating public debate and stimulating debate within the govern- Notable examples of ESW that had important ef- ment, with the latter being more important for gen- fects despite limited dissemination are the FSAPs erating results. Another such example is the Impact and the AML/CFT (see chapter 4 for examples of of HIV/AIDS on Household Poverty TA in Lesotho. the effects of these products), although they are This TA was not publicized in the media, although confidential and can only be released publicly it did influence the views of stakeholders. with the consent of the authorities. There are also examples of ESW being similarly influential In contrast, the country reviews also provided ex- within the same country despite very different dis- amples of wide dissemination enhancing the effects 58 FA C T O R S I N F L U E N C I N G T H E E F F E C T I V E N E S S O F E S W A N D TA of ESW and TA. In Peru, the TTL for the Evaluation Strategy. In Jordan it led the Ministry of In some cases, and Design of Concessions ESW noted that wide Industry and Trade to reduce the number government interest was dissemination in many different forums, including of days it takes to start a business. In more important for ESW private sector ones, was a factor in its impact. In Mauritius, the Bureau of Investments set and TA effects than Bank Guyana, the reports that had the greatest effects itself the goal of being ranked among the dissemination. were those that came into the public domain, top 10 places where it is easiest to do whether by design or not. For instance, the CFAA, business by 2010 (Mauritius is ranked 27 as of which was meant to be confidential, was leaked to 2008). In Vietnam, the report was credited with the press and resulted in important effects. In con- helping the private sector raise its voice. In two trast, the ICA, which was not meant to be confi- countries, however, the report was criticized as mis- dential but the release of which had been held up leading (Peru) or not entirely conforming to the by the government, had less effect. country situation (Guyana; see IEG 2008b). In the case of the forest sector review in the Dem- Examples from the country reviews of Regional ocratic Republic of Congo, there was wide dis- ESW and TA that had significant effects because cussion and debate at various stages during its of wide dissemination include A Time to Choose: production. The efforts to reach a consensus Caribbean Development in the 21st Century among stakeholders (government, the logging (World Bank 2005d) and the Regional Confer- industry, donors, and CSOs) on the conclusions ence on Pensions TA in the Middle East and North and recommendations of the report delayed its Africa. The wide dissemination of the former was completion by nearly two years. In the end, al- responsible for influencing donor activities and its though it was impossible to satisfy all the parties use by academia in Guyana. The reporting of the (in particular, most of the logging industry and latter in the media in Jordan made the task known some CSOs), there was a strong influence on the to a government minister. views and actions of government, of nearly all donors, and of most CSOs. Some stakeholders criticized the Bank’s dissem- ination efforts. For example, Bangladesh stake- Wide dissemination has also been useful for in- holders considered the Bank unsuccessful in troducing new concepts. In Mauritius, for in- stimulating public debate or in influencing the stance, the awareness campaign that accompanied opinion of key stakeholders, as it only disseminates the issuance of the corporate governance code (an its views to a preselected group of like-minded outcome of the Corporate Governance ROSC) people. Civil society and private sector repre- has filtered the concept into the minds of the sentatives in the Democratic Republic of Congo population. In Jordan, the new corporate gover- expressed frustration with the difficulty in ob- nance code (resulting from the Corporate Gov- taining copies of Bank analytical work and the lim- ernance ROSC) was posted on the Internet for ited notice that the Bank took of their comments. comments. There was widespread belief there (where the effects of ESW and TA were Wide dissemination Wide dissemination has also helped with the ef- generally rated very low) that nearly all has been useful for fects of regional and global ESW . A prominent ex- ESW was produced by the Bank for the introducing new concepts ample of wide dissemination of a global ESW Bank, with little interest in wider dis- and for the effects of leading to significant effects was the Doing Busi- semination and debate. regional and global ESW. ness reports. In 6 of the 12 countries reviewed, without prompting, stakeholders cited Doing In Guyana, a donor indicated that when the Bank Business as an example of global ESW that has had reports are not released, it creates suspicion and important effects. distrust among donors. In Jordan, a CSO repre- sentative indicated that when the Bank has a pol- In Guyana, a donor indicated that the report in- icy of not disseminating (such as with the Poverty formed the country’s National Competitiveness Assessment), it creates a negative image for the 59 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Bank. Some stakeholders in Bangladesh, Peru, The fifth most mentioned mode of dissemination Serbia, and Vietnam indicated that dissemination in the country reviews was hard copies of reports. should also go outside of the capital. This was mentioned in all the countries reviewed, much more so in IDA than in IBRD countries (29 Mode of dissemination percent of the IDA stakeholders mentioned this, A majority of in-country stakeholders selected compared to 10 percent of the IBRD stakeholders). e-mail with links or attachments to reports as the For the other four modes of dissemination, how- most effective mode of dissemination. On aver- ever, there was no difference in preferences for dis- age, around two-thirds of those interviewed and semination modes between stakeholders in IBRD surveyed selected this mode (table 5.6). A good and in IDA countries (table 5.7). example of this is Caribbean News in the Latin America and the Caribbean Region, an electronic In addition, the country reviews also provided ex- newsletter that includes links to and attached re- amples of other modes of dissemination that ports. There may be a bias in this finding, how- worked well. The Jordan Securities Commission ever, because three of the sources of information and the Vietnam State Securities and Exchange were from electronic surveys (that is, from stake- Commissions, respectively, put the corporate gov- holders with access to e-mail). A much smaller ernance code (the outcome of the Jordan Cor- share of in-country stakeholders who were in- porate Governance ROSC) and the Vietnam terviewed in the country reviews gave first rank Corporate Governance ROSC on their Web sites, to this mode of dissemination. which generated attention. In Peru, the Bank of- ficially presented the Policy Notes to the president The second largest number of stakeholders (around on television, which entailed two and a half hours 51 percent) selected conferences/seminars/work- live on two channels and video links to other shops as the most effective mode of dissemination. cities to allow for questions from the public. In Stakeholders indicated that when they do not have Vietnam, the Food Safety and Agricultural Health time to read reports (especially some of the Bank’s Action Plan (ESW) was disseminated through a tel- large ESW products), seminars or workshops are evision program on food safety and the quality of effective ways of dissemination. They also pro- life. In Bangladesh, some TTLs wrote op-ed pieces vide opportunities to ask questions and provide for newspapers. feedback. In-country stakeholders also made suggestions The third most selected mode of dissemination was for improving dissemination. In Bangladesh, some posting on the Bank’s external Web site (selected indicated that dissemination must go beyond a by 48 percent of in-country stakeholders). This was one-off launch or workshop, should be sustained followed by translation (selected by 22 percent). over a longer period, and should involve focus Table 5.6: Modes of Dissemination Selected as Most Useful, by Shares of Stakeholders (percent) Specific Specific ESW/TA Country ESW TA user Weighted Mode reviews survey survey survey average E-mail with link/attachment to report 31 90 75 80 64 Conferences/seminars 56 49 75 45 51 World Bank external Web site 38 46 40 55 48 Translation 29 23 — 19 22 Source: IEG surveys and country reviews. Note: ESW = economic and sector work; TA = technical assistance. 60 FA C T O R S I N F L U E N C I N G T H E E F F E C T I V E N E S S O F E S W A N D TA Table 5.7: Modes of Dissemination Selected as Most Useful, by Shares of IBRD and IDA Stakeholders (percent) Specific Specific ESW/TA Country ESW TA user Weighted reviews survey survey survey average Mode IBRD IDA IBRD IDA IBRD IDA IBRD IDA IBRD IDA E-mail with link or attachment to report 32 31 87 95 88 60 87 77 68 59 Conferences/seminars 59 53 67 55 50 80 49 53 53 56 World Bank external Web site 39 38 53 40 38 33 55 48 49 42 Translation 18 38 40 24 13 20 25 19 22 26 Source: IEG surveys and country reviews. Note: ESW = economic and sector work; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; TA = technical assistance. group meetings and discussions in informal settings continuous communication channel with the with various groups (grass roots, private sector, press and other stakeholders. academia, ministries, CSOs, and so on). They also indicated that the Bank’s Web site needed im- Translation and language ability provement to ease access to reports on Bangladesh. Although translation was rated the fourth most ef- fective mode of dissemination, it was cited as im- In-country stakeholders in Serbia made the same portant in a majority (8 of 12) of the countries suggestion, indicating that the Bank’s Web site reviewed (Bangladesh, Democratic Republic of needs a mechanism to announce the arrival of new Congo, Jordan, Mali, Peru, Romania, Serbia, and materials. The importance of being up to date was Vietnam). Stakeholders in these countries men- also mentioned with respect to distribution lists, tioned the need not only for translation but also in the context of the previously mentioned for better quality and quicker translation. They Caribbean newsletter, as well as in Peru, where even suggested directly writing ESW in the lan- stakeholders indicated that the Bank’s distribution guage of the country. In the Democratic lists need to be inclusive and current. Republic of Congo, translation was the Stakeholders in some most important improvement urged by countries identified a In two countries without Bank offices some in- in-country stakeholders, and in Vietnam need for quicker and country stakeholders indicated that the existence it was ranked as the second most useful better translations of ESW. of a World Bank publications office was not well means of dissemination (after work- advertised and that the location was not con- shops/ conferences). In the Democratic Repub- venient. Some stakeholders in Peru indicated that lic of Congo, in-country stakeholders indicated it is important to have a local partner for effective that translation took so long that when docu- dissemination (as the International Finance Cor- ments finally became available in French, many poration [IFC] has done with Doing Business). had lost their relevance to policy making. Finally, the media in Lesotho indicated that the The quality of translation also frequently left much Bank should establish good relations with the to be desired. This sentiment was echoed by a gov- media and have regular press conferences, even ernment official in Mali, who indicated that the if not very much is happening. A similar com- French version of the report was so bad that he ment was made by stakeholders in Mauritius, went back to the English version to figure out who indicated that press misinterpretation of the what the report was trying to say. The ESW with Bank’s messages had resulted in negative effects. the most impact in the Democratic Republic of To avoid this, the Bank should try to establish a Congo (forest sector review, the country status 61 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS report on education [World Bank 2005b], and the the criticism of the “one-size-fits-all” approach (such health country status report [World Bank 2005a]) as from stakeholders in Bangladesh and Peru). were all written directly in French by francophone Bank staff or consultants. The same criticism was voiced by some respon- dents in Mali, who indicated that the Bank has a Some also expressed a A counterpart in Peru commented that tendency to impose “off-the-shelf ” policies that preference for TTLs who the fact that reports were done in English Mali does not have the technical capacity to resist are fluent in the native indicated that they were really not writ- but that many Malians know are not going to language. ten for the country, but for the Bank. In work in practice (cotton sector privatization, rail- Bangladesh, in-country stakeholders in- way concession, and energy sector privatization). dicated that translations of summaries of key re- Some Bank TTLs also emphasized the impor- ports and their distribution through widely tance of providing policy options in their tasks, accessible outlets would be helpful. In Romania, which is not always encouraged by the Bank. a higher proportion of respondents in the gov- ernment than in civil society thought that trans- Regarding the ranking of efforts, TTLs generally lation was important. In Serbia, translation was ranked efforts exerted on providing a menu of pol- seen as very important. icy options in the middle—below efforts on tech- nical quality and partnership, but above timeliness Stakeholders also indicated the importance of and dissemination (table 5.3). This is more or less having TTLs that speak the native language flu- consistent with their perception of incentives in ently. Some of the most highly rated tasks in Jor- the Bank (tables 5.4 and 5.5). dan, the Poverty Institutional Framework TA and the AML/CFT Assessment, were led by Arabic- Adequate resources speaking TTLs. In the Democratic Republic of Adequate resources are needed for both the tasks Congo, donors noted that Bank communication themselves and for sustained follow-up. In skills were very poor, with many TTLs not speak- Malaysia, a government official gave a rating of 3.5 ing adequate French. (just average) to the International Best Practice Note for Financing Tertiary Education TA because Efforts and incentives the interaction with Bank consultants working TTLs ranked efforts on dissemination close to on the TA was not sustained enough. In Jordan, the bottom and translation at the very bottom of government officials indicated that sending some- the scale (table 5.3). Such low rankings are con- one in for a couple of days once in a while, as hap- sistent with the incentives TTLs perceive that pened with the programmatic PER, is not enough. they face in the Bank (tables 5.4 and 5.5). Some of the ESW and TA TTLs who responded to Other Relevant Dimensions the surveys also shared these views. Of the 320 In addition to technical quality, partnership, orig- TTLs who responded to the open-ended question ination, and dissemination, around which the of the most effective way for the Bank to change evaluative questions of this evaluation were for- staff incentives (to deliver a product to maximize mulated, other dimensions of ESW and TA also impact), 78 (around 25 percent) indicated that emerged as being important. budget and funding are important, with 26 of those 78 specifying budget for dissemination as im- Stakeholders and TTLs Menu of policy options portant. In the electronic survey of TA TTLs, of the emphasized the Stakeholders in a few of the countries in- 111 TTLs who responded to the same question for importance of providing dicated that presentation of policy op- TA, 22 (around 20 percent) indicated that budget policy options, which tions was important for enhancing the is important, with 4 of the 22 specifying budget for could help overcome effects of ESW and TA. Providing policy op- dissemination as important. criticism of the Bank’s tions would allow for the consideration of perceived “one-size-fits- political economy issues (as suggested by Underfunding of tasks more generally had forced all” approach. respondents in Serbia) and would address TTLs to leverage funding of ESW and TA tasks 62 FA C T O R S I N F L U E N C I N G T H E E F F E C T I V E N E S S O F E S W A N D TA with that of other activities (usually project prepa- they gave dissemination). In-country Sustained follow-up and ration and supervision, or with activities in other stakeholders also ranked timeliness next the tasks themselves countries) or to rely on trust funds. Both of these to the bottom with respect to the attrib- require adequate solutions are highly unsatisfactory. Leveraging utes they consider important for en- resources. often means that ESW was shunted into second hancing the effects of ESW and TA. This place in claims on the TTLs’ time and attention. does not necessarily mean that they considered Trust funds are very time consuming to manage, timeliness unimportant, just that it was not as even after the consolidation of all bilateral trust important as the higher-ranked attributes of tech- funds to mitigate the variety of individual donor nical quality, partnership, and relevance. In some requirements. Further, trust funds could also bias of the countries, poor timeliness in the delivery the selection of sectors to study (see chapter 2). of certain products was cited as the reason for their Also, they cannot fund Bank staff, thus possibly poor impact. missing out on considerable expertise (including from the network anchor units and the research TTLs have varying views on the importance of complex) and also potentially compromising the timeliness. A TA TTL working on Jordan indi- quality of the reports (there was some evidence cated that the Bank places too much emphasis on of this; see chapter 2). quality, at the cost of timeliness. This TTL said that “the best is the enemy of the good” and that the Adequate funding is also important for enhanc- Bank does not understand that there is a trade- ing the Bank’s agility in responding to clients’ off between timeliness and quality (that is, the last needs. For example, IFC-Latin America and the 4–5 percent of quality does not really matter, but Caribbean has resources earmarked for free TA, it comes at the expense of timeliness). with budget assured for five years, and allows sig- nificant discretion of their use in the field. As a re- However, some TTLs (in Jordan, Mali, Timeliness was generally sult, IFC can be more agile than the Bank in Mauritius, Romania, and Serbia) think seen as less important providing TA. This was found to be the case in Peru there is too much emphasis on timeli- than technical quality, (in the context of the country review), where the ness. They feel the emphasis is about partnership, and budget for Bank TA is controlled at Bank head- meeting internal Bank deadlines (fiscal relevance, although in quarters, the funds are limited, and the govern- year bean counting) rather than client pri- some instances the lack of ment finds it easier to go to other donors for TA orities, sometimes even at the expense timeliness undermined than through the Bank. of quality. the effects of ESW. Agility is important because when countries are Shorter pieces confronted with specific problems and ask the Government officials in Bangladesh and the Dem- Bank for assistance, a quick response shows that ocratic Republic of Congo indicated that they the Bank is relevant. That can generate demand preferred shorter pieces. These are also the coun- for further business. Mauritius provides an ex- tries that gave some of the lowest ratings to Bank ample of just-in-time TA that had great effects: ESW and TA overall in terms of their achievement the Aid for Trade TA. According to Bank staff, this of objectives. is an example of “exactly what to do in MICS, just- in-time-support, cutting edge, and done by top In Bangladesh, government officials indicated that people.” When budget constraints affect agility, it shorter, focused policy notes with concrete rec- limits the extent to which the Bank can respond ommendations for policy were more useful. In to requests, and business opportunities are lost. the Democratic Republic of Congo, there was a de- sire not only for shorter pieces but also for greater Timeliness reliance on dialogue than on reports. A focus on In all the surveys, in-country stakeholders rated dialogue in the case of the forest sector review was timeliness about the same as dissemination (the probably a key factor in its influence. A donor in exception is in the country reviews, where stake- Mali indicated that Bank reports should be shorter holders gave this significantly higher ratings than and more quickly available in French, which would 63 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS In a few countries the also reduce delays involved in translation work more with other ministries (not just the Min- Bank was criticized for and dissemination. istry of Finance). In Mauritius, for instance, the Min- not working enough with istry of Finance was much more positive about ministries other than the TTLs in the Democratic Republic of the policy impact than sectoral ministries, which Ministry of Finance. Congo, Mauritius, and Serbia indicated were not as familiar with the Bank’s ESW . This was preferences for shorter pieces. TTLs work- partly because there was no country office in Mau- ing on the Democratic Republic of Congo indi- ritius at the time, although a small one has now cate that there is too much emphasis on producing been established. lengthy documents that contain sophisticated analyses and too little on shorter policy notes The neglect of other ministries relative to the geared to the practical needs of senior policy mak- Ministry of Finance was also found in Mali, where ers. Echoing this sentiment is a TTL working on capacity building efforts (with respect to public Mauritius, who indicated that the product should financial management) need to be extended to be broken up into short policy notes, focusing on sectoral ministries. In Romania, one stakeholder what the government wants instead of big pol- suggested that in preparing ESW , Bank staff, when ished reports. This TTL noted that Bank internal requesting data, should work with the various processes are not geared to this. In general, TTLs government agencies directly rather than work- ranked this dimension very low with respect to ef- ing through the Ministry of Finance, which at fort (table 5.3). times was treated by the Bank staff as a secretariat of the Bank. Working more with sectoral ministries In a few of the countries reviewed, in-country stakeholders indicated that the Bank needs to 64 Chapter 6 In Peru, the World Bank report “Restoring the Multiple Pillars of Old Age Income Security” influenced reforms in the pension system. Photo by Scott Wallace, courtesy of the World Bank Photo Library. Main Findings and Recommendations Effectiveness of ESW and TA verall, World Bank ESW and TA delivered to clients during fiscal O 2000–06 met their stated objectives of informing lending, informing/ assisting in the implementation of policy, building capacity/ institutions, stimulating public debate, and facilitating knowledge exchange. On the whole, users of ESW and TA rated achievement of these objectives average or higher. The extent to which ESW and met their objectives in client countries ranged TA objectives were met from substantially above average to substantially Among the stated objectives, those pertaining to below average. The differences in the effects of Bank activities received higher ratings than those ESW and TA between countries could be attrib- that directly affected client countries. Specifically, uted partly to country-specific factors and partly the objective of informing lending was met to a to other factors that the Bank could address. greater extent, on average, than the other objec- tives. Interestingly, users of ESW gave the highest Factors in ESW and TA effectiveness rating to informing Bank strategy, although this A country-specific factor that seemed to have had was not even one of the stated objectives of Bank a bearing on the effects of ESW and TA in client ESW . The evaluation also found that TA met the countries was the level of government capacity. ESW objectives of informing Bank lending and in- This was the case for the postconflict Democratic fluencing the development community, neither of Republic of Congo, where limited government ca- which were stated objectives of TA. pacity seemed to have negatively affected the ex- tent to which ESW and TA influenced policy and ESW and TA also had other effects besides the built capacity. stated objectives. The more prominent effects were on countries other than those for which In Bangladesh, Guyana, and Lesotho, in-country they were intended, on private investment deci- stakeholders indicated that the capacity level in sions by domestic and international investors, on the countries was too low for them to benefit support for local research, and on the creation of from ESW and TA. High turnover of government informal networks between Bank staff and policy officials (including ministers) in Bangladesh, Jor- makers. dan, and Peru (in the sector ministries in the last) negatively affected the extent to which ESW and Although the various ESW and TA objectives were TA influenced policy or built capacity. Additionally, met on average, they were met to different extents political and other constraints were found to have in different countries. For the 12 countries re- affected the receptivity of governments to Bank viewed in depth, the extent to which ESW and TA advice stemming from ESW and TA. 67 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS At the same time, some country-specific factors pieces still need to be underpinned by rigorous had a positive influence on the effectiveness of analysis. ESW and TA. For instance, Romania’s transition from a closed to an open society seemed to have How to enhance effectiveness been an important factor behind the high ratings Technical quality. Maintaining and striving for for almost all the objectives. Romania appreci- high technical quality of Bank ESW and TA is im- ated Bank involvement after its opening up. In par- portant, given the correlation between quality ticular, it appreciated Bank assistance in the EU and effectiveness of these products. Further, in accession process, which provided Romania with light of the correlation between cost and quality more room to maneuver by making its dialogue and the fact that ESW is more poorly resourced with the EU less one sided. Stakeholders also in IDA than in IBRD countries (chapter 2), there appreciated the opportunity to debate issues is a need for better-resourced ESW for IDA coun- publicly after Romania’s transition and gave high tries. This may need to be accompanied by greater ratings on stimulating public debate. selectivity of ESW tasks for IDA countries to reduce the burden on the more limited government ca- In addition to country-specific factors, other fac- pacity in some of them and to free resources for tors pertaining to Bank processes and corporate the greater collaboration and follow-up needed incentives had a bearing on the effectiveness of to enhance the effects of ESW . Bank ESW and TA in client countries. Specifically, the technical quality of the product had a posi- Partnership with clients. Closer collaboration tive influence on the extent to which ESW and TA with Bank clients, from task initiation through objectives were met. Close collaboration with the formulation of conclusions and recommen- clients from task initiation through the formula- dations, is important. This may or may not entail tion of conclusions and recommendations was im- actual client involvement in producing parts of the portant, whether clients actually produced part of task. In countries with higher levels of capacity, the task or not. Whether clients requested the task partnership in production should be the norm, did not generally make a significant difference to as there is capacity to draw on.1 In countries with ESW and TA achieving objectives, although the capacity constraints, joint production in narrowly clients still needed to be genuinely interested in targeted short pieces of work tailored to the prac- the tasks for them to be effective. tical needs of policy makers has been found to help build capacity. One of the most important factors for enhancing the effects of ESW was sustained follow-up that Consideration should also be given to the com- went beyond one-off dissemination events. That plications arising from whether civil servants have follow-up could be in the form of lending (by the the time to be involved in drafting reports and how Bank or other donors) or nonlending TA (formal they could be compensated for the tasks. In ad- or informal) and typically lasted for some period dition to time and budget, closer collaboration also after the delivery of the ESW and TA to clients. requires that the Bank recognize such efforts, which can be accomplished through client feed- Dissemination to the right audience was some- back on the task. times more important than wide dissemination for ESW and TA to be effective. Translation is im- Origination. In-country stakeholders in most of portant to client countries, and having Bank staff the countries reviewed indicated the benefits of and consultants that speak the local language en- wide consultation—both inside and outside the hanced the effectiveness of these tasks. Clients in government—in determining the topics for ESW two capacity-constrained countries indicated a (and to a lesser extent for TA, because these are preference for shorter pieces with concrete pol- mostly client requested). Consultation with civil so- icy recommendations, or even the use of dia- ciety could be useful where there are tough areas logue rather than reports, although such short of reforms and where such reforms could only be 68 M A I N F I N D I N G S A N D R E C O M M E N D AT I O N S sustainable if there were demand for them from thought of as an implementation strategy, although within the country. Greater interaction will also dissemination is certainly a part of implementation. help improve the image of the Bank, such as in Bangladesh, where the Bank’s image is negative. The implementation strategy should be clearly spelled out, with all the intended activities bud- Wide consultation could also help in the identi- geted for and incorporated into the concept paper fication of local stakeholders who are knowl- as an integral part of the task. The starting point edgeable about the needs and priorities of the of the strategy would be to determine the in- country, as well as the reforms that could be im- tended audience of the task by taking into account plemented at any given point in time, such as in the stated task objective. For tasks that are mainly Romania. Consultation with donors at the outset intended to inform Bank activities (lending and of a task could enhance synergies and avoid du- strategy), the target audience would clearly be in- plication. In some countries, governments indi- ternal. For tasks that are mainly directed toward cated that ESW could be more likely to make a the client countries, in some instances a stake- difference if donors were involved from the start.2 holder analysis may be useful for determining the target audience. For the latter tasks, the im- At the same time, ESW and TA do not necessar- plementation strategy would also need to spell out ily have to be requested by the client to have ef- the mode of dissemination, as well as the in- fects. In fact, in-country stakeholders in some of tended follow-up, whether lending or nonlend- the countries indicated that the Bank has a use- ing, formal or informal. The strategy should also ful role in proposing topics, as the Bank can draw include translation, where applicable. on its own knowledge and international experi- ence to analyze and propose a vision for actions Dissemination should include to those in the that the country needs to take. country who have opposite views. A case could be made for the dissemination to be managed by When the Bank proposes tasks, however, it still the country unit, which would ensure that dis- needs to ascertain first that there is genuine re- semination is coordinated at the Regional and ceptivity on the part of the government and sec- institutional levels to avoid saturating the public ond that the tasks are rooted in the country’s and to coordinate messages from different parts priorities. This could be done by tailoring tasks to of the Bank. Given the different considerations dis- the needs and interests of the clients and to sub- cussed in chapter 5 with respect to how broad or stantial collaborative efforts. narrow dissemination should be, it seems ap- propriate to leave such decisions to the country For countries with capacity constraints, where teams (led by the country director); they are best government officials do not have time to read placed to determine the political economy as- much of the Bank ESW produced, greater selec- pects pertaining to the particular tasks. tivity could be useful. This could be done by giv- ing priority to ESW that is needed to inform Adequate resources. Many of the suggestions lending and strategy and that is requested and proposed in the preceding paragraphs require needed by clients. This is particularly important funding. Genuine collaboration with clients and sus- where there is a Bank budget constraint. tained follow-up all require adequate staff time and hence resources. Adequate resources are also Dissemination and follow-up. It is not dis- needed for the tasks themselves, especially in Sub- semination per se but sustained follow-up for Saharan Africa, which has the poorest resourced some period (from six months to a year, for in- country-specific ESW (as discussed in chapter 2), stance) after the delivery of the tasks to clients that as well as the tasks of lowest rated quality. is important for achieving ESW and TA objectives in client countries. In this light, a dissemination Adequate funding is also important for enhanc- strategy could perhaps be more appropriately ing the agility of the Bank’s response to clients’ 69 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS needs, especially for rapid-response TA. Related First, there is a strong preference for nonlending to this is the need to devolve greater decision- over lending activities in IBRD countries. The making responsibility and budget flexibility to preference is more pronounced if only new lend- the country offices, because they are the frontline ing activities are taken into consideration (that is, recipients of such government requests. if supervision of existing loans is excluded). Substantive presence of ESW and TA task Second, the preference for nonlending over lend- teams in country offices. To the extent al- ing (or grant) activities is not as strong or as un- lowed by budget, the Bank should try to ensure ambiguous in IDA countries. Third, TA is more substantive task team presence in country of- important than ESW in both IBRD and IDA coun- fices. This will facilitate closer collaboration with tries. It is particularly striking that among those clients in the production of the task (especially TA) who are familiar with specific ESW tasks of the as well as follow-up. Bank, a much larger share gave first ranking to TA than to ESW . What Do Clients Want from the Bank? In-country stakeholders were asked to rank the The stronger preference for nonlending over lend- importance of the various Bank activities for their ing (and grant) activities in IBRD than in IDA country (figure 6.1). Three observations emerged countries came through clearly in the country re- from their responses. views. MICs value advice from the Bank, and some of them borrow from the Bank to obtain this ad- vice (through dialogue in the context of loan preparation). For instance, some government of- ficials in Mauritius indicated that they are bor- Figure 6.1: First-Ranked Bank Activity with Respect rowing from the Bank only because they want to Importance to Clients, by Shares of IBRD and IDA access to Bank technical support to implement re- In-Country Stakeholders forms but do not want to pay outright for the Bank’s AAA. In Romania, where accession to the 100 EU has obviated the need for external borrowing 90 21 26 31 (including from the Bank), there is a clear view 80 40 among in-country stakeholders that Bank ESW 53 55 57 (and to a lesser extent TA) should be de-linked 70 18 24 from the Bank’s lending operations. The same 60 88 26 view was expressed in Jordan and Serbia. Percent 50 13 29 40 13 7 26 In contrast, some IDA countries need the Bank’s 20 30 19 funds. In the Democratic Republic of Congo, for 12 7 21 example, where there are huge needs, especially 20 43 33 in infrastructure, all stakeholder groups (inside and 28 24 10 18 20 14 outside the Bank and the government) indicate 13 0 that funding is most important for the country. IBRD IDA IBRD IDA IBRD IDA IBRD IDA Country reviews Specific ESW Specific TA ESW/TA The preference for TA over ESW also came user survey user survey user survey through clearly in the country reviews, from both Response source higher- and lower-capacity countries. For the TA ESW Supervision Lending countries with higher capacity, such as Malaysia, TA is appreciated more than ESW because they can Source: IEG surveys and country reviews. Note: Totals may not equal 100 percent because of rounding. ESW = economic and sector work; IBRD = In- see the results quickly; they demand TA accord- ternational Bank for Reconstruction and Development; IDA = International Development Association; TA = ing to their needs, so the TA is tailored to their technical assistance. needs.3 In Jordan, government officials indicated 70 M A I N F I N D I N G S A N D R E C O M M E N D AT I O N S that they request Bank TA because Jordan needs Figure 6.2: First-Ranked Institutions with Respect help with the “nuts and bolts” of implementa- to Usefulness of Reports and TA, by Shares of tion—it is not what to do that they need advice In-Country Stakeholders on, but how to do it. The authorities in Malaysia and Mauritius found Bank TA so useful that they 100 paid for it. 90 80 9.0 7.0 At the same time, for countries such as Guyana, 9.4 TA is preferred because the country lacks capac- 70 14.9 25.3 ity. The urgent need to build capacity was also the 60 Percent reason TA was ranked second, after funding, in the 50 Democratic Republic of Congo. 28.4 22.1 40 68.8 Clients’ views of usefulness of reports 30 and TA by the Bank versus other institutions 20 32.8 31.3 In the three user surveys conducted for this eval- 10 uation, in-country stakeholders were asked to rank the usefulness of Bank reports and TA com- 0 pared with those provided by other institutions. Specific ESW Specific TA ESW/TA user survey user survey user survey Two observations emerged from the responses. Response source First, more in-country stakeholders gave first Nongovernmental Private Regional development bank rank to reports and TA produced by the Bank than organization to those produced by other institutions. Second, Bilateral aid agency Other multilateral IMF more in-country stakeholders gave first rank to and regional TA than to ESW produced by the Bank (see rank- Local think-tank Government World Bank ing from specific ESW survey and specific TA sur- university institution vey in figure 6.2). Source: IEG surveys. Note: ESW = economic and sector work; IMF = International Monetary Fund; NGO = nongovernmental or- ganization; TA = technical assistance. Types of products A range of ESW and TA was found to have effects. These include tasks that were not requested by indicated that two kinds of assistance are needed clients yet have important effects, in particular from the Bank—just-in-time TA and long-term some corporate and IDA-mandated tasks and engagement in key areas of reform; he cited the core diagnostics. Among these, CEMs and PERs PESW as a good instrument for the latter. were cited most often as having informed the Bank’s CASs and DPLs. In-country stakeholders es- TTLs echoed this sentiment; they see the PESW pecially appreciated the FSAP and the associated as allowing for deeper analysis and partnerships ROSCs, the ICA, and Doing Business for their with clients that can enhance effects in the coun- benchmarking functions. try. They also see the PESW as allowing for the pro- duction of intermediate inputs and more flexibility A relatively new instrument, the programmatic with the delivery calendar. The Bank’s adminis- ESW (PESW), was cited by some countries as use- trative database does not currently allow for the ful. These are multiyear, multiactivity (a combi- tracking of this type of task, so it is difficult to as- nation of analytical work and TA) tasks that allow certain how important they are. objectives and activities to be reset during the course of the task. In Jordan, the PESW on pen- Finally, policy notes provide an alternative to re- sions is providing inputs for the reform of the so- ports for just-in-time advice to governments. cial security law. In Peru, a government official These have increased as a share of all ESW (from 71 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS 14 to 21 percent over the evaluation period) and terparts, they still received above average ratings are useful instruments for clients (such as Bangla- for usefulness. desh and the Democratic Republic of Congo) that have indicated their preference for shorter Among the countries reviewed, in-country stake- pieces of analytical work. By and large, there were holders in the Democratic Republic of Congo no significant differences between policy notes were particularly skeptical of the practical value and other ESW with respect to their various di- of regional and global studies for their country. mensions (technical quality, relevance, timeli- Contrary to all the other countries, stakeholders ness, partnership, or dissemination) or the extent in Romania ranked regional over country-specific to which they met various objectives.4 ESW in terms of usefulness. This is because they perceived that Romania, as a transition country, In two countries, Peru and Serbia, policy notes— could learn from other transition countries. In- mainly syntheses of existing analytical work— country stakeholders in Serbia, another transi- were produced during the evaluation period with tion country, also appreciated regional tasks that the specific intent of informing incoming gov- compared Serbia to best practices in other Balkan ernments on a wide range of economic and sec- countries. toral issues. In Peru, this was the Policy Notes, which most officials were aware of; some had In addition to transition country issues, the country read chapters that were of interest to them. Al- reviews found a variety of other issues that were though no specific policy changes could be iden- particularly amenable to being addressed tified as having originated from the Policy Notes, regionally: transport issues for landlocked coun- the general view (with some exceptions) was that tries (Lesotho and Mali), HIV/AIDS and climate the document was useful as a general reference change (Guyana), environment (Lesotho), for- for orienting the new government’s economic estry (Democratic Republic of Congo), regional and social policies. In Serbia, policy notes for the trade (Lesotho, Mauritius, and Peru), electricity new government were cited as having kept the (Lesotho), water sharing (Bangladesh), and anti- government on a reform path. money laundering (Guyana). For small countries (such as Guyana and Lesotho), regional studies also Country, regional, and global ESW and TA make sense for reasons of economies of scale.5 In-country stakeholders in all but 1 of the 12 countries ranked country-specific over regional Global ESW on certain issues was cited as useful and global ESW and TA in terms of importance to by in-country stakeholders in the country reviews, their countries (the exception is Romania). This including issues with a global dimension (climate ranking is validated by responses to the various change in Bangladesh, water and forestry in surveys of in-country stakeholders (see table 6.1). Guyana). For countries pursuing a globalization Although regional and global ESW and TA were strategy, global ESW, such as Doing Business, is ranked lower than their country-specific coun- important for understanding global trends (Mau- Table 6.1: Average Ratings of In-Country Stakeholders on the Usefulness of Country, Regional, and Global ESW and TA From specific From specific From ESW/TA ESW user survey TA user survey user survey Country 5.4 5.0 4.8 Regional 4.5 4.6 4.5 Global 4.3 4.0 4.6 Source: IEG survey. Note: The rating scale is from 1 to 6, with 1 denoting “no extent” and 6 denoting “great extent.” ESW = economic and sector work; TA = technical assistance. 72 M A I N F I N D I N G S A N D R E C O M M E N D AT I O N S ritius) or for forcing changes in the country because ally offered a prediction of the extent to which in- of global comparisons (Guyana). In-country stake- ternational integration would have been retarded holders also noted that global ESW pertaining to without these Bank products (Romania would learning new mechanisms, such as micro-insurance have joined the EU by 2014, together with Ukraine, and micropensions, are useful. and would have joined the North Atlantic Treaty Organization by 2020). What Would Have Happened Without Bank ESW and TA: Clients’ Views In some instances, in-country stakeholders indi- For the country reviews, in-country stakeholders cated that Bank ESW and TA were critical for the were asked the counterfactual question, What reforms in their countries. Representatives from would have happened if the Bank had not un- the civil society, private sector, and academia in dertaken ESW and TA for their countries? Guyana said outright that if the Bank had not un- dertaken the ESW , no other institution would have In 10 of the 12 countries, a majority of the re- done it, and the changes that resulted would not spondents indicated that Bank ESW and TA had have happened. In Mali, respondents indicated that made a difference to the country’s reform pro- without the CFAA, the CPAR, and the PER TA, it gram. This view was expressed by respondents in would not have been possible to carry out the pub- countries with a wide range of income, capacity, lic financial management reforms. In Mauritius, one and policy orientation. In Malaysia (among the donor said that the country’s reforms “would not richest of the countries reviewed), 14 of 17 re- have happened at all.” In Peru, a respondent in- spondents indicated that Bank involvement had dicated that reforms in the education sector would improved things over what they would otherwise not have happened at all. have been. In Bangladesh (among the poorest of the countries reviewed), the typical response to Respondents in some countries indicated that the question—even from those who disagreed Bank ESW helped their reforms be more com- strongly with the Bank—was that there would prehensive and less piecemeal (Jordan, Lesotho, have been an intellectual void. Malaysia, and Peru). Respondents in some coun- tries also indicated that Bank ESW and TA helped In the Democratic Republic of Congo and Jor- reduce the cost of policy mistakes. In Romania, dan, the responses were not quite as unequivo- some respondents indicated that the Bank, hav- cal. The majority of those who responded to this ing accumulated a lot of international experience question (13 of 18 stakeholders) in the Democratic regarding best practices, was in a good position Republic of Congo indicated that it would have to advise the country on how to avoid costly mis- made little or no difference; the majority of those takes. Similar sentiments—of Bank advice help- in Jordan (14 of 20) indicated that it “would not ing to reduce costly policy mistakes—were echoed have been a big issue” or “would not have stopped by respondents in Bangladesh, Serbia, and Viet- reforms” or that the country “will survive” or nam. Having the support of Bank analysis and “would have managed eventually.” However, 13 of advice also provided a level of comfort or con- those 14 in Jordan also indicated that Bank ESW fidence for countries to move forward. This was and TA had helped; comments included that they cited by respondents in Mauritius and Malaysia. “benefited from it” and that Bank ESW and TA helped accelerate reforms and improve the qual- In some countries, respondents indicated that ity of reforms. Bank ESW lent credibility to their reforms. In Peru, for example, government officials indicated For the 10 countries that attributed Bank ESW and that they used arguments presented in Bank stud- TA with making a difference to their reform pro- ies to bolster their positions and to persuade oth- grams, many credited the Bank products for hav- ers. In Romania, respondents noted that, given ing accelerated and improved the quality of Romania’s recent political history, the authori- reforms. A government official in Romania actu- ties (including those who were democratically 73 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS elected) lacked credibility with their own popu- The poor quality of information extends to the re- lation. The involvement of the Bank, which has a sults tracking framework itself. The framework very good reputation there, in the government’s consists of the Activity Initiation Summary and the initiatives helped bridge the government’s cred- Activity Completion Summary that TTLs have to ibility gap with the population. fill out for each task. TTLs indicate the objectives for the tasks in the former and the extent to The key characteristics of Bank ESW and TA val- which the objectives were met in the latter. Al- ued by in-country stakeholders were high tech- though all TTLs fill out the former summary, not nical quality, independence, neutrality and all fill out the latter. Since fiscal 2005, when it be- objectivity, and provision of international per- came mandatory to follow all ESW and TA with a spectives. In Malaysia, it was pointed out that the completed Activity Completion Summary (the high technical quality was in part due to the guideline is that this completion needs to happen Bank’s ability to draw from a large pool of ex- within six months of delivery to clients), this only pertise, from both inside and outside the Bank. happened for around two-thirds of the ESW and around 62 percent of the TA tasks, the majority Regarding neutrality, one example was provided of which were completed after six months of by respondents in Romania. They indicated that delivery. the Bank’s neutrality depoliticized the process of addressing corruption. Neutrality of Bank ad- Another weakness is the absence of systematic vice was cited by bilateral donors in Guyana as one feedback from clients on ESW and TA after their reason the government is more likely to take ad- completion. The exception is the FSAP , which is vice from the Bank than from other donors, who followed by a questionnaire to clients on various tend to have specific interests. Neutrality was also aspects of the tasks, including their views on qual- cited as useful in a politically divided country ity and their expectations of usage of the report. such as Guyana. A high-level government official in Jordan cited neutrality/objectivity as among Conclusions and Recommendations the key attributes of Bank ESW and TA that are val- for the Way Forward ued, compared to analysis and advice that could The Bank’s ESW and TA are found to be useful to be purchased from the private sector. different extents by different stakeholders. Within the Bank, ESW is particularly useful for informing The Bank’s Results Tracking Framework Bank strategies. ESW is also useful for informing for ESW and TA lending and is associated with (certain aspects of) Although the Bank has a results tracking framework the quality of lending. In the Bank’s client coun- for reporting and monitoring ESW and TA activi- tries, there is a stronger preference in IBRD coun- ties, the implementation of the framework is tries for the Bank’s nonlending activities (ESW and poor. This is reflected in the poor quality of the TA) than for its lending activities; the preference information on ESW and TA in the Bank’s admin- is not as strong or as clear in IDA countries. Both istrative database.6 Around 13 percent of ESW and groups of countries find TA more useful than nearly 20 percent of TA delivered during fiscal ESW . In-country stakeholders generally find reports 2000–06 was misclassified with respect to sector. and TA produced by the Bank more useful than Nearly one-third of the ESW delivered during the those produced by other institutions, particularly period was not assigned any report type. The poor for TA. information in the Bank’s database reflects the poor monitoring and reporting of these tasks in In light of these findings, the Bank’s current focus the Bank. If there were proper monitoring and re- on knowledge and learning as one of the six strate- porting—that is, if the information were actually gic directions is well placed. Some recommenda- used for such purposes—then Bank staff would tions have emerged from the findings for enhancing have an incentive to enter accurate information. the effects of Bank ESW and TA, which would also 74 M A I N F I N D I N G S A N D R E C O M M E N D AT I O N S support the knowledge and learning and other team presence in country offices, particularly strategic directions. They are as follows. in countries with low institutional capacity. This will facilitate closer collaboration with 1. Reinvigorate the mandate—which un- clients, from task initiation through follow-up. derpinned the fiscal 1999 ESW reforms— In addition, formulate a dissemination and im- for country teams to maintain a strong plementation strategy for ESW and TA at the knowledge base for countries and sectors concept paper stage. Such a strategy should where the Bank is providing or planning identify the target audience, the mode of dis- to provide funds. Bank country strategies semination, and the follow-up arrangements and lending activities need to continue to be after dissemination, all of which should be ex- supported by requisite analysis, although a plicitly budgeted for as integral parts of the task. return to strictly defined “core diagnostic” ESW is unnecessary. 4. Recognize and build on client prefer- ences, whether for nonlending over lend- 2. Ensure that ESW tasks in IDA countries ing services or for TA over ESW. Institute are adequately resourced, even if it a mechanism to obtain client feedback on a pe- means fewer ESW in some countries. This riodic basis on delivered ESW and TA products. will help to address the lower level of re- Such feedback should include clients’ views on sources for individual ESW tasks in IDA coun- collaboration, follow-up, and usefulness of the tries than in IBRD countries and is supported tasks (including specifics of how the tasks were by the findings that cost matters for quality and used). The client feedback should be requested quality matters for effectiveness. Greater se- at a set period (for instance, around 1 year) after lectivity will also help reduce the burden on the delivery of the task to the client to allow limited government capacity in some IDA time for follow-up, and it should be the last countries and free resources for more collab- milestone for ESW and TA. Obtaining client oration and follow-up needed to enhance the feedback would encourage a stronger results effectiveness of ESW . Selectivity could be en- focus for ESW and TA and help counterbal- hanced by giving priority to ESW that informs ance current Bank incentives for lending over Bank lending and strategy or that is clearly de- nonlending and for ESW over TA. sired and needed by the client. 5. Finally, take the results tracking frame- 3. Enhance the institutional arrangements work seriously, including by incorporating for undertaking ESW and TA. To the extent systematic client feedback as noted in the allowed by budget, ensure substantive task above recommendation. 75 Appendixes Malaysia dock. The Investment Climate Assessment on Malaysia informed the government on issues pertaining to the private sector. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. APPENDIX A: ESW AND TA TYPES ESW Report Types Commodities Study Debt and Creditworthiness Study Diagnostic reports Economic Updates and Modeling Core diagnostic reports Energy Study Country Economic Memorandum/ Foreign Trade, Foreign Direct Investment, Development Policy Review and Capital Flows Study Country Financial Accountability Assessment General Economy, Macroeconomics, Country Procurement Assessment Review and Growth Study Poverty Assessment Law and Justice Study Public Expenditure Review Mining/Oil and Gas Integrative Fiduciary Assessment Other Agricultural Study Other Education Study Other diagnostic reports Other Environmental Study Accounting and Auditing Assessment (Report Other Financial Accountability Study on the Observance of Standards and Codes) Other Financial Sector Study City Development Strategy Other Health Study Corporate Governance Assessment (Report Other Infrastructure Study on the Observance of Standards and Codes) Other Poverty Study (Non-PORs) Country Environmental Analysis Other Procurement Study Country Gender Assessment Other Public Sector Study Country Infrastructure Framework Other Rural Study Education Sector Review Other Social Protection Study Energy-Environment Review Other Urban Study Financial Sector Assessment Program PSD, Privatization and Industrial Policy Health Sector Review Public Environmental Expenditure Institutional and Governance Review Public Investment Review Insolvency Assessment (Report on the Strategic Environmental Assessment/Analysis Observance of Standards and Codes) Women in Development and Gender Study Investment Climate Assessment Knowledge Economy Study TA Output Types Legal and Judicial Sector Assessment • Institutional Development Plan—Advice on Recent Economic Development client-owned blueprint for institutional in Infrastructure strengthening, ranging from broad-based strate- Risk and Vulnerability Assessment gies to fully implementable plans, which could Rural Development Assessment include the sequencing of activities, resource Social Analysis allocation, key issues, objectives, and other ac- tion items Advisory reports Anti-Money Laundering and Combating the • “How-To” Guidance—Operational advice Financing of Terrorism in policy/program formulation and hands-on 79 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS implementation, including technical notes, step- include draft legislation, regulation, Poverty by-step instructions, “best practice” manuals, and Reduction Strategy Paper inputs, and grant procedural guidelines proposals • Model/Survey—Advice for client-driven data • Knowledge-Sharing Forums—Bank-facili- collection and/or analysis used for projections, tated, client-managed meetings aimed at shar- forecasting, and simulations ing international “best practice,” exchanging knowledge, catalyzing reform, and building • Client Document Review—Advice and re- consensus on a particular issue. view of client-owned documents, which could 80 APPENDIX B: METHODOLOGY Country Reviews that evaluation findings that emerge from the Twelve countries were selected for in-depth re- country reviews would have controlled for dif- views. The in-depth reviews entailed structured ferences between countries arising from these interviews of Bank staff, Bank management, and criteria. To reduce “evaluation fatigue,” the In- in-country stakeholders in the field (government dependent Evaluation Group (IEG) sought to officials and representatives from the private sec- minimize overlap with countries reviewed in re- tor, civil society, academia, the media, and the cent or ongoing IEG thematic or country evalu- legislature). The structured interviews sought ations, as well as overlap with countries covered stakeholder views (with quantitative ratings) on under the two phases of Quality Assurance Group various dimensions of a portfolio of Bank eco- (QAG) country analytical and advisory activity nomic and sector work (ESW) and technical as- (AAA) assessments. sistance (TA) (quality, partnership, relevance, dissemination, timeliness, origination, and so on) Portfolio selection and the extent to which these tasks met various For countries with large ESW and TA programs, objectives (lending, policy, institution and ca- which was the case for 7 of the 12 countries pacity building, public debate, donor activities, and (Bangladesh, Jordan, Mali, Peru, Romania, Ser- knowledge exchange). bia, and Vietnam), the sectors in which ESW and TA were concentrated were selected for review: The interviews also sought the views of stake- economic policy, financial sector, private sector de- holders on the relative importance (with rank- velopment (PSD), public sector governance, and ings) of the following for the country concerned: rural sector.1 These are also the top sectors for all (1) the different Bank instruments (lending and country-specific ESW delivered by the Bank dur- nonlending), (2) the different types of Bank ing fiscal 2000–06 (and are also among the top sec- ESW/TA (country, regional, and global), and (3) the tors for all country-specific TA except the rural various dimensions of ESW and TA. The country sector; see chapter 2). reviews also included desk reviews of ESW , docu- ments on TA and loans, Country Assistance Strate- For those countries for which Bank activities in se- gies, and poverty-reduction strategies. The country lected sectors were still too numerous for a de- reviews were conducted between January 2007 and tailed review, there was a further narrowing of ESW, July 2007. The interview questionnaires are avail- TA, and loans selected for review. This was the case able on request. for Bangladesh, Peru, and Vietnam. In instances where the Bank’s work in certain sectors had Country selection been particularly important, those sectors were The 12 countries were selected to be broadly rep- also included for review. This was the case of the resentative of Bank clients with respect to Re- social protection sector both in Jordan and Peru: gion, income, population, Country Policy and the Bank did substantial work on this sector in Institutional Assessment, size of lending (per Jordan, and the sector was featured as a central capita), and share of ESW cost in the cost of coun- element of the Country Assistance Strategy in try services (see tables B.1 and B.2). This means Peru. For the other six countries (Democratic 81 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS The evaluation of these Bank tasks (ESW , TA, and Table B.1: Countries Selected for Review loans) entailed conducting structured interviews with 568 stakeholders (table B.3 presents the Region IBRD IDA Blenda composition of the stakeholders). Sub-Saharan Africa Mauritius Democratic Republic of Congo, Lesotho, Mali Surveys of ESW and TA Task Team East Asia and Pacific Malaysia Vietnam Leaders Europe and All task team leaders (TTLs) of ESW and TA during Central Asia Romania Serbiab fiscal 2000–06 were surveyed (electronically) on a Latin America and specific task that they had undertaken; this meant the Caribbean Peru Guyana 1,258 ESW TTLs and 499 TA TTLs. TTLs who man- Middle East and aged more than one task were surveyed on only North Africa Jordan — one of the tasks (selected on a random basis). The South Asia — Bangladesh surveys were launched in January 2007 and lasted Notes: IBRD, IDA, and blend designation as of fiscal 2006. IBRD = International Bank for Reconstruction and Development; IDA = International Development Association. for three weeks. See table B.4 for survey results. a. Blend countries are those that are eligible for IDA resources on the basis of per capita income but also have limited creditworthiness to borrow from IBRD. The surveys sought information on various di- b. Serbia was Serbia and Montenegro from 2003 and became independent on June 5, 2006. mensions of ESW and TA (origination, partnership, and dissemination) as well as the perceptions Republic of Congo, Guyana, Lesotho, Malaysia, (with quantitative ratings) of TTLs on the extent Mali, and Mauritius), the entire portfolio of AAA to which the various objectives of ESW and TA and loans was reviewed. were achieved. The surveys also sought the views of TTLs on the relative importance of the various Based on the selection criteria indicated above, dimensions for the effectiveness of ESW and TA the IEG team evaluated 129 ESW and 64 TA tasks, and their views on the incentives provided by as well as 88 loans that could have been informed the Bank that could affect the various dimen- by the ESW evaluated (that is, the loans were in sions. The survey questionnaires are available on the same sectors as the ESW and were approved request. either in the same fiscal year as or after the com- pletion of the ESW). Bias check for survey responses One of the most important potential biases in The sectoral distributions of the ESW and TA that survey responses would stem from TTLs re- were evaluated were largely in line with the sec- sponding to the survey only if they have positive toral distributions of all such (country-specific) perceptions of the effects of their tasks. To check Bank tasks delivered during the period (figures B.1 for this bias, the IEG team divided the TTLs into and B.2). two groups: those who responded to the survey Table B.2: Size of Lending and Budget for ESW for Selected Countries Share of ESW cost in the Large loans per capita Medium loans per Small loans per cost of country services ($63–$400) capita ($34–$63) capita ($0–$34) Large (0.2–1.00) Serbia Peru Mauritius, Malaysia Medium (0.14–0.19) Jordan Lesotho, Vietnam Bangladesh Small (0–0.13) Romania, Guyana Democratic Republic of Congo, Mali Source: IEG, based on World Bank administrative data. Note: Loans per capita denote cumulative loan commitments during fiscal 2000–06 normalized by population size of the country. Costs of ESW and country services are the cumulative costs during fiscal 2000–06. All data are from the World Bank. ESW = economic and sector work. 82 APPENDIX B: METHODOLOGY Figure B.1: Sectoral Distribution of ESW in Country Reviews and of All Bank Country-Specific ESW (Fiscal 2000–06) 25 20 15 Percent 10 5 0 y ion ng nt t r T ion t s re on t t e r t on rt t ion en to en en en to en en lic ce IC nc po me tu ini cti cti ec ec at lat at em pm pm em pm pm Po rvi na al uc ns M du te uc on lS lS nit ob pu Se er ag elo str elo ur elo Tra elo ic ro Ed Re vir & cia ra Sa ov om Gl Po oc an ra lP al ev ev ev ev Ru gy En rG an rty Pr ion & Inf & on lM cia nD rD lD nD er Fin ve cto ly n, Ec at En So cia cia cto pp ma ba tio Po er Se So Su an Ur Se tri Op Hu c Fin Nu er bli te at iva Pu h, W alt Pr He Sector Sectoral shares of ESW in country reviews Sectoral shares of all country ESW Source: IEG and World Bank administrative data (adjusted by IEG for misclassification). Note: ESW = economic and sector work; Global ICT = global information, communications, and technology. Figure B.2: Sectoral Distribution of TA in Country Reviews and of All Bank Country-Specific TA (Fiscal 2000–06) 35 30 25 Percent 20 15 10 5 0 y ion ng nt t r T ion t s re on t nt e r t on rt t ion en cto ion men en to en en lic ce IC nc po me ctu me ini cti cti ec at at at em pm pm pm Po rvi na Se al ns M du te uc on ma opul p ru e lS nit ob Se er ag elo elo ur elo Tra o ic ro l t Ed Re vir l & cia ra Sa s ov om Gl ve oc an ra lP al P ev ev ev Ru gy En rG De an rty Pr & Inf on lM & cia nD rD lD er Fin ve to ly n, Ec an at En So cia cia cto pp ec tio Po er b So Su an cS Ur Se tri Op Hu Fin Nu er bli te at iva Pu h, W alt Pr He Sector Sectoral shares of TA in country reviews Sectoral shares of all country TA Source: IEG and World Bank administrative data (adjusted by IEG for misclassification). Note: Global ICT = global information, communications, and technology; TA = technical assistance. 83 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS tasks than those who did not respond—at any Table B.3: Profile of Stakeholders Interviewed significance level. Stakeholder type Number For TA, the team did not find a statistically sig- Bank management 21 nificant difference between the two groups of Bank staff (ESW, TA, and loan TTLs) 187 TTLs (those that responded to the survey and those that did not) with respect to three of the Staff of other institutions 7 QAG ratings for their tasks: overall quality, dialogue Authorities (and ex-authorities) 191 and dissemination, and strategic relevance. The Legislature 9 average ratings were lower for internal quality Donors 63 (significant at the 5 percent level) and likely im- Civil society 39 pact (significant at the 10 percent level) for those Private sector 33 TTLs who did not respond to the survey. Academia/research 18 Total 568 Another bias could arise from the sectoral and Source: IEG country reviews. Regional distributions of the ESW and TA tasks to Note: ESW = economic and sector work; TA = technical assistance, TTL = task team leader. which the TTLs have responded. No biases were found, however. Both the sectoral and Regional dis- tributions for the ESW tasks to which there were Table B.4: Survey of ESW and TA TTLs responses matched very closely with those of all such tasks delivered during fiscal 2000–06 (fig- Number of Corrected ures B.3 and B.4). The match was also close for TA TTLs number of Number of Response tasks, although there was some underrepresen- e-mailed TTLsa responses rate (%) tation of TA tasks (by 7 percent) in the financial sec- ESW 1,258 1,143 696 60.9 tor (figure B.5) and some overrepresentation (by TA 499 419 220 52.5 8 percent) in Sub-Saharan Africa (figure B.6), com- Total 1,757 1,562 916 58.6 pared with all TA delivered during the period. Source: IEG surveys. Note: ESW = economic and sector work; TA = technical assistance; TTL = task team leader. Surveys of In-Country Users of Specific a. Corrections are for: TTLs who have retired; tasks not really ESW or TA; TTLs inherited the tasks; TTLs in name only, while some others know the tasks better; TTLs not sure what the tasks were; TTLs out of ESW and TA office. Information on in-country users of specific ESW and TA was obtained through the surveys of ESW and those who did not. The team then matched and TA TTLs described above. Specifically, the QAG ratings to the tasks managed by the TTLs. For surveys asked the TTLs to provide contact and ESW , approximately 17 percent of the tasks man- other information for each of the following cate- aged by TTLs in each of the two groups (that is, gories of in-country users of ESW: government those that responded to the survey and those official, donor, and civil society representative. that did not) had QAG ratings, whereas for TA, TTLs provided addressable samples of 351 users around 8 percent of the tasks in each of the two of specific ESW and 172 users of specific TA. groups had QAG ratings. Electronic surveys were sent to these users. In- For ESW , the team found no statistically significant formation on net addressable samples (that is, ex- differences in the average QAG ratings between cluding those who were out of the office or who the two groups for all five QAG ratings (overall indicated they have not used or are not aware of quality and each of the subratings on internal the specific task on which they were surveyed) and quality, dialogue and dissemination, strategic rel- responses are presented in table B.5. The margin evance, and likely impact). In other words, those of error is 11.7 percent for the specific ESW user ESW TTLs who responded to the survey did not survey and 17.1 percent for the specific TA user have significantly different QAG ratings for their survey. The stakeholder profiles of the respon- 84 APPENDIX B: METHODOLOGY Figure B.3: Sectoral Distribution of ESW to Which TTLs Responded and of All Bank ESW (Fiscal 2000–06) 18 16 14 12 Percent 10 8 6 4 2 0 y ion ng nt t r T ion t re on t t e elo r t on t t ion en to en en en to en or en lic IC nc me tu ini cti cti ec ec sp at lat at em pm pm em pm pm Po na al uc M du te uc on lS lS n nit ob pu er ag elo str elo ur Tra elo ic ro Ed Re vir & cia ra Sa ov om Gl Po oc an ra lP ev ev ev ev Ru gy En rG an rty Pr Inf & on lM & cia nD rD lD nD er Fin ve to ly n, Ec En So cia cia cto pp ma ba ec tio Po So Su an cS Ur Se tri Hu Fin Nu er bli te at iva Pu h, W alt Pr He Sector Sectoral shares covered by ESW TTL survey Sectoral shares of all all ESW Source: IEG and World Bank administrative data (adjusted by IEG for misclassification). Note: ESW = economic and sector work; Global ICT = global information, communications, and technology; TTL = task team leader. Figure B.4: Regional Distribution of ESW to Which TTLs Responded and of All Bank ESW (Fiscal 2000–06) 25 20 15 Percent 10 5 0 Sub-Saharan East Asia Europe and Latin America Middle East and South Asia Africa and Pacific Central Asia and the Caribbean North Africa Region Regional shares covered by ESW TTL survey Regional shares of all ESW Source: IEG survey and World Bank administrative data. Note: ESW = economic and sector work; TTL = task team leader. 85 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Figure B.5: Sectoral Distribution of TA to Which TTLs Responded and of All Bank TA (Fiscal 2000–06) 14 12 10 Percent 8 6 4 2 0 y ion ng nt t r T ion t re on t t e r t on rt t ion en to en en en to en en lic IC nc po me tu ini cti cti ec ec at lat at em pm pm em pm pm Po na al uc ns M du te uc on lS lS nit ob pu er ag elo str elo ur elo Tra elo ic ro Ed Re vir & cia ra Sa ov om Gl Po oc an ra lP ev ev ev ev Ru gy En rG an rty Pr Inf & & on lM cia nD rD lD nD er Fin ve to ly n, Ec En So cia cia cto pp ma ba ec tio Po So Su an Ur cS Se tri Hu Fin Nu er bli te at iva Pu h, W alt Pr He Sector Sectoral shares covered by TA TTL survey Sector shares of TA Source: IEG and World Bank administrative data (adjusted by IEG for misclassification). Note: TA = technical assistance; Global ICT = global information, communications, and technology; TTL = task team leader. Figure B.6: Regional Distribution of TA to Which TTLs Responded and of All Bank TA (Fiscal 2000–06) 35 30 25 20 Percent 15 10 5 0 Sub-Saharan East Asia Europe and Latin America Middle East and South Asia Africa and Pacific Central Asia and the Caribbean North Africa Region Shares covered by TA TTL survey Shares of Bank TA Source: IEG and World Bank administrative data. Note: TA = technical assistance; TTL = task team leader. 86 APPENDIX B: METHODOLOGY dents are presented in table B.6 and the country Table B.5: Survey Statistics of Specific ESW profiles in tables B.7 and B.8. There were re- and Specific TA User Surveys sponses on specific ESW for 41 countries and on Specific ESW User Specific TA User specific TA for 17 countries. The surveys were conducted in September–October 2007 for about Addressable sample 351 172 three weeks. Less— Out of office 15 8 The survey questions are similar to the ones asked Have not used ESW/TA 39 20 in the country reviews. Additionally, the surveys Net addressable sample 297 144 sought the views of the stakeholders on their Number of responses 70 33 willingness to pay for Bank ESW/TA. The survey Response rate 24% 23% questionnaires are available on request. Source: IEG surveys. Note: ESW = economic and sector work; TA = technical assistance. Bias checks for specific ESW and specific TA user survey responses Biases in responses could arise from the fact that only respondents who feel strongly (either posi- Table B.6: Stakeholder Profile of Respondents tively or negatively) about the Bank would re- to Specific ESW and Specific TA User Surveys spond to the surveys. One way to check for such Shares of Shares of potential bias is to compare the profile of the re- respondents to respondents to spondents to the profile of those to whom the sur- specific ESW specific TA vey was sent. user survey user survey Total number of respondents 70 33 There does not seem to be a significant bias with of which: respect to the Regional distribution of responses Public sector (%) 30.0 53.1 to the specific ESW and the specific TA user surveys. With respect to the specific ESW survey, Donor agency (%) 21.4 9.4 there is quite a close correspondence between the Academia/research (%) 20.0 12.5 Regional distribution of the respondents and Multilateral (%) 10.0 6.3 those to whom the survey was sent, with a slight Civil society (%) 8.6 6.3 overrepresentation of respondents from Sub- Private sector (%) 8.6 9.4 Saharan Africa (of around 8 percent); the differ- Consultants (%) 1.4 3.1 ences for the other Regions are quite small (be- Source: IEG surveys. tween 2 and 3.5 percent; figure B.7). With respect Note: ESW = economic and sector work; TA = technical assistance. to the correspondence between the Regional dis- tribution of the responses to the TA user survey and those to whom the survey was sent, the Re- gional shares for Sub-Saharan Africa, East Asia responded in the specific ESW user survey, com- and Pacific, Europe and Central Asia, and Latin pared with such distributions for all Bank ESW. The America and the Caribbean are very close, al- sectoral distribution of the respondents to the though there is an overrepresentation of re- specific ESW user survey (based on the sectoral sponses from the Middle East and North Africa (by classification of the ESW to which they were re- about 10 percent), and no responses from South sponding) generally matched that of all Bank Asia (figure B.8). ESW (figure B.9). The Regional distribution matched somewhat less well, with a somewhat Sectoral and Regional distributions of tasks larger representation from Sub-Saharan Africa (by to which in-country users responded about 12 percent) and smaller representation There were no significant biases in the sectoral or from Europe and Central Asia (also by about 12 Regional distributions of the tasks to which users percent; figure B.10). 87 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Table B.7: Country/Region of ESW to Which In-Country Stakeholders Responded in the Specific ESW User Survey Number of Number of Country/territory respondents Country/territory respondents Africa Europe and Central Asia Angola 1 Bulgaria 1 Burundi 1 Hungary 1 a Cape Verde 3 Kosovo 1 Ethiopia 1 Moldova 1 Kenya 1 Russian Federation 1 Lesotho 1 Regional EU accession 1 Mauritania 2 Regional 1 Niger 1 Subtotal 7 Nigeria 3 Latin America and the Caribbean Sierra Leone 4 Argentina 1 Swaziland 1 Caribbean 1 Togo 2 Colombia 1 Uganda 2 Ecuador 2 Regional 1 Guatemala 1 Subtotal 24 Mexico 2 East Asia and Pacific Venezuela, R.B. de 1 Cambodia 2 Subtotal 9 Indonesia 3 Middle East and North Africa Lao People’s Dem. Rep. 2 Iran, Islamic Rep. of 1 Pacific Islands 1 Lebanon 1 Philippines 1 Subtotal 2 Solomon Islands 1 South Asia Thailand 1 Afghanistan 1 Tonga 1 Bangladesh 1 Vietnam 2 India 3 Subtotal 14 Nepal 1 Sri Lanka 2 Subtotal 8 World 6 Total 70 Source: IEG survey. Note: ESW = economic and sector work; EU = European Union. a. Under UN Security Council Resolution 1244 (1999), Kosovo is administered by the United Nations Interim Administration Mission in Kosovo. For TA, the match was not as close, although the were no responses on TA tasks delivered in South biases were not hugely significant. The Regional Asia). With respect to the sectoral distribution, distribution of the TA to which users responded none of the differences in distribution were greater in the survey was quite close to that of all Bank than 10 percent, except for the financial sector, TA (figure B.11), although Sub-Saharan Africa was for which there were no responses on TA, and overrepresented by about 22 percent and South the rural sector, for which there was an over- Asia was underrepresented by 7 percent (there representation of about 11 percent (figure B.12). 88 APPENDIX B: METHODOLOGY Table B.8: Country/Region of TA to Which In-Country Stakeholders Responded in the Specific TA User Survey Number of Number of Country and Region respondents Country respondents Sub-Saharan Africa Europe and Central Asia Angola 2 Kyrgyz Republic 2 Burkina Faso 2 Turkey 1 Comoros 2 Regional 3 Mauritius 1 Subtotal 6 Mozambique 1 Latin America and the Caribbean South Africa 1 Brazil 1 Sudan 1 Subtotal 1 Regional 4 Middle East and North Africa Subtotal 14 Iraq 2 East Asia and Pacific Qatar 1 China 2 Regional 2 Indonesia 1 Subtotal 5 Mongolia 1 Vanuatu 2 Vietnam 1 Subtotal 7 Total 33 Source: IEG survey. Note: TA = technical assistance. Figure B.7: Regional Distribution of Specific ESW User Survey Mailing List and of Respondents 40 35 30 25 Percent 20 15 10 5 0 Sub-Saharan East Asia Europe and Latin America Middle East and South Asia Africa and Pacific Central Asia and the Caribbean North Africa Region Regional shares in mailing list Regional shares of respondents Source: IEG survey. Note: ESW = economic and sector work. 89 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Figure B.8: Regional Distribution of Specific TA User Survey Mailing List and of Respondents 45 40 35 30 Percent 25 20 15 10 5 0 Sub-Saharan East Asia Europe and Latin America Middle East and South Asia Africa and Pacific Central Asia and the Caribbean North Africa Region Regional shares in mailing list Regional shares of respondents Source: IEG survey. Note: TA = technical assistance. In sum, there is quite a close correspondence be- evance, timeliness, partnership, dissemination) of tween the sectoral and Regional distributions of specific ESW; the extent to which and specifics of ESW tasks to which users responded in the survey, how the ESW had informed lending in the coun- and those of all ESW delivered by the Bank. The try; what would have happened if the ESW had not correspondence is slightly less close for TA tasks. been produced; and their rankings of various Bank instruments (lending and nonlending) for Surveys of Loan TTLs (Within-Bank the country, various types of reports for loan Users of ESW) preparation, and the different dimensions of ESW Respondents to the ESW TTL survey provided intended to make ESW more useful to their work. 205 names of loan TTLs who would know about or who would have used the specific ESW on Comparing the Regional distribution of the ESW which the ESW TTLs were being surveyed. Those to which the loan TTLs were responding with ESW TTLs who were also loan TTLs were excluded that of all ESW indicates there are no significant from the survey. Of the 205 loan TTLs who were biases on this front (figure B.13). The differences surveyed, 4 were out of the office, 13 indicated they in Regional shares were smaller than 10 percent did not use the ESW, and 2 indicated they were not for all the Regions and much smaller than that for the appropriate people to be surveyed. This some Regions. There is even less bias on the sec- brought the addressable sample to 186. Of the ad- toral front with a greater correspondence be- dressable sample, 91 responded, which is a 49 tween the sectoral shares (figure B.14). percent response rate and reflects a margin of error of 10.3 percent. The survey was conducted Survey of Users of ESW and TA in September–October 2007 for about three weeks. In addition to surveying in-country stakeholders on specific tasks, this evaluation also undertook The survey sought information on loan TTLs’ a survey of potential users of Bank ESW and TA views on various dimensions (technical quality, rel- (text continues on page 94) 90 APPENDIX B: METHODOLOGY Figure B.9: Sectoral Distribution of Respondents to Specific ESW User Survey and of All Bank ESW (Fiscal 2000–06) 20 15 Percent 10 5 0 y ion ng nt t r T ion t re on t t e r t on t ion rt en to en en en to en en lic IC nc po me tu ini cti cti ec ec at lat at em pm pm em pm pm Po na al uc ns M du te uc on lS lS nit ob pu er ag elo str elo ur elo elo ic Tra ro Ed Re vir & cia ra Sa ov om Gl Po oc an ra lP ev ev ev ev Ru gy En rG an rty Pr Inf & lM & on cia nD rD lD nD er Fin ve to ly n, Ec En So cia cia cto pp ma ba ec tio Po So Su an cS Ur Se tri Hu Fin Nu bli er te at iva Pu h, W alt Pr He Sector Sectoral shares covered by specific ESW user survey Sector shares of all ESW Source: IEG and World Bank administrative data (adjusted by IEG for misclassification). Note: ESW = economic and sector work; Global ICT = global information, communications, and technology. Figure B.10: Regional Distribution of Respondents to Specific ESW User Survey and of All Bank ESW (Fiscal 2000–06) 40 35 30 25 Percent 20 15 10 5 0 Sub-Saharan East Asia Europe and Latin America Middle East and South Asia Africa and Pacific Central Asia and the Caribbean North Africa Region Shares of ESW by Region Regional shares of respondents Source: IEG survey and World Bank administrative data. Note: ESW = economic and sector work. 91 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Figure B.11: Regional Distribution of Respondents to Specific TA User Survey and of All Bank TA (Fiscal 2000–06) 45 40 35 30 Percent 25 20 15 10 5 0 Sub-Saharan East Asia Europe and Latin America Middle East and South Asia Africa and Pacific Central Asia and the Caribbean North Africa Region Regional shares covered by specific TA user survey Shares of all TA Source: IEG survey and World Bank administrative data. Note: TA = technical assistance. Figure B.12: Sectoral Distribution of Respondents to Specific TA User Survey and of All Bank TA (Fiscal 2000–06) 25 20 15 Percent 10 5 0 y ion ng nt t r T ion t re on t t e elo r t on t t ion en to en en en cto en or en lic IC nc me ctu ini cti cti ec sp at lat at em pm pm em pm pm Po na l e a M du te uc on lS tru lS n nit ob pu er ag elo elo ur Tra elo c ro Ed Re mi vir & cia ra Sa s ov Gl Po oc an ra lP ev ev ev ev Ru o gy En rG an rty Pr Inf & on lM & cia nD rD lD nD er Fin ve to ly n, Ec En So cia cia cto pp ma ba ec tio Po So Su an cS Ur Se tri Hu Fin Nu er bli te at iva Pu h, W alt Pr He Sector Sectoral shares covered by specific TA user survey Shares of all TA Source: IEG and World Bank administrative data (adjusted by IEG for misclassification). Note: Global ICT = global information, communications, and technology; TA = technical assistance. 92 APPENDIX B: METHODOLOGY Figure B.13: Regional Distribution of ESW to Which Loan TTLs Responded, and of All ESW 40 35 30 25 Percent 20 15 10 5 0 Sub-Saharan East Asia Europe and Latin America Middle East and South Asia Africa and Pacific Central Asia and the Caribbean North Africa Region Shares of loan TTL Shares of all ESW Source: IEG survey and World Bank administrative data. Note: ESW = economic and sector work; Global ICT = global information, communications, and technology; TTL = task team leader. Figure B.14: Sectoral Distribution of ESW to Which Loan TTLs Responded, and of All ESW 18 16 14 12 Percent 10 8 6 4 2 0 y ion ng nt t r T n t re on t nt e elo r t on rt t ion en to en en to en en lic IC tio nc n D nspo me ctu me ini cti cti ec ec at at em pm pm pm pm Po na al ula M du te uc on lS tru re lS nit ob er ag elo elo Tra elo c p ro u Ed Re mi vir & cia ra Sa s ov Gl Po oc an ra lP ev ev ev ev Ru o gy En rG an rty Pr Inf & lM & on cia nD rD lD er Fin ve to ly n, Ec En So cia cia cto pp ma ba ec tio Po So Su an cS Ur Se tri Hu Fin Nu er bli te at iva Pu h, W alt Pr He Sector Shares of ESW to which loan TTLs responded Shares of all ESW Source: IEG and World Bank administrative data (adjusted by IEG for misclassification). Note: ESW = economic and sector work; Global ICT = global information, communications, and technology; TTL = task team leader. 93 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS on their general views of various dimensions and which 394 (77 percent) were from 91 Bank client effects of such Bank products. The survey was countries and 115 (23 percent) from developed sent electronically to 2,700 users, with the e-mail countries. The stakeholder profiles of the respon- list being compiled from recent client surveys un- dents are presented in table B.9 and the country dertaken by IEG (the Financial Sector Assessment profiles in table B.10. Program [FSAP] and low-income countries under stress surveys), the World Bank (Operation and Pol- Bias checks for the ESW/TA user icy Country Services’s Conditionality Survey; re- survey response spondents to the World Bank Reader Survey by To check for potential biases arising from the External Affairs), and the Independent Evaluation possibility that only those who feel strongly (ei- Office of the International Monetary Fund (IMF) ther positively or negatively) about the Bank re- (the African Survey). sponded to the survey, various aspects of the respondents’ profiles were compared against The survey questions were similar to those asked those from the original mailing list. There does not in the specific ESW and specific TA user surveys, appear to be any significant Regional or stake- except they did not pertain to specific ESW or holder type bias, given the close correspondence TA tasks and they sought the views on ESW and between these two groups on both those fronts TA as a whole, rather than separately. The survey (figures B.15 and B.16). The Regional distributions questionnaire is available on request. of the two groups were very similar. For stake- holder type there was an overrepresentation of re- Of the 2,700 to whom the survey was sent, 20 were spondents from academia (by 11 percent) and out of office and 262 indicated they had not used an underrepresentation from civil society (by 14 Bank ESW or TA; this resulted in a net addressable percent). sample of 2,418. Of this sample, 550 people re- sponded, which is a 23 percent response rate and The sectoral distribution of Bank client respon- reflects a margin of error of 4.5 percent. The sur- dents (based on responses to the question of vey was done in September–October 2007 for the main sector in which they work) is very sim- about three weeks. Of the 550 respondents, 509 ilar to that of the sectoral distribution of all Bank provided information on their country of origin, of ESW and TA (figure B.17), with some overrep- resentation of the economic policy sector (by 9 percent) and the education sector (by 11 per- Table B.9: Stakeholder Profile of Respondents cent). The overrepresentation of the education to ESW/TA User Survey sector could be attributed to the fact that 64 of the 86 respondents who selected education as Shares of Shares of respondents respondents their main sector are academics/researchers from Bank from developed (which implies there may not be an overrepre- client countries countries sentation of the education sector). Given the Total number of respondents 389 114 (otherwise) close correspondence in the sec- Academia/research (%) 40.9 40.4 toral distributions, there are not likely to be bi- Civil society (%) 7.2 3.5 ases on this front. Donor agency (%) 2.8 5.3 Private sector (%) 17.2 24.6 Compared with the sectoral distribution, the Re- Public sector (%) 27.8 14.9 gional distributions of Bank client respondents and of all Bank ESW and TA are less close, but Multilateral (%) 1.8 5.3 there do not seem to be significant biases on this Consultants (%) 2.1 6.1 front (figure B.18). All the differences in shares Combination (research/private sector) (%) 0.3 — were less than 10 percent, and for some Regions Source: IEG survey. (Sub-Saharan Africa and East Asia and Pacific) Note: ESW = economic and sector work; TA = technical assistance. they were much smaller. 94 APPENDIX B: METHODOLOGY Table B.10: Country Profile of Bank Client Respondents to User ESW/TA Survey Regions Regions Sub-Saharan Africa Europe and Central Asia Benin 2 Albania 2 Burkina Faso 1 Armenia 2 Cameroon 1 Azerbaijan 3 Chad 1 Bosnia and Herzegovina 1 Congo, Dem. Rep. of 2 Bulgaria 4 Côte d’Ivoire 1 Croatia 4 Ethiopia 2 Czech Republic 3 Gambia, The 1 Hungary 2 Ghana 2 Kazakhstan 2 Kenya 8 Kyrgyz Republic 2 Liberia 1 Lithuania 1 Mauritius 3 Poland 4 Mozambique 1 Romania 5 Nigeria 12 Russian Federation 3 Rwanda 2 Slovak Republic 3 Senegal 3 Slovenia 2 Sierra Leone 1 Turkey 19 South Africa 14 Turkmenistan 1 Sudan 2 Ukraine 3 Tanzania 2 Subtotal 66 Togo 1 Latin America and the Caribbean Uganda 3 Zambia 3 Argentina 6 Zimbabwe 1 Bolivia 2 Subtotal 70 Brazil 14 Chile 1 East Asia and Pacific Colombia 3 Cambodia 13 Costa Rica 3 China 12 Dominican Republic 1 Fiji 2 Ecuador 4 Indonesia 10 El Salvador 1 Lao PDR 2 Grenada 1 Malaysia 1 Guatemala 1 Mongolia 1 Haiti 5 Papua New Guinea 2 Honduras 4 Philippines 13 Mexico 16 Korea, Republic of 2 Nicaragua 1 Solomon Islands 1 Paraguay 2 Thailand 9 Peru 10 Vietnam 11 Suriname 2 Subtotal 79 Trinidad and Tobago 1 Uruguay 1 Middle East and North Africa Venezuela, R.B. de 7 Algeria 2 Subtotal 86 Egypt, Arab Republic of 3 South Asia Iran, Islamic Rep. of 1 Jordan 4 Afghanistan 3 Lebanon 3 Bangladesh 3 Morocco 3 India 54 Tunisia 1 Nepal 1 Yemen, Republic of 2 Pakistan 7 Subtotal 19 Sri Lanka 6 Subtotal 74 Total 394 Source: IEG survey. Note: ESW = economic and sector work; TA = technical assistance. 95 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Figure B.15: Regional Distribution of ESW/TA User Survey Mailing List and of Respondents 40 35 30 25 Percent 20 15 10 5 0 Sub-Saharan East Asia Europe and Latin America Middle East and South Asia Africa and Pacific Central Asia and the Caribbean North Africa Region Regional shares of respondents to user survey Regional shares of contacts in user survey mailing list Source: IEG survey. Note: ESW = economic and sector work; TA = technical assistance. Figure B.16: Stakeholder Profile of ESW/TA User Survey Mailing List and of Respondents 45 40 35 30 25 Percent 20 15 10 5 0 Academia/research Civil society Donor agency Private sector Public sector Respondent profile User survey respondents User survey mailing list Source: IEG survey. Note: ESW = economic and sector work; TA = technical assistance. 96 APPENDIX B: METHODOLOGY Figure B.17: Sectoral Distribution of ESW/TA User Survey Respondents, and Sectoral Shares of All Bank ESW and TA 25 20 Percent 15 10 5 0 y ion ng nt t r t T ion t re on t t e r t on rt t ion en to en en en en to en en lic IC nc po me tu ini cti cti ec ec at lat at em pm pm pm em pm pm Po na al uc ns M du te uc on lS lS nit ob pu er ag elo elo str elo ur elo Tra elo ic ro Ed Re vir & cia ra Sa ov om Gl Po oc an ra lP ev ev ev ev ev Ru gy En rG an rty Pr & Inf lM & on cia dD nD rD lD nD er Fin ve to ly n, Ec En So cia cia cto pp an ma ba ec tio Po So Su an cS Ur Se tri er Hu Fin nd Nu er bli te Ge at iva Pu h, W alt Pr He Sector Shares of respondents Shares of all AAA Source: IEG and World Bank administrative data (adjusted by IEG for misclassification). Note: AAA = analytical and advisory activities; ESW = economic and sector work; Global ICT = global information, communications, and technology; TA = technical assistance. Matching ESW to Loans and closed during fiscal 2003–06 would have IEG This exercise began with the selection of a sam- ratings, and these would have been mostly de- ple of loans from two portfolios of loans: those ap- velopment policy loans (DPLs). proved during fiscal 2003–05 that have quality at entry (QAE) ratings and those approved during For each of the two portfolios of loans, 50 percent fiscal 2003–06 that have IEG loan ratings. Only were selected randomly by the IEG team, strati- loans approved from fiscal 2003 were selected to fied by sector and Region. This resulted in a sam- account for the possibility that ESW delivered up ple of 119 loans with QAE ratings and 90 loans with to three years prior to loan approval could have IEG ratings, among which 13 have both QAE and informed the loan; hence, this would cover ESW IEG ratings. The sectoral distribution of the se- delivered during the evaluation period of fiscal lected sample of loans with QAE ratings corre- 2000–06. sponds closely to that of the portfolio of all loans, but this was not the case for the selected sample During fiscal 2003–06, the Bank approved 1,106 of loans with IEG ratings, for reasons mentioned loans, of which 152 have IEG ratings (14 percent in the above paragraph. of all loans). During fiscal 2003–05, the Bank ap- proved 809 loans, of which 196 have QAE ratings The Regional distribution of the selected sample (24 percent of all loans). The sectoral distribution of loans with QAE ratings corresponds closely to of loans with QAE ratings closely corresponds to that of the portfolio of all development policy that of the portfolio of all loans (figure B.19); and all investment loans, as well as all loans (fig- that of loans with IEG ratings does not corre- ures B.21 and B.22). For loans with IEG ratings, spond very well (figure B.20). The latter is not sur- however, the Regional distribution of the selected prising, given that only loans that were approved sample of loans corresponds closely only to that 97 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Figure B.18: Regional Distribution of ESW/TA User Survey Respondents, and Regional Shares of All Bank ESW and TA 40 35 30 25 Percent 20 15 10 5 0 Sub-Saharan East Asia Europe and Latin America Middle East and South Asia Africa and Pacific Central Asia and the Caribbean North Africa Region Regional shares of respondents Regional shares of ESW and TA Source: IEG survey and World Bank administrative data. Note: ESW = economic and sector work; TA = technical assistance. of the portfolio of all DPLs (figure B.23) but not QAG Quality Ratings so closely to that of all investment loans (figure During fiscal 2000–06, QAG assessed 725 ESW B.24). This is again not surprising, given that loans and 157 TA tasks, including country-specific, re- with IEG ratings are those that had closed during gional, and global tasks. The distribution of TA the evaluation, and these are mainly DPLs. In tasks that were assessed by QAG closely reflects light of the above discussion, that is, the close ap- the distribution of TA tasks delivered by the Bank proximation of both the Regional and sectoral during the period. The match is not quite as close distribution of loans with QAE ratings (but not that for ESW tasks, with an oversampling of country- of loans with IEG ratings) to that of all loans, only specific tasks and an undersampling of regional inferences drawn from loans with QAE ratings and global tasks that QAG assessed (table B.11). were used in the discussion in chapter 3. For country-specific tasks, there is a close match The matching entailed the review of loan docu- between the Regional distribution of ESW tasks ments and ESW to identify those ESW that could that were assessed by QAG and of all such tasks have informed the loans (the relevant ESW). For delivered by the Bank during the period, but a less DPLs, which are generally multisectoral, ESW in close match for TA tasks (table B.12). all sectors were reviewed. For investment loans, the review included ESW in the economic policy The Regional and sectoral distribution of ESW and financial management sector in addition to tasks with QAG ratings on both the four- and six- the sector of the loan. Based on this review, only point scales are very close to such distributions those ESW that could actually have informed the for all Bank ESW , although they are somewhat loans were selected for inclusion in the analysis. less close for TA tasks (figures B.25–B.30). 98 APPENDIX B: METHODOLOGY Figure B.19: Sectoral Distribution of All Loans and Selected Sample of Loans with QAE Ratings, Fiscal 2003–05 16 14 12 10 Percent 8 6 4 2 0 y ion ng t r T ion on t e r t on rt t ion n to en to en en lic IC nc po me ini cti cti ec ec at lat at pm pm pm Po na al ns M du te uc on lS nit lS ob pu er elo elo Tra elo ic ro Ed Re vir & cia Sa ra ov om Gl Po lP ev ev ev Ru gy En rG an rty & & on cia rD lD nD er Fin ve ly to n, Ec En So cia cto pp ba ec tio Po Su So Ur cS Se tri Nu er bli te at iva Pu h, W alt Pr He Sector Share of all loans Share of loans with QAE ratings Source: IEG and World Bank administrative data (adjusted by IEG for misclassification). Note: Global ICT = global information, communications, and technology; QAE = quality at entry. Figure B.20: Sectoral Distribution of All Loans and Selected Sample of Loans with IEG Ratings, Fiscal 2003–06 30 25 20 Percent 15 10 5 0 y ion ng nt r T ion on nt e r t on rt t ion cto to en en lic IC nc po me me ini cti cti ec at lat at pm pm Po na Se al ns M du te uc on lop lS nit ob pu er elo Tra elo ic ro l Ed Re vir & cia ra Sa ov om Gl Po ve lP ev ev Ru gy En rG e an ty & ,& on cia rD lD nD er er Fin to ly Ec n En So v cia cto pp ba ec tio Po So Su cS Ur Se tri Nu bli er te at iva Pu h, W alt Pr He Sector Share of all loans Share of loans with IEG ratings Source: IEG and World Bank administrative data (adjusted by IEG for misclassification). Note: Global ICT = global information, communications, and technology. 99 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Table B.11: Country-Specific, Regional, and Global Tasks Assessed by QAG, Fiscal 2000–06 ESW TA Shares of Shares of ESW Shares of Shares of TA all ESW assessed by QAG all TA assessed by QAG (%) (%) (%) (%) Country 78 89 80 79 Regional 16 8 18 19 Global 6 3 2 3 Source: IEG staff calculation based on World Bank and QAG data. Note: ESW = economic and sector work; QAG = Quality Assurance Group; TA = technical assistance. Table B.12: Regional Distribution of Country-Specific ESW and TA Tasks Assessed by QAG, Fiscal 2000–06 ESW TA Shares of Shares of ESW Shares of Shares of TA all ESW assessed by QAG all TA assessed by QAG Region (%) (%) (%) (%) Sub-Saharan Africa 23 24 17 23 East Asia and Pacific 15 12 27 15 Europe and Central Asia 21 22 24 25 Latin America and the Caribbean 15 17 7 11 Middle East and North Africa 11 11 18 14 South Asia 15 15 8 12 Source: IEG staff calculation based on World Bank and QAG data. Note: ESW = economic and sector work; QAG = Quality Assurance Group; TA = technical assistance. 100 APPENDIX B: METHODOLOGY Figure B.21: Regional Distribution and Selected Sample of All Development Policy Loans with QAE Ratings, Fiscal 2003–05 50 40 30 Percent 20 10 0 Sub-Saharan East Asia Europe and Latin America Middle East and South Asia Africa and Pacific Central Asia and the Caribbean North Africa Region All adjustment loans Sample adjustment loans with QAE ratings Source: IEG, based on World Bank administrative data and QAG data. Note: QAE = quality at entry. Figure B.22: Regional Distribution and Selected Sample of All Investment Loans with QAE Ratings, Fiscal 2003–05 40 30 Percent 20 10 0 Sub-Saharan East Asia Europe and Latin America Middle East and South Asia Africa and Pacific Central Asia and the Caribbean North Africa Region All investment loans Sample investment loans with QAE ratings Source: IEG, based on World Bank administrative data and QAG data. Note: QAE = quality at entry. 101 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Figure B.23: Regional Distribution and Selected Sample of All Development Policy Loans with IEG Ratings, Fiscal 2003–06 40 30 Percent 20 10 0 Sub-Saharan East Asia Europe and Latin America Middle East and South Asia Africa and Pacific Central Asia and the Caribbean North Africa Region All adjustment loans Sample adjustment loans with IEG ratings Source: IEG and World Bank administrative data. Note: IEG = Independent Evaluation Group. Figure B.24: Regional Distribution and Selected Sample of All Investment Loans with IEG Ratings, Fiscal 2003–06 40 30 Percent 20 10 0 Sub-Saharan East Asia Europe and Latin America Middle East and South Asia Africa and Pacific Central Asia and the Caribbean North Africa Region All investment loans Sample adjustment loans with IEG ratings Source: IEG and World Bank administrative data. Note: IEG = Independent Evaluation Group. 102 APPENDIX B: METHODOLOGY Figure B.25: Sectoral Shares of ESW with 4-Point QAG Ratings and All Bank ESW 18 16 14 12 Percent 10 8 6 4 2 0 y ion ng nt t r T ion t re on t t e r t on rt t ion en to en en en to en en lic IC nc po me tu ini cti cti ec ec at lat at em pm pm em pm pm Po na al uc ns M du te uc on lS lS nit ob pu er ag elo str elo ur elo Tra elo ic ro Ed Re vir & cia ra Sa ov om Gl Po oc an ra lP ev ev ev ev Ru gy En rG an rty Pr Inf & lM & on cia nD rD lD nD er Fin ve to ly n, Ec En So cia cia cto pp ma ba ec tio Po So Su an cS Ur Se tri Hu Fin Nu bli er te at iva Pu h, W alt Pr He Sector Share of observations with QAG 4-point ratings Share of ESW Source: IEG, based on World Bank administrative data (adjusted by IEG for misclassification) and QAG data. Note: ESW = economic and sector work; Global ICT = global information, communications, and technology; QAG = Quality Assurance Group. Figure B.26: Sectoral Shares of ESW with 6-Point QAG Ratings and All Bank ESW 18 16 14 12 Percent 10 8 6 4 2 0 cy ion ng nt t r T ion t re on t nt e elo r t on t t ion en to en en cto en or en IC nc me ctu me li ini cti cti ec sp at lat at em pm pm pm pm Po na l e a M du te uc on lS tru re lS n nit ob pu er ag elo elo Tra elo ic ro u Ed Re vir & cia ra Sa s ov om Gl Po oc an ra lP ev ev ev ev Ru gy En rG an rty Pr & Inf on lM & cia nD rD lD nD er Fin ve to ly n, Ec En So cia cia cto pp ma ba ec tio Po So Su an cS Ur Se tri Hu Fin Nu er bli te at iva Pu h, W alt Pr He Sector Share of observations with QAG 6-point scale ratings Share of ESW Source: IEG, based on World Bank administrative data (adjusted by IEG for misclassification) and QAG data. Note: ESW = economic and sector work; Global ICT = global information, communications, and technology; QAG = Quality Assurance Group. 103 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Figure B.27: Sectoral Shares of TA with Four-Point QAG Ratings and All Bank TA 18 16 14 12 Percent 10 8 6 4 2 0 y ion ng nt t r T ion t re on t t e r t on rt t ion en to en en en to en en lic IC nc po me tu ini cti cti ec ec at lat at em pm pm em pm pm Po na al uc ns M du te uc on lS lS nit ob pu er ag elo str elo ur elo Tra elo ic ro Ed Re vir & cia ra Sa ov om Gl Po oc an ra lP ev ev ev ev Ru gy En rG an rty Pr & Inf on lM & cia nD rD lD nD er Fin ve to ly n, Ec En So cia cia cto pp ma ba ec tio Po So Su an cS Ur Se tri Hu Fin Nu er bli te at iva Pu h, W alt Pr He Sector Share of observations with 4-point QAG ratings Share of TA Source: IEG, based on World Bank administrative data (adjusted by IEG for misclassification) and QAG data. Note: Global ICT = global information, communications, and technology; QAG = Quality Assurance Group; TA = technical assistance. Figure B.28: Sectoral Shares of TA with Six-Point QAG Ratings and All Bank TA 16 14 12 10 Percent 8 6 4 2 0 y ion ng nt t r T ion t re on t t e elo r t on t t ion en to en en en cto en or en lic IC nc me tu ini cti cti ec sp at lat at em pm pm em pm pm Po na al Se c M du te uc on lS u n nit b pu er ag elo str elo ur Tra elo ic o ro l Ed Re vir & cia ra Sa ov Gl Po om oc an ra lP ev ev ev ev Ru gy En rG an ty Pr Inf & lM & on cia nD rD lD nD er er Fin to ly n, Ec En So v cia cia cto pp ma ba ec tio P o So Su an cS Ur Se tri Hu Fin Nu bli er te at iva Pu h, W alt Pr He Sector Share of observations with QAG 6-point scale ratings Share of TA Source: IEG, based on World Bank administrative data (adjusted by IEG for misclassification) and QAG data. Note: Global ICT = global information, communications, and technology; QAG = Quality Assurance Group; TA = technical assistance. 104 APPENDIX B: METHODOLOGY Figure B.29: Regional Shares of ESW with Four- and Six-Point QAG Ratings and All Bank ESW 35 30 25 20 Percent 15 10 5 0 Sub-Saharan East Asia Europe and Latin America Middle East and South Asia Africa and Pacific Central Asia and the Caribbean North Africa Region QAG 4-point QAG 6-point All ESW Source: IEG, based on World Bank administrative and QAG data. Note: ESW = economic and sector work; QAG = Quality Assurance Group. Figure B.30: Regional Shares of TA with Four- and Six-Point QAG Ratings and All Bank TA 30 25 20 Percent 15 10 5 0 Sub-Saharan East Asia Europe and Latin America Middle East and South Asia Africa and Pacific Central Asia and the Caribbean North Africa Region QAG 4-point QAG 6-point All TA Source: IEG, based on World Bank administrative and QAG data. Note: QAG = Quality Assurance Group; TA = technical assistance. 105 APPENDIX C: PORTFOLIO OF WORLD BANK ESW AND TA, FISCAL 2000–06 Table C.1: Portfolio of World Bank ESW and TA, Fiscal 2000–06 A: Number of ESW Products Delivered to Clients Fiscal Fiscal Fiscal Fiscal Fiscal Fiscal Fiscal 2000 2001 2002 2003 2004 2005 2006 Total Country 309 295 366 536 569 525 471 3,071 Regional 64 47 92 133 104 86 95 621 Global 2 57 61 83 35 238 Total 373 342 460 726 734 694 601 3,930 Shares Country (%) 82.8 86.3 79.6 73.8 77.5 75.6 78.4 78.1 Regional (%) 17.2 13.7 20.0 18.3 14.2 12.4 15.8 15.8 Global (%) 0.0 0.0 0.4 7.9 8.3 12.0 5.8 6.1 B: Cost of ESW Products Delivered to Clients (US$ 000) Fiscal Fiscal Fiscal Fiscal Fiscal Fiscal Fiscal 2000 2001 2002 2003 2004 2005 2006 Total Country 46,745 46,423 51,185 71,455 86,010 91,153 97,916 490,887 Regional 15,610 4,275 7,766 16,093 16,055 27,916 21,338 109,053 Global 54 4,319 8,044 13,838 4,114 30,368 Total 62,355 50,698 59,004 91,867 110,108 132,907 123,368 630,308 Shares Country (%) 75.0% 91.6 86.7 77.8 78.1 68.6 79.4 77.9 Regional (%) 25.0% 8.4 13.2 17.5 14.6 21.0 17.3 17.3 Global (%) 0.0% 0.0 0.1 4.7 7.3 10.4 3.3 4.8 C: Number of NLTA Products Delivered to Clients Fiscal Fiscal Fiscal Fiscal Fiscal Fiscal Fiscal 2000 2001 2002 2003 2004 2005 2006 Total Country 95 147 219 276 235 282 233 1,487 Regional 28 15 34 57 56 67 71 328 Global 5 1 12 6 12 2 3 41 Total 128 163 265 339 303 351 307 1,856 Shares Country (%) 74.2 90.2 82.6 81.4 77.6 80.3 75.9 80.1 Regional (%) 21.9 9.2 12.8 16.8 18.5 19.1 23.1 17.7 Global (%) 3.9 0.6 4.5 1.8 4.0 0.6 1.0 2.2 (table continues on next page) 107 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Table C.1: Portfolio of World Bank ESW and TA, Fiscal 2000–06 (continued) D: Cost of NLTA Products Delivered to Clients (US$ 000) Fiscal Fiscal Fiscal Fiscal Fiscal Fiscal Fiscal 2000 2001 2002 2003 2004 2005 2006 Total Country 9,615 14,908 36,800 35,557 26,974 34,886 34,159 192,898 Regional 6,314 4,582 5,992 13,150 14,492 16,213 17,913 78,657 Global 432 40 1,835 1,095 2,174 420 1,978 7,973 Total 16,360 19,530 44,626 49,802 43,640 51,520 54,050 279,528 Shares Country (%) 58.8 76.3 82.5 71.4 61.8 67.7 63.2 69.0 Regional (%) 38.6 23.5 13.4 26.4 33.2 31.5 33.1 28.1 Global (%) 2.6 0.2 4.1 2.2 5.0 0.8 3.7 2.9 Source: World Bank administrative data. Note: n.a. = not applicable; NLTA = nonlending technical assistance. Average costs of country, regional, and global ESW and TA, fiscal 2000–06 Figure C.1: Average Cost of Country, Regional, and Global ESW, Fiscal 2000–06 350,000 300,000 250,000 Cost (US$) 200,000 150,000 100,000 50,000 0 2000 2001 2002 2003 2004 2005 2006 Fiscal year Country Regional Global All Source: Derived by IEG from Bank administrative data. Note: ESW = economic and sector work. 108 A P P E N D I X C : P O R T F O L I O O F W O R L D B A N K E S W A N D TA , F I S C A L 2 0 0 0 – 0 6 Figure C.2: Average Cost of Country, Regional, and Global TA, Fiscal 2000–06 700,000 600,000 500,000 Cost (US$) 400,000 300,000 200,000 100,000 0 2000 2001 2002 2003 2004 2005 2006 Fiscal year Country Regional Global All Source: Derived by IEG from Bank administrative data. Note: TA = technical assistance. Sectoral distribution of ESW and TA, fiscal 2000–06 Figure C.3: Number of ESW by Sector Board Classification, Fiscal 2000–06 700 600 500 400 Number 300 200 100 0 nt re al n nt rt t t t g n t on ion n r nt e r y en en en en to to lic tio nin tio tio nc o ob me tu me me cti sp ec ec at m em pm em Po na ita ula uc uc Gl i M te p p n on uc lS op lS er d elo str elo Tra ur elo ag n ic p ro Ed Re vir l Sa & ra cia ov Po e om oc an ra lP ev ev ev ev Ru En gy rG Pr rty an & Inf & lM on cia nD nD lD rD er Fin ply ve to n, Ec En So cia cia cto ma ba ec tio Po up So Ur an cS Se tri Hu S Fin Nu er bli te at iva Pu h, W alt Pr He Sector Source: World Bank administrative data, adjusted by IEG for misclassification. Notes: The number of ESW includes country, Regional, and global ESW. ESW = economic and sector work; Global ICT = global information, communications, and technology. 109 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Figure C.4: Number of TA by Sector Board Classification, Fiscal 2000–06 250 200 150 Number 100 50 0 t re t t on ion on T t r y ng t nt e t r ion rt ion en en en en to en en to lic IC nc po tu me cti cti ini ec ec at at pm em em pm pm pm Po na lat al uc ns du te M uc lS on lS nit ob er elo str ur ag pu elo elo elo ic Tra ro Re Ed vir ra & cia Sa ov Gl om oc an Po ra lP ev ev ev ev Ru gy En rG Pr rty an Inf & lM on cia nD nD lD rD & er Fin ve to ly Ec n, So En cia cia cto pp ma ba ec Po tio So Su an Ur cS Se Hu tri Fin bli er Nu te at iva Pu h, W Pr alt He Sector Source: World Bank administrative data, adjusted by IEG for misclassification. Note: The number of TA includes country, regional, and global TA. Global ICT = global information, communications, and technology; TA = technical assistance. Table C.2: The Top ESW Sectors (by numbers) by Region, Fiscal 2000–06 East Europe Latin Middle Sub- Asia and America East and Saharan and Central and the North South Sector Africa Pacific Asia Caribbean Africa Asia Global Total Economic policy Financial sector Public sector governance Private sector development Rural sector Energy and Mining Poverty reduction Education * Social protection * Health, nutrition, and population Source: IEG. Note: Shaded cells are top ESW sectors for Bank. = a top five sector by the number of ESW. * = equal numbers of ESW in the marked sectors for the particular Region. 110 A P P E N D I X C : P O R T F O L I O O F W O R L D B A N K E S W A N D TA , F I S C A L 2 0 0 0 – 0 6 Table C.3: The Top TA Sectors (by numbers) by Region, Fiscal 2000–06 East Europe Latin Middle Sub- Asia and America East and Saharan and Central and the North South Sector Africa Pacific Asia Caribbean Africa Asia Global Total Financial * * Private sector development * * Public sector governance ** Environment * ** *** * Social development * ** Energy and Mining ** Economic policy * *** Rural sector * Water Supply and Sanitation ** Urban development * ** Social protection * Transport Global ICT * * Health, nutrition, and population * Source: IEG. Note: Shaded cells are top TA sectors for Bank. = top sector by the number of TA. *, **, and *** indicate equal numbers of TA in the marked sectors for the particular Regions. Figure C.5: Average Cost of Country-Specific ESW by Sector, Fiscal 2000–06 300,000 250,000 200,000 Cost (US$) 150,000 100,000 50,000 0 t cia tion ra or r e y vir ent nt ion on t nt t Tra re rt y & CT ng Po ent ion ion t en cto en en en lic nc po t me l M pme tu cti ini lI ec at lat at pm m em em m pm Po na c e uc ns a M Fin rote du uc lS lS lop on nit lop b pu er elo ur elo ag str elo ic o Re Ed Sa ov om e Gl e oc an lP ra ev ev v ev ev u En rG e rty an Pr & R Inf rg & on cia Nu r D nD lD dD nD ne to ly ve n, Ec So cia cia to pp ba an ma ec tio E Po ec So Su Ur cS an tri er Hu He te S Fin nd bli er at Ge iva Pu h, W alt Pr Sector Source: World Bank administrative data. Note: ESW = economic and sector work; Global ICT = global information, communications, and technology. 111 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Figure C.6: Average Cost of Country-Specific TA by Sector, Fiscal 2000–06 250,000 200,000 Cost (US$) 150,000 100,000 50,000 0 t t t t re T nt n nt om ce y n n ion n Ru ctor or ng elo t elo t t en en en en r n en lic IC tio tio tio tio po tu me me er Sect n D pme n ini at em em pm pm pm Po na al a So educ ec ca e uc ns M on l lop nit lS ob pu er ot u ur ag elo elo str Tra ic l Ed vir Sa cia ra & ov Gl Po c S eve Pr oc R an ra ev ev ev ev En gy rG Pr rty an ial & Inf lM & on dD nD lD rD Fin to ve c ly n, Ec En cia cia cto pp an ma ba ec tio Po So Su an Ur Se tri er Hu Fin nd Nu er bli te Ge at iva Pu h, W alt Pr He Sector Source: World Bank administrative data. Note: Global ICT = global information, communications, and technology; TA = technical assistance. Figure C.7: Ratio of Average Cost of Country-Specific ESW to TA Tasks, Fiscal 2000–06 Human Development Procurement Environment Poverty Reduction Social Development Financial Management Water Supply & Sanitation Gender and Development Public Sector Governance Economic Policy Sector Health, Nutrition, & Population Global ICT Rural Sector Social Protection Financial Sector Infrastructure Education Private Sector Development Energy & Mining Urban Development Transport 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 Ratio of cost Source: IEG calculations based on World Bank administrative data. Note: ESW = economic and sector work; Global ICT = global information, communications, and technology; TA = technical assistance. 112 A P P E N D I X C : P O R T F O L I O O F W O R L D B A N K E S W A N D TA , F I S C A L 2 0 0 0 – 0 6 Regional distribution of ESW and TA Figure C.8: Number of ESW per Country by Region, Fiscal 2000–06 70 60 50 40 Number 30 20 10 0 Sub-Saharan East Asia Europe and Latin America Middle East and South Asia Africa and Pacific Central Asia and the Caribbean North Africa Region Number of country ESW per country Number of regional ESW per country Total number of ESW per country Source: IEG calculations based on World Bank administrative data. Note: ESW = economic and sector work. Figure C.9: Number of TA per Country by Region, Fiscal 2000–06 25 20 15 Number 10 5 0 Sub-Saharan East Asia Europe and Latin America Middle East and South Asia Africa and Pacific Central Asia and the Caribbean North Africa Region Number of country TA per country Number of regional TA per country Total number of TA per country Source: IEG calculations based on data from World Bank administrative data. Note: TA = technical assistance. 113 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Although Sub-Saharan Africa delivered the small- gional ESW to country-specific ESW (followed est number of Regional ESW (per country), it closely by East Asia and Pacific), and South Asia spent the most on regional ESW as a share of total the lowest (figure C.10). spending on ESW—27 percent compared with an average of 18 percent for all Regions (with South Although Sub-Saharan Africa did not deliver the Asia spending the smallest share, 6 percent). largest number of TA per country, it spent by far the These differences in the cost shares of Regional most on Regional TA as a share of total spending ESW were due entirely to differences in the on TA (as it did for ESW). Fifty-nine percent of average costs of such ESW—they were the most Sub-Saharan Africa TA spending was on Regional expensive in Sub-Saharan Africa, costing an aver- TA, compared with an average of 29 percent for all age of around $201,000, and the least expensive Regions (with East Asia and Pacific and South Asia in South Asia, costing an average of around spending the least—12 and 13 percent, respec- $105,000. tively). The average cost of Regional TA in Sub- Saharan Africa was the most expensive of all the Re- Conversely, country-specific ESW cost the least in gions (as it was in the case of Regional ESW)— Sub-Saharan Africa, around $141,000, compared $308,200 compared with the average of $239,805 with the average of around $160,000 for all Re- for all regional TA. Latin America and the Caribbean gions. South Asia had among the most expen- and South Asia had the least expensive Regional TA, sive country-specific ESW, averaging around which cost an average of around $110,000 and $171,000. That was topped by East Asia and Pacific, $125,000, respectively. Sub-Saharan Africa had the for which the average cost of country-specific highest ratio of average cost of Regional TA to ESW was $177,000. This means that Sub-Saharan country-specific TA (as it did with ESW), and Latin Africa had the highest ratio of average cost of Re- America and the Caribbean the lowest (figure C.11). Figure C.10: Ratio of Average Cost of Regional ESW to Country-Specific ESW, Fiscal 2000–06 1.60 1.40 1.20 1.00 Ratio of cost 0.80 0.60 0.40 0.20 0.00 Sub-Saharan East Asia Europe and Latin America Middle East and South Asia All Africa and Pacific Central Asia and the Caribbean North Africa Region Source: IEG calculations based on World Bank administrative data. Note: ESW = economic and sector work. 114 A P P E N D I X C : P O R T F O L I O O F W O R L D B A N K E S W A N D TA , F I S C A L 2 0 0 0 – 0 6 Misclassification of ESW, TA, and Loans Figure C.11: Ratio of Average Cost of Regional TA in the Bank Administrative Database to Country-Specific TA, Fiscal 2000–06 For the purpose of this evaluation, the IEG team reviewed the administrative database for ESW , 3.00 TA, and loans for fiscal 2000–06 to correct for 2.50 misclassification of sector board. The review in- 2.00 dicated that 13 percent of ESW , 20 percent of TA, Ratio of cost and 10 percent of loans were misclassified (tables 1.50 C.4–C.6). Table C.7 provides numbers of ESW by 1.00 report type. 0.50 0.00 Sub-Saharan East Asia Europe and Latin America Middle East South Asia All Africa and Pacific Central Asia and the and Caribbean North Africa Region Source: IEG calculations based on World Bank administrative data. Note: TA = technical assistance. Table C.4: Misclassification of ESW Number of Number of Total Percent of correctly misclassified number of misclassified Sector classified ESW ESW ESW ESW Economic policy 561 90 651 13.8 Education 202 5 207 2.4 Energy and mining 144 14 158 8.9 Environment 104 28 132 21.2 Financial management 176 16 192 8.3 Financial sector 369 26 395 6.6 Gender and development 44 1 45 2.2 Global ICT 23 0 23 0.0 Health, nutrition, and population 171 5 176 2.8 Operational services 1 17 18 94.4 Poverty reduction 198 82 280 29.3 Private sector development 304 26 330 7.9 Procurement 121 5 126 4.0 Professional development 2 0 2 0.0 Project finance and guarantees 1 0 1 0.0 Public sector governance 251 31 282 11.0 Resource management 2 0 2 0.0 Rural sector 254 35 289 12.1 Sector board not applicable 0 24 24 100.0 Social development 71 32 103 31.1 Social protection 163 28 191 14.7 Transport 103 8 111 7.2 Urban development 95 27 122 22.1 Water supply and sanitation 64 6 70 8.6 Total 3,424 506 3,930 12.9 Source: IEG and World Bank administrative data. Note: ESW = economic and sector work; global ICT = global information, communications, and technology. 115 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Table C.5: Misclassification of TA Number of Number of Percent of correctly misclassified Total number misclassified Sector classified TA TA of TA TA Administrative and client support 2 0 2 0.0 Economic policy 102 28 130 21.5 Education 65 3 68 4.4 Energy and mining 98 10 108 9.3 Environment 133 15 148 10.1 Financial management 13 34 47 72.3 Financial sector 195 21 216 9.7 Gender and development 18 2 20 10.0 Global ICT 35 0 35 0.0 Health, nutrition, and population 68 4 72 5.6 Operational services 7 6 13 46.2 Poverty reduction 48 36 84 42.9 Private sector development 191 128 319 40.1 Procurement 9 2 11 18.2 Professional development 3 0 3 0.0 Project finance and guarantees 1 0 1 0.0 Public sector governance 110 19 129 14.7 Resource management 1 0 1 0.0 Rural sector 71 15 86 17.4 Sector board not applicable 0 9 9 100.0 Social development 96 7 103 6.8 Social protection 57 13 70 18.6 Transport 49 0 49 0.0 Urban development 79 12 91 13.2 Water supply and sanitation 38 3 41 7.3 Total 1,489 367 1,856 19.8 Source: IEG and World Bank administrative data. Note: Global ICT = global information, communications, and technology; TA = technical assistance. 116 A P P E N D I X C : P O R T F O L I O O F W O R L D B A N K E S W A N D TA , F I S C A L 2 0 0 0 – 0 6 Table C.6: Misclassification of Loans Number of Number of Share of correctly misclassified Total number misclassified Sector classified loans loans of loans loans Economic policy 104 21 125 16.8 Education 160 3 163 1.8 Energy and mining 125 2 127 1.6 Environment 49 8 57 14.0 Financial sector 89 9 98 9.2 Gender and development 1 0 1 0.0 Global ICT 16 0 16 0.0 Health, nutrition, and population 171 4 175 2.3 Poverty reduction 22 7 29 24.1 Private sector development 58 22 80 27.5 Public sector governance 149 41 190 21.6 Rural sector 239 18 257 7.0 Social development 36 4 40 10.0 Social protection 90 22 112 19.6 Transport 163 6 169 3.6 Urban development 91 5 96 5.2 Water supply and sanitation 85 10 95 10.5 Total 1,648 182 1,830 9.9 Source: IEG and World Bank administrative data. Note: Global ICT = global information, communications, and technology. 117 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Average cost of ESW and TA in IBRD versus IDA countries Figure C.12: Ratio of Average Cost of ESW for IBRD versus IDA Countries, Fiscal 2000–06 Water Supply & Sanitation Transport Private Sector Development Global ICT Education Infrastructure Social Protection Poverty Reduction Public Sector Sector Urban Development Financial Sector Health, Nutrition, & Population Economic Policy Environment Social Development Human Development Energy & Mining Rural Sector Financial Procurement 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 Ratio: IBRD versus IDA Source: IEG calculations based on World Bank administrative data. Note: ESW = economic and sector work; Global ICT = global information, communications, and technology; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association. Figure C.13: Ratio of Average Cost of TA for IBRD versus IDA Countries, Fiscal 2000–06 Urban Infrastructure Social Protection Health, Nutrition, & Population Economic Policy Public Sector Financial Social Development Financial Sector Sector Poverty Reduction Water Supply & Sanitation Environment Rural Sector Energy & Mining Global ICT Private Sector Development Transport Education Human Procurement 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 2.20 2.50 2.75 3.00 Ratio: IBRD versus IDA Source: IEG calculations based on World Bank administrative data. Note: Global ICT = global information, communications, and technology; IBRD = International Bank for Reconstruction and Development; IDA = International Develop- ment Association; TA = technical assistance. 118 A P P E N D I X C : P O R T F O L I O O F W O R L D B A N K E S W A N D TA , F I S C A L 2 0 0 0 – 0 6 Trust fund contribution to ESW costs, by Region 58 percent of all trust funds for TA) than to Re- and sector gional (38 percent) or global tasks (4 percent). As Sub-Saharan Africa received the largest amount of with ESW , Sub-Saharan Africa received more trust trust funds for ESW among all the Regions, al- funds for Regional than for country-specific TA. though it was not the top recipient of such funds on a per country basis. On a per country basis, As in the case for ESW, Sub-Saharan Africa re- South Asia received the largest amount, and Latin ceived the largest amount of trust funds for TA America and the Caribbean received the smallest among all the Regions, although again it was not amount. the top recipient of these funds on a per country basis. On a per country basis, East Asia and Pacific Of the trust funds that went to ESW , 65 percent was the top recipient, and Latin America and the went to country-specific ESW , 31 percent to re- Caribbean received the lowest amount (as it also gional ESW, and 4 percent to global ESW . More trust did in the case of ESW). funds went to country-specific than to Regional ESW in all the Regions except for Sub-Saharan For global TA, trust funds favored PSD and global Africa, where it was the reverse. ICT (information, communications, and tech- nology), which received 34 and 22 percent, re- The bulk of trust funds for global ESW (28 percent) spectively, of all the trust funds that financed went to PSD. Smaller amounts went to energy and global TA in the Bank. Global TA tasks in global mining and urban development, but trust funds ICT were virtually all trust funded, as were global played a very important role in these sectors, TA in environment and in water supply and san- contributing 98 and 89 percent, respectively, of the itation; trust funds contributed to around 95 per- costs of ESW . cent of all the costs of all these global TA. Global ICT and environment were also among the top With respect to Regional ESW, PSD was favored sectors (in numbers) for global TA Bank-wide. by trust funds, receiving 27 percent of all trust funds that went to Regional ESW in the Bank, For Regional TA, social development was the top followed by the rural sector; education; energy and sector favored by trust funds, receiving 15 percent mining; and health, nutrition, and population. of all trust funds that financed Regional TA in the All the sectors favored by trust funds except for Bank; this was followed by energy and mining, en- energy and mining were among the top sectors vironment, water supply and sanitation, and the for Regional ESW Bank-wide. rural sector. Of the sectors of trust fund focus, so- cial development and environment were among Trust funds spread across more sectors with re- the top sectors for Regional TA Bank-wide. spect to country-specific ESW than Regional ESW, focusing on the rural, poverty reduction, envi- For country-specific TA, the sectors favored by ronment, economic policy, water supply and san- trust funds were PSD, urban development, energy itation, and education sectors. Three of these and mining, water supply and sanitation, and overlapped with the top sectors for country- environment. Compared with Regional TA, there specific ESW Bank-wide: economic policy, rural was greater overlap between the top sectors for sector, and poverty reduction. country-specific Bank-wide and those that were of trust fund focus, with PSD, energy and mining, Trust fund contribution to TA costs, by Region and environment among the top sectors for both. and sector Similar to ESW , a larger amount of trust funds went to country-specific tasks (which received 119 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Table C.7: Numbers of ESW by Report Types, Fiscal 2000–06 Core diagnostic Advisory reports Integrative Fiduciary Assessment 14 Public Investment Review 3 Development Policy Review 35 Strategic Environmental Assessment/Analysis 3 Country Financial Accountability Assessment 104 Commodities Study 5 Country Economic Memorandum 105 Law and Justice Study 8 Poverty Assessment 107 Other procurement study 9 Country Procurement Assessment 109 Women in Development and Gender Study 11 Public Expenditure Review 148 Debt and Creditworthiness Study 14 Subtotal 622 Other financial accountability study 14 Mining/Oil and Gas 27 Other diagnostic Economic updates and modeling 28 City Development Strategy 1 AML/CFT 34 Energy-Environment Review 2 Other urban study 53 Recent Economic Developments in Infrastructure 2 Energy study 54 Country Infrastructure Framework 6 Other rural study 61 Knowledge Economy Study 9 Foreign Trade, FDI, and Capital Flows 72 Country Environmental Analysis 14 Other education study 74 Country Gender Assessment 16 Other social protection study 78 Risk and Vulnerability Assessment 17 Other environmental study 82 Institutional and Governance Review 18 Other health study 82 Legal and Judicial Sector Assessment 20 General economy, macroeconomics, and growth 83 Rural Development Assessment 22 Other agricultural study 85 Insolvency Assessment (ROSC) 27 Other poverty study 95 Health Sector Review 32 PSD, Privatization and Industrial Policy 99 Education Sector Review 40 Other public sector study 109 Social analysis 42 Other infrastructure study 131 Corporate Governance Assessment (ROSC) 48 Other financial sector study 141 Accounting and Auditing Assessment (ROSC) 54 Subtotal 1,455 Investment Climate Assessment 80 Financial Sector Assessment Program 100 Other report types (not classified by the Bank) Subtotal 550 Human capital development 6 Social and structural review 2 Not assigned 1,240 Other 55 Total 3,930 Source: World Bank data. Note: AML/CFT = anti-money laundering and combatting the finacing of terrorism; FDI = foreign direct investment; ROSC = Report on the Observance of Standards and Codes. 120 APPENDIX D: ECONOMETRIC ANALYSIS Effects of ESW on Lending fiscal 2000–06, only those loans approved from fis- An econometric exercise was conducted to ana- cal 2003 onward were selected to allow for ESW lyze the effects of ESW on the quality of lending. delivered up to three years prior to loan approval The exercise looked at several dimensions of ESW to be considered for informing the loan. that could have an effect on lending: Two sets of regressions were run. The first set es- • The existence of ESW (that is, whether there timates the effects of various dimensions of ESW was at least one ESW that could have informed on lending quality ratings for those ESW that the loan) could have informed the loan (see Data section, • The number of ESW below). The second set estimates the effects of var- • The unit cost of ESW ious dimensions of ESW on loan ratings for all the • The quality of ESW ESW in the same sector of the loan delivered up • Whether partnership in the production of ESW to three years prior to loan approval. mattered for ESW to have an effect on lending • Whether origination of ESW mattered for ESW Data to have an effect on lending. Lending quality The first three dimensions are self-explanatory. As mentioned, QAE ratings given by QAG were The quality dimension is based on the hypothe- used to proxy for lending quality. These ratings are sis that ESW that have been rated highly (for in- given to a randomly selected sample of lending ternal quality or strategic relevance or likely impact) operations soon after Board approval. The rat- could lead to better quality of lending. The last two ings are based on QAG’s assessments of loan ob- dimensions are predicated on the hypothesis that jectives, the likelihood of achieving development ESW that were undertaken in partnership with objectives, and the underlying logic and results or requested by clients could be more pertinent framework. to the needs of the countries and that such ESW would be more relevant for lending and would im- Ratings are given for nine categories: strategic prove the quality of lending. relevance and approach; technical, financial, and economic aspects; poverty, gender, and social QAE ratings provided by QAG were used to proxy development; environmental aspects; fiduciary for the quality of lending. The regressions were aspects; policy and institutional aspects; imple- run on a selected sample of 119 loans with QAE mentation arrangements; risk assessment; and ratings. This sample represents approximately bank inputs and processes. There are also ratings 50 percent of all the loans approved during fis- on subcategories under each of the categories. Ad- cal 2003–051 that have QAE ratings (196 loans), ditionally, there is an overall assessment rating that stratified by Region and sector. The 119 loans is a synthesis of the ratings for the first eight cat- with QAE ratings consist of 97 investment loans egories (excluding the ratings for Bank inputs and 22 DPLs. Given that the evaluation period is and processes). 121 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS For fiscal 2003, the QAE ratings are on a four- d. Policy and institutional aspects point scale, with a lower number associated with e. Quality of risk assessment—The subrating on a better rating. From fiscal 2004 onward, the rat- financial management capacity ings are on a six-point scale. For the purpose of f. Overall assessment of projects. this analysis, the ratings on a six-point scale are converted to ratings on a four-point scale using The explanatory variables the conversion formula used by QAG. These include the numbers (costs) and the av- erage unit costs of ESW supporting a loan. When the number of ESW was found to have no signif- ESW that could have informed the loans icant effect on the lending quality rating, the eval- The portfolio of ESW delivered up to three years uation also looked into the possible effect of the prior to loan approval was reviewed to identify existence of ESW on lending quality, that is, those ESW that could have actually informed the whether the loans that are supported by at least loan (the relevant ESW). The matching entailed re- one ESW have better quality. view of loan documents as well as ESW to identify the relevant ESW . For DPLs, which are generally Two sets of origination data were used, sepa- multisectoral, ESW in all sectors were reviewed. For rately: Bank administrative data and responses investment loans, the review included ESW in the to the ESW TTL survey. The partnership data are economic policy and financial management sec- from responses to the ESW TTL survey. For qual- tors in addition to those in the same sector as the ity of ESW , four QAG quality ratings were used, sep- loan. Based on this review, only those ESW that arately: overall quality, strategic relevance and could actually have informed the loans were se- timeliness, dialogue and dissemination, and likely lected for inclusion in the analysis. impact. Because the regressions are at the loan level but these variables are at the ESW task level, Econometric Specification the latter are averaged for the regressions. And be- cause these task-level data are available only for Dependent variables: Lending quality ratings a limited number of ESW , the averages are not nec- A subset of the QAE ratings and subratings was se- essarily representative of all the ESW supporting lected for the econometric exercise. The selection a loan. was based on the possibility of their being influ- enced by ESW , as well as data availability. Specifi- The controls cally, the following QAE ratings were selected: The regressions controlled for: a. Strategic relevance and approach, and all • Size of the loan four subratings: • Loan preparation cost • Coherence and consistency of development • QAE rating on task team’s composition in re- rationale and results framework underpin- lation to the operation’s complexity as proxy ning the project for the quality of the task team (the QAE sub- • Consistency of the project’s objectives with ratings of this category are for skill mix, conti- the country and sector strategies nuity, experience, and staff/consultants mix) • Clarity, realism, and scope of the project’s de- • Whether the loans are investment or DPLs velopment objectives • Whether the country is an IDA country • Adequacy of country and sector knowledge • Region of the loan. underpinning the project b. Technical, financial and economic aspects Apart from ESW , loans can also be informed by c. Fiduciary aspects—two subratings: Bank research. However, no reliable systematic in- • Financial management formation was available on the size and costs of • Adequacy and quality of financial manage- these research activities to be included in the ment arrangements econometric analysis. 122 A P P E N D I X D : E C O N O M E T R I C A N A LY S I S In specifications where the variables of interest are ing categories 1 and 2 versus 3 and 4 or rating cat- origination, partnership, and the quality of the egories 1, 2, and 3 versus 4. When the parallel lines ESW , no control variables were used because of assumption is violated for some explanatory vari- the low number of observations. ables, using an ordered probit model can lead to erroneous conclusions because one set of coef- Estimation Methodology ficients can no longer describe the relationship be- Ordered probit models were used to estimate tween different groups of rating categories. the specifications because the quality of lending ratings are ordinal rather than cardinal (an ordi- Accordingly, the parallel regression assumption nary least square model is not appropriate, as it was tested for each ordered probit specification. requires the dependent variable to be cardinal). Whenever the assumption is violated, a general- ized ordered probit model was used. This allows The validity of the ordered probit model crucially for the relaxation of that assumption. hinges on the parallel regressions (or parallel lines) assumption, which implies that the rela- Specifications have been estimated with and with- tionship between all pairs of rating categories is out the control variables. Control variables in dif- the same. In other words, an ordered probit ferent combinations were introduced to ensure model assumes that the coefficients describing the that the results are not sensitive to the specifica- relationship between rating category 1 and rating tion. Some lending quality rating variables did categories 2, 3, and 4 combined are the same as not have enough observations in every rating cat- those that describe the relationship between rat- egory. For specifications involving those variables, Table D.1: Generalized Ordered Probit Regressions on Lending Quality QAE overall QAE policy and QAE financial Dependent variable assessment institutional aspects management capacity Independent variables 1➔2,3 1,2➔3 1➔2,3 1,2➔3 1➔2,3 1,2➔3 Existence of ESW –0.536** –0.536** –0.709*** –0.709*** (0.270) (0.270) (0.206) (0.206) Financial management ESW –0.404*** –0.404*** (0.118) (0.118) Size of the loan –0.000676 –0.000676 –0.000159 –0.000159 –0.000116 –0.000116 (0.000760) (0.000760) (0.000921) (0.000921) (0.000617) (0.000617) Loan preparation cost 0.00125*** –0.000484 0.000423 0.000423 –0.000104 0.000985 (0.000474) (0.000818) (0.000653) (0.000653) (0.000528) (0.000662) Task team quality 1.530*** 1.530*** 1.433*** 1.433*** 0.147 0.147 (0.181) (0.181) (0.345) (0.345) (0.214) (0.214) Dummy for IDA countries 0.156 0.156 0.612 –0.910*** 0.382 0.382 (0.332) (0.332) (0.519) (0.168) (0.319) (0.319) Dummy for investment loans –0.284 –0.284 –0.171 3.619*** –0.193 –0.193 (0.333) (0.333) (0.652) (0.280) (0.135) (0.135) Number of observations 116 116 115 Wald chi-square 82.70 3365.5 21.92 Prob > chi 2 0.00 0.00 0.00 Source: IEG calculations. Note: Standard errors are in parentheses. ESW = economic and sector work; IDA = International Development Association; QAE = quality at entry. *, **, and *** indicate significance at 10%, 5%, and 1% level, respectively. 123 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS • Overall assessment—Loans that are preceded Table D.2: Ordered Probit Regressions on Loan by at least one ESW are more likely to get a bet- Quality ter rating, although the actual number of ESW QAE strategic QAE adequacy of did not matter. Loans with high task team qual- Dependent variable relevance and country and sector ity are more likely to get better ratings. Independent variables approach knowledge • Policy and institutional aspects—Loans that Existence of ESW –0.250*** are preceded by at least one ESW are more likely (0.0696) to get a better rating. The actual number of ESW Number of ESW –0.132* did not matter. Loans with high task team qual- (0.0700) ity are more likely to get better ratings. DPLs are Size of loan –0.00129 0.00173 more likely to get a rating of 2 or better. (0.00116) (0.00124) • Strategic relevance and approach—Loans that are preceded by at least one ESW are more Loan preparation cost –0.000163 0.000964 likely to get a better rating. The actual number (0.000400) (0.000766) of ESW did not matter. Loans with high task Loan supervision cost team quality are more likely to get better ratings. Task team quality 1.357*** 1.602*** • Adequacy of country and sector knowl- (0.266) (0.419) edge—The number of ESW has a significant Dummy for IDA countries –0.340 0.395 effect on the rating, although the coefficient is (0.230) (0.248) significant only at the 10 percent level.2 Loans Dummy for investment loans –0.0486 –0.353 that are preceded by a greater number of sup- (0.216) (0.327) porting ESW are more likely to have better rat- Inflation rate ings. Loans with high task team quality are also GDP growth rate more likely to get better ratings. Initial level of GDP per capita • Financial management capacity—Loans Number of observations 116 115 that actually cited Country Financial Accounta- bility Assessment (or other similar financial man- Wald chi-square 337.91 131.17 agement ESW) in the project appraisal Prob > chi2 0.00 0.00 documents are more likely to get a better rat- Source: IEG calculations. Note: ESW = economic and sector work; GDP = gross domestic product; IDA = International Development ing. The mere existence of a Country Financial Association; QAE = quality at entry. * and *** indicate significance at 10% and 1% level, respectively. Accountability Assessment has no effect on the rating. None of the control variables is significant. the ratings scale was transformed by combining ESW (existence, number, average unit cost) did not the ratings category in a way that preserves the have any effects on the QAE ratings on three of the ordering across categories. For specifications in- four subratings under the strategic relevance and volving the number and the average unit cost of approach dimension (specifically coherence and ESW , a robustness check was performed by drop- consistency of development rationale and results ping some extreme observations to ensure that framework underpinning the project; consistency the results are not driven by those observations. of project objectives with country and sector strate- gies; and clarity, realism, and scope of project’s de- Results velopment objectives). ESW also did not have any effects on the QAE ratings on technical, financial, Regressions of loans on ESW that could have and economic aspects or the fiduciary aspects. informed the loans The regression results are presented in tables Regressions of loans on ESW D.1 and D.2. For the purpose of brevity, only in the same sector those results for which ESW had a significant ef- A similar econometric exercise was carried out for fect on lending quality are presented. all loans approved during fiscal 2003–05 that 124 A P P E N D I X D : E C O N O M E T R I C A N A LY S I S have QAE ratings. For this exercise, all ESW (com- pleted up to three years prior to loan approval) Table D.3: Cost and Quality for ESW in the same sector board as the loans were in- cluded, not just those ESW that could have in- Specifications Specification 1 Specification 2 formed the loans. Bank budget –1.388*** –1.298*** (0.271) (0.271) The results for the overall QAE assessment and ad- Trust fund 0.420 0.482* equacy of country and sector knowledge ratings (0.260) (0.274) were very similar to the results discussed above. Regional ESW dummy –0.319** However, there were some counterintuitive re- (0.140) sults. This reflects the shortcomings of including Global ESW dummy –0.392 ESW based purely on sector boards without as- (0.260) certaining whether the ESW could have actually East Asia and Pacific –0.135 informed the loans. The results have not been re- (0.161) ported for the sake of brevity. Europe and Central Asia –0.231* (0.134) For all the regressions Latin America and the Caribbean –0.297** The average cost of ESW has no effect on lending (0.145) quality. Further, no significant association was Middle East and North Africa –0.004 found between origination, partnership, or qual- (0.159) ity of ESW and different dimensions of lend- South Asia –0.099 ing quality. However, given the limited data avail- (0.155) ability, the regression results involving origination, Number of observations 725 725 partnership, and quality of ESW variables are not Wald chi-square 26.23 36.79 conclusive. Prob > chi2 0.000 0.000 Source: IEG calculations. Note: Standard errors are in parentheses; *, ** and *** indicate significance at 10%, 5%, and 1% level, Bank Budget versus Trust Fund respectively. ESW = economic and sector work. Cost and Quality Regression analysis was undertaken to determine the association between technical quality and the cost of ESW and TA. The cost data are from the found for the trust fund component of the total Bank’s administrative database and the quality cost (table D.3). More specifically: data are from QAG (specifically, the internal qual- ity rating). The independent variables are the • The Bank budget component of total cost was Bank budget component of total cost, the trust significantly (at the 1 percent level) associated fund component of total cost, dummies for re- with the quality of the ESW . In other words, as gional and global products, and dummies for more Bank budget is spent on ESW , the more each Region. The specifications were estimated likely it is that the ESW will have a higher qual- using ordered probit models for ESW and TA ity rating. separately. • The trust fund component of total cost of ESW was not significantly associated with the qual- Overall, the regressions indicate that the posi- ity of the ESW . tive and significant association between cost and • There was no significant association between quality for ESW only holds for the bank budget the cost (trust fund or Bank budget compo- component of the total cost; no association was nents) and the quality of TA. 125 APPENDIX E: SPECIFIC EXAMPLES OF EFFECTS OF ESW AND TA FROM COUNTRY REVIEWS Table E.1: Selected Examples of FSAP Informing Policy Country Effects Bangladesh Defined the agenda for reforms in the financial sector (of both the Central Bank and the commercial banks), including improving risk-management systems, capital adequacy norms, and changes in the governance structure of the Central Bank. Guyana Some recommendations are being implemented, for example, strengthening oversight of the insurance industry and the Bank of Guyana having regular meetings with commercial banks. Jordan Led to the adoption of the supervisory ladder by the Insurance Commission; informed the authorities’ Strategy for Promoting Foreign Investments in Securities; led to the implementation of the prompt corrective action framework (although staff from the Central Bank of Jordan indicated that these recommendations were adopted but not implemented after the termination of the IMF program); led to the introduction of cross-border supervision with other central banks and publication of guidelines on corporate governance. Mauritius Helped strengthen banking supervision and led to the creation of the Financial Sector Commission. Peru Almost all the recommendations have been implemented: at the Central Bank, it helped introduce improvements in the payments and exchange systems; at the Superintendency of Banks, Insurance and Pension Fund Administrators, it confirmed that they were on the right track, led to the issuance of legislation on legal protection for supervisors, helped define strategies to deal with bank resolutions, and helped establish commercial judges. The “seal of approval of the Bank” enhanced the reputation of the superintendency, which facilitated its pushing changes through legislation. Serbia Together with the Financial Sector Policy Note and the post-FSAP TA, the FSAP has been associated with major reforms of and improvements in banking supervision (including improvements in regulatory procedures, education and development of staff, relations with third parties, and actuarial professions), bank privatizations, and bank closures. Source: IEG country reviews. Note: FSAP = Financial Sector Assessment Program; TA = technical assistance. 127 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Table E.2: Selected Examples of ESW on Public Sector Governance Informing Policy Country ESW task Effects Guyana PER Led to changes in the budget, including timely budgets, and greater transparency through the removal of discretions (all related to heavily indebted poor countries). Jordan Work Program for Resulted in the setting of cash expenditure ceilings for each line ministry and did away with Consolidating Budget annual budget negotiations for each ministry; also resulted in results-oriented budgeting and the Management Reforms removal of subsidies. PER Resulted in the consolidation of expenditures by sectors (rather than by ministries) using the IMF GFS 2001 classification, and the development of a Medium-Term Expenditure Framework to replace annual budgeting. Peru PER Focused attention on the quality of expenditures, especially in the social sectors (for example, the “Vaso the Leche” program); made mergers of social programs possible; influenced the budget law in moving toward results budgeting; and led to the issuance of public expenditure norms. Romania PEIR (2002) Became a basis for macroeconomic stabilization policies of the Natase government. Public Expenditure Contributed to the modification of the financial structure of the public sector in 2006, including Management (2005) the introduction of stress testing of insolvency conditions for local authorities. Serbia PEIR (2003) Resulted in improvements in public sector governance. Review of Public Helped launch reforms of the state administration. Expenditures Vietnam PER (various years) Have been very important for strengthening budget and expenditure policy, and balancing the goals of growth and poverty reduction; the 2005 PER produced a comprehensive evaluation of expenditure and now the government has a more sustainable fiscal policy; PERs have been used regularly by the government for policy-making and for preparing resolutions to reduce weaknesses in public expenditure management. Source: IEG country reviews. Note: ESW = economic and sector work; IMF GFS = International Monetary Fund government finance statistics; PEIR = Public Expenditure and Institutional Review; PER = Public Expenditure Review. 128 A P P E N D I X E : S P E C I F I C E X A M P L E S O F E F F E C T S O F E S W A N D TA F R O M C O U N T R Y R E V I E W S Table E.3: Selected Examples of ESW Informing Policy in the Economic Policy, Private Sector Development, and Social Protection Sectors Country ESW task Effects Economic policy Malaysia Assessing Fiscal Incentives Provided information on the value of fiscal incentives and inputs for public policy in the areas of investment climate, service delivery, agriculture, higher-end manufacturing, and human resources; helped convince the government to lower the corporate income tax rate in the 2007 budget. Romania Restructuring for European Became a blueprint for negotiating accession with the EU, especially regarding chapters that Union (EU) Integration— were not closed (public institutions and agriculture); instrumental in introducing top-up The Policy Agenda ESW subsidies that resolved one of the open issues with the EU; recommendations for the (CEM) financial sector led to the creation of the Securities Commission. Serbia Debt report Assisted in working out limits for foreign borrowing. Vietnam World Trade Organization Contributed to the country’s accession to the World Trade Organization, which constituted the Accession Preparation and third round of big reforms in the country; the ESW helped set the agenda for trade reform. Poverty, Trade Reform, and Export Policy and Performance Study Private sector development Malaysia ICA Led the government to establish a committee to ensure that deregulation and improvements in public service delivery are carried out smoothly; contributed to changing the approach to skills development and supporting small and medium enterprises, for which government programs were too cumbersome. Serbia ICA Together with the Private Sector Note was credited as being important for regaining the momentum in the privatization process and in attracting foreign investment and in helping determine priorities in reforming the registration process for new businesses and identifying areas where further action was needed. Social protection Jordan Note and policy dialogue Together with the Programmatic Economic Sector Work on Pension Reform (TA), led to a on pensions variety of changes in the pension system (although there is still an outstanding reform agenda), including raising the age of eligibility to get a pension; widening the membership of the committee (from only the military) that decides who is disabled and who is not; closing civil service funds to new entrants since 1996 and to the military since 2003 and shifting them to the Social Security Corporation (which, according to one in-country stakeholder, is a clear case of impact, “as clear a case as can be documented”); restructuring and changing the management of the Social Security Corporation; and preparing instructions to regulate the pension industry. Peru “Peru: Restoring the Cited by in-country stakeholders as one of the most useful reports; led to the creation of a Multiple Pillars of Old Age commission to make pensions sustainable; led to reforms in the second pillar and helped Income Security” (World reduce the costs of the system and introduce more competitiveness for pension fund Bank 2004d) management companies; led to the issuance of norms on disability and survivors, introduction of competition, and the reduction in the cost of premiums; currently informing the discussions of reforms to the first pillar. Source: IEG country reviews. Note: CEM = Country Economic Memorandum; ESW = economic and sector work; ICA = Investment Climate Assessment; TA = technical assistance. 129 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Table E.4: Selected Examples of ESW and TA Informing Strategy Country ESW and TA task Effects Human development Congo, Dem. Rep. of Country Status Reports on Education Informed the government’s strategy for education and health. and Health (ESW) Lesotho Development Impact of AIDS (ESW) Helped identify the focus of the Poverty Reduction Strategy Paper by showing the AIDS trend and the country situation. Malaysia International Best Practice Note for Recommendations incorporated into the Ninth Five-Year Plan Financing Public Tertiary Education (TA) of Malaysia. Private sector development Guyana ICA Informed the country’s National Competitiveness Strategy. Lesotho Private Sector Development Strategy Led to the adoption of a new government strategy to promote the (ESW) private sector: influenced business licensing procedures; involved government in partnerships with private sector in construction, telecommunications, and health. Transport Mali Transport report Together with the Mali Transport Corridor Improvement Project, contributed to a coherent transport strategy (government cannot distinguish between effects of ESW and those of loan). Mauritius Transport report Together with follow-up TA, resulted in an 18-point program. Economic policy Serbia Breaking with the Past (ESW) Helped formulate a reform strategy for the rural sector by, among other things, proposing solutions for cooperatives and land privatization. Rural Romania Forest governance study (TA) Helped establish the National Strategy for the Forestry Sector. Financial sector Romania Financial Sector Reform Program (TA) Assisted in the preparation of a strategy for banking reforms. Source: IEG country reviews. Note: ESW = economic and sector work; ICA = Investment Climate Assessment; TA = technical assistance. 130 A P P E N D I X E : S P E C I F I C E X A M P L E S O F E F F E C T S O F E S W A N D TA F R O M C O U N T R Y R E V I E W S Table E.5: Selected Examples of ESW and TA Informing Legislation ESW task Country Effects IDA mandates/core diagnostics Financial accountability Guyana Informed the Procurement and Tender Act, the Budget Law, the Audit and Accountancy Act. Procurement assessment Guyana Influenced modification of the Procurement Law (although not to the extent the World Bank had wanted). Procurement assessment Mauritius Resulted in a new procurement act, which is being implemented; the study was cited by a former government minister in Mauritius as perhaps the most effective study of the Bank for the country. Peru Providing the basis for the reform of the procurement legislation (one and a half years after completion). ICA Malaysia Propelled changes in labor law and in the registration of property. PER Guyana Informed the Fiscal Management Accountability Act (although a number of things under the Act are still not done). Peru Led to the Law of Fiscal Decentralization, Legislative Decree 955 (supported by DPLs). PER Update Serbia Led to a new law on wages in state administration (2006), an amended law on civil servants (2006), and amendments to the social security law (2006). FSAP and associated ROSCs AML/CFT Bangladesh Cited as a catalyst for the anti-money-laundering legislation that the government has implemented. Jordan Resulted in new improved draft legislation that has replaced preexisting legislation that had been presented (though not passed) in Parliament; the Jordanian authorities are now in the process of issuing instructions for the new AML Law. Mauritius Informed new AML/CFT legislation; as a result, a strong anti-money- laundering framework is now in place. FSAP Serbia Associated with changes to and implementation of the Banking Law (2005) (together with the Financial Sector Policy Note and the post-FSAP TA). Corporate Governance ROSC Jordan Resulted in a draft corporate governance code that was put on the Securities’ Commission Web site in February 2005. Romania Resulted in the amendment of the Law on Companies (2005–06), the adoption of the Law on Insolvency, and the promulgation of the Code of Conduct for the Bucharest Stock Exchange, which was later adopted by the Securities Commission. Accounting and Auditing ROSC Jordan Led to the preparation of a law for accounting. Mauritius Led to a new draft Financial Reporting Law (drafted with Bank TA), which provides legal backing to the accounting body and creates oversight for the accountant profession, and created the Financial Reporting Council. Other ESW Economic policy Breaking with the Past ESW Serbia Starting point for major subsequent reforms, especially in the areas of banking and privatization (Law on Privatizations 2001), mortgages, and business registrations; led to improvements in the investment climate via changes in the FDI law, cadastre law, mortgages law, and regulations affecting small and medium enterprises. (table continues on next page) 131 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Table E.5: Selected Examples of ESW and TA Informing Legislation (continued) ESW task Country Effects Economic policy (continued) World Trade Organization Accession Vietnam Had large effects on the legal agenda that was required for World Trade Preparation and Poverty, Trade Reform, Organization accession. and Export Policy and Performance Study Financial sector Capital markets study Romania Resulted in the modification of the Law on Capital Markets (2004), which included the establishment of the National Securities Commission (also a EU requirement); also instrumental in regulations regarding the insurance and reinsurance frameworks (Law 136 was amended in 2004–05 and 2006). Private sector development Private Sector Development Strategy Lesotho Led to changes in the Company Act, in regulations, and in licensing and registration. Private sector study Vietnam Was useful as an underpinning for the Enterprise Law, which was cited by a stakeholder (from a government think tank) as perhaps the most important change in Vietnam in recent years. The law introduced the freedom to do business by private parties. Public sector governance Anticorruption study Romania Majority of recommendations adopted in Law 161 on Public Sector Institutions; basis of all anticorruption efforts in Romania since it was prepared in 2001. Rural Forest sector study Congo, Dem. Informed the decree that froze new forestry concessions to halt speculation Rep. of and curb illegal logging, resulting in 25 million hectares being taken back into the public domain as many illegal concessions were cancelled. Food Safety and Agricultural Health Vietnam Helped prepare a basic guideline for meeting food safety standards, which is Action Plan a requirement for World Trade Organization members. Water supply and sanitation Water Sector Review Update Jordan Led to changing of Article 28 in the Water Authority of Jordan Law, which allowed the private sector to participate in managing the projects; also led to the 2002 groundwater bylaw to regulate the use of wells that put limits on how much water people could pump from the ground without paying. TA Forest Governance Study Romania Assisted the authorities in preparing a new forest code, which was submitted to Parliament. Civil Service and Anticorruption Work Serbia Provided a basis—through regular consultations and drafting suggestions— for the Law on Civil Service. Banking Sector Review Vietnam Helped the state bank in drafting two basic laws, the Central Bank Law and the Credit Institutions Law. Telecom Reform and ICT/E-Commerce Vietnam The expert consultant supported by the TA presented alternative models from Development Agenda several countries, read Vietnamese laws and regulations, and offered suggestions on what to do on telecom laws and related matters. Source: IEG country reviews. Note: AML/CFT = anti-money laundering and combating the financing of terrorism; DPL = development policy lending/loan; ESW = economic and sector work; EU = European Union; FDI = foreign direct investment; FSAP = Financial Sector Assessment Program; ICA = Investment Climate Assessment; ICT = information, communications, and technology; IDA = International Development Association; PER = Public Expenditure Review; ROSC = Report on the Observance of Standards and Codes; TA = technical assistance. 132 A P P E N D I X E : S P E C I F I C E X A M P L E S O F E F F E C T S O F E S W A N D TA F R O M C O U N T R Y R E V I E W S Table E.6: Selected Examples of TA Assisting in the Implementation of Government Policies and Programs Country TA tasks Effects Financial sector Romania Capital Market Development Program Was instrumental in establishing a regulatory regime for the pension, insurance, and mortgage markets (in conjunction with the European Commission and the IMF). Financial Sector Reform Program Assisted in dealing with the serious problem of large state banks through privatizations, closures, and restructuring. Public sector governance Romania Anticorruption Diagnostic Was the basis for Romania’s anticorruption efforts through initiating a public debate and educational efforts and helped privatize the former RomTelecom monopoly and establish the National Agency for Regulation and Communication, which arbitrated among private operators and was charged with protecting the public interest. Infrastructure Jordan Private Participation in Infrastructure Led to the launching of the Public-Private Partnership Policy. Peru Public-Private Infrastructure Helped jump start the concessions program and in particular resulted in the revision of the tariffs in the port system; the estimate from the TA that poor logistics constituted the equivalent of a 34 percent tariff on exports propelled reforms in transport; also, regulatory accounting and financial modeling provided by the TA are currently used to monitor concessioned enterprises. Rural Romania Challenges of EU Accession in Romanian Was very helpful in closing the chapter on agriculture in preaccession Agriculture discussions with the EU. Vietnam Forest Sector Support Helped put in place a system of forest certification for plantations. Private sector development Malaysia International Experience in Privatization Resulted in the government’s review of its privatization policy and, among other things, has led to concessions becoming widespread. International Best Practice Note for Financing Tertiary Education Contributed to the government’s emphasis on quality and relevance, including the need to benchmark universities to international standards and the need to coordinate policy and actions in tertiary education with those in primary and secondary education. Source: IEG country reviews. Note: EU = European Union; IMF = International Monetary Fund; TA = technical assistance. 133 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Table E.7: Selected Examples of TA and ESW Facilitating Knowledge Exchange Country Tasks Effects Mauritius Real Time Gross Allowed the Monetary Authority to know the liquidity position of banks and enabled better debt Settlement Initiative TA management for the Treasury. Aid for Trade TA Helped the government prepare a trade reform package. Corporate Governance Mainstreamed the concept of corporate governance in the country. ROSC Romania Agricultural Sector TA Translation of the TA into Romanian provided the Ministry of Agriculture with Romanian equivalents of EU technical agricultural terminology. Vietnam Corporate Governance Government official noted that many new concepts on corporate governance were learned. ROSC (ESW) Telecom Reform and Government counterpart indicated that a lot was learned from the conceptual framework ICT/E-Commerce provided by the Bank consultant, with applications of the framework at every opportunity. Development Agenda TA PER Facilitated the transfer of knowledge to universities and research institutes. Source: IEG country reviews. Note: ESW = economic and sector work; EU = European Union; ICT = information, communications, and technology; PER = Public Expenditure Review; ROSC = Report on the Observance of Standards and Codes; TA = technical assistance. 134 A P P E N D I X E : S P E C I F I C E X A M P L E S O F E F F E C T S O F E S W A N D TA F R O M C O U N T R Y R E V I E W S Table E.8: Selected Examples of ESW Informing the Development Community Donor ESW Effects on donor activities Guyana DFID Financial accountability Started working on procurement issues. European Financial accountability One of three conditions required for eligibility to European Commission Commission poverty-reduction budget support; European Commission issues annual reports on the implementation of CFAA recommendations. AML/CFT Assessment for All Eastern Informed regional program. Caribbean Currency Union IDB Financial accountability Informed Technical Cooperation Project on Strengthening Auditor General’s Office. New combined financial accountability Informing new loans that are being developed. and procurement assessment FSAP Informed Financial Sector Reform Program ICA Informed Strengthening Competitiveness Program; Agricultural Diversification Program; electricity loan. IMF Financial accountability Used it to track relief conditions for heavily indebted poor countries. DPR, PER Informed the Poverty Reduction Growth Facility (which was essential for other donors to go forward with their activities). ICA Developing program to address constraints identified in ICA. Jordan Danida Gender Assessment Provided funding for the Jordan National Women’s Commission. GTZ PER Providing funding. USAID Work Program for Consolidating Budget Led to provision of support to improve budget management. Management Reforms, PER FSAP Used as a blueprint for TA financing for USAID to support implementation of Prompt Corrective Action Framework and Corporate Governance of Banks. US Treasury AML/CFT Led to provision of TA on outreach campaign on AML/CFT. Lesotho MDC Private Sector Development Strategy Designed an intervention after participating in the private sector development forum, which was held to disseminate the report. Mauritius AFD Transport Action Plan, PER Mobilized additional resources and shifted priorities. European CEM Led to additional budget support through cofinancing of the Trade and Commission Competitiveness DPL. Procurement Made recommendations that provided conditions for European Commission financial support. (Table continues on next page) 135 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS Table E.8: Selected Examples of ESW Informing the Development Community (continued) Donor ESW Effects on donor activities Mali European Transport Support to Sustainable Led to increased funding for road subsector; agreed to directly finance Commission Economic Growth routine road maintenance for the first time. Peru IDB PER Informed policy-based loan to support Reform of Poverty Reduction and Human Capital Development Program. IMF FSAP, PER Informed conditionalities of IMF programs. Source: IEG country reviews. Note: AFD = Agence Française de Développement; AML/CFT = anti-money laundering and combatting the financing of terrorism; CEM = Country Economic Memorandum; CFAA = Country Financial Accountability Assessment; Danida = Danish International Development Agency; DFID = Department for International Development of the United Kingdom; DPL = development policy lending/loan; DPR = Development Policy Review; FSAP = Financial Sector Assessment Program; GTZ = German Agency for Technical Cooperation; ICA = Investment Climate Assess- ment; IDB = Inter-American Development Bank; IMF = International Monetary Fund; MCD = Millennium Development Corporation; PER = Public Expenditure Review; TA = technical assistance; USAID = U.S. Agency for International Development. Table E.9: Selected Examples of TA Informing the Development Community TA Donor Effects on donor activities Jordan Skills and Vocational Training Workshop CIDA, European Commission Led to provision of funds Jordan Vision for Education Forum CIDA, European Commission, USAID Led to provision of funds Poverty Institutional Framework United States Led to provision of funds Mauritius Aid for Trade Multiple donors Led to provision of funds Source: IEG reviews. Note: CIDA = Canadian International Development Agency; TA = technical assistance; USAID = U.S. Agency for International Development. 136 ENDNOTES Chapter 1 5. The ranking by cost is the same as with numbers 1. According to Operations Policy and Country Ser- except for economic policy, which—although ranked in vices, in fiscal 2004, ESW products budgeted at more the top seven for the number of TA—only ranked in the than $50,000 were required to have objectives speci- top nine by cost, being replaced by urban development fied at task initiation; in fiscal 2005, all ESW products as one of the top seven sectors for TA expenditures. were required to have objectives specified. Prior to fis- 6. The Bank’s administrative data are used for the cal 2004, some tasks had objectives specified accord- discussion in this section. ing to a simpler results framework. 7. For details on the Bank ESW TTL survey, see 2. Though these objectives were only made ex- appendix B. plicit in May 2004, they are used in this evaluation to 8. The 8 percent figure is derived from survey re- evaluate ESW and TA produced prior to that date also; sponses of Bank team leaders. This is much higher than these objectives had already been implicitly underlying the 2 percent figure reported in the Bank’s administra- such Bank tasks prior to that date, and this evaluation tive database. Given the errors in Bank administrative data considers them reasonable objectives for ESW and TA. (see endnote 4 and chapter 6), this evaluation uses data obtained from the Bank TTL surveys. Chapter 2 9. Because of the relatively low number of data 1. See appendix C for details on the Bank ESW and points per sector from the survey responses, the eval- TA portfolio. uation only draws definitive conclusions where the 2. Costs are defined as the cumulative costs for survey responses are consistent with Bank adminis- completed tasks delivered in a given fiscal year. Costs trative data. include both Bank budget and trust funds. 10. For the same reason provided in endnote 9, this 3. The exception is South Asia, for which this com- evaluation only draws definitive conclusions where the parison cannot be made because there are no purely survey responses are consistent with Bank adminis- IBRD countries there, although its largest country, trative data. India, can access IBRD funds. 11. The same rating scale is used throughout this 4. The source is Bank administrative data, adjusted evaluation. by IEG for misclassification. The IEG review of the port- 12. QAG switched from a four-point to a six-point folio of ESW and TA during fiscal 2000–06 indicated scale in fiscal 2003, although some products delivered that around 13 percent of ESW tasks and 20 percent of prior to fiscal 2003 but assessed later by QAG have rat- TA tasks were classified under an incorrect sector board ings on a six-point scale. This evaluation uses a six- (see appendix C). For analytical purposes for this eval- point rating scale. Hence, for the purpose of comparison, uation, IEG created two sector board classifications: this section only includes those tasks with QAG ratings infrastructure (which combines transport, energy and on a six-point scale. This biases the comparison slightly: mining, and water supply and sanitation) and human the country reviews covered ESW and TA tasks for the development (which combines health, nutrition, and period fiscal 2000–06, and over that period QAG ratings population and education). This is because some ESW have improved somewhat, such that tasks that have four- and TA are in these combined sectors rather than the point ratings have slightly lower ratings than those with individual ones to which they had been classified. six-point ratings. The portfolios of tasks with four-point 137 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS and tasks with six-point QAG ratings are representative 6. These are averages of loan TTL ratings; there of those of all Bank products on both a sectoral and Re- were 69 ratings for country-specific ESW , 8 ratings for gional basis (see appendix B). regional ESW , and 14 for global ESW . 13. Significance here means statistically significant 7. Ratings on loan design are given by QAG, which at the 5 percent level. This definition of significance is provides QAE ratings for eight dimensions of loan de- used henceforth in this report. Significance was as- sign and implementation arrangements, and an over- certained by conducting t-tests that compare the means all assessment rating (which synthesizes ratings on the (averages) of two sets of ratings, taking into account eight dimensions), for a selected sample of loans after their variability (or spread). approval by the Bank’s Board (see appendix D). 14. The correlation was significant at the 5 percent 8. Loans with QAE ratings used for this analysis are level for ESW on a four-point QAG quality rating scale representative on a Regional and sectoral basis of all and at the 10 percent level for those on a six-point rat- Bank loans (see appendix B). ing scale. This could mean that as the rating scale be- 9. Bank management contended that CFAAs are comes more finely tuned, the correlation is weaker, not expected to guide financial management arrange- and/or that the cost-quality correlation is weaker for the ments at the project level. more recent tasks (that is, those with ratings on a six- 10. All the results reported in this section are sta- point rating scale compared with those with ratings on tistically significant at either the 1 or 5 percent level un- a four-point scale). less mentioned otherwise. 15. The exceptions were infrastructure, social de- 11. One of the Bangladesh country directors indi- velopment, urban development, and private sector cated that Bank ESW and TA have influenced policy and development; the average cost of ESW in these four sec- dialogue through lending, but not otherwise. tors fell during fiscal 2000–06. Chapter 4 Chapter 3 1. The in-country stakeholder ratings presented in 1. The sample of loans was selected from those this chapter for all the objectives except “informing the with QAE ratings. The fiscal 2006 QAE ratings were development community” are the averages provided not available in time for this evaluation. by all the stakeholders (government officials, private 2. Given the evaluation period of fiscal 2000–06, the sector, and civil society). The ratings presented for “in- sample of loans was selected only from those approved forming the development community” are those pro- during fiscal 2003–05. This allowed for ESW that was de- vided by the donors only. livered up to three years prior to the approval of the 2. This refers to statistically significantly different at loan to be considered for informing the loans. There the 5 percent level. The report uses this definition of is a difference between “ESW that could have informed difference throughout. the loans” and ESW that are just in the same sector as 3. The ratings are on a six-point scale, with 1 being the loans. For instance, an agriculture ESW may or may low and 6 being high, 4 being slightly above average, not inform an irrigation loan, although both belong to and 3 being slightly below average. The report uses the the rural sector board. The IEG team took this into ac- same rating scale throughout. count by its in-depth review of loan documents and ESW 4. Development Results in Middle-Income Coun- contents to ensure that only those ESW that could tries (IEG 2007, pp. 40–41) also found lower impact of have actually informed the loan are included. Bank AAA on capacity building and influencing public 3. The rating is on a 6-point scale, as mentioned in debate compared with informing policy in MICs. chapter 2, with 1 being low and 6 being high. 5. The IEG team also undertook multivariate re- 4. This was not significantly different statistically gression analysis to determine the association (if any) from the ratings given by loan TTLs in the country between ESW (existence, number, and cost) and pol- reviews. icy (as proxied by the Country Policy and Institutional 5. Bank staff who were also TTLs of the ESW on Assessment). The unstable and counterintuitive re- which they were being surveyed were excluded from sults raised doubts about the appropriateness of the the loan TTL survey. Country Policy and Institutional Assessment as a proxy 138 ENDNOTES for policy. Hence, the results of the regressions are and TA objectives from all the different sources of not reported in this evaluation. information. 6. Ratings for this objective were not available for 10. Development Results in Middle-Income Coun- all the countries reviewed. The reasons include no TA tries (IEG 2007, pp. 40–41) also found that when the undertaken during the evaluation period (Guyana), Bank’s knowledge services were ineffective, the lack of in-country stakeholders not recalling the TA (Bangla- dissemination was the most frequently cited reason. desh), and no ratings being provided for this objective 11. Public Sector Reform: What Works and Why? (Lesotho). (IEG 2008c, p. 40) also found much smaller impact on 7. The success indicators the Bank has specified governance (as measured by the Country Policy and In- for this objective pertain to donor activities, including stitutional Assessment governance score) when AAA was additional resources being mobilized by donors and a not followed by lending. shift in donor policy/priorities, although typically the de- 12. The Education for All–Fast-Track Initiative is a velopment community encompasses a wider concept global partnership between donor and developing than just the donor community. This evaluation focuses countries to ensure accelerated progress toward the specifically on ESW’s influence on the donor commu- Millennium Development Goal of universal primary nity, rather than on the wider development community. education by 2015. All low-income countries that demonstrate serious commitment to achieve universal Chapter 5 primary completion can receive support from the Fast- 1. Development Results in Middle-Income Coun- Track Initiative. tries (IEG 2007, pp. 38–39) also found that Bank AAA 13. At the beginning of 2008, the Bank opened a for those countries was of high technical quality. small liaison office in Mauritius. 2. This refers to correlation coefficients that are significant at the 5 percent level. The positive and sig- Chapter 6 nificant correlation was found for almost all the ESW 1. The same recommendation was made in Devel- and TA objectives from all the different sources of in- opment Results in Middle-Income Countries (IEG 2007, formation. p. 71). 3. Assessing World Bank Support for Trade, 2. Engaging with Fragile States (IEG 2006b, p. 19) 1987–2004 (IEG 2006a, p. 26) also found that high- also found that analytical work done in collaboration quality ESW supports policy dialogue. with other donors can improve the Bank’s policy in- 4. Since 2006, the Sub-Saharan Africa Region has fluence. In addition, “Country Financial Accountability adopted an ESW Quality Improvement Program to Assessments and Country Procurement Assessment improve the impact of ESW and its role in capacity Reviews: How Effective Are World Bank Fiduciary Di- building. agnostics?” (IEG 2008a, p. xi) found that greater col- 5. This refers to correlation coefficients that are sig- laboration with relevant donors in the preparation of nificant at the 5 percent level. The positive and signifi- these ESW would enhance their impact. cant correlation was found for almost all the ESW and TA 3. Although several of the Malaysia TA tasks are in objectives from all the different sources of information. the form of reports, they are considered TA rather 6. The information on origination (whether it was than ESW because the Bank’s role in those tasks was client requested or not), by specific task, came from re- advising on the content of the study (the terms of ref- sponses to the ESW and TA TTL surveys. This infor- erence) and assisting with the identification of con- mation was applied, for the same specific task, to the sultants, rather than writing the report. responses to the specific ESW and the specific TA sur- 4. This is based on ratings provided by in-country veys, respectively. stakeholders in the specific ESW user survey. This find- 7. This was significant at the 10 percent level. ing is consistent with QAG ratings on policy notes ver- 8. This was significant at the 10 percent level. sus those on other types of ESW . 9. This refers to correlation coefficients that are 5. Small States: Making the Most of Development significant at the 5 percent level. The positive and sig- Assistance (IEG 2006c, p. 18) also found that the Bank nificant correlation was found for almost all the ESW has been able to achieve some economies of scale 139 USING KNOWLEDGE TO IMPROVE DEVELOPMENT EFFECTIVENESS through Regional ESW that addresses common prob- Appendix D lems of small states. 6. The poor quality of the Bank’s administrative 1. The fiscal 2006 QAE ratings were not available in data on ESW and TA was also found by QAG. time for this exercise. 2. Because only three loans in the sample have rat- Appendix B ings of 3 and lower, the ratings were rescaled by com- 1. For Mali, in addition to the five sectors, AAA and bining ratings 2, 3, and 4 into one category. However, loans in financial management; procurement; health results do not change even when these loans are nutrition, and population; and transport were also dropped from the sample. The results in Table D.2 are reviewed. based on the rescaled rating. 140 BIBLIOGRAPHY Bannerjee, Abhijit, Angus Deaton, Nora Lustig, Ken Ro- ———. 2007b. “Lesotho: Managing Government Fi- goff, and Edward Hsu. 2006. “An Evaluation of the nances for Growth and Poverty Reduction.” Public World Bank Research, 1998–2005.” External Evalu- Expenditure Management and Financial Accounta- ation, World Bank, Washington, DC. bility Review No. 36359, World Bank, Washington, IEG (Independent Evaluation Group). 2008a. “Country DC. Financial Accountability Assessments and Country ———. 2005a. “Democratic Republic of Congo: Health, Procurement Assessment Reviews: How Effective Nutrition and Population.” Country Status Report No. Are World Bank Fiduciary Diagnostics?” Report No. 35626-ZR, World Bank, Washington, DC. 43395, World Bank, Washington, DC. ———. 2005b. Education in the Democratic Repub- ———. 2008b. Doing Business: An Independent Eval- lic of Congo: Priorities and Options for Regenera- uation—Taking the Measure of the World Bank–IFC tion. Washington, DC: World Bank. Doing Business Indicators. IEG Study Series. Wash- ———. 2005c. Lecciones del TLCAN para América ington, DC: World Bank. Latina y el Caribe” [Lessons from the North Amer- ———. 2008c. Public Sector Reform: What Works and ican Free Trade Agreement for Latin America and the Why? IEG Study Series. Washington, DC: World Caribbean]. Washington, DC: World Bank. Bank. ———. 2005d. A Time to Choose: Caribbean Devel- ———. 2007. Development Results in Middle-Income opment in the 21st Century. Washington, DC: World Countries. IEG Study Series. Washington, DC: World Bank. Bank. ———. 2004a. “Mauritius: Modernizing an Advanced ———. 2006a. Assessing World Bank Support for Trade, Pension System.” World Bank Social Policy Protec- 1987–2004: An IEG Evaluation. IEG Study Series. tion Study No. 29588, Washington, DC. Washington, DC: World Bank. ———. 2004b. “Mauritius: The New Economic Agenda ———. 2006b. Engaging with Fragile States: An IEG Re- and Fiscal Sustainability.” Public Expenditure Re- view of World Bank Support to Low-Income Coun- view No. 26152, World Bank, Washington, DC. tries Under Stress. IEG Study Series. Washington, DC: ———. 2004c. “Peru: Restoring the Multiple Pillars of World Bank. Old Age Income Security.” Report No. 27618, World ———. 2006c. Small States: Making the Most of De- Bank, Washington, DC. velopment Assistance—A Synthesis of World Bank ———. 2004d. World Development Report 2004: Mak- Evaluation Findings. Washington, DC: World Bank. ing Services Work for Poor People. Washington, DC: ———. 2003. Sharing Knowledge: Innovations and Re- World Bank. maining Challenges. Washington, DC: World Bank. ———. 2003. “Breaking the Conflict Trap: Civil War Integrated Framework. 2003. Integration into the World and Development Policy.” World Bank Research Re- Trading Environment—Lesotho, Vol. 1. Diagnostic port No. 26121, World Bank, Washington, DC. Trade Integration Study. Geneva: World Trade Or- ———. 2002. “Post-Conflict Recovery in Africa: An gainzation. Agenda for the Africa Region.” Working Paper No. World Bank. 2007a. “Catalyzing the Future: An Inclusive 24651, World Bank, Washington, DC. and Sustainable Globalization.” Annual Meeting Re- ———. 2001. “HIV/AIDS in the Caribbean: Issues and marks by Robert B. Zoellick. http://www.go.world- Options.” World Bank Country Study, World Bank, bank.org/1OYEN7PF60. Washington, DC. 141 RECENT IEG PUBLICATIONS 2006 Annual Report on Operations Evaluation Annual Review of Development Effectiveness 2006: Getting Results Addressing the Challenges of Globalization: An Independent Evaluation of the World Bank’s Approach to Global Programs Assessing World Bank Support for Trade, 1987–2004: An IEG Evaluation Books, Buildings, and Learning Outcomes: An Impact Evaluation of World Bank Support to Basic Education in Ghana Brazil: Forging a Strategic Partnership for Results—An OED Evaluation of World Bank Assistance Bridging Troubled Waters: Assessing the World Bank Water Resources Strategy Capacity Building in Africa: An OED Evaluation of World Bank Support China: An Evaluation of World Bank Assistance The CGIAR at 31: An Independent Meta-Evaluation of the Consultative Group on International Agricultural Research Committing to Results: Improving the Effectiveness of HIV/AIDS Assistance—An OED Evaluation of the World Bank’s Assistance for HIV/AIDS Control Country Assistance Evaluation Retrospective: OED Self-Evaluation Debt Relief for the Poorest: An Evaluation Update of the HIPC Initiative A Decade of Action in Transport: An Evaluation of World Bank Assistance to the Transport Sector, 1995–2005 The Development Potential of Regional Programs: An Evaluation of World Bank Support of Multicountry Operations Development Results in Middle-Income Countries: An Evaluation of the World Bank’s Support Economies in Transition: An OED Evaluation of World Bank Assistance Engaging with Fragile States: An IEG Review of World Bank Support to Low-Income Countries Under Stress The Effectiveness of World Bank Support for Community-Based and -Driven Development: An OED Evaluation Evaluating a Decade of World Bank Gender Policy: 1990–99 Evaluation of World Bank Assistance to Pacific Member Countries, 1992–2002 Extractive Industries and Sustainable Development: An Evaluation of World Bank Group Experience Financial Sector Assessment Program: IEG Review of the Joint World Bank and IMF Initiative From Schooling Access to Learning Outcomes: An Unfinished Agenda—An Evaluation of World Bank Support to Primary Education Hazards of Nature, Risks to Development: An IEG Evaluation of World Bank Assistance for Natural Disasters How to Build M&E Systems to Support Better Government IEG Review of World Bank Assistance for Financial Sector Reform Improving Investment Climates: An Evaluation of World Bank Group Assistance Improving the Lives of the Poor Through Investment in Cities Improving the World Bank’s Development Assistance: What Does Evaluation Show? Maintaining Momentum to 2015: An Impact Evaluation of Interventions to Improve Maternal and Child Health and Nutrition Outcomes in Bangladesh New Renewable Energy: A Review of the World Bank’s Assistance Pakistan: An Evaluation of the World Bank’s Assistance Pension Reform and the Development of Pension Systems: An Evaluation of World Bank Assistance Poland Country Assistance Review: Partnership in a Transition Economy The Poverty Reduction Strategy Initiative: An Independent Evaluation of the World Bank’s Support Through 2003 The Poverty Reduction Strategy Initiative: Findings from 10 Country Case Studies of World Bank and IMF Support Power for Development: A Review of the World Bank Group’s Experience with Private Participation in the Electricity Sector Putting Social Development to Work for the Poor: An OED Review of World Bank Activities Small States: Making the Most of Development Assistance—A Synthesis of World Bank Findings Social Funds: Assessing Effectiveness Sourcebook for Evaluating Global and Regional Partnership Programs Water Management in Agriculture: Ten Years of World Bank Assistance, 1994–2004 World Bank Assistance to the Financial Sector: A Synthesis of IEG Evaluations The World Bank in Turkey: 1993–2004—An IEG Country Assistance Evaluation World Bank Lending for Lines of Credit: An IEG Evaluation All IEG evaluations are available, in whole or in part, in languages other than English. For our multilingual section, please visit http://www.worldbank.org/ieg ISBN 978-0-8213-7725-3 90000 9 780821 377253 SKU 17725 THE WORLD BANK