103710 WORLD BANK OPERATIONS EVALUATION DEPARTMENT Number 34 OED REACH May 29, 2001 Rating IDA’S Development Performance OED evaluates development interventions by assessing • desk reviews of Bank documents and working how their results stack up against their own stated papers, project operations databases, extant studies, objectives.1 From an accountability perspective, this and OED Country Assistance Evaluations; goal-based approach is attractive since it relates results • staff surveys, with more than 200 respondents; to objectives agreed to by the Board of Executive • in-country and international consultations with repre- Directors.2 sentatives of government, civil society, the private Specifically, OED evaluates outcomes by considering sector, and other assistance agencies in nine focus three factors: countries involving more than 680 participants; • the relevance of the intervention’s objectives in rela- • two international workshops bringing together tion to country needs and institutional priorities; nearly 60 experts from both borrower and donor • efficacy, i.e. the extent to which the developmental countries; objectives have been (or are expected to be) • wide-ranging interviews with relevant Bank man- achieved; agers and staff; and • efficiency, i.e. the extent to which the objectives • where the evidence allowed, results-based analyses have been (or are expected to be) achieved without tracing IDA inputs (policies, alignment of resources), using more resources than necessary. outputs (volume and composition of lending and non- The assessment of relevance is especially critical. lending services), and reach (stakeholder participa- When done well, it nets out excessively or inadequately tion, coordination with other donors), to outcomes ambitious objectives. Combining these three factors, and results. overall outcome is rated on a six-point scale, ranging from highly satisfactory to highly unsatisfactory. (See box 1 for details.) Outcome is not a simple average of Box 1. OED’s Outcome Rating Scale the factors because the factors interact: they are cumu- lative and interdependent, e.g., satisfactory relevance is Highly satisfactory: All relevant developmental useless if efficacy is poor. objectives are (or are expected to be) achieved and/or exceeded efficiently, with no shortcomings. How Did OED Evaluate IDA 10-12? Satisfactory: most of the relevant development Compliance. The review concentrated on IDA’s compli- objectives are (or are expected to be) achieved ance with replenishment undertakings and develop- efficiently with only minor shortcomings. ment contributions in six thematic development Partially satisfactory: Substantial shortcomings priorities: poverty reduction, social sector develop- are observed, even though most of the major rele- ment, private sector development, governance, environ- vant objectives, on balance, are (or are expected to mentally sustainable development, and gender. It also be) met. addressed four priority process reform objectives: per- Partially unsatisfactory: Some of the major rele- formance based allocations, enhanced CAS design and vant objectives are not (or are not expected to be) implementation, aid coordination, and participation. met, and/or many of the objectives are not relevant. Outcomes. OED assessed the extent to which IDA- Unsatisfactory: Most major, relevant objectives supported projects, programs, and other activities were are not (or not expected to be) met and/or most relevant to these concerns, the degree to which they objectives are not relevant. have been or are expected to be achieved (efficacy), Highly unsatisfactory: None of the relevant and at what cost (efficiency). objectives is (or is expected to be) met or is not Design. OED’s review used a multifaceted evaluation relevant. design, including: This work was done with the support of an interna- How Well Did IDA Meet Agreed Undertakings? tional advisory group of six prominent international Overall, IDA’s compliance with the replenishment under- development experts, three each from developing and takings has been satisfactory, with important qualifica- developed countries, including a number with experi- tions. During the IDA10-12 period, IDA made significant ence in high-level government, private sector, civil soci- advances in sharpening the poverty focus of investment ety, and MDB positions. and adjustment lending and analytical work, established Because few activities initiated under IDA 10, 11, and a strong presence in the social sectors, brought gover- 12 are complete, the evaluation had to consider a number nance issues to the fore, and enhanced processes related of proxies for assessing IDA’s performance: the quality of to country assistance strategies, performance-based allo- the analysis underlying the program, its coherence at the cations, participation, and aid coordination. However, country and sector (or thematic) levels, the responsive- compliance was uneven across and within areas of pro- ness of lending and non-lending services to the country gram and process emphasis.Though recently accelerat- context, the selectivity of resource allocations and choice ing, IDA’s implementation of undertakings related to of instruments, and, where possible, IDA’s impact on coun- gender, environment, and private sector development try policies, institutions and actions.This analysis was was modest in depth and pace in relation to the replen- made more difficult by the lack of a corporate scorecard ishment agreements. or fully-implemented results-based management system The compliance rating is based on the findings of from which performance data could be extracted. Box 2 IDA Review background studies which identified the highlights the methodological challenges. extent and pace of implementation to date of over 150 separate IDA10-12 undertakings.The rating reflects the overall extent of substantive compliance, rather than Box 2. What Were the Main Challenges of the the effectiveness or outcomes of IDA’s actions.The indi- IDA Evaluation? vidual replenishment undertakings varied widely, rang- ing from broad redirections in operations to calls for Conducting evaluations is almost always compli- specific reports. Findings on the degree of IDA’s compli- cated, but this evaluation was especially demand- ance made use of a system of ratings on individual ing and complex. undertakings, which was reviewed with management. First, neither the replenishment reports nor This system also served as input into the substantive management instructions provided specific discussion of implementation summarized in the back- benchmarks against which compliance or out- ground paper entitled, IDA10-12 Replenishment comes could be measured. OED therefore looked Undertakings Implementation Matrix. at the extent to which IDA took actions to change its processes and programs and to institutionalize What Were IDA’s Contributions to Development Out- those changes. comes? Second, the temporal scope of the evaluation was limited.At the request of the IDA Deputies, OED finds the development outcomes of IDA the evaluation focused on performance under IDA programs—influenced by exogenous factors and bor- replenishments 10 and 11 and the first year of rower and partner performance, as well as IDA IDA 12. Many of the programs supported during performance—to be partially satisfactory, with notable this period are still under way, and their develop- improvements over the period. Much progress can be ment outcomes are not fully known. seen in project-level outcomes as well as in quality indi- Finally,since many factors affect country program cators compiled by QAG. For example, as the figure outcomes in a country context,attributing success or shows, projects completed in the IDA 11 period (mostly failure to IDA’s actions posed a particular difficulty.To initiated in earlier periods) are significantly more likely deal with this,OED took account of external factors than those completed in IDA 9 or 10 to be rated by OED that might have affected IDA’s development perform- as having satisfactory outcomes, likely sustainability and ance,such as fluctuations in the world economy,bor- substantial institutional development impact. (Too few rower’s and partner’s performance,war and civil cases are available from the IDA 12 exiting year for analy- disturbances,and natural disasters. sis.) This is a welcome development and the hypothesis In addressing this last issue, OED conducted in- of increasingly satisfactory outcomes is well supported country consultations in nine focus countries, cho- by other evidence displayed throughout the IDA Review. sen according to the following criteria: size of IDA Project-level performance is an important indicator, program, inclusion of blend and IDA only countries, but does not tell the full story of IDA’s contribution to regional diversity, country performance ratings (i.e., development outcomes. In recent years, IDA has shifted countries across the CPIA spectrum, excluding the its focus to the “higher plane” of country programs.The very lowest, where the IDA program was small results from OED’s CAEs for 23 IDA and IDA blend coun- because of very low performance), availability of an tries (adjusted to match the rating scale in Box 1) found OED Country Assistance Evaluation (CAE), and spe- that 13 of these country programs were rated partially cial circumstances (e.g., post conflict). satisfactory, compared to 8 that were rated fully satisfac- tory.3 (Two others were rated unsatisfactory.) While CAEs Table: Compliance and Outcomes Related to IDA Undertakings Compliance Outcomes Development objectives Relevance Efficacy/Efficiency Poverty reduction Satisfactory Highly satisfactory Partially satisfactory Social sector development Highly satisfactory Highly satisfactory Satisfactory Private sector development Partially satisfactory Partially satisfactory Partially satisfactory Governance Satisfactory Satisfactory Satisfactory Environmentally sustainable development Partially satisfactory Partially satisfactory Partially satisfactory Gender Partially satisfactory Satisfactory Partially satisfactory Process reform objectives CAS enhancement Highly satisfactory Highly satisfactory Satisfactory Performance-based allocations Satisfactory Highly satisfactory Partially satisfactory Aid coordination Satisfactory Satisfactory Partially satisfactory Participation Satisfactory Satisfactory Satisfactory cover a longer period than the IDA Review, most do level, and promote aid coordination fell short of admit- cover the most recent CAS periods, make use of up-to- tedly demanding objectives. date QAG data, assess efficiency using country budget As noted earlier, since many IDA-supported projects coefficients, and draw lessons for improving program and programs from the period under review are still performance going forward.These results support the ongoing and their results not fully known, OED consid- finding that the performance of IDA has been partially ered a number of proxies for performance that satisfactory in terms of development outcomes. informed its assessment.This analysis found many areas An important indicator of performance across coun- of progress, but some significant shortcomings as well: tries is the allocation of aid. Recent work by DEC (box Quality of analysis underlying the program: IDA’s 3) suggests that the allocation of IDA resources across analytical work is one its strengths, highly regarded by countries is highly aligned with poverty reduction and borrowers and other development partners.The increase policy performance. Given the terms of reference of the Review, of particu- lar importance to this assessment is performance against Box 3. Evaluating Aid Effectiveness across the objectives embedded in the replenishment undertak- Countries ings.The results of that analysis are shown in the table in terms of their relevance, efficacy, and efficiency. A major methodological difficulty is distinguishing To assess relevance, OED evaluated the corporate the contribution of IDA from all other activities. strategies in each of the priority areas, and their transla- OED’s approach is to assess country assistance tion in country assistance programs. Overall, the rele- strategies down to the project level and see vance of IDA’s efforts in most areas targeted by IDA whether the results of lending and non-lending undertakings has been highly satisfactory. operations were satisfactory.This approach has But the bottom line on development is results on the limitations because of fungibility. ground.This calls for an assessment of the efficacy of An additional approach, which has different limi- IDA’s activities.To this end, OED asked the extent to tations, is the aid effectiveness analysis that has which, given the corporate strategy, implementation is been done in DECRG (“Aid Allocation and Poverty likely to achieve program objectives.As the table shows, Reduction,” forthcoming in European Economic in 6 of the 10 priority areas likely progress against objec- Review). It estimates the contribution of aid to tives is only partially satisfactory, while aggregate effi- poverty reduction econometrically across coun- ciency has fallen short of the targets set forth in the tries.This approach evaluates major aid programs— Strategic Compact. bilaterals and IDA—for its allocation across On the most important dimension—poverty countries, but cannot distinguish different effects reduction—evaluations of IDA performance at the coun- of ‘rival’ aid programs within a country.This is com- try level point to a generally positive record in assisting plementary to OED’s approach, which is much bet- countries to lay the foundations for economic growth ter in-country than cross-country. Specifically, DEC and poverty reduction.Yet the record of IDA countries in finds that IDA’s allocation is superior to any of the sustaining growth high enough and long enough to ben- bilateral programs, and indeed, that its allocation efit the majority of the poor has been disappointing in rule is good in absolute terms. IDA deviates from most IDA countries due in part to many factors beyond the poverty—policy’ rule (direct aid to low income IDA’s control.The review found that IDA’s efforts to fos- countries with reasonable policies) on an ad hoc ter private sector development, enhance opportunities basis, but not systematically. Its deviations usually for women, promote environmentally sustainable devel- reflect additional knowledge. opment, use its resources more selectively at the country Trends in Performance of IDA Projects Closing in the and many OED evaluations stress weaknesses in institu- Years Covered by Replenishments 9, 10 and 11 tional analysis and capacity development. Overall, then, OED finds that IDA has complied satis- Percent factorily with the replenishment undertakings, and 100 given factors not wholly under IDA control, made a par- 90 tially satisfactory contribution to development out- 80 Outcomes (percent satisfactory) comes against ambitious goals.This reflects accelerating 70 improvements in IDA and country performance but still 60 limited progress in achieving the over-arching goal of 50 Sustainability (percent likely) poverty reduction. 40 30 Conclusions 20 OED’s evaluation of the IDA 10-12 program and the Institutional Development 10 Impact (percent substantial) rating of partially satisfactory outcomes represents a 0 careful application of an objectives-based methodol- IDA 9 IDA 10 IDA 11 ogy to a complex evaluation challenge, which has tried to take account of IDA’s performance as well as in the quantity of data and analysis, especially on poverty exogenous factors and borrower and partner perform- is a major achievement. But the quality of the data and in- ance. Overall, OED found that the progress achieved country capacity building on data collection and analysis has accelerated.Although shortcomings remain, imple- ( e.g., on poverty and on gender) remain important chal- mentation trends augur positively for the future.To lenges (as borne out in the PRSP progress reports). realize this potential, however, the next IDA replenish- Coherence at country and sector (or thematic) lev- ment discussion should develop a longer-term vision els: Although IDA has strengthened its poverty orienta- focused on results, engage developing countries in set- tion, more needs to be done to integrate that orientation ting replenishment conditions, and define those com- into macroeconomic and sectoral priorities and policies. mitments in terms of monitorable and achievable While CASs have shown substantial improvement in objectives with realistic costing. More immediately, to their coverage of poverty over the last seven years, with deepen and broaden the gains from the existing pol- most occurring since 1997, they have made less improve- icy framework IDA needs to: ment in setting priorities and identifying country-specific • focus on implementation in areas of emphasis; poverty-focused assistance strategies. • align resources to strategic priorities; and Responsiveness of lending and non-lending services • consolidate the IDA mandates. to country context: While the innovations in lending instruments are an important feature of the period, there Notes also has been a decline in support for ESW, and continu- 1. The results may represent substantial progress relative to the status quo ante (or what would have happened in the absence of Bank/IDA inter- ing weakness in M&E, which denies IDA and its partners vention) but still lead to less than fully satisfactory outcomes if they had not of adequate knowledge on cost effective interventions in met their stated objectives or had done so inefficiently. This is especially support of core objectives. likely when the objectives are particularly demanding, as was the case with Strategic selectivity: OED found—and it is com- the IDA replenishments. 2. The alternative—goal-free evaluation—is used by OED whenever it is monly agreed that—strategic selectivity remains a chal- feasible to weigh the net benefits of the intervention in relation to the oppor- lenge for IDA and its partners. tunity cost of the resources used. Thus, economic and financial rates of Impact on country policies, institutions, and actions: returns are routinely used to assess the justification of projects where appro- Portfolio performance shows improvements on outcomes, priate data can be obtained. Unfortunately, cash flow analysis is not practical and QAG ratings—which are useful leading indicators—are for the complex programs funded by IDA. Even for projects, the implementa- tion of policy and capacity building objectives (which are often the most cru- improving.But institutional development impact,though cial) cannot be quantified precisely enough to allow cash flow analysis. improving,still remains too low.In addition,the Annual 3. The CAE ratings were adjusted to match the rating scale in Box 1. Review of Development Effectiveness for 1999 and 2000 The methodology used in the CAEs has not been agreed with Management. Task Manager: Catherine Gwin: (202) 473–1103 Director: Gregory Ingram (202) 473-1052 Brief produced by OEDPK: ecampbellpage@worldbank.org