T H E W O R L D B A N K O P E R A T I O N S E V A L U A T I O N D E P A R T M E N T Agriculture Policy Reform in the ECA Transition Economies, 1991-2002 An Assessment of the World Bank's Approach John R. Heath Director-General, Operations Evaluation: Gregory K. Ingram Acting Director: Nils Fostvedt Manager: Alain Barbu Task Manager: John R. Heath, Senior Evaluation Officer, OEDST 2003 The World Bank This paper is available upon request from OED. Washington, D.C. ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION The Operations Evaluation Department (OED) is an independent unit within the World Bank; it reports directly to the Bank's Board of Executive Directors. OED assesses what works, and what does not; how a borrower plans to run and maintain a project; and the lasting contribution of the Bank to a country's overall development. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank's work, and to provide accountability in the achievement of its objectives. It also improves Bank work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. OED Working Papers are an informal series to disseminate the findings of work in progress to encourage the exchange of ideas about development effectiveness through evaluation. The findings, interpretations, and conclusions expressed here are those of the author(s) and do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent. The World Bank cannot guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply on the part of the World Bank any judgment of the legal status of any territory or the endorsement or acceptance of such boundaries. Contact: Operations Evaluation Department Partnerships & Knowledge Programs (OEDPK) e-mail: eline@worldbank.org Telephone: 202-458-4497 Facsimile: 202-522-3125 http:/www.worldbank.org/oed iii ABBREVIATIONS AND ACRONYMS ALP Agricultural Labor Productivity ASAL Agriculture Sector Adjustment Loan CDF Comprehensive Development Framework CEE Central and Eastern Europe CIS Commonwealth of Independent States CPIA Country Policies and Institutions Assessment DO Development Outcome EBRD European Bank for Reconstruction and Development ECA Europe and Central Asia Region ECSSD Environmentally and Socially Sustainable Development Sector Unit EC European Community GAO Gross Agricultural Output GDP Gross Domestic Product HD Human Development Network IFC International Finance Corporation IMF International Monetary Fund OECD Organization for Economic Cooperation and Development OED Operations Evaluation Department PPP Purchasing Power Parity PREM Poverty Reduction and Economic Management Network SAL Structural Adjustment Loan USSR Union of Soviet Socialist Republics iv Contents Acknowledgments........................................................................................................... vii Executive Summary......................................................................................................... ix 1. Background and Approach.........................................................................................1 Approach..................................................................................................................1 2. Relevance ......................................................................................................................4 Was the Overall Scale of Assistance Appropriate? .................................................4 Was the Coverage of Individual Reform Themes Adequate?...................................6 3. Efficacy..........................................................................................................................7 How Effective Were the Different Instruments Used?.............................................7 Analytic Work..............................................................................................7 Adjustment vs. Investment Lending ............................................................9 Stand-alone Projects...................................................................................13 Developing Institutions..............................................................................13 Was Lending Contingent on Previous Progress on Policy Reform?.....................15 Has the Bank Addressed Rural Poverty Effectively?.............................................17 4. Outcome......................................................................................................................19 Does Policy Reform Lead to Productivity Growth? ..............................................19 Does the Policy Environment Influence Project Outcomes?.................................20 Which Reform Themes Show Most Progress? .......................................................21 5. Conclusions.................................................................................................................27 Annex 1. Method..............................................................................................................29 Annex 2. Status of Agricultural Reforms in CEE and CIS Countries at end of 200133 Annex 3. ECA Strategy for Agriculture ........................................................................35 Annex 4. Project Cohort..................................................................................................39 Annex 5. Rating the Project Cohort...............................................................................42 Annex 6. Lessons from the Project Cohort....................................................................49 Annex 7. Poverty Indicator Ratings...............................................................................59 v Annex 8. Does ECSSD Lending Discriminate in Favor of Countries that Lead on Agriculture Policy Reform?......................................................................................60 Boxes Box 1. The Food Sector Study ­ A Pioneer Effort ..........................................................8 Box 2. A Staff Member's Frustration with New IDF Funding Procedures................15 Tables Table 1. Per Capita Bank Assistance to Rural Areas: By Region ................................5 Table 2. Level of Bank Assistance to ECA Rural: By Period........................................5 Table 3. Coverage of Reform Themes by Period ............................................................6 Table 4. ECA: Investment and Adjustment Lending Bearing on Agriculture..........10 Table 5. OED Outcome Rating for Different Lending Instruments: ECA Cohort ...10 Table 6. ECSSD Staff Response to Hypotheses About Strategy (N=21 staff) ............12 Table 7. Adjustment Loans: Number of Tranches and Outcome Rating FY90-01...13 Table 8. Association of Reform with Productivity Growth..........................................20 Table 9. Association Between Country Reform Rating and Project Outcome Rating.................................................................................................................21 Table 10. Association of Commitments with Productivity Growth ............................21 Table 11. Rating of Completed Projects by Reform Theme........................................22 Table 12. Outgoing Investment Projects with Strong Evidence of Results ................22 Table 13. The Gap Between CEE and CIS Countries..................................................23 vii Acknowledgments ECSSD management and staff participated in the development of this study and their cooperation is gratefully acknowledged, as is that of the other persons outside ECSSD who gave generously of their time. The task team leader for this study was John R. Heath. Consultants Elliott Hurwitz and Zvi Lerman helped analyze the data and prepared background reports. The following Bank staff provided valuable comments on earlier drafts: Csaba Csaki, Joseph Goldberg, Mark Lundell, John Nash and Iain Shuker. The OED peer reviewers were Alice Galenson and Christopher Gerrard. Staff assistance was provided by Rose Wairimu Gachina. ix Executive Summary 1. This desk study evaluates the Bank's contribution to policy reform in the agriculture sector of the ECA transition economies, covering the period 1991-2002. Compared to other regions, agriculture is a small sector in ECA; but it accounts for a major share of the Bank's regional assistance--second only to multi-sector projects in the number of projects financed. 2. The evaluation is based mainly on sources internal to the Bank: documents, databases and staff interviews. Data from these sources are compared to annual ratings of agriculture reform progress by ECCSD, and statements of the Bank's rural strategy for ECA. 3. Agriculture policy reform is organized by the five themes captured for each of the transition economies by the ECSSD ratings: · Trade liberalization and market development · Land administration and reform · Privatization of agro-processing and input supply · Rural finance · Institutional reform. 4. The main findings were as follows. First, ECSSD's assistance program has grown substantially. ECA-wide, per head of the rural population, the Bank's lending and administrative spending are twice the mean for all Bank regions. Between 1991-96 and 1997- 02, in per capita terms, there was an increase of 12 percent in commitments and 35 percent in Bank administrative spending to the transition economies of ECA. Taking the evaluation period as a whole (1991-02), of the five policy reform themes, land policy and rural finance received, respectively, the first and second greatest coverage. Between the first and second half of the evaluation period the number of land policy projects almost tripled. 5. Second, there has been substantial variation in the progress made on the five themes. Of the completed projects, operations devoted to land policy and rural finance received the largest proportion of satisfactory ratings. Of the ongoing projects, most progress is being made by those devoted to land policy and institutional reform (specifically, of agriculture services). Also, based on citations of lessons learned, these themes have also generated the largest amount of learning, which may be conducive to more effective Bank support. 6. Third, ECSSD staff show a growing appreciation of the need to tackle policy reform with a variety of instruments, and to abstain from doctrinaire approaches. More ECSSD staff support adjustment operations than oppose them but there is a lack of consensus about their relative importance for policy reform. Most staff said that institutional development cannot be achieved without investment lending. There is some evidence that even small levels of investment lending can promote significant institutional development. 7. Fourth, monitoring of rural poverty has been neglected. Between the beginning and the end of the evaluation period there was an increase in absolute poverty in the ECA transition economies. Recent statements of ECA rural strategy give greater prominence to poverty reduction than was the case at the start of the transition. The study examined the x adequacy of poverty monitoring by projects and found that, despite an overall improvement in the use of performance indicators, there are few poverty-specific indicators and even fewer data. Also, there are few references to poverty reduction in the citation of lessons learned. 8. In addition, the following specific findings were made: · Between 1991-96 and 1997-02, ECA was the onlyBank region where there was an increase in rural commitments per capita per year (paragraph 2.4). · Analytic work grew rapidly in strength and depth--rapidly building from near zero in the early 1990s--and was made extensive use of outside the Bank; but there was more demand for it from CEE than CIS governments (paragraph 3.4). · Many ECSSD staff perceive one-tranche adjustment operations to be more effective but the data show that they are not more likely to lead to satisfactory outcome ratings than multi-tranche operations (paragraph 3.18). · Contrary to staff perceptions, stand-alone projects (covering only one reform theme) have not been associated with a higher frequency of satisfactory outcome ratings than projects covering several themes (paragraph 3.19). · Pilot projects have been successfully scaled up (paragraph 3.22). · Strategically-located policy analysis units have helped to champion reform (paragraph 3.24); but the Bank's institutional development funding mechanism no longer provide ready support for these units (Box 2). · On sequencing, because institutions take so long to nurture (paragraph 3.20) experience suggests that institutional reform needed to be tackled at the beginning of the transition; and land reform should also have been broached at the start, parallel to price and trade reforms (paragraph 4.16). · Agribusiness privatization has not been served by an effective partnership between the Bank and IFC (paragraph 4.18) · There has been considerable innovation in approaches to rural finance, including successful experiments with village-based group lending and easing of systemic constraints (paragraphs 4.19-4.21) · Overall, project monitoring was better in 1997-02 than in 1991-96 (paragraph 3.31); since 1997 particularly good monitoring has been developed for agriculture service projects (paragraph 4.22); but monitoring of poverty indicators has been neglected (3.31). 1 1. Background and Approach 1.1 Ten years or so after most countries in ECA began their transition from command to market economies, it is an opportune moment to examine progress made and the Bank's contribution to it. This assessment concentrates on the agriculture sector, narrowly defined; it does not cover rural infrastructure (including irrigation) or natural resource management. Other sectors are covered in a larger OED study that is now underway.1 1.2 There is geographical and thematic sequence to the Bank's work on ECA. Before the dissolution of the Soviet Union in 1991, lending for agriculture in the socialist countries was confined initially to Yugoslavia and Romania, followed (from 1985 onwards) by Hungary and (after 1990) by Poland. In the early years of the post- socialist transition (1991-94), agriculture lending was limited to Eastern Europe and Russia. After1995, the other countries of the former Soviet Union joined the fold. In the earlier period, the Bank's assistance leaned towards trade liberalization and agribusiness privatization (informed by copious analytic work). This was followed by work on farm and land privatization and restructuring, rural finance, and (most recently) agricultural research and extension. As the compass of the Bank's work has extended geographically and thematically so the challenges have become greater, embracing poorer countries with less well trained government staff and more intractable issues (broadly, "institutional development"). APPROACH 1.3 This assessment is to a large extent a meta-analysis--a desk-based synthesis of previous evaluative reports that adds value by presenting a big picture, a whole that is intended to be larger than the sum of its parts. This is supplemented by interviews with the staff who are (or have been) most directly engaged with ECA agriculture and sector policy reform. Owing to the limited budget for the study it was not possible to interview development partners or country counterparts, or to survey beneficiaries. This potentially introduces a bias: staff comment may be self-serving. On the other hand, staff are generally better informed than others about what the Bank is doing; and the purpose of the exercise was as much to capture differences in staff perceptions of the Bank's effectiveness as it was to extract an aggregate assessment. 1.4 The assessment gives particular emphasis to three principles of OED method--relevance, efficacy, and outcome. In view of data and resource limitations, no attempt is made to evaluate impacts--that is, to isolate the specific contribution of the Bank either to progress on agriculture policy reform, or to the possible growth and poverty reduction payoff to reform. 1.5 Relevance poses the question: Did the Bank do the right thing?--in terms of country needs and the Bank's overall mission. For this study, the specific questions are 1. "Evaluation of World Bank Assistance to the Transition Economies". An entry workshop for this study was held on May 15, 2002. The study's web address is: (http://www.worldbank.org/OED/transitioneconomies/index.html). 2 (1) Were the resources committed by the Bank commensurate with the relative importance of the agriculture sector in the transition economies? (2) Did the Bank pay sufficient attention to the diverse facets of policy reform? In addressing this question, the study uses the typology developed by ECSSD which distinguishes five themes: · Trade liberalization and market development; · Land administration and reform; · Privatization of agro-processing and input supply; · Rural finance; · Institutional reform. 1.6 Efficacy asks whether the Bank did what it said it would do (whether its actions were in line with its statements of strategy) and whether the best approach was taken. The specific questions are (3) How effective were the different instruments the Bank used to promote policy reform? Instruments include analysis, technical assistance, adjustment lending and investment lending. (4) Did the Bank adhere to its stated principle of getting the country's policy framework right before making big loans? The 1997 rural strategy for ECA (Annex 3) emphasizes this principle. In the wake of the 2002 Monterrey conference the same principle has become a cornerstone of the Bank's development assistance strategy. (5) Was the Bank effective in tackling rural poverty in ECA? The poverty issue bears on relevance as well as efficacy: the 1997 ECA rural strategy indicates that poverty is not a major issue for the region. By downplaying poverty was the Bank's approach fully responsive to client needs? Efficacy is tackled from the angle of project monitoring of poverty: on the assumption that "what gets measured gets done"--if income, employment and living standards effects of projects are not kept track of, the Bank is likely to be less effective in helping to reduce poverty. Also, consideration is given to whether the lessons cited in appraisal and evaluation reports reflect an engagement with the issue of poverty. 1.7 Outcome refers to the good or bad results of a development initiative-- without pretending to attribute a precise weight to the Bank's contribution. (6) How strong is the association between policy reform and agriculture productivity growth? The aim is to consider the payoff to the Bank's work on behalf of policy reform. Does the (limited) available data demonstrate that countries with a good macroeconomic and sector policy framework are better able to achieve agriculture productivity growth--which may help to raise rural living standards--than countries that lag on policy reform? 3 (7) Do agriculture projects have a higher proportion of satisfactory outcome ratings in reform-leader countries than in countries that are policy reform laggards? (8) What differences are there between the five reform dimensions (see Question (2) above) in terms of progress made? This question is addressed by considering the progress reports on ongoing projects, as well as evaluations of completed projects. 1.8 The method used for addressing these eight questions, and the sources of data, are described in Annex 1. 4 2. Relevance WAS THE OVERALL SCALE OF ASSISTANCE APPROPRIATE? 2.1 Is Bank spending and lending on agriculture commensurate with the relative importance of the sector in the economy--allowing for the inevitable shrinkage that the transition entailed? During the socialist period the agriculture sector was larger than it would have been under a market economy. "Centrally planned economies channeled an inordinate proportion of resources into food production. In a perverse economic triumph they managed, despite inefficiency, to deliver a better diet than consumers could afford in the long run, or would choose to buy at unsubsidized prices."2 The transition necessarily entailed an initial collapse of food production. 2.2 ECA-wide, the average share of agriculture value added in GDP declined from 17 percent in 1990 to 10 percent in 2000 (compared to a fall from 9 to 7 percent in Latin America, a region that is comparable to ECA in terms of PPP gross national income per capita). Over the same period, the proportion of the population living in rural areas fell from 41 percent to 35 percent.3 This was a major turnaround. Of the six Bank regions, only Latin America and the Caribbean had a smaller agriculture/rural share in 2000. 2.3 Yet agriculture's role in the ECA portfolio has been substantial. Of all the projects exiting the portfolio since FY1989, agriculture is second only to multi-sector projects in the number of operations financed (27 out of a total of 209 projects); and it is third, after multi-sector and economic policy, on net commitments (US$1.7 billion out of US$19.2 billion).4 With respect to sector-specific analytic work, agriculture is second to energy in the number of reports produced, accounting for 18 of the 126 reports to appear in gray cover. 2.4 Per head of the rural population in the region, The Bank's rural lending and spending in the ECA region was twice the mean for all regions (Table 1).5 Between 1991-96 and 1997-02, ECA was the only region where there was an increase in rural commitments per capita per year: they rose by 36 percent (from US$1.65 to US$2.24), compared to a Bank-wide decline of 20 percent (from US$0.98 to US$0.79 per head). ECA has also done more analytic work. A 2002 OED study found that between 1993 and 1999, ECA spending on sector work amounted to one-third of spending on project preparation, compared to a cross-regional mean of one-quarter. Because ECA was new to the Bank and because the transition from command to market economy was a new 2. Karen M. Brooks, "Decollectivization and the Agricultural Transition in Eastern and Central Europe", World Bank Policy Research Working Papers, WPS No. 793, October 1991. 3. In 1998-2000, 21 percent of the labor force (the same proportion for males and females) was employed in agriculture (World Development Indicators 2002, p. 58). 4. Excluding Bosnia, Cyprus, Portugal, Turkey and Yugoslavia. These numbers do not include structural adjustment loans with agriculture policy conditions. 5. In this paragraph the lending and spending figures cover all activities mapped to the Bank's rural units, not just those bearing on agriculture. 5 departure, there was a need to invest heavily in acquiring the knowledge that would underpin lending operations. Table 1. Per Capita Bank Assistance to Rural Areas: By Region Rural Rural Rural Spending on Rural Population, commitments commitments rural analytic spending per 1995 approved, approved per work, project head of the (millions)/1 FY1991-2002 head of the preparation rural (US$ million)/2 rural and population population per supervision, per year year FY1993-1999 (US$) (US$) (US$'000)/3 AFR 399.6 3,967 0.83 104,140 0.04 EAP 1,176.8 10,059 0.71 61,892 0.01 ECA 168.9 3,938 1.94 46,658 0.04 LCR 128.1 5,242 3.41 54,807 0.06 MNA 113.1 2,659 1.96 30,825 0.04 SAR 907.5 5,163 0.47 52,139 0.01 All Regions 2,894.2 31,028 0.89 350,461 0.02 /1 World Development Indicators. /2 World Bank, Business Warehouse data. /3 World Bank, Corporate Resource Management database. 2.5 How did the level of assistance vary between the first and the second half of the evaluation period? To make the analysis more comprehensive, in Table 2, the definition "rural" commitments is increased to include 20 percent of all structural adjustment loans which include agriculture policy conditions.6 Average annual loan approvals went up by 12 percent per capita while spending increased by 35 percent. Bank spending per dollar lent rose from 6.2 to 7.4 cents. Table 2. Level of Bank Assistance to ECA Rural: By Period Rural Rural Rural Spending on Spending per population commitments commitments rural analytic head of rural (millions)/1 approved approved per work, project population (US$ million)/2 head of rural preparation per year population per and (US$) year supervision (US$) (US$'000)/2 FY1991-1996 168.9 (in 1,745 1.72 108,158 0.11 1995) FY1997-2002 167.1 (in 1,940 1.93 144,226 0.14 2000) Change -1.1% 11.2% 12.5% 33.3% 34.8% /1 World Development Indicators. /2 World Bank, Business Warehouse data. Note. The definition of ECA is limited to the transition economies (unlike in Table 3, which refers to all ECA countries). 2.6 In summary, overall Bank assistance to ECA agriculture was high in relation to the attention to other sectors, in comparison with other regions, and in view of the substantial contraction of the sector that took place in most ECA countries after the 6. The estimate of spending also includes 20 percent of the preparation cost of structural adjustment loans that are not mapped to Rural but which include agriculture conditions. 6 transition. However, the level of assistance given was probably more than justified by the size of the structural adjustment required; strong support for agriculture policy reform facilitated the adjustment WAS THE COVERAGE OF INDIVIDUAL REFORM THEMES ADEQUATE? 2.7 Taking the evaluation period as a whole (1991-2002), land and rural finance received, respectively, the first and the second greatest project coverage. Between the first and the second half of the period there was some change in the emphasis given to the five reform themes (Table 3). 2.8 The number of "land" projects almost tripled. Of the five themes, only agribusiness witnessed little growth in project coverage. The credit lines associated with rural finance projects explain why this theme generates the largest lending volume. As a share of all projects, the number of adjustment operations increased for the trade liberalization, agribusiness and finance themes and decreased for land and institutional reform (agriculture services). In both periods, trade liberalization and market development received more attention in analytic work (sector reports) than any other theme. In each period, country assistance strategies (the first of which appeared in 1994) gave roughly equal attention to each theme (Annex 7). Table 3. Coverage of Reform Themes by Period Agricultural Reform Themes Trade Land Agribusiness Rural Institutional Liberalization Admini- Privatization Finance Reform and Market stration Development and Reform 1991-96 N of projects covering each theme/1 7 8 8 10 8* Share of adjustment ops. in N of 57% 63% 38% 10% 13% projects Project cost specific to theme (US$ 255.9 249.5 256.5 410.0 158.0* million) N of analytic reports covering each 15 10 7 6 8 theme/1 N of CASs referring to each theme/1 7 8 6 7 6 1997-02 N of projects covering each theme/1 13 24 12 17 12* Share of adjustment ops. in N of 77% 42% 75% 24% 8% projects Project cost specific to theme (US$ 276.1 391.6 224.9 414.0 86.4* million) N of analytic reports covering each 17 13 7 8 12 theme/1 N of CASs referring to each theme/1 11 12 10 12 13 Source: Annexes 4 and 6. /1 A project or report may cover more than one theme; therefore, summing across the row gives a number greater than the sum of individual projects or reports. * Agriculture services only. 7 3. Efficacy HOW EFFECTIVE WERE THE DIFFERENT INSTRUMENTS USED? 3.1 Based on outcome ratings, lessons cited in project documents, and the comments by staff who were interviewed, a picture emerges of a dynamic program of assistance that has evolved continuously since the beginning of the transition, effectively exploiting the complementarities of the instruments available. Analytic Work 3.2 Since 1991 a substantial investment has been made in analytic work and related technical assistance and policy dialogue. The number of reports produced (Annex 7) speak to the importance of this investment. In the early 1990s more than half of all Bank agriculture sector reviews were devoted to ECA countries. The cost figures for sector work are far less reliable than they are for projects, mainly because the accounting system is less rigorous. Business Warehouse data is patchy for the period before 1997 and not easily reconciled with earlier data sources. Also, much analysis is done under the cover of project preparation and supervision and does not result in formal sector reports: many staff commented that this trend has intensified in recent years. It is therefore difficult to assess long-run trends in the relative importance of sector work. 3.3 How has this work evolved? Throughout the evaluation period the sector work carried out by ECSSD and its precursors has helped underpin structural adjustment operations and has fed into the preparation of country assistance strategies and economic memoranda. Against this backdrop, sector work progressed through four phases. In the early 1990s comprehensive reviews provided a benchmark assessment for various ECA economies, setting out an agenda for policy reform. One of the largest of these was a food sector study (Box 1), begun before the collapse of the Soviet Union. In the mid-1990s, the focus shifted to specific themes. A substantial body of work on land reform and farm restructuring led to the Bank becoming the leading source of knowledge about the these issues in central and eastern Europe. Other studies were devoted to rural finance, trade and research and extension. During a third phase (the latter part of the 1990s), a series of sub-regional reports were produced, focused mainly on accession to the European Union and benefiting from Strategic Compact funding. Regional seminars based on this work brought the Bank into closer dialogue with the European Commission. Another multi-country initiative tackled the question of farm debt in the CIS countries, influencing debt settlement legislation in Moldova and Ukraine. Since 2000, a fourth phase has involved the next cycle of comprehensive sector reviews, including Moldova, Macedonia and Serbia. 8 Box 1. Pioneering Work on Food and Agriculture Policy Reform A 1994 OED study on the Bank's technical assistance program to the former Soviet Union praised the work on developing an action plan for food and agriculture policy reforms. The Bank organized a team of international experts with the participation of the IMF, the EBRD, the EC and the OECD. The original focus of the work was the whole of the USSR and international consultants visited a number of the Republics in December 1991. Following the breakup of the USSR the work focused primarily on Russia. A major seminar was organized jointly with the Government of the Russian Federation to discuss the report in the Spring of 1992 and the Report as well as a number of background papers were subsequently published. The experts' wide-ranging recommendations formed the basis for the beginning of a dialogue with Russian authorities on agricultural sector reform. Some reform efforts in the sector have been implemented consistent with the Report's recommendations. For example, significant price liberation and reduction of producer and consumer subsidies occurred in 1992-93; also, in October 1993 a Presidential decree was issued which eliminated virtually all restrictions on the sale, mortgage, leasing, inheritance or granting of land. At the same time, politically strong conservative elements in the sector retarded implementation of recommendations in the areas of firm restructuring and privatization. 3.4 This analytic work involved much effort and was highly influential. Within the space of a few years the Bank became a major repository of knowledge on ECA. An 1997 OED evaluation found that donors, firms and academics were major users of the Bank's ECA studies--to a greater extent than in other regions. The market for sector work was more developed in the CEE than the CIS. In CEE countries government staff were more highly trained and more receptive to western ideas: there was a ready market for the Bank's sector work and, in many cases, the analytic work was more critical for development than lending. In the CIS, governments tended to be less sophisticated and less well placed to express a demand for sector work: their primary interest was in the money, not the advice, the Bank brought to the table. 3.5 Assessing the overall quality of the work is beset with difficulties. Unlike projects, sector work entails sketchy self-evaluation, dating back only to the late 1990s: compared to implementation completion reports, the activity completion reports on sector work are thin on description and largely devoid of evaluation (a Bank-wide problem). In FY2000-02, the database contains 15 activity status reports for completed ECSSD sector work tasks; of these only 8 contain evaluative comments by the sector manager. Despite the gaps in description and evaluation, the record does show a very rich and varied program of sector work with a marked problem-solving, practical orientation. 3.6 The annual reviews of sector work by the Quality Assurance Group have so far picked up only five ECSSD reports that fit the scope of this evaluation: four of these were rated satisfactory and one (FY 00 Belarus Farm Restructuring) highly satisfactory. The reports score highly on the involvement of counterparts and other donors and the contribution made to policy dialogue. In two cases reservations are expressed about insufficient treatment of rural poverty and incomplete signaling of priorities. The difficulty of managing sector work together with project work-- leading to late delivery of the former--is also mentioned. 9 3.7 The staff interviewed for this evaluation were asked to comment on the quality of sector work. The overall assessment was positive, with particular emphasis on the rigor and depth of the analysis. There is some concern by ECSSD managers that in the early years the recommendations were too economically purist--almost theological in the insistence on liberalization and privatization--paying insufficient attention to political realities, or the implication of those realities for priority setting and sequencing. Sometimes this amounted to monolog not dialog. Many staff commented that the focus on recommendations for increasing the competitiveness of the agriculture sector took pride of place over concerns about the impact--or, more correctly, lack of immediate impact--of the reforms on poverty reduction. Several people noted that in the early years sector work was poorly connected to project work and insufficiently problem-oriented. But even if the format was perceived as unwieldy by many project- focused staff, the comprehensive sector reviews were essential for building a knowledge base and widely drawn upon. A certain lack of articulation may have resulted from treating sector work as an independent activity conducted by specialized staff. The 2002 strategy sends a clear message that sector work should not be handled by specialized sector economists but by persons active in project work.7 Recent experience shows that this approach is not without its problems: the unending press of multiple project management tasks makes it difficult for staff to find the large blocks of uninterrupted time that are needed for rigorous analysis. 3.8 Two trends are evident. First, a handful of staff with a sophisticated background in policy analysis have departed and have not been replaced. One staff member observed that this has weakened the capacity of ECSSD to champion the rural strategy at the critical meetings where resources are allocated; for this person it helped to explain why, despite rural lending's prominent place in the ECA portfolio, there is little treatment of the rural dimension in the recent flagship study on the first ten years of transition.8 3.9 A second trend, in line with other regions, is a greater emphasis on short "sector policy notes", with a rapid turn-over time and timely delivery. Many country directors favor this format and tend to reserve spending on formal sector work for the PREM unit, rather than the sector units. There is an expectation that the analysis be done under the cover of project preparation and supervision, rather than as an independent activity. Adjustment vs. Investment Lending 3.10 In terms of average levels per period, adjustment lending for agriculture grew in importance between 1991-96 and 1997-02 (Table 4). But adjustment operations are very unevenly distributed between years and there is no evidence of a steady annual increase. Table 4. ECA: Investment and Adjustment Lending Bearing on Agriculture 7. ECSSD, Reaching the Rural Poor in the Europe and Central Asia Region, October 2002, p. 68. 8. World Bank, Transition: The First Ten Years. Analysis and Lessons for Eastern Europe and the Former Soviet Union, Washington, D.C., 2002. 10 FY1991-96 FY1997-FY02 US$ millions % US$ millions % Investment/1 1,219 70 1,062 55 Adjustment: 322 18 776 40 "SECAL"/2 Adjustment: 204 12 102 5 "SAL/3 Total 1,745 100 1,940 100 Source: Business Warehouse. /1 Investment loans mapped to Rural. /2 Adjustment loans mapped to Rural. /3 Adjustment loans not mapped to Rural but with agriculture policy conditions. 3.11 Bank-wide, across all sectors, 84 percent of adjustment operations had a satisfactory outcome rating in FY00-02, compared to 77 percent for investment operations.9 For completed agriculture projects in this evaluation's cohort (Annex 4), adjustment operations also receive a satisfactory outcome rating more frequently than investment projects. (Table 5). But this does not necessarily mean that adjustment operations produce better development results than investment projects; it may mean that the Bank's methods of evaluation are better adapted, or more rigorously applied, to investment projects.10 Table 5. OED Outcome Rating for Different Lending Instruments: ECA Cohort 1991-02 Investment operations Adjustment operations N of operations 14 21 Percent satisfactory 67% 79% Source: Annex 4. 3.12 The relevance of adjustment operations to policy reform was vigorously debated around the end of 1998 in ECSSD staff retreats. Arguments were made for and against the use of sector-specific adjustment operations (ASALs) as the primary instrument of reform. Proponents argued that although ASALs may be imperfect they are the best available tool for reforming the policy environment (and reform is essential because if the policy environment is not adequate investment operations will not result in growth). It was argued that adjustment operations work if the government is genuinely committed. They are also important because the resources they entail make it possible for the Bank to "buy a seat at the table". The Bank's capacity to make adjustment loans gives it an edge in relation to other donors. To the extent that the Bank and the European Union are promoting different agendas, it is important to conserve the adjustment option if the Bank is to remain influential in the region. 3.13 A number of counter-arguments were made. First, Ministries of Agriculture invariably oppose or give only lukewarm support to adjustment operations because the funds do not directly benefit the sector. Second, experience shows that many of the reforms made under the adjustment aegis have been weak and superficial: capable of being reversed at the stroke of a pen. (This was truer for the earlier round of 9. OED, Annual Review of Development Effectiveness, 2002 (Table A1). 10. See presentation by C. Gerrard and S. Ramachandran, OED Staff Retreat, February 2003. 11 adjustment loans; later operations paid more attention to reforms that could not so easily be reversed). Third, the sector management unit is often pressured by country management units into releasing the funds for macroeconomic and political purposes, even if the sector-specific reform conditions have not been met. 3.14 In a rejoinder, the ASAL proponents argued that the alternatives to this type of operation were not very compelling. First, the option of attaching sector conditions to multi-sector and macroeconomic adjustment loans (SALs) is problematic: although SALs can help concentrate the minds of Finance Ministries they are not a good vehicle for fostering sector dialogue; and sector conditions are even more likely to be waived in the face of political pressure than is the case with ASALs. Second, attaching policy conditions to investment loans is not a good alternative because when conditions are not met the whole operation grinds to a halt, preventing the realization of valid physical objectives (such as rehabilitation of fast-depreciating irrigation works). Moreover, it is difficult to achieve nationwide reforms through investment projects because these are typically limited to certain regions; this is a particular problem in big countries. Third, sector investment loans (which involve the funding of part of a line ministry's program) have run into procurement difficulties, are often too complex to be workable, and may actually substitute for government funds rather than supplementing them. 3.15 Clearly, adjustment operations are not sufficient. On the other hand, one-third of those interviewed were unconvinced that they were even necessary (Table 6, Hypothesis Number 3). Some of these people stated that in countries with the weakest policy frameworks what counts is continuous coaching: quick-disbursing operations do not change mindsets; what is needed is persistent dialogue. In Ukraine, for example, the sector adjustment operation met with only moderate success in pushing through policy reform; a bigger impact was achieved with the policy analysis unit staffed by Ukrainian economists with Western training. This unit is ready on call to support reformers with analysis and prepare comments on regressive policy proposals. 3.16 While more ECSSD staff support the principle of adjustment than oppose it, stronger than any endorsement of adjustment is the call to work simultaneously with several complementary instruments: using analytic work and regular conversation with diverse interest groups to build support for reform; employing technical assistance to work out the legal and regulatory details; and offering investment loans--whether parallel, or subsequent, to the adjustment operation--as carrots to entice the skeptics (particularly line ministries that do not benefit directly from adjustment funding). Staff particularly emphasize the need for well-supervised and well-coordinated technical assistance: this is one of the lessons learned that is most frequently cited (Annex 6). Table 6. ECSSD Staff Response to Hypotheses About Strategy (N=21 staff) 12 HYPOTHESES Accept Reject Neither Accept Nor Reject Number 1. At the beginning of the transition, rather than 57% 14% 29% trying to reduce rural poverty through specific interventions it is more important to take measures to raise the productivity of agriculture. Number 2. It is generally unwise to make large investment 76% 14% 10% loans before a satisfactory macro and sector policy framework has been established. Number 3. In countries where the policy framework is 48% 33% 19% poor, policy-based lending (SALs, SECALs, loans with substantial conditionality) is necessary in order to achieve reform. Number 4. In the absence of a substantial lending 62% 33% 5% program it is difficult to bring about institutional reform. Number 5. Single-sector (agriculture) projects are more 57% 33% 10% likely to achieve their development objectives than multi- sector (rural development) projects. Source: OED interviews, October 2002. The question put to staff was "Please indicate whether your specific experience in ECA would lead you to accept or reject the following hypotheses". 3.17 In summary, ECSSD has had an open, vigorous and healthy debate about the merits of adjustment. A categorical statement about when, or when not, to launch these operations would be unreasonable to expect--the large differences between countries make generalization hazardous. What has emerged is a greater appreciation of the complexities. It is hard to discern a particular trend. For example, although the argument has been made for inserting agriculture policy conditions in structural adjustment loans rather than designing sector-specific policy operations, project approvals do not yet show a clear trend in this direction. (A Bulgaria sector adjustment loan--which received a satisfactory outcome rating--was approved as recently as FY01). 3.18 In order that implementation not be held up by reform wrangling, there is some suggestion of a move toward one-tranche operations--meaning that all policy conditions must be met before the loan is approved. A lesson has been drawn that single-tranche operations are preferable (Annex 6). However, the overall data for ECA (all sectors) do not show that one-tranche operations are more likely to result in a satisfactory outcome rating (Table 7). Taking all countries together the rate of satisfactory outcomes is higher for single-tranche loans; but the relationship disappears when the countries are broken down by level of policy reform and institutional development. This is because single-tranche loans are more likely to go to countries scoring highest on the reform and institutional ratings; weaker performers tend to get multi-tranche loans. Country rating influences the propensity for 13 satisfactory outcome much more than the number of tranches; this holds for all regions, not just ECA.11 Table 7. Adjustment Loans: Number of Tranches and Outcome Rating FY90-01. Share of operations CPIA Grouping* with satisfactory outcome All Countries 3.5 or lower 3.6 or higher All Regions One tranche 89% 75% 95% Two tranches 87% 70% 92% Three or more tranches 68% 60% 86% ECA One tranche 85% 69% 92% Two tranches 80% 73% 100% Three or more tranches 73% 60% 100% Source: Operations Policy and Country Services, 2002. * CPIA Country Policies and Institutions Assessment (World Bank indicator). Stand-alone Projects 3.19 Surprisingly perhaps, projects that are dedicated exclusively to one of the five reform themes have a significantly weaker outcome rating (50 percent satisfactory) than projects that cover more than one theme (67 percent satisfactory). This runs counter to staff perceptions. For land administration projects, one of the most frequently cited lessons (based on the findings of a 1992 OED evaluation of land titling projects) is that "stand-alone operations work best". In the staff's response to Hypothesis Number 5 (Table 6), there is also a hint that multifaceted projects are thought less likely to work; although some pointed out that they can be made to work if the project does not extend beyond a small area of the country. Between the first and the second half of the evaluation period the number of stand-alone land projects approved rose from one to six. (For the other themes there was no clear trend, except for agribusiness where it was the opposite, the number of stand-alones falling from five to zero). Developing Institutions 3.20 A much-cited "lesson learnt" is that institutional development takes a long time--much longer than was originally expected. Does this mean that quick- disbursing adjustment operations are less appropriate vehicles than slower-disbursing investment projects? In ECA, as for other regions, adjustment operations rate 11. There is more evidence to bear this out. In a study of 182 World Bank adjustment loans it was found that the allocation of conditions between upfront conditionality and first, second, and third tranches plays no role in determining the success rate (David Dollar & Jakob Svensson, "What Explains the Success or Failure of Structural Adjustment Programs?", World Bank Policy Research Working Papers, No. 1938, April 1998. 14 somewhat lower than investment operations on institutional development impact.12 But the difference is not statistically significant. More important than the vehicle is the context into which it is inserted: CEE countries rate higher on institutional development impact than CIS countries, irrespective of whether adjustment or investment lending are employed. 3.21 Can institutional development occur in the absence of lending? Almost two- thirds of ECCSD staff are inclined not to think so (Table 6, Hypothesis 4), arguing that "if you don't lend you can't get a seat at the table". And investment projects-- because they are slower disbursing--buy a longer stay at the table than adjustment operations. They also bring staff into closer contact with the people whose ways are most likely to be in need of reform: the staff of the line ministries. Only a minority of staff interviewed said that institutional reform could be achieved by dialogue and analytical work alone. Most agree that lending is necessary; but a more relevant question--about which there is less agreement--is what scale of lending is needed. 3.22 There is some spectacular evidence that a lot can be achieved with small, pilot projects. In Azerbaijan, the FY97 Farm Privatization project supported the distribution of the land of six farms to their members. Plans to scale up have been fully realized. Almost all the state and collective farms in the country have been privatized, based on the model used for the six pilot farms. Not only were all 10,500 hectares of arable land of the six pilot farms distributed among 6,645 eligible farm families with land titles and ownership certificates, but 1.4 million hectares of arable lands formerly belonging to 1,979 state and collective farms were transferred to some 870,000 farm families. 3.23 The case of Ukraine illustrates one of many possible trajectories of institutional development. Here there were a number of half-starts before real progress was made. An investment project (Seed Development, FY95) had an unsatisfactory outcome. Except for some improvements in seed certification, there was no discernible change in the management and business conduct of participating government agencies. 3.24 A loan for a Pre-Export Guarantee Facility (FY97) never became effective. The Agriculture Sector Adjustment operation (FY97) was also less successful than hoped, partly because the Ministry of Agriculture gave little support: although grain elevators were privatized, little headway was made in liberalizing domestic agricultural markets and restructuring state agricultural institutions. Simultaneously, there was an unsuccessful attempt to use a Soviet-era survivor, the Institute of Agrarian Economics, as a vehicle for analytic work and a springboard for policy dialogue. Finally, what succeeded was a policy analysis unit, a multi-donor initiative fortuitously housed at a high level of government (the Vice Prime Minister's Office), staffed by young Ukrainians with Western training. This is not to say that the previous less-than-satisfactory initiatives had no bearing on the unit's ultimate success in promoting policy reform: there was a lengthy but indispensable process of 12. Zvi Lerman, "Policy-Based Lending and Agricultural Policy Reform in ECA", July 2002, pp. 36- 36 (OED Background paper. Photocopy). 15 learning from past mistakes and building on half-achievements. It is of some concern that a recent revision of the terms of Institutional Development Fund grants makes it harder now to use them for supporting policy analysis units (Box 2). Box 2. A Staff Member's Frustration with New IDF Funding Procedures "The Bank has a very successful policy analysis unit in Ukraine funded by an Institutional Development Fund grant: a small investment with a huge payoff in terms of policy results. We thought we would replicate this activity in Georgia, and submitted the application to the IDF panel. We were turned down, despite having an excellent Minister in Georgia who really wanted to reform the Ministry and a model that had already been shown to work in Ukraine. The problem was that the eligibility rules for IDF had changed since the Ukraine proposal was approved. The IDF grant program has essentially been captured by the lawyers, accountants and procurement establishment in the Bank. (you may be able to come up with a better collective name) The new eligible activities are as follows: Strengthened public expenditure systems, well managed financial systems, improved procurement systems, results oriented monitoring and evaluation, and systematic changes in legal and judicial systems. Nothing else qualifies, and so our grant was rejected." Source: Personal communication from Iain Shuker. WAS LENDING CONTINGENT ON PREVIOUS PROGRESS ON POLICY REFORM? 3.25 How did the level of Bank assistance vary with the progress countries had made on agricultural policy reform? The principle of lending selectivity--favoring countries most advanced on policy reforms--was a cornerstone of the 1997 ECA strategy. At an ECSSD staff retreat in December 1998 a case was made that the size of the agricultural and rural lending programs should depend on progress with relevant policy reform. Investment support would be forthcoming for reforming countries, with little or nothing for non-reforming countries.13 3.26 What actually happened with lending is more complex. Based on ECSSD's own ratings of reform progress, there is no relation between the level of reform in a given year and the volume of lending per capita, or between progress on reform (1997-2001) and per capita lending (Annex 9). Superficially, lending was less selective after 1997 than before it, contrary to the trend in the region's statements of strategy. Presented with this finding some staff commented that lending was demand- driven. This can be read in both a positive and a negative way. On the one hand, countries that are substantially reformed (e.g. Hungary), no longer request Bank assistance: the "graduation" of these countries pushes down the average reform rating so that over time lending appears to become less selective (favoring reform laggards over reform leaders). On the other hand, some staff argue that the Bank's country management units are often captured by their clients (a tendency not peculiar to ECA that has grown stronger since the shift to matrix management): wanting to be responsive to the country counterparts (and under pressure from their superiors) 13. This emerged from the staff interviews conducted for this study. 16 country managers may swallow their principles and lend to reform laggards. In this case, the lack of selectivity is more real than perceived. 3.27 The Bank's experience in Kyrgyz Republic is illustrative. Although the absolute lending volume is small, the country is an extreme outlier in terms of per capita lending and administrative spending (project preparation costs vary little in relation to loan volume so a large number of small projects--the case for Kyrgyz Republic--represents a heavy spending burden for the Bank). In the late 1990s the country was selected as a pilot for the Comprehensive Development Framework, raising pressure to lend. At that time, as Central Asia went, Kyrgyz was a reform leader: between 1997 and 1998 its overall ECSSD rating moved from 5.8 to 6.6. But, as of 2001, it had fallen back to 6.2, mainly because of corruption and weakening governance. Thus, lending expanded substantially at a time when the country was backtracking. Practically speaking, this would have been hard to avoid, even without the CDF pressure. Given that it takes time to prepare investment projects, that projects under preparation acquire inertia, and that the country assistance cycle entails at least a three-year lock-in, it is impossible to fine tune the lending volume to match reform progress. Perhaps the high-profile CDF exercise raised expectations for Kyrgyz unduly, perhaps too many projects were financed; but the larger point about the difficulty of engineering a selective lending strategy still holds.14 3.28 Leaving aside logistical--engineering--problems, there are issues of principle when it comes to lending selectivity; as voiced in the 2002 rural strategy, these principles suggest that ECSSD's approach to lending is now less dogmatic than it was in 1997. The strategy states that "Delaying investments until policies are right can reduce the ability to help large numbers of poor and can sometimes significantly increase the eventual costs of investments (e.g., to rehabilitate deteriorated irrigation and drainage infrastructure)."15 Taken to a logical extreme, this could be read to mean that even in unstable macroeconomic environments a case can be made for project lending, a position that is clearly contrary to the one taken by the Bank at Monterrey. One ECSSD manager took Albania as an example: in the heat of the "pyramid" crisis of 1997 when banks were collapsing and life savings were being lost, the village credit associations supported by the Bank continued to perform well; an OED field-based evaluation concurs. There are other reasons for not being dogmatic. First, poverty tends to be more acute in relatively-unreformed countries: should the most needy be cut off from lending? Second, it is hard to agree on a threshold level of reform beyond which lending might be deemed appropriate. Anyway, in the case of ECA, it could be argued that the threshold has already been exceeded--in most countries trade has been liberalized and assets have been privatized. The main source of reform lag is institutional development, a slippery concept for which thresholds are hard to define; 14. Others have made similar findings about the lack of selectivity. Nunnenkamp uses regression analysis to show that, in the course of the 1990s, the Bank did not increase targeting of financial aid toward poor countries pursuing sound economic policies, contrary to the claims it made at the Monterrey Conference in March 2002 (Peter Nunnenkamp, "Shooting the Messenger of Good News: A Critical Look at the World Bank's Success Story of Effective Aid", Kiel Working Papers, No. 1103 (Kiel Institute for World Economics, Germany), April 2002. 15. ECSSD, Reaching the Rural Poor in the Europe and Central Asia Region, October 2002, p. 65. 17 and for which the ECSSD rating is imprecise. Finally, there is plenty of scope for common sense about lending volume: lending before countries are fully reformed does not necessarily entail big-ticket items; there should be ample room for pilot projects. HAS THE BANK ADDRESSED RURAL POVERTY EFFECTIVELY? 3.29 A 2000 report notes The magnitude of the increase in poverty would probably suffice to distinguish the experience of the ECA region from other regions in recent times. Between 1988 and 1998, absolute poverty rates in Europe and Central Asia increased from 2 to 21 percent. While these estimates are an approximation given serious data deficiencies, there is little doubt that absolute poverty increased dramatically in the region.16 3.30 ECA is also distinguished from other regions by the lower share of the poor living in rural areas. Nevertheless, the countryside houses more than half of the poor, not only in Central Asia, Moldova, Lithuania, Romania, but also in Hungary and Poland.17 Moreover, throughout the region, rural households have a higher risk of poverty. This, coupled with the increase in absolute poverty between the start and the end of the 1990s, would seem to justify the change in the Bank's rural strategy for ECA--the shift from the statement in 1997 that poverty was not the most urgent issue in this region to the statement in 2002 that poverty reduction is, indeed, a priority. 3.31 Is this shift manifest in project design? One test of the centrality of the commitment to reducing poverty is to examine the proportion of projects that contain relevant indicators and, more to the point, whether data is cited for these indicators in supervision and completion reports. (This measure cannot be legitimately applied to adjustment operations because there is no formal requirement that these include key performance indicators). Overall, monitoring of investment projects improved substantially between the first and second half of the evaluation period: the proportion of these projects in the cohort that had good indicators rose from 17 percent in 1991- 96 to 62 percent in 1997-02 (Annex 5). But of the twenty investment projects with good indicators only three included measures directly relevant to poverty reduction. One of these was Moldova Rural Investment and Services (FY02), which included a baseline survey and an indicator to capture the change in rural incomes in project areas relative to the baseline. Two of the three projects were located in Albania and date back to the earlier period. 3.32 Another approach is to examine the frequency with which poverty impacts are discussed in the lessons learned section of appraisal and completion reports. (Unlike the indicators filter, this sweep includes adjustment operations). Out of a total of 272 16. World Bank, Making Transition Work for Everyone: Poverty and Inequality in Europe and Central Asia, Washington, DC, 2000, p. 31. 17. World Bank, 2000, ibid., p. 97. 18 lessons cited for projects in the evaluation cohort (Annex 6) only 7 bear directly on poverty.18 3.33 It could be argued that agriculture projects should first and foremost address growth; and that growth is the single best way to tackle poverty. The literature review found no ECA-specific studies to support this hypothesis. Staff interviewed tended to favor productivity-enhancing over targeted interventions (there is a hint of this in the response to Hypothesis 1, Table 6). This is a legitimate point; but it is still of concern that poverty impact is not being monitored. Even if it is accepted that an explicit focus on poverty reduction is not appropriate for all projects bearing on agriculture, the limited treatment of the issue is still striking. Moreover, even for projects that do not directly tackle poverty, some measurement of income change may be appropriate. It may well be that rural poverty is being adequately addressed by other projects outside the evaluation cohort. But there it is equally possible that the rural dimension of poverty may not be sufficiently captured. 3.34 One staff member suggested that by privileging agricultural productivity and boosting competitiveness, ECSSD has taken a valid tack that is nevertheless not highly relevant for the large number of marginal farm households, those whose land and resources are so limited that they have few prospects of ever being competitive farmers. It might be objected that responsibility for welfare interventions and provision of safety nets for this group can be left to non-agriculture projects. But there is a danger that the rural poor may fall between the cracks, particularly in a region where poverty is often regarded as more of an urban issue. Some staff noted that coordination between ECSSD and the Human Development unit is weak; and that HD's health and education projects did not always pay due attention to the rural population. 18. The projects in which "poverty" lessons are cited are: Albania, Rural Development (FY95); Bulgaria, Agriculture Sector Adjustment (FY01); Moldova, Second Structural Adjustment (FY98); Poland, Agriculture Sector Adjustment (FY93). 19 4. Outcome DOES POLICY REFORM LEAD TO PRODUCTIVITY GROWTH? 4.1 The payoff to agricultural policy reform is better measured by agricultural productivity growth than by output growth or reduction in rural poverty. Output response is inappropriate because, under communism, many agricultural sectors were larger than a market economy would bear. Transition necessarily entailed shrinkage of the farm sector: in many countries output collapsed in the short run.19 In countries where output did not collapse this could reflect either some continuance (or legacy of) the old policies (negative); or a response to the new reforms (positive). Thus, output response is an ambiguous indicator. 4.2 Poverty is also a problematic measure because the data (for rural areas in particular) are patchy. Also, because rural households do not derive all their income from agriculture a reduction in poverty may not necessarily derive from farm growth- enhancing policy reform. Agricultural productivity measures are not without their problems either, but they are at least a more direct measure of response to policy reform. 4.3 Three productivity paths have been identified: Path I. A strong decline in gross agricultural output (GAO) coincides with a strong increase in agricultural labor productivity (ALP). Examples: Czech Republic, Hungary, Slovak Republic; Path II. A strong decline in GAO coincides with a strong decline in ALP. Examples: Azerbaijan, Belarus, Kazakhstan, Kyrgyz Republic, Moldova, Russia, Tajikistan, Ukraine; Path III. A strong increase in GAO coincides with a (slower) increase in ALP. Example: Albania (also China and Vietnam).20 4.4 In Central and Eastern Europe output declines were halted by the mid-1990s, stabilizing at 70 to 100 percent of the pre-reform level. But in the Commonwealth of Independent States, the fall in output continued to the late 1990s, with production steadying at around 60 percent of the level before the transition. 4.5 There is a positive association between policy reform and productivity (Table 8). The association is stronger for total factor productivity than for labor productivity, the former being a more complete measure and therefore a superior indicator. But the 19. Data series may exaggerate the scale of the collapse because, under socialism, there was an incentive for the private sector to under report and the state sector to over report. In Central and Eastern Europe it has been estimated that up to 15 percent of the crop output fall is a result of these biases (K. Macours and J.F.M. Swinnen, "Causes of output decline in economic transition: The case of Central and Eastern European agriculture", Journal of Comparative Economics, Vol. 28, March 2000, pp. 172-206. 20. K.Macours & J.F.M. Swinnen, "Patterns of agrarian transition", Economic Development and Cultural Change, 2002, pp. 365-394. 20 association is not strongly positive: many factors come into play in determining productivity change and multivariate analysis (beyond the scope of this paper) would be needed to measure the impact of policy change more precisely. Table 8. Association of Reform with Productivity Growth. Agricultural Labor Agriculture Total Factor Productivity Productivity Number of countries 23 15 Time period: Productivity data 1990-1999 1992-1997 Policy Reform Rating: FY1999 FY1997 Reference Year Median Change in Productivity 69.5% 2.4% Correlation coefficient 0.448 0.636 Note. The productivity data are from Z. Lerman, Y. Kislev, A. Kriss and D. Bitton, "Agricultural output and productivity in the former Soviet republics", Economic Development and Cultural Change (forthcoming). 4.6 The nature of the output and productivity response was influenced by Taxation Output only increased in countries where there was a significant reduction in the taxation of agriculture. Rights Clear individual property rights to land and other assets (rather than changes in farm scale) were essential for strong productivity growth. Exit Productivity was more likely to increase where opportunities and social benefits improved outside the farm sector, encouraging labor to leave agriculture.21 DOES THE POLICY ENVIRONMENT INFLUENCE PROJECT OUTCOMES? 4.7 Are OED project outcome ratings more likely to be satisfactory in countries that are reform leaders, compared to countries that are laggards? There is a significant positive association for non-rural projects but not for rural projects (Table 9). Taking the evaluation period as a whole (1991-02), rural projects in countries that rate high on the 1997 scale of agricultural policy reform did not have more satisfactory outcomes than those implemented in countries that rated low. But this may be a statistical artifact, reflecting the small size of the rural cohort and the lower variance (proportionately fewer projects implemented in laggard countries). 21. Based on the findings of Macours & Swinnen, 2002, op. cit., p. 387. 21 Table 9. Association Between Country Reform Rating and Project Outcome Rating 1991-02 exits Rural projects Non-rural projects Number of projects rated 25 171 % of projects rated satisfactory 88.0 81.3 % of projects implemented in "reform 72.0 59.6 leader" countries (1997 ECSSD rating)/1 Chi Square 1.729 12.915 Significance (NS) p=0.1885 p=0.0003 /1 The same rating is applied to rural and non-rural projects. This is because the ECSSD rating correlates closely (r=0.91 in both cases) with the Bank's Country Performance and Institutional Assessment Ratings and the Heritage Foundation's Economic Freedom Index; these last two ratings are not specific to the agriculture sector. 4.8 What is the ultimate impact of Bank assistance? The correlation between 1991-96 Bank commitments and the change in two measures of agricultural productivity is equivocal (Table 10).22 A more sophisticated analysis of this relationship would need to control for multiple intervening variables and lags in the disbursement of Bank loans and in the time needed to adopt improved techniques. Table 10. Association of Commitments with Productivity Growth Agricultural Labor Agriculture Total Factor Productivity Productivity Number of countries 23 15 Time period: Productivity data 1990-1999 1992-1997 Time period: Commitments 1991-96 1991-96 Median Change in Productivity -3.5% 2.4% Correlation coefficient -0.206 0.487 Note. The productivity data are from Z. Lerman, Y. Kislev, A. Kriss and D. Bitton, "Agricultural output and productivity in the former Soviet republics", Economic Development and Cultural Change (forthcoming). WHICH REFORM THEMES SHOW MOST PROGRESS? 4.9 Which reform themes have shown greatest progress, as measured by the outcome ratings of Bank projects? Put another way, how has the satisfactoriness of project outcome varied by reform theme? 4.10 Completed projects. Land administration and rural finance receive, respectively, the first and the second highest ratings (Table 11). 22. Here the data refer to the sum of all loans approved in 1991-96 mapped to the Rural Sector Board, plus 20 percent of the value of structural adjustment loans (not mapped to Rural) that include agriculture conditions. 22 Table 11. Rating of Completed Projects by Reform Theme N of Percent Percent projects Satisfactory Unsatisfactory Completed projects/1 75 68% 32% Land administration and reform 18 83% 17% Rural finance 17 71% 29% Trade liberalization and market development 18 61% 39% Agribusiness privatization 18 61% 39% Institutional reform: Ag services 4 50% 50% Note. Refers to sub-rating (specific to each theme), not overall project rating. /1 OED outcome rating. 4.11 Ongoing projects. Of the 26 investment operations, only 8 combine a satisfactory DO rating, well-designed performance indicators, and long enough time under implementation for results to be significant (Table 12). Between them these projects give the densest coverage to land policy and agriculture services. Table 12. Ongoing Investment Projects with Strong Evidence of Results/1 Country Project Themes Covered Name Trade Land Agribusiness Rural Institutional (Year Liberalization Administration Privatization Finance Reform: Approved) and Market and Reform Agriculture Development Services Albania Micro-credit X (FY99) Armenia Ag Reform X X X Support (FY98) Azerbaijan Farm X X Privatization (FY97) Azerbaijan Agricultural X X Development (FY99) Georgia Agriculture X X X (FY97) Georgia Agricultural X Research, Extension & Training (FY00) Kyrgyz Land & Real X Republic Estate Registration (FY00) Romania Ag Support X Services (FY00) /1 Satisfactory DO rating plus good performance indicators plus data on indicators showing progress. (FY01-02 approvals excluded on the assumption that not enough time has elapsed). 4.12 Progress on all the themes is much greater for the Central and Eastern European Countries than for the Commonwealth of Independent States. Between 1997 and 2001 23 the gap between these sub-regions narrowed on land policy but widened on institutional reform, the one area where CIS actually went backwards (Table 13). Table 13. The Gap Between CEE and CIS Countries ECSSD Trade and Land Admin. Agribusiness Rural Institutional Rating Market & Reform Privatization Finance Reform (Out of 10) Reform 1997 CEE 7.7 7.4 7.2 6.2 6.9 CIS 5.7 4.6 5.0 4.2 4.5 CIS/CEE (%) 74% 62% 69% 68% 65% 2001 CEE 8.2 8.2 8.5 7.5 8.3 CIS 6.0 5.7 5.5 4.7 4.2 CIS/CEE (%) 73% 70% 65% 63% 51% Source: Annex 2. 4.13 The sub-regions are closest on measures of trade liberalization. The 2002 ECSSD strategy notes that "Most ECA countries have liberalized prices, including reduction or elimination of distortionary taxes and subsidies, have de-regulated domestic marketing restrictions, including elimination of state orders that require farmers to sell a minimum quantity of certain commodities, and have reduced state participation in productive and commercial activities such as input supply and processing."23 Further progress is hampered by the volatility of world commodity prices--specifically, the absence of hedging mechanisms--and the forthcoming accession of some CEE countries to the European Union, raising farmer expectations of benefiting from the multiple subsidies offered to existing members under the Common Agricultural Policy. This is the theme where the scope for backtracking is greatest given that many of the reforms are stroke-of-the-pen and may easily be rescinded. 4.14 Of the five reform themes, this receives by far the thinnest treatment in terms of citation of lessons learned (Annex 6). While there is general agreement within the Bank that trade liberalization is a priority and that structural adjustment operations are an appropriate vehicle, there is much less certainty about the role that the Bank should play in market development. This was highlighted in recent discussions between OED and the Region on the rating of two wholesale market projects in Poland. There are certain positive externalities (better product standards, improved hygiene, reduced traffic congestion) so that a public goods argument could be made for Bank intervention; but many24 are not convinced by this argument. Returns were lower than anticipated because supermarkets and discount wholesalers moved in from Western Europe capturing much of the trade; the Bank identified this risk at appraisal 23. ECSSD, Rural Development Action Plan, August 21, 2002, p. 33. 24. Including most of the evaluators at an external workshop where one of the projects was used as a case study in OED rating methods. 24 but still proceeded. Reviewing its own wholesale market projects, the European Bank has also concluded that "in several instances demand has been overestimated."25 4.15 On land policy, the CIS region has been much slower to introduce individualized land rights and private farms. Because the CIS land allocation strategies have centered on distributing paper shares rather than physical plots the extent of individual cultivation is much lower than in the CEE. On average, 16 percent of agricultural land is cultivated individually in household plots and family farms across the CIS, compared with 63 percent across the CEE countries (up from 4 percent and 14 percent respectively before the transition began).26 4.16 This is one area where the long gestation period justifies an early start-- earlier than originally anticipated. In the reform sequence, the 1997 strategy places land ownership change in second place after trade liberalization; but the lessons contained in project documents, and staff comments, suggest that land reform should be broached at the start of the transition, parallel to price and trade reforms. (In many cases it was discussed early on but was not politically feasible). A frequently-cited lesson learned is that, when farms are restructured, the members of collectives should be given titles, rather than paper shares, to land and property, and allowed to dispose of these as they wish. Substantial support services are needed to make restructuring work and it is the relative absence of these in CIS which helps explain why the sub- region lags behind the CEE countries. 4.17 With respect to cadastre and land registration projects, it is widely recognized that effective implementation depends on a clear legal framework, a simple institutional structure with a single registry for rural and urban areas, long-term government commitment and open access to the information that is collected. There should be a gradual increase in cost recovery, involving fee-for-service charges. Costs should be contained by avoiding over investment in computerized technology and over specification of mapping and survey standards. 4.18 Agribusiness privatization has succeeded insofar as most state-owned processing firms and input suppliers have been sold off; but the great majority of them remain short of capital and are technically bankrupt, with outmoded technologies and poor quality products. Post-privatization support is critical and the 2002 strategy concedes that there is a shortage of replicable models to draw on. Although less so than for trade liberalization, there is still a certain thinness in the citation of lessons learned. This is one area where partnership with the International Finance Corporation would seem to make sense. But staff report that there is very little practical cooperation. The Bank is hampered by procurement and disbursement procedures that prove cumbersome and costly for private investors. This, in principle, would suggest a division of labor, with the IFC, rather than the Bank, doing all the lending for 25. European Bank for Reconstruction and Development, Special Study Review: The Bank's Wholesale Market Programme, 2002, p. 3. 26. Zvi Lerman, "Land Policies and Evolving Farm Structures in Transition Countries in Eastern Europe and Central Asia" (Paper presented at Annual Bank Conference on Development Economics, April 2002). 25 agribusiness. But staff--in both institutions--report that the IFC is more risk averse than the Bank, preferring to invest in other sectors. Thus, the scope for meaningful partnership is remarkably limited. The Bank does contribute by sponsoring analytic and project work intended to promote an enabling environment for business. Rather than sponsoring seed production, for example, it can make a bigger difference by adopting a systemic approach--examining why farmer demand for improved varieties often falls short of projections. But one staff member commented that the Bank could do much more to support simplification of procedures for businesses, such as registration of new firms and the enforcement of contracts. 4.19 The rural finance portfolio is much larger and shows better results. It is here that lesson learning has been most intense. In the first place, much has been learnt about the scope and limitations of credit-line operations: early experience (for example, in Albania) showed that lines of credit should be not be channeled through state institutions which, because of their softer budget constraint, are inclined to be less rigorous in appraising business plans. On-lending by a number of competing private intermediaries, without credit targeting, is normally what works best. A continuing problem is that commercial banks have a very limited branch network and little or no expertise in assessing the viability of farms. 4.20 The most innovative work in this area of reform is not the facilitating of on- lending but rather the easing of systemic constraints: creating the institutions for rural financial markets. This approach is reinforced by the Bank's work on farm restructuring and land titling--because the thinness of land markets and the lack of title has limited the scope for mortgaging rural land. In addition, important steps have been taken to improve the legal and regulatory framework bearing on the use of moveable property to secure transactions. The Bank's sponsorship of a secured transactions registry in Romania has met with considerable success and is a pioneering experience--the first time that an approach long advocated by the Bank has actually been acted on (the key to success was using the leverage provided by a structural adjustment loan). 4.21 In Albania and Moldova another approach has worked well. This involves substituting physical collateral by mutual guarantees, focusing on cohesive village communities where peer pressure can be relied upon to ensure loan repayment. Savings and credit associations require lengthy nurturing--intensive training is needed--but they can bear fruit even in remote and resource-poor regions and, in Albania, they have demonstrated their capacity to survive in the face of a nationwide crisis (the pyramid schemes) which bankrupted many lenders. However, projects have been less successful in getting these village lending institutions to mobilize savings. 4.22 It is still rather early to assess progress on institutional reform. As the ECSSD ratings indicate, here lies the greatest challenge. With respect to agriculture services, there were few projects in the first half of the evaluation period, consistent with the sequencing recommendations contained in the 1997 strategy. Most of the ongoing projects are too recent to show big results but if it is the case that "what gets measured gets done", prospects seem good: many of the projects with sound monitoring frameworks belong in this category (Table 12). The quality of the 26 monitoring reflects the tightness of the design. Earlier projects failed because they tried to build on the centralized, supply-driven model inherited from the Soviet era. The latest projects are demand-driven--recognizing that farmers will only make use of new technologies if the incentive environment is right and the technologies are economically viable. Competitive grant schemes are being used; these are an efficient, demand-driven way to generate new technologies, although they require intensive training and follow-up. Another area of institutional reform where there has been success is the development of water user associations; this was alluded to in the staff interviews but lies outside the project cohort selected for the evaluation. 27 5. Conclusions 5.1 With respect to the eight questions posed at the outset (paragraphs 1.5-1.7), this assessment proposes the following answers: Q1. Overall Bank assistance to ECA agriculture was high in relation to the attention to other sectors, in comparison with other regions, and in view of the substantial contraction of the sector that took place in most ECA countries after the transition. However, the level of assistance given was probably more than justified by the size of the structural adjustment required; strong support for agriculture policy reform facilitated the adjustment (paragraphs 2.2-2.5). Q2. Land policy and rural finance captured more Bank resources than the other three themes. These are also the areas where the Bank obtained the best project results. If more had been invested in the other areas maybe better results would have been forthcoming; but this is impossible to verify (paragraphs 2.7-2.8). Q3. Rather than concluding that particular instruments of Bank assistance were more effective than others, the assessment emphasizes the need to tackle policy reform with a battery of complementary instruments, and to abstain from doctrinaire approaches (paragraphs 3.2-3.24). Q4. Contrary to stated principles, the Bank's lending to ECA agriculture has not been closely aligned with the progress that countries have made on policy reform; but there are compelling justifications for this lack of alignment (paragraphs 3.25- 3.28). Q5. Despite the greater attention paid to rural poverty in recent strategy statements there is not yet evidence of improved project monitoring of the living standard indicators bearing on poverty; nor is there discussion of poverty issues in the "lessons learned" sections of appraisal and completion reports (paragraphs 3.31-3.32). Q6. The limited available data shows a moderately positive relation between the level of agriculture policy reform and agriculture productivity growth (paragraph 4.5). Q7. For agriculture at least, there is no association with a country's policy reform status and project outcome ratings; but statistical caveats apply (paragraph 4.7). Q8. There is substantial variation between the five reform themes in the progress made: trade liberalization is mainly complete; there has been much progress on land reform and rural finance; privatization follow-up and institutional development lag (paragraph 4.10-4.22). 5.2 In addition, the following specific findings were made: 28 · Between 1991-96 and 1997-02, ECA was the only region where there was an increase in rural commitments per capita per year (paragraph 2.4). · Between these periods, the number of "land" projects almost tripled (paragraph 2.8 ), and a high proportion of these projects had satisfactory outcomes (paragraph 4.10). · Analytic work grew rapidly in strength and depth--rapidly building from near zero in the early 1990s--and was made extensive use of outside the Bank; but there was more demand for it from CEE than CIS governments (paragraph 3.4). · Many ECSSD staff perceive one-tranche adjustment operations to be more effective but the data show that they are not more likely to lead to satisfactory outcome ratings than multi-tranche operations (paragraph 3.18). · Contrary to staff perceptions, stand-alone projects (covering only one reform theme) have not been associated with a higher frequency of satisfactory outcome ratings than projects covering several themes (paragraph 3.19). · Pilot projects have been successfully scaled up (paragraph 3.22). · Strategically-located policy analysis units have helped to champion reform (paragraph 3.24); but the Bank's institutional development funding mechanism no longer provides ready support for these units (Box 2). · On sequencing, because institutions take so long to nurture (paragraph 3.20) experience suggests that institutional reform needed to be tackled at the beginning of the transition; and land reform should also have been broached at the start, parallel to price and trade reforms (paragraph 4.16). · Agribusiness privatization has not been served by an effective partnership between the Bank and IFC (paragraph 4.18) · There has been considerable innovation in approaches to rural finance, including successful experiments with village-based group lending and easing of systemic constraints (paragraphs 4.19-4.21) · Overall, project monitoring was better in 1997-02 than in 1991-96 (paragraph 3.31 ); since 1997 particularly good monitoring has been developed for agriculture service projects (paragraph 4.22); but, as noted above, monitoring of poverty indicators has been neglected. 29 Annex 1 Annex 1. Method 1. Limitations of the Approach Taken. In the interests of manageability and owing to resource constraints, the assessment concentrates on the agriculture sector, narrowly defined; it does not cover rural infrastructure (including irrigation) or natural resource management. Other sectors are covered in a larger OED study that is now underway.27 The assessment is to a large extent a meta-analysis--a desk-based synthesis of previous evaluative reports that adds value by presenting a big picture, a whole that is intended to be larger than the sum of its parts. This is supplemented by interviews with the staff who are (or have been) most directly engaged with ECA agriculture and sector policy reform. Owing to the limited budget for the study it was not possible to interview development partners or country counterparts, or to survey beneficiaries. This potentially introduces a bias: staff comment may be self-serving. On the other hand, staff are generally better informed than others about what the Bank is doing; and the purpose of the exercise was as much to capture differences in staff perceptions of the Bank's effectiveness as it was to extract an aggregate assessment. 2. Evaluation Template. The actions of the Bank in support of agriculture policy reform are assessed in relation (a) to the Bank's own strategy statements and (b) the Bank's assessment of reform progress. With respect to (a), there are formal statements of the Bank's rural strategy for ECA, issued in 1997 and 2002 (these are summarized in Annex 3). With respect to (b), in 1997 the Bank unit responsible for agriculture in the Europe and Central Asia Region (ECSSD) began to rate progress in the transition economies on five dimensions of agriculture policy: · Trade liberalization and market development · Land administration and reform · Privatization of agro-processing and input supply · Rural finance · Institutional reform. 3. There is now a series of ratings (henceforth, the "ECSSD ratings") on each of these policy dimensions for the five years, 1997-2001 (Annex 2). 4. Challenges in Rating the Projects. There are three major issues. First, a single project may address multiple reform themes (particularly in the case of adjustment operations). The overall outcome rating may conceal substantial variation in the progress made on the various themes that a project covers. The projects were sorted by the five ECSSD reform themes (Annex 5). All the projects corresponding to a given theme were rated for outcome on that theme, as inferred from a close reading of evaluation and supervision reports. Aggregating these sub-ratings, it was estimated that 65 percent of completed projects have a satisfactory outcome rating; this compares to an OED's overall outcome rating of 74 percent satisfactory. However, the difference between sub-rating and overall rating is not statistically significant. On 27. "Evaluation of World Bank Assistance to the Transition Economies" (Web address: http://www.worldbank.org/OED/transitioneconomies/index.html) 30 Annex 1 the other hand, for ongoing projects there is a moderately significant difference between the overall (Development Objective) rating and the rating for theme (inferred from component ratings) (Table 1). Table 1. Contingency Test for Significant Difference: Overall vs. Sub-Rating Completed projects Ongoing projects (1) Number of projects 75 51 (2) Share of satisfactory 74% 98% projects, overall rating (3) Share of satisfactory 65% 88% projects, sub-rating/1 (4) Difference between (4) and X2=1.565 X2=3.835 (5) (5) Significance level NS P=0.05 /1 Sub-rating is specific to one of the five reform themes. NS =Not significant 5. A second problem is the need to adjust for the lack of equivalency between the ratings for completed projects (the OED outcome rating) and for ongoing projects (the Development Outcome rating). Supervision reports are more likely to accentuate the positive than evaluation reports. In this cohort, 98 percent of the DO ratings for ongoing projects were satisfactory compared to the 74 percent of the outcome ratings for completed projects. Because, from the evaluation standpoint, the DO rating is a baser currency than the outcome rating, when reporting on progress by theme it is important not to lump together ongoing and completed projects. 6. The challenge is to get as much evaluative input from the ongoing projects without compromising the overall integrity of the findings. One way to do this is to define ongoing projects as satisfactory only when, in addition to a satisfactory DO rating, they boast sound monitoring results (as reflected in the annex on Key Performance Indicators). But this expedient only works for investment operations. This is the third problem: adjustment operations are still not required to apply performance indicators, a failing that was highlighted by OED in 1997.28 Subtracting adjustment operations reduces the cohort of ongoing operations from 29 to 26. Data Sources Used in this Assessment (a) ECSSD Ratings (see Annex 2). The original framework is presented in Csaba Csaki and John Nash, "The Agrarian Economies of Central and Eastern Europe and the Commonwealth of Independent States: Situation and Perspectives, 1997", World Bank Discussion Paper, No. 387, 1998. 28. This failing was to be addressed by revising Operational Directive 8.60; but the revision is still pending (OED, Reforming Agriculture: The World Bank Goes to Market, Washington, DC: World Bank, 1997) 31 Annex 1 (b) ECA Agriculture Strategy (see Annex 3). The main references are: Rural Development: From Vision to Action, 1997; and Reaching the Rural Poor: An Updated Strategy for Rural Development, 2002 (c) Lending and Bank Administrative Spending. These figures are based on what is currently available in the Business Warehouse, plus an earlier set of tabulations that Corporate Resource Management prepared for the OED Rural Development study. (d) Project Cohort (see Annex 4). A cohort of 65 projects was selected, 35 completed and 30 ongoing . The cohort includes a subset of ECSSD projects bearing most directly on agriculture policy: environment, infrastructure (including irrigation) and social development projects are excluded; the definition of operations corresponding to institutional reform is confined to agriculture service projects. The cohort also includes 10 structural adjustment operations containing agriculture policy conditions. In addition, Annex 3 contains a complete list of dropped projects. (e) Project Ratings (see Annex 5). In addition to reporting the overall project outcome (or, in the case of ongoing projects, the Development Objective rating), the evaluation ascribes a rating to those aspects of each project bearing on the five reform dimensions, based on an inspection of project status reports, implementation completion reports, and OED reports. This same annex includes an assessment of the quality of performance indicator monitoring, based on the relevant sections of project status reports and completion reports (see, also, Annex 7 for rating of poverty indicators). (f) Lessons Learned (see Annex 6). For each project in the cohort, the lessons learned--as cited in appraisal, completion and evaluation reports--were extracted. These were sorted into generic lessons and lessons specific to the reform themes. (g) Country Strategy and Analysis. As well as examining the contribution of projects, the evaluation reviewed the extent to which sector reports and country assistance strategies covered the five reform dimensions. (h) Agricultural Productivity. The data are taken from Z. Lerman, Y. Kislev, A. Kriss and D. Bitton, "Agricultural output and productivity in the former Soviet republics", Economic Development and Cultural Change (forthcoming). The issue is also addressed in: K.Macours & J.F.M. Swinnen, "Patterns of agrarian transition", Economic Development and Cultural Change, 2002, pp. 365-394. (i) Literature Review (OED background paper. Photocopy). Zvi Lerman, "Policy- Based Lending and Agricultural Policy Reform in ECA", July 2002. (j) Staff Interviews. Interviews were conducted with 28 persons, mainly current or former ECSSD staff. Based on their experience in ECA--concentrating on the countries they have worked on--staff were asked to comment on progress made on the reform dimensions, and to assess the effectiveness of the various instruments, strategies and procedures that the Bank has deployed in support of agriculture policy reform. They were also invited to accept or reject a set of hypotheses bearing on Bank strategy (Instrument attached below). 32 Annex 1 ECA Hypotheses Please indicate whether your specific experience in ECA would lead you to accept or reject the following hypotheses. Place an "X" next to Accept or Reject. Feel free to expand on the reasons for your choice if you wish: Hypothesis 1. At the beginning of the transition, rather than trying to reduce rural poverty through specific interventions it is more important to take measures to raise the productivity of agriculture. Accept Reject Comment Hypothesis 2. It is generally unwise to make large investment loans before a satisfactory macro and sector policy framework has been established Accept Reject Comment Hypothesis 3. In countries where the policy framework is poor, policy-based lending (SALs, SECALs, loans with substantial conditionality) is necessary in order to achieve reform Accept Reject Comment Hypothesis 4. In the absence of a substantial lending program it is difficult to bring about institutional reform Accept Reject Comment Hypothesis 5. Single-sector (agriculture) projects are more likely to achieve their development objectives than multi-sector (rural development) projects Accept Reject Comment Thanks for your cooperation. 33 Annex 2 Annex 2. Status of Agricultural Reforms in CEE and CIS Countries at end of 2001 1 = Centrally Planned Economy 10 = Completed Market Reforms * Country Price & Market Land Reform Agro-processing Rural finance Institutional Total Score 01 00 99 98 97 01 00 99 98 97 01 00 99 98 97 01 00 99 98 97 01 00 99 98 97 01 00 99 98 97 Czech Rep 9 8 9 9 9 9 9 8 8 8 10 10 9 9 8 9 9 9 9 8 9 9 8 8 8 9.2 9.0 8.6 8.6 8.2 Hungary 9 8 8 9 9 9 9 9 9 9 10 10 10 10 9 9 9 9 9 8 9 9 8 8 8 9.2 9.0 8.8 9.0 8.6 Slovenia 9 9 9 9 8 9 9 9 9 9 10 10 8 8 8 8 8 7 7 8 10 8 8 8 9 9.2 8.8 8.2 8.2 8.4 Estonia 9 9 9 9 10 9 9 8 6 6 9 9 8 8 7 9 9 8 8 7 9 9 9 9 9 9.0 9.0 8.4 8.0 7.8 Latvia 9 9 9 8 7 9 9 9 9 9 9 9 8 8 7 9 9 8 8 7 9 9 8 8 8 9.0 9.0 8.4 8.2 7.6 Slovak Rep 8 8 7 7 7 8 8 8 8 7 9 9 8 8 8 8 8 8 8 8 8 7 7 7 7 8.2 8.0 7.6 7.6 7.4 Bulgaria 9 9 9 8 6 8 8 8 8 7 8 8 8 7 5 7 7 6 5 4 8 8 7 6 5 8.0 8.0 7.6 6.8 5.4 Poland 8 8 7 8 9 8 9 8 8 8 9 9 9 8 7 7 7 7 7 6 8 8 8 8 8 8.0 8.2 7.8 7.8 7.6 Lithuania 7 8 8 8 7 9 9 8 8 8 8 8 8 8 7 7 7 6 6 6 9 8 8 8 7 8.0 8.0 7.6 7.6 7.0 Albania 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 7 6 5 3 3 7 6 5 5 5 7.6 7.2 6.8 6.4 6.4 Armenia 8 8 7 7 7 8 8 8 8 8 8 8 7 7 7 7 7 7 7 7 6 6 7 8 8 7.4 7.4 7.2 7.4 7.4 Romania 7 7 7 6 7 8 8 8 7 7 8 8 7 6 6 7 6 6 5 6 7 6 5 5 4 7.4 7.0 6.6 5.8 6.0 Croatia 7 7 6 6 6 7 7 6 6 5 7 7 7 7 6 6 6 6 6 6 8 8 8 8 6 7.0 7.0 6.6 6.6 5.8 Azerbaijan 8 8 7 7 6 8 8 8 8 6 6 6 6 5 5 6 5 5 5 4 5 5 5 5 4 6.6 6.4 6.2 6.0 5.0 Georgia 9 8 8 7 7 7 7 6 6 7 5 5 5 5 5 7 6 6 6 6 5 5 5 5 6 6.6 6.2 6.0 5.8 6.2 Macedonia 8 8 8 8 7 7 7 7 7 7 6 6 7 7 8 5 5 4 4 4 7 7 7 7 6 6.6 6.6 6.6 6.6 6.4 Bosnia HG 7 6 6 6 - 7 6 6 6 - 6 6 6 6 - 6 6 6 6 - 5 5 5 5 - 6.2 5.8 5.8 5.8 - Kyrgyz Rep 7 7 7 7 6 7 7 7 7 6 6 6 6 6 6 6 6 6 7 6 5 5 6 6 5 6.2 6.2 6.4 6.6 5.8 Moldova 7 7 7 8 7 7 8 7 7 6 6 6 6 7 7 6 6 5 5 5 4 5 5 5 4 6.0 6.4 6.0 6.4 5.8 Ukraine 7 6 6 6 7 6 6 6 5 5 7 7 6 6 7 6 6 5 5 5 4 4 4 4 3 6.0 5.8 5.4 5.2 5.4 Kazakhstan 6 6 6 5 7 5 5 5 5 5 7 6 6 6 7 6 6 6 6 5 5 5 5 5 5 5.8 5.6 5.6 5.4 5.8 Russia 6 6 6 6 7 5 5 5 5 5 8 7 7 7 7 5 5 5 5 6 5 5 5 5 5 5.8 5.6 5.6 5.6 6.0 Tajikistan 6 6 5 5 4 6 6 5 5 2 5 5 5 4 5 3 2 2 2 3 4 4 4 4 5 4.8 4.6 4.2 4.0 3.8 Yugoslavia 6 3 - - - 5 5 - - - 5 3 - - - 3 3 - - - 5 4 - - - 4.8 3.6 - - - Uzbekistan 4 3 3 3 4 4 4 2 2 1 4 3 1 1 1 2 2 1 1 1 3 3 3 3 4 3.4 3.0 2.0 2.0 2.2 Turkmenistan 2 2 2 2 2 3 3 3 3 2 2 1 1 1 1 1 1 1 1 1 2 2 3 3 3 2.0 1.8 2.0 2.0 1.8 Belarus 2 2 2 2 3 2 2 2 1 1 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1.8 1.8 1.8 1.6 1.8 AVERAGE 7 7 7 6 6 7 7 6 6 6 7 7 6 6 6 6 6 5 5 5 6 6 6 6 5 6.7 6.5 6.1 6.0 5.5 Source: ECSSD staff estimates. The original source was Csaba Csaki and John Nash, "The Agrarian Economies of Central and Eastern Europe and the Commonwealth of Independent States: Situation and Perspectives, 1997", World Bank Discussion Paper, No. 387, 1998. 34 Annex 2 Key to Numerical Ratings Used in Table 1 Market Conforming Land Reform Privatization of Rural Financial Institutional Policy Environment Agroprocessing Systems Framework and Input Supply 1-2. Direct state 1-2. System 1-2. Monopolistic 1-2. Soviet type 1-2. Institutions of control of prices and dominated by state owned system, with command markets. large-scale farms. industries. "Agrobank" as the economy. sole financing channel. 3-4. Deregulation with 3-4. Legal 3-4. Spontaneous 3-4. New banking 3-4. Modest indicative prices, and framework for land privatization and regulations are restructuring of price controls; privatization and mass privatization introduced; little or government and significant NTB on farm restructuring in design of early no commercial public institutions. imports or exports. in place, implementation banking. implementation stage. launched only recently 5-6. Mainly 5-6. Advanced 5-6. 5-6. Restructuring 5-6. Partly liberalized markets stage of land Implementation of of existing banking restructured constrained by the privatization, but privatization system, emergence governmental and absence of large-scale farm programs in of commercial local institutions. competition and some restructuring is not progress. banks. remaining controls on fully complete. trade policy. 7-8. All command 7-8. Most land 7-8. Majority of 7-8. Emergence of 7-8. Government economic type privatized, but industries financial institutions structure has been interventions are titling is not privatized with a serving agriculture. refocused while removed. Market and finished and land framework research, trade policies are in market not fully conducive for extensions, and compliance with functioning. foreign direct education is being WTO, however, investment. reorganized. domestic markets are not fully developed. 9-10. Competitive 9-10. Farming 9-10. Privatized 9-10. Efficient 9-10. Efficient markets with market structure based on agro-industries and financial system for public institutions conforming trade and private ownership input supply, with agriculture, agro- focused on the agricultural policies, and active land improved industries, and needs of private and no more than markets. international services. agriculture. modest protection. competitiveness. Source: ECSSD staff estimates. The original source was Csaba Csaki and John Nash, "The Agrarian Economies of Central and Eastern Europe and the Commonwealth of Independent States: Situation and Perspectives, 1997", World Bank Discussion Paper, No. 387, 1998. 35 Annex 3 Annex 3. ECA Strategy for Agriculture 1. According to various reports by Bank staff produced shortly after the transition began,29 reform of the agricultural sector in ECA called for a combination of macroeconomic stabilization and structural reform. Macroeconomic stabilization hinged mainly on fiscal adjustment, entailing cuts in government financing of enterprises, either through subsidies or through the banking system. This imperative had to be balanced against the possibility that if spending cuts were too deep absolute poverty would increase, and investment in critical infrastructure would collapse. In addition to stabilization, wide-ranging structural reforms of the agriculture sector were called for. These reforms entailed the break-up of monopolies, permitting the entry of new firms and economic diversification, based on the creation of viable factor and product markets, physical marketing infrastructure and market information systems, and an appropriate legal and regulatory framework. 2. The ECA strategy for tackling the structural aspects of agricultural policy reform has been grouped under the following five headings30 (i) Liberalization of prices and markets. Before the transition governments supported agriculture by maintaining favorable terms of trade and an ad hoc system of differentiated producer prices. Food prices were heavily subsidized, leading to a high level of consumption compared to economies of similar GDP per capita in other regions. Among the reforms needed were · alignment of domestic prices with world prices · elimination of government procurement (state orders) · substitution of quotas by tariffs (ii) Privatization of land and reorganization of large farm units. Under communism, farm structure and agricultural support systems were geared almost entirely to large-scale state and collective farms, where production costs were high. Property rights were poorly defined, and management practices precluded effective linking of remuneration with productivity. Reforms involved · establishing rights to own, buy, sell and mortgage land · creating the land registries and cadastres to underpin operation of land markets · allocating land and physical assets of state and collective farms among workers · revising legislation to permit the formation of voluntary, commercial cooperatives 29. Including, Karen Brooks, J. Luis Guasch, Avishay Braverman & Csaba Csaki, "Agriculture and the transition to the market", Journal of Economic Perspectives, Fall 1991; and Csaba Csaki, "Transformation of Agriculture in Central Eastern Europe and the Former USSR", World Bank Policy Research Working Papers, WPS No. 888, April 1992. 30. Csaba Csaki & John Nash, The Agrarian Economies of Central and Eastern Europe and the Commonwealth of Independent States: Situation and Perspectives, 1997, World Bank Discussion Paper No. 387, 1998. 36 Annex 3 · transferring social services from state and collective farms to local governments (iii) Privatization of the food industry and of trade in farm produce and capital inputs. In the command economies, monopolies predominated in input manufacturing and supply and in food procurement, processing and distribution. This led to high transport, processing and marketing margins. The food system was not consumer responsive: neither packaging, nor product quality, variety and availability were based on buyer preference. Also, farmers' preferences were largely ignored by input manufacturers and distributors. Reforms entailed · Privatization of food processing and marketing, input supply and trucking · Scaling back of licensing requirements and other impediments to new firm entry · Regulations to head off re-monopolization · Reduced interference by local government in food markets (iv) Rural finance. Transition countries inherited a credit and banking system in dire straits. Credit discipline in agriculture was extremely low because for decades the government had used the credit system to channel subsidies to inefficient farms and allowed the banking system to regularly write off outstanding farm debts. Because of its remoteness, the rural economy posed special problems for the development of a diversified banking system. The reform agenda included · Encouraging the development of new rural banking institutions · Making deposit and lending interest rates positive in real terms · Sponsoring institutional development and training · Establishing procedures for prudential regulation of rural financial institutions · Strengthening the framework for secured transactions (v) Institutional reform. The Ministry of Agriculture and affiliated agencies were poorly equipped to help enhance the competitiveness of agriculture. There was limited awareness of the desirable role of the state. Underpaid and unmotivated staff strived to supplement their incomes by corrupt dealings. The institutional framework was extremely centralized. Reform focused on: · Making research and extension more responsive to user needs · Encouraging the financial autonomy of water user associations 3. A summary statement of regional strategy is contained in annexes to the Bank's corporate rural strategy of 1997 and 2002.31 Between these dates the strategy remained broadly the same. But the 2002 strategy is different in the following ways 4. The principle that investment lending should be conditional on previous policy reform has been qualified. On lending, the 1997 strategy states that "A minimum amount of reform in the macroeconomic and agricultural policy framework is 31. Rural Development: From Vision to Action, 1997; Reaching the Rural Poor: An Updated Strategy for Rural Development, 2002 37 Annex 3 required before there can be any intervention in the sector". In 2002, the position is more nuanced: "where the economic rates of return for an investment are good, where the investment would help substantial numbers of rural poor, where delays would further increase the costs of that investment, and where the rural sector is "net-taxed", our strategy suggests that we support modest investments". 5. Greater political realism. The 2002 strategy emphasizes that "Our strategy also takes into account political economy and social well-being and acknowledges that some reforms may have to be crafted differently or go more slowly than pure efficiency considerations would dictate." 6. More attention to risk. In 2002, it is acknowledged that "Our borrowers have legitimate concerns that the move to a market economy from the previous socialist system brings considerable risks and uncertainty that require specific attention, particularly when many of the risk mitigation instruments of market economies are not yet available". 7. Major focus on institutional development. The 2002 strategy states that "We will seek to emphasize institutional development, capacity building, the involvement of local participation and sound governance in all our work" 8. Realistic expectations for partnerships. In 1997 there was a pledge to strengthen partnerships. In response to the "inadequate cooperation and liaison with key partner organizations, particularly with the EU", attempts will be made to strengthen links, including "inviting other donors and the EU to participate in Bank missions". In 2002, the strategy again underscores the value of partnership but is more candid about the prospects: "Partnerships, even with IFC, are not cost-free or easy". 9. Adjustment lending to be married to investment. In 2002, the strategy argues that "We gain more commitment from stakeholders when reforms are accompanied by investment...adjustment lending needs to be complemented by investments to achieve lasting impact and to nurture the kind of local involvement, development of local institutions and capacity building that we believe are critical". Adjustment and investment can run in parallel, rather than the one necessarily preceding the other. This is a departure from the following sequencing envisaged in 1997. Sequence of reform areas: (1) Price liberalizations and trade reform (2) Ownership change (land and rural assets) (3) Demonopolization of marketing services (4) Financial reforms. Sequence of operational initiatives: (1) Adjustment operations (2) Long lead-time projects (e.g. ag research) (3) Cadastre and technical assistance (4) Public goods (e.g. irrigation , research and extension) 38 Annex 3 (5) Credit lines for the emerging sector.32 10. Combine investment lending and policy reform. In a related departure, the 2002 strategy states: "We will emphasize project lending that combines investment with support for sub-sector policy reform [including]...support to decentralization of water management and water users' associations. 11. Make sector work more efficient. The 2002 strategy says: "We will aim to complete our ESW tasks more promptly, to have them carried out, to the extent possible, by the person who is responsible for lending in a particular country, and to have local ownership, to improve effectiveness". 12. Finally, the 2002 strategy pays more explicit attention to poverty reduction. This is at variance with the following statements in the 1997 strategy: Rural poverty "is not the key issue in ECA countries"; "Equity concerns will not be a primary justification for intervening in the sector in most cases". 32. World Bank, Rural Development: From Vision to Action, 1997, p. 127 (Appendix Figure B1). 39 Annex 4 Annex 4. Project Cohort Country Project Project Name Where Operations Fall in the Evaluation Period (FY91-02), ID 91 92 93 94 95 96 97 98 99 00 01 02 Hungary Agro-processing Modernization Hungary Integrated Ag Export Poland Agro-industries Export Development Poland P008564 Ag Development Romania P008772 Private Farmer & Enterprise Support Albania P008264 Rural Poverty Alleviation Pilot Albania P008256 Ag Sector Adjustment Hungary P008482 Product Market Development Poland P008583 Ag Sector Adjustment Bulgaria P008309 Ag Development Latvia P008527 Ag Development Russia P008811 Ag Reform Implementation Support Russia P034579 Land Reform Implementation Support Albania P008273 Rural Development Kazakhstan P008502 SAL Kyrgyz Rep P008516 Ag Privatization & Enterprise Adjustment Moldova P008554 SAL Ukraine P009117 Ag Seed Developmen Uzbekistan P009122 Cotton Sub-sector Albania P035787 Agro-processing Development Croatia P008335 Farmer Support Services Estonia P008403 Agriculture Kyrgyz Rep P008513 Sheep & Wool Improvement Lithuania P008538 Private Ag Development Macedonia P010001 Private Farmer Support Moldova P008556 Agriculture Azerbaijan P040544 Farm Privatization Georgia P008415 Agriculture Kazakhstan P046044 Real Property Registration Kyrgyz Rep P008520 Rural Finance 1 Lithuania P044056 SAL Romania P008795 Ag Sector Adjustment Tajikistan P008863 Ag Recovery & Social Protection 40 Annex 4 Country Project Project Name Where Operations Fall in the Evaluation Period (FY91-02), ID 91 92 93 94 95 96 97 98 99 00 01 02 Ukraine P009113 Ag Sector Adjustment Armenia P035806 Ag Reform Support Kazakhstan P008503 Agricultural Post Privatization Assistance Kyrgyz Rep P040721 Ag Support Services Macedonia P038492 SAL Moldova P0044147 SAL 2 Moldova P035781 Rural Finance Romania P034213 General Cadastre Albania P055160 SAC Albania P051310 Micro-credit Armenia P057560 Title Registration Azerbaijan P035813 Ag Development Bulgaria P057925 Ag Sector Adjustment 1 Georgia P052153 SAC 3 Kyrgyz Rep P064585 Rural Finance 2 Latvia P044804 Rural Development Poland P008616 Wholesale Markets 1 Poland P055988 Wholesale Markets 2 Slovenia P055304 Real Estate Registration Tajikistan P049718 Farm Privatization Support Tajikistan P047438 SAC Georgia P065715 Ag Research, Extension & Training Kyrgyz Rep P049719 Land Registration Romania P043882 Ag Support Services Albania P054736 Ag Services Development Bulgaria P057926 Ag Sector Adjustment 2 Bulgaria P055021 Registration and Cadastre Lithuania P068706 SAL 2 Moldova P056891 Rural Investment & Services Moldova P060434 SAC 3 Romania P057960 Rural Finance Uzbekistan P046043 Rural Enterprise Support 41 Annex 4 ECSSD Dropped Projects Country Project ID Title Year Dropped Total Spent US$'000 Albania P056444 Rural Finance 1999 56.6 Albania P019041 N District Rural Development 20002 75.4 Belarus P008296 Agriculture 1993 29.2 Belarus P008301 Pilot Agriculture 1998 12.4 Bulgaria P008304 Biodiversity Conservation 1995 48.1 Bulgaria P008311 Irrigation Restructuring 2000 1087.6 Croatia P008331 Ag Sector Adjustment 1999 181.58 Czecl Republic P037903 Biodiversity 2 1999 11.2 Hungary P048391 Ag. Standards 1999 60.7 Hungary P008490 Ag. Modernization 1995 258.3 Kazakhstan P040719 Rural Finance 1999 2.6 Kyrgyz Republic P035789 Rural Poverty 1997 459.5 Kyrgyz Republic P050189 People's Initiative 1999 315.6 Latvia P064594 Rural Poverty Alleviation 2001 1.2 Macedonia, for P038492 Ag. Sector Adjustment 1998 0.4 Macedonia, for P044608 Ag. Guarantee 1999 19.1 Maldova P039928 Water Resources Development. 1998 226.9 Maldova P040559 Agriculture 2 1999 1.6 Maldova P060658 Rural Finance 2 2001 27.3 Poland P008591 State Farm Privatization 1995 107.0 Poland P038140 State Farm Privatization 1996 166.8 Poland P055994 Wroklaw Wholesale Market 1998 3.4 Poland P050659 N. Poland Development 1998 162.0 Poland P044451 Warehouse Receipts 1999 119.3 Poland P056947 Off-Farm Rural Employment 1999 34.9 Poland P073110 Ag. Competitiveness 2002 153.0 Romania P008790 Grain Markets and Exports 2000 1108.9 Romania P043432 Agriculture Guarantee 2001 209.2 Russian Fed. P008813 Ag. Sector Adjustment 1995 20.8 Russian Fed. P008812 Rural Credit 1996 16.0 Russian Fed. P008819 Regional Agriculture 1996 1879.4 Russian Fed. P036972 Regional Agriculture 2 1996 8.1 Russian Fed. P040722 Land Reform Implementation 2 1996 0.2 Russian Fed. P035819 Forestry 1996 7.5 Russian Fed. P008841 Ag. Research 1996 106.4 Russian Fed. P043433 Input Guarantee 1999 22.2 Russian Fed. P038965 Ag. Sector Adjustment 1999 461.8 Russian Fed. P058342 Nishny Pilot Technical Asst. 2000 0.4 Russian Fed. P062485 Ag Sector Adjustment 2000 77.4 Russian Fed. P060045 Forestry 2001 289.3 Slovak Republic P036059 Land Market Development 1997 19.3 Slovak Republic P008845 Forestry 1998 472.8 Slovenia P045026 Small-Scale Irrigation 1999 144.2 Tajikistan P008859 Ag Sector Adjustment 1998 0.6 Tajikistan P008861 Infrastructure Rehabilitation 1999 631.6 Turkmenistan P040724 Ag. Privatization 1996 11.0 Turkmenistan P049720 Ag. Development 2001 147.9 Turkmenistan P008866 Karakum Canal 2002 178.1 Turkmenistan P051336 Crop Protec. & Animal Health 2002 241.2 Ukraine P044813 Agribusiness Development 2000 200.9 Ukraine P0056847 Rural Finance Pilot 2002 302.4 Uzbekistan P009124 Ag Sector Adjustment 1993 13.7 Uzbekistan P035790 Ag Extensión & Research 2 1998 669.9 Uzbekistan P049722 Ag Sector Adjustment 1999 55.1 Uzbekistan P043077 Ag. Enterprise Restructuring. 2 2001 10.7 Yugoslavia for P009294 Private Ag. Sector Development 1995 109.5 42 Annex 5 Annex 5. Rating the Project Cohort Note on Table Columns Costs. The cost refers to the project component bearing on a particular reform (rather than project as a whole). In the case of adjustment operations where there is no breakdown of costs by component, the figure cited represents 20 percent of the total cost of the operation (indicated by "*") Outcome Rating. For completed projects, the outcome cited is the most recent (whether EVM, ES or PPAR). For ongoing projects, the rating refers respectively to the last and the current Development Objectives rating cited in the most recent Project Status Report ("P plus the sequence number of the report"). Sub-Rating. Refers to the rating for the reform component. In the case of completed projects, the sub-rating is inferred from the description of the achievement of project objectives given in the ICR. For on-going projects, it refers to the component rating bearing on the reform in question. Key Performance Indicators. Refers to the Indicators section of Project Status Report or ICR. Monitoring is characterized as "Good" when there is a full specification of targets and the "Actual" column contains data. "Poor" can mean either that indicators are non-existent or incomplete or that there is no data. (Projects not approved before FY01 are rated "good" if indicators and targets are specified, even if no actual data is yet cited: on the assumption that it may be too early for data to have been collected). Adjustment operations are uniformly rated "NA" because Bank operational guidelines do not require indicators to be prepared for adjustment operations. Lessons. Refers to whether or not lessons specific to the reform theme in question are cited in appraisal or evaluation reports. Note on presentation and acronyms Adjustment operations are in bold font. * A notional 20 percent of the total project cost is assigned to cover this theme. P Project Status (Sequence Number) HS Highly Satisfactory S Satisfactory MS Moderately Satisfactory MU Moderately Unsatisfactory U Unsatisfactory HU Highly Unsatisfactory NA Not Applicable 43 Annex 5 Trade Liberalization and Market Development Country Project Name Stand Cost Overall Sub- Key Performance Specific (Approval year) Alone? US$m Rating Rating Indicators Lessons HUN Agro-processing No NA MU MU Poor No Modernization (FY89) HUN Integrated Ag Export No NA S S Good No (FY90) ALB Ag Sector Adjustment No 9.0* MS S NA No (FY93) HUN Product Market Yes 88.3 S NA Poor No Development (FY93) POL Ag Sector Adjustment No 60.0* S U NA No (FY93) RUS Ag Reform No 50.1 MS S Poor No Implementation Support (FY94) KYR Ag Privatization & No 8.4* S S NA No Enterprise Adjustment (FY95) MOL SAL No 13.1* MU U NA No (FY95) UZB Cotton Sub-sector No 27.0 P20 U/S Poor None Improvement S/S (FY95) LIT SAL No 16.0* S S NA No (FY97) ROM Ag Sector Adjustment No 60.0* MU U NA No (FY97) TAJ Ag Recovery & Social No 10.0* MU MU NA No Protection (FY97) UKR Ag Sector Adjustment No 60.0* MS U NA No (FY97) MAC SAL No 12.6* S S NA No (FY98) MOL SAL 2 No 14.0* MS S NA No (FY98) BUL Ag Sector Adjustment No 15.2* S S NA No 1 (FY99) POL Wholesale Markets 1 Yes 26.0 U NA Poor Yes (FY99) POL Wholesale Markets 2 Yes 22.2 MS NA Poor Yes (FY99) ALB Ag Services No 4.1 P04 S/S Good No Development S/S (FY01) BUL Ag Sector Adjustment No 10.0* MS MS NA No 2 (FY01) LIT SAL 2 No 20.0* P04 U/U NA No (FY01) S/S MOL SAC 3 No 6.0* P02 S/S NA No (FY02) S/S 44 Annex 5 Land Administration and Reform Country Project (Approval Stand Cost Overall Sub- Key Specific FY) Alone? (US$m Rating/2 rating Performance Lessons /1 /1 Indicators/3 Cited?/4 Hungary Integrated Agricultural No 7.0 S MS Good No Export (FY90) Albania Ag Sector No 9.0 S MS NA Yes Adjustment (FY93) Poland Ag Sector No 60.0 S MS NA No Adjustment (FY93) Latvia Agricultural No 2.5 HS HS Good No Development (FY94) Russia Land Reform Yes 115.0 P21 S/S NA Poor Yes Implementation Support (FY94) Kazakhstan SAL No 36.0 S S NA Yes (FY95) Kyrgyz Ag Privatization & No 8.4 S MS NA Yes Republic Enterprise Adjustment (FY95) Moldova SAL No 13.1* MU U NA No (FY95) Estonia Agriculture No 5.5 P16 S/HS Poor No (FY96) S/HS Azerbaijan Farm Privatization No 2.3 P12 S/S HS/HS Good Yes (FY97) Georgia Agricultural No 6.0 P14 S/S HS/HS Good No Development (FY97) Kazakhstan Real Estate Yes 8.0 MS MS Poor Yes Registration Pilot (FY97) Lithuania SAL No 16.0 S S NA No (FY97) Romania Ag Sector No 60.0* MU S NA No Adjustment (FY97) Tajikistan Ag. Recovery & No 10.0 MU U NA No Social Protection (FY97) Ukraine Ag Sector No 60.0 MS MS NA Yes Adjustment (FY97) Kyrgyz Agricultural Support No 4.0 P14 S/S S/U Poor Yes Republic Services (FY98) Macedonia SAL No 12.6 S U NA Yes (FY98) Moldova SAL 2 No 20.0 MS S NA Yes (FY98) Romania General Cadastre Yes 37.3 P13 S/S NA Poor Yes (FY98) Albania SAC No 9.0 S S NA No (FY99) Armenia Title Registration Yes 10.6 P09 S/S NA Good Yes (FY99) Azerbaijan Agricultural No 5.4 P09 S/S S/S Good Yes Development and Credit (FY99) Bulgaria Ag Sector No 15.2 S S NA No Adjustment (FY99) Georgia SAC 3 No 12.0 P07 S/S S/S NA No (FY99) Latvia Rural Development No 2.6 S HS Good Yes (FY99) Slovenia Real Estate Yes 28.9 P11 S/S NA Poor Yes Registration Modernization (FY99) 45 Annex 5 Country Project (Approval Stand Cost Overall Sub- Key Specific FY) Alone? (US$m Rating/2 rating Performance Lessons /1 /1 Indicators/3 Cited?/4 Tajikistan Farm Privatization No 4.7 P11 S/S S/S Poor Yes Support (FY99) Kyrgyz Land and Real Estate Yes 11.8 P08 S/S NA Good Yes Republic Registration (FY00) Albania Ag Services No 2.1 P04 S/S S/S Good No (FY01) Bulgaria Ag Sector No 10.0 MS S NA No Adjustment 2 (FY01) Bulgaria Registration and Yes 37.1 P04 S/S NA Poor Yes Cadastre (FY01) Moldova SAC 3 No 6.0* P02 S/S S/S NA No (FY02) 46 Annex 5 Agribusiness Privatization Country Project Name Stand Cost Overall Sub- Key Performance Specific (Year Approved) Alone? US$m Rating Rating Indicators Lessons HUN Agro-processing Yes 76.2 MU NA Poor Yes Modernization (FY89) HUN Integrated Ag Export Yes 199.9 S NA Good No (FY90) POL Agro-industries Export Yes 125.3 HU NA Poor Yes Development (FY90) ALB Ag Sector Adjustment No 9.0* S MS NA No (FY93) POL Ag Sector Adjustment No 60.0* S MS NA No (FY93) RUS Ag Reform No 70.1 MS U Poor No Implementation Support (FY94) KYR Ag Privatization & No 8.4* S MS NA Yes Enterprise Adjustment (FY95) MOL SAL No 13.1* MU U NA No (FY95) UKR Ag Seed Development Yes 63.3 U NA Poor Yes (FY95) UZB Cotton Sub-sector No 26.8 P20 S/S Poor No (FY95) S/S ALB Agro-processing Yes 5.8 U NA Poor No Development (FY96) ROM Ag Sector Adjustment No 60.0* MU S NA No (FY97) TAJ Ag Recovery & Social No 10.0* MU MU NA No Protection (FY97) UKR Ag Sector Adjustment No 60.0* MS S NA No (FY97) ARM Ag Reform Support No 0.8 P11 S/S Good No (FY98) S/S KYR Ag Support Services No 8.7 P14 S/S Poor No (FY97) S/S MAC SAL No 12.6* S MS NA Yes (FY98) MOL SAL 2 No 14.0* MS U NA No (FY98) BUL Ag Sector Adjustment No 15.2* S MS NA No (FY99) TAJ SAC No 11.6* S S NA Yes (FY99) BUL ASAL 2 No 10.0* MS MS NA No FY01) MOL SAC 3 No 6.0* P02 S/S NA No (FY02) S/S UZB Rural Enterprise No 16.0 P03 S/S Good No Support S/S (FY02) 47 Annex 5 Rural Finance Country Project Name Stand Cost Overall Sub- Key Performance Specific Alone? US$m Rating Rating Indicators Lessons POL Ag Development Yes 81.0 MS NA Poor Yes (FY91) ROM Private Farmer & Yes 166.1 S NA Poor Yes Enterprise Support (FY92) ALB Rural Poverty No 1.7 S HS Good Yes Alleviation Pilot (FY93) ALB Ag Sector No 9.0* S U NA Yes Adjustment (FY93) BUL Ag Development Yes 63.4 HU NA Poor Yes (FY94) LAT Ag Development No 52.5 HS HS Good Yes (FY94) ALB Rural Development No 0.8 S HS Good Yes (FY95) ALB Agro-processing No 4.0 U U Poor Yes Development (FY96) LIT Private Ag Yes 30.0 U NA Poor Yes Development (FY96) MAC Private Farmer No 1.5 P16 S/S No info Poor Yes Support (FY96) AZE Farm Privatization No 13.4 P12 S/S S/S Poor No (FY97) GEO Agriculture No 19.5 P14 S/S MS/MS Good No (FY97) KYR Rural Finance 1 Yes 21.9 MS NA Good Yes (FY97) ROM Ag Sector No 60.0* MU S NA No Adjustment (FY97) ARM Ag Reform Support No 11.0 P11 S/S S/S Good No (FY98) KAZ Ag Post Privatization No 19.4 P12 S/S S/S Poor Yes Assistance (FY98) MOL Rural Finance Yes 7.0 MS NA Good Yes (FY98) ALB SAC No 9.0* S S NA No (FY99) ALB Micro-credit No 8.1 PS12 S/S Good Yes (FY99) S/S AZE Ag Development No 15.2 P09 S/S U/U Poor Yes (FY99) BUL Ag Sector No 15.2 S U NA No Adjustment 1 (FY99) KYR Rural Finance 2 Yes 22.9 P11 S/S NA Poor Yes (FY99) LAT Rural Development No 14.2 S S Poor Yes (FY99) BUL Ag Sector No 10.0* MS S NA No Adjustment 2 (FY01) MOL Rural Investment & No 13.7 S S Good Yes Services (FY02) ROM Rural Finance Yes 147.6 P05 U/U NA Good Yes (FY02) UZB Rural Enterprise No 5.9 P03 S/S S/S Poor No Support (FY02) 48 Annex 5 Institutional Reform: Agriculture Services Country Project Name Stand Cost Overall Sub- Key Specific Alone? US$m Rating Rating Performance Lessons Indicators POL Ag Sector No 60.0* S U NA Yes Adjustment (FY93) RUS Ag Reform No 19.0 MS S Poor No Implementation Support (FY94) UZB Cotton Sub-sector No 1.9 P20 S/S S/S Poor No (FY95) CRO Farmer Support Yes 30.0 P16 S/S NA Poor No Services (FY96) EST Agriculture No 6.0 P16 S/S Poor No (FY96) S/HS KYR Sheep & Wool Yes 16.8 P15 S/S NA Poor Yes Improvement (FY96) MAC Private Farmer Support No 5.8 P16 S/S S/S Poor Yes (FY96) MOL Agriculture Yes 18.5 P18 S/S NA Poor No (FY96) AZE Farm Privatization No 4.3 P12 S/S HS/HS Good No (FY97) GEO Agriculture No 0.5 P14 S/S S/S Good No (FY97) ARM Ag Reform Support No 4.4 P11 S/S U/U Good No (FY98) KAZ Ag Post-Privatization No 2.4 P12 S/S S/S Poor Yes Assistance (FY98) KYR Ag Support Services No 15.6 P14 S/S S/S Poor No (FY97) AZE Ag Development No 9.9 P09 S/S S/S Good Yes (FY98) GEO Ag Development & Yes 12.4 P06 S/S MS/S Good Yes Training (FY00) ROM Ag Support Services Yes 17.8 P06 S/S S/S Good Yes (FY00) ALB Ag Services No 2.5 P4 S/S S/S Good Yes Development (FY01) BUL Ag Sector No 10.0* MS U NA No Adjustment 2 (FY01) MOL Rural Investment & No 5.4 P01 S S Good Yes Services (FY02) UZB Rural Enterprise No 1.2 P03 S/S S/S Good No Support (FY02) 49 Annex 6 Annex 6. Lessons from the Project Cohort In this section, the "messages" are paraphrases from Bank appraisal and evaluation documents (a) Generic Lessons Summary Theme Messages Frequency of citation/1 Political commitment · Policy reform and adjustment operations only work when the government is fully committed. · Multi-party dialogue may be needed to build support. · Stakeholder incentives need to be understood. · Early results are needed to sustain commitment. · Line ministries often have little incentive to support adjustment operations because funds do not flow to them directly. · Governments may comply with the letter but not the spirit of the proposed reforms. 15% Conditionality and · Policy conditions should be few in number and enforceable. tranche strategy · Single-tranche operations (with up-front conditionality) are superior to multi-tranche operations. 10% Technical Assistance · Timely, well-coordinated technical assistance is often an essential prerequisite for development effectiveness, particularly in the case of adjustment operations. 10% Institutional · Sustainable policy reform presupposes institutional development, development which often takes a long time to achieve. · Some argue against waiting for institutional development before intervening, making a case for "big bang": starting with the (quick- to-do) incentive reforms before the (slower) institutional reforms are achieved. 7% Donor coordination · Donors must speak with one voice if adjustment operations are to succeed. 7% Project management · A strong project implementation unit is essential for project success. · It is important to provide intensive training, early on, to staff in the project implementation unit. 7% Complementarity · The prospects for policy reform will be much greater when adjustment operations are complemented by technical assistance and investment projects · Offering line ministers the prospect of a parallel or subsequent investment project may help win their support for an adjustment operation 5% /1 Refers to the frequency with which this theme was cited, based on 101 lessons from 29 projects. 50 Annex 6 Detail Country Project Name Lessons Cited (Approval year) HUN Agro-processing 1. There is a need for close follow up and flexibility. Modernization (FY89) HUN Integrated Ag Export 1. Stick to a few, well-defined objectives. 2. Limit the number of policy conditions. 3. External (FY90) financing of technical assistance calls for close supervision. POL Agro-industries 1. Bank projects need strong institutional components to complement the provision of funds. Export Development 2. As a performance measure, the rate of disbursement needs to be used carefully because (FY90) rapid disbursement does not necessarily mean substantial development progress. ALB Rural Poverty 1. Learning by doing is an effective method for developing a flexible, demand-driven project in Alleviation Pilot a transition economy while providing for immediate and visible results. 2. The pilot project's (FY93) bottom-up approach for rural infrastructure rehabilitation created some competition and bureaucratic rivalries with the central government ministries. 3. Creation of autonomous government foundations may be resisted because they operate outside the regular salary framework; their soft budget constraint may create perverse incentives. ALB Ag Sector 1. Adjustment ops are most likely to succeed where political groups favor reform. 2. Adjustment (FY93) ESW is important. 3. Hybrid operations (inc. technical assistance) can facilitate policy reform. 4. Sector adjustment operations and investment operations are complementary. 5. Sector adjustment loans and their associated conditions can be an effective instrument for policy reform when there is a reform-minded government in power. 6. It is important to be realistic about the amount of time required to implement institutional reforms for the provision of rural services. HUN Product Market Good technical assistance, especially during project preparation, can significantly contribute Development (FY93) to the development effectiveness of Bank operations. POL Ag Sector 1. The agriculture sector's growth and export potential cannot be easily harnessed to Adjustment (FY93) lead transition economies out of recession. 2. Agriculture is less resilient than other sectors when faced with inflation and macroeconomic instability. 3. Ag adjustment operations need to address rural poverty by providing safety nets. 4. Ag adjustment should not be restricted to quick-disbursing operations. 5. To help improve rural infrastructure there should be greater provision for the Bank to finance local costs. LAT Ag Development 1. Government ownership is key. 2. A strong project implementation unit is essential. 3. (FY94) Frequent supervision is needed. 4. Continuity of staffing in PIU and Bank team helps. RUS Ag. Reform 1. The Bank should require Project Agreements and covenants with participating regional Implementation governments to improve its ability to suspend and cancel project investments that are going Support (FY94) off track. 2. Project preparation missions should more rigorously investigate the likely demand for goods and services projects will provide. 3. A stakeholder workshop should be held at Mid- Term Review. 4 The Bank should be willing to adjust the content and weight of components during implementation if this will help achieve project objectives. KAZ SAL (FY95) 1. There is a time-lag in adjusting to economic reforms the length of which is hard to predict. 2. Institutions take longer to build than is typically expected. 3. Detailed planning and scheduling is ultimately less important than government commitment and flexibility. KYR Ag Privatization & 1. A sound legal framework is a necessary but not a sufficient condition for policy Enterprise reform; institutional reform is also needed and takes much longer. 2. Effective Adjustment (FY95) economic reform requires sufficient attention to be paid to establishing an appropriate institutional framework. MOL SAL (FY95) 1. To ensure implementation of reforms, it is essential that parliamentary leaders be brought into the process early. 2. Operations should require strong up-front conditionality and tranche conditions should be framed strictly. 3. The Bank should only adopt such conditions that it is willing to enforce. 4. Multi-sector adjustment operations are often held hostage by lack of progress in one or more sectors. 5. Structural adjustment operations and sector operations can be mutually reinforcing, especially when compliance with adjustment conditions leads to the removal of key constraints to sector development, and when sector operations help develop institutional capacity. 6. A system of quarterly performance reviews could be used to indicate, sector by sector, whether conditions had been met and whether disbursement should be made. This would bring greater credibility and realism to decisions about when and in which sectors disbursement may proceed. UKR Ag Seed ES 1. Successful restructuring of government institutions requires government commitment . Development (FY95) 2. When sector policy reforms are needed for Bank supported investments to be profitable, Bank management needs to provide adequate support to supervision teams lacking in the necessary skills. EST Agriculture (FY96) There is a need to fully train project implementation unit staff early on. MOL Agriculture (FY96) It is essential to train counterparts in Bank procurement and disbursement procedures. LIT SAL (FY97) 1. A sustained program of information and education can bring about deep shifts in counterpart thinking. 2. Pertinent technical assistance and effective donor coordination can be critical to project success. 3. Adjustment operations are most effective when part of an integrated country strategy. 4. Challenging reform targets can be an important stimulus to action. 5. A participatory approach in the design of the SAL helped achieve consensus. 6. The SAL team may help to win support by briefing multi-party committees in the run up to parliamentary elections. 7. Effective donor coordination, particularly of technical assistance, is critical to project success. ROM Ag Sector I1. Successive single tranche operations would have been preferable to one comprehensive Adjustment (FY97) two-tranche operation. 2. Early positive results are needed to ensure political sustainability. 3. Generating early positive results requires rapid, efficient implementation. 4. Ministries of Agriculture may see adjustment operations as all pain and no gain. 5. The Romania ASAL, at 51 Annex 6 Country Project Name Lessons Cited (Approval year) US$350 million, was too large. ES. Significant delays in the privatization program may limit the impact and sustainability of the initial incentive reforms. TAJ Ag Recovery & 1. In conflict torn countries sponsorship of a peace process is needed before adjustment can Social Protection be attempted. 2. Adjustment operations with substantial sector (in addition to macroeconomic) (FY97) components need sector-specific action plans and sector-intensive supervision. 3. Strong government commitment is needed if adjustment operations are to be effective. 4. Without a close-knit core team of senior officials with whom the Bank can work, support for policy reforms will be ineffective. 5. The Bank's advice on project design and lessons of experience is as important as the funds it lends. 6. The Field Office can play an active role in monitoring progress with policy reforms and helping to prevent slippage. 7. Disbursement by tranches subject to the government meeting loan conditions may help to ensure that reforms are made. UKR Ag Sector 1. Build a stable base through policy dialogue and anticipate changes in leadership.2. Adjustment (FY97) Remember that governments can satisfactorily comply with loan conditions while failing to champion the spirit of reforms. 3. Favor ambitious quantifiable targets over legal steps with unmonitorable results: define the final results deemed to indicate development progress and leave the amendment of laws and regulations needed to achieve these results entirely to the government's discretion. 4. When adjustment operations are prepared it is important to analyze the receptivity of major political groups to the changes being sought, to specify the public education and outreach programs needed to convince stakeholders, and to identify the elements of the operation most likely to appeal to stakeholders. 5. When framing tranche conditions assess whether there exists the parliamentary support needed for these conditions to be met. KYR Ag Support Services 1.In KYR, project implementation will be ineffective without foreign technical assistance. 2. (FY98) Institution building is top priority MAC SAL (FY98) 1. There is a need for a post-completion monitoring mechanism to verify that the spirit of the reforms is complied with as well as the letter. 2. Pertinent technical assistance and effective donor coordination can be critical to project success. MOL SAL 2 (FY98) 1. Front-loading of conditions and single-tranche operations may help avoid impasse on policy reform. 2. Good donor coordination is needed. 3. Improved public expenditure management typically entails a fundamental reduction in government's range of commitments and clearer setting of priorities. 4. Post-privatization support is essential. MOL Rural Finance Where there are multiple sources of donor funding, poorly coordinated and geared to multiple (FY98) objectives, there will be weaker budgeting and cost controls. ALB SAC (FY99) 1. A reform program can be greatly enhanced when the strategy is to combine structural adjustment with follow-up investment projects. 2. An inter-ministerial committee with support of the Prime Minister can be a good vehicle for reducing resistance to reform. 3. At times of crisis, donors must speak with one voice if the operation is to succeed. BUL Ag Sector 1. "Big bang" can be made to work: incentive reforms (quick to do) need not wait for (slower) Adjustment 1 (FY99) institutional reforms; Bulgaria opted for gradualism at first but got nowhere. 2. In a poor policy environment there is value to be had from constructive disengagement. 3. Good sector work is needed to lay the foundation for lending operations 4. Where there is political consensus for reform, sector adjustment loans can be effective. 5. Process is as important as the substance of reform. 6. Good sector work is an essential foundation. 7. One-tranche adjustment loans are effective. TAJ SAC (FY99) A technical assistance credit that is tailored to the envisaged reform program and is sufficiently flexible can help to ensure the success of a subsequent adjustment operation. GEO Ag Research, 1. Long-term commitment is needed to address ag issues through phased programs of Extension & Training intervention and broad-based participation. 2. It is important to provide specialized training to (FY00) counterparts, especially in procurement and disbursement. ALB Ag Services 1. Set up strong implementation unit. 2. Set up decentralized delivery structure, including Development (FY01) private sector. BUL Ag Sector 1. Single-tranche adjustment can work when there is strong government commitment and Adjustment 2 (FY01) when it is undertaken within a well-elaborated medium-term framework. 2. Single-tranche loans are more flexible than multi-tranche loans since they avoid locking into legal conditions and policy targets that may not be fully understood or supported by Government, and for which implementation capacity may be inadequate or hard to foresee. 3. Line ministries have little incentive to collaborate with adjustment operations: the operations impose costs on ministry clients but the direct benefits flow only to the Ministry of Finance. 4. One way to increase line ministry support for adjustment is to include a technical assistance or investment component; but ECA experience with these hybrid operations has been poor. 5. Policy- relevant analytical work is valuable in formulating major policy changes and when discussed with government and major stakeholders can help to build consensus. 6. Market and resource allocation responses to reform can be weak in the absence of complementary investments, technical assistance, and capacity building measures; given the long time required to prepare Bank investment loans and the desirability that they run in parallel to adjustment operations, preparation of investment projects needs to begin early. 7. Policy reform may be a weak instrument for poverty reduction. 8. The Ministry of Agriculture needs to focus on providing public goods with a view to improving market response to reform and extending the benefits of reform to poor rural inhabitants and smaller farmers. 52 Annex 6 (b) Theme-Specific Lessons Summary Theme Messages (Listed by descending frequency of citation) Trade Liberalization and Market · Farmers respond rapidly to new market opportunities. Development (N=3 projects, 9 · Newly-created wholesale markets may be vulnerable to competition from lessons cited) older marketplaces and emerging supermarkets and discount wholesalers. Land Administration and Reform · Effective implementation of cadastre and land registration projects (N=18 projects, 42 lessons cited) depends on a clear legal framework, a simple institutional structure, long- term government commitment and open access to the resulting land information. · Stand alone projects work best. · The time needed for reform and restructuring is often underestimated. · Costs should be contained by avoiding over investment in technology and over specification of mapping and survey standards. · There should be a gradual increase in cost recovery, involving fee-for- service charges. · Members of collectives should be given titles [not paper shares] to land and property and allowed to dispose of these as they wish. · Substantial support services are needed to make farm restructuring work. Agribusiness Privatization · Successful restructuring of state-owned enterprises cannot be (N=7projects, 18 lessons cited) accomplished without sweeping reform of ownership and governance. · The Bank's procedures are cumbersome and costly for private investors and severely hamper its ability to assist privatization. · Post-privatization support is essential if the new firms are to be viable. Rural Finance (N=20 projects, 76 · Attempts to strengthen intermediation will fail if macroeconomic and lessons cited) structural reforms (such as privatization and land titling) have not already been undertaken. · Credit targeting is ineffective. · Lines of credit should not be channeled through state-owned banks. · If credit line operations are to work there needs to competition between private intermediaries and an experienced apex institution. · Rural finance projects may be constrained by the lack of profitable opportunities for lending, banks' lack of information about the clientele, and borrowers' lack of experience in preparing business plans. · Village savings and loan associations can be both efficient and resilient, maintaining a high level of loan repayment in the face of political unrest; but they need to make stronger efforts to mobilize savings. · The availability of collateral needs to be increased, not only by land titling but also by creating a framework for financial transactions secured with moveable property. Institutional Reform: Agriculture· Some services lend themselves to private provision, others do not; the Services (N=8 projects, 24 rationale for public service provision needs to be made explicit. lessons cited) · Farmers will only make use of new technologies if the incentive environment is right and the technologies are economically viable. · To the extent possible, the cost of these services should be recovered · Competitive Grant Schemes represent an efficient, demand-driven way to generate new technologies but they entail intensive training and follow- up. 53 Annex 6 Trade Liberalization and Market Development Lessons Country Project Name Specific Lessons Cited KAZ SAL (FY95) 1. Demonopolizing agriculure marketing--which had been expected to bear fruit only over the medium term--proved immediately effective; contrary to expectations there was a much slower response to the change of [land] ownership. MAC Private Farmer Support Private farmers prefer private to social sector marketing channels. (FY96) POL Wholesale Markets 1 (FY99) 1. Competition from newly-established supermarkets and discount wholesalers reduces the viability of traditional wholesale marketplaces. 2. If the traditional marketplaces are to survive they must offer more than perishable food. 3. The sustainability of a wholesale market largely depends on the adaptive skills of its management. 4. Supervisory boards can be captured by management interests, failing to represent those of the shareholders. 5. Wholesale marketplaces combine the features of public and private goods and therefore it is not reasonable to expect private firms to be solely responsible for covering the investment cost. 6. Loan agreements should contain commitments by local authorities to close older wholesale marketplaces once the new facilities are ready for operation. 7. In rapidly-evolving economies, market niches may disappear very quickly. POL Wholesale Markets 2 (FY99) Same as above. 54 Annex 6 Land Administration and Reform Lessons Country Project Specific Lessons (Approval FY) ALB Ag Sector Technical assistance is key for developing land tenure legislation. Adjustment (FY93) RUS Land Reform 1. Projects work where a single agency is responsible for implementation. 2. Project Implementation Implementation Unit must be adequately staffed, trained and funded. 3. Projects which register Support (FY94) existing rights are more successful than those that seek to reallocate rights. 4. Project must be consistent with government's own plans to address land use questions. 5. Land cadastre and titling costs can greatly exceed appraisal estimates where due attention is not paid to procedures for adjudication, survey and dispute settlement. KAZ SAL (FY95) 1. Demonopolizing agriculure marketing--which had been expected to bear fruit only over the medium term--proved immediately effective; contrary to expectations there was a much slower response to the change of [land] ownership. KYR Ag Privatization & Development of land markets was too ambitious an objective to be realized within 12-15 Enterprise months of the program implementation. Adjustment (FY95) AZ Farm Privatization 1. Land privatization and farm restructuring is best phased in line with the development of (FY97) markets, financial intermediation and macroeconomic stabilization; the process is accelerated as the distortions in the economy are eliminated, procedures refined, legal systems developed and supporting institutions evolve to provide timely support services. 2. Farm members should be given titles to land and property and allowed to dispose of these as they wish. 3. Substantial support services are needed to compensate for lack of information on new laws and procedures, weak institutional capacity, lack of working capital, and poor roads and irrigation works. KAZ Real Estate The lack of a firm legal basis for land registration, the existence of strong inter-agency rivalry Registration Pilot for control of legislation, and the lack of policy consensus on the goal of markets in land (FY97) complicate project development. ICR 1. Land registration procedures need to reflect clients' expressed views. 2. Good registration systems are based on fee for service. 3. Project implementation units (PIUs) need to be strictly controlled by implementing agencies. 4. Effective inter-agency coordination is problematic (and PIUs do not have the leverage to make coordination effective). 5. Registration can be over automated resulting in the purchase of sophisticated equipment that is not fully used and is unlikely to be maintained. UKR Ag Sector It was unrealistic to expect the government to achieve all the immediate institutional outcomes Adjustment (FY97) (such as changes in legislation) during the short time frame of the project, let alone the expected results of the project (such as the dissolution of 1000 collective agricultural enterprises). KYR Agricultural Land reform has a high payoff if the policy environment is right and the approach is Support Services participatory. (FY98) MAC SAL (FY98) 1. It would have been more effective to specify the details of the law on land use and the break- up of the agro-kombinats in the Loan Agreement. 2. Follow-up is needed to ensure that government's commitment to the letter of agro-kombinat divestiture (inc. land leasing) is actually translated into action. MOL SAL 2 (FY98) Although path breaking the agriculture privatization program has not yet led to a highly productive agricultural sector. Rather it has tended to provide a means of subsistence for rural Moldovans...several complementary measures are needed including land consolidation and ag services. ROM General Cadastre Effective implementation of cadastre and land registration projects depends on a clear legal (FY98) framework, a simple institutional structure, long-term government commitment, and open access to the resulting land information. ARM Title Registration It can be difficult reaching agreement on which agency should manage an integrated registry of (FY99) rural and urban land. AZR Agricultural Based on the Thai experience, successful registration projects entail simple project design, Development and clear institutional responsibility, strong government commitment, gradual increase in cost Credit (FY99) recovery, and efficient use of new technologies. LAT Rural Improvement of a collateral system based on legal ownership of property is a prerequisite to a Development successful credit operation. (FY99) SLO Real Estate 1.Stand alone projects work best. 2. Government should be strongly committed. 3. Institutional Modernization framework should be simple (Project includes inter-agency service agreements and (FY99) development of a data interface between the cadastre, court registry office and other users). TAJ Farm Privatization (Repeats Azerbaijan Farm Privatization). 1. Land privatization and farm restructuring is best Support (FY99) phased in line with the development of markets, financial intermediation and macroeconomic stabilization; the process is accelerated as the distortions in the economy are eliminated, procedures refined, legal systems developed and supporting institutions evolve to provide timely support services. 2. Farm members should be given titles to land and property and allowed to dispose of these as they wish. 3.Substantial support services are needed to compensate for lack of information on new laws and procedures, weak institutional capacity, lack of working capital, and poor roads and irrigation works. KYR Land and Real 1. Stand alone projects work best. 2. Strong government commitment is essential. 3. Avoid Estate over investing in computers. 4. Do not set over costly standards for surveying and mapping.5. Registration Give as much benefit to rights holders as to governments. 6. Minimize conflict between (FY00) cadastral and legal agencies in managing registration system. BUL Registration and 1. Stand alone projects work best. 2. Solid government support is needed. 3. A simple Cadastre (FY01) institutional framework is best. 4. A single, strong project implementation unit is needed. 55 Annex 6 Agribusiness Privatization Lessons Country Project Name Specific Lessons Cited HUN Agro-processing 1. Successful restructuring of medium to large state owned enterprises cannot be accomplished Modernization without sweeping reforms that address questions of enterprise ownership and governance, and (FY89) sustained exposure to domestic and foreign competition. 2. Bank's assistance in the early stages of privatization is hampered by restrictions on the use of Bank funds for financing transfers of existing, used assets. 3. The Bank's procedures for sub-loan application, evaluation and reporting are a burden on private sector participants. POL Agro-industries 1. Successful restructuring of medium to large state-owned enterprises cannot be accomplished Export without sweeping reforms that address questions of enterprise ownership and governance. 2. Development Credit lines are inappropriate when financial intermediaries are weak and inflation is high. (FY90) HUN Product Market 1. Restrictions on use of project funds to finance used assets and real estate seriously restricts Development Bank's ability to support privatization. 2. The Bank's reporting requirements, disbursement (FY93) categories and procurement guidelines are perceived by most private businesses (on-lending banks and sub-borrowers) to be too cumbersome and costly. KYR Ag Privatization & 1. Before proceeding with privatization the debts of the grain marketing and processing Enterprise enterprises should have been dealt with; lumbered with these debts the newly privatized Adjustment enterprises were not viable. 2. Liberalization and privatization measures and elimination (FY95) of legal impediments is not sufficient for the creation of a competitive environment for agribusiness. The new enterprises suffered from lack of working capital, lack of access to modern technologies, and weak managerial skills. 3. Post-privatization support is essential to ensure that newly formed enterprises are viable. UKR Ag Seed 1. Pre-selection of firms should be avoided to the extent possible; in Ukraine it was necessary Development because there were no viable commercial banks capable of vetting business plans. 2. Newly- (FY95) created firms operating in an uncertain market environment require a substantial learning period before they reach full production and efficiency. MAC SAL (FY98) ICR. 1. Post-privatization assistance is needed to complete and strengthen reforms. 2 Agreement by government that Agro-Kombinants will divest assets and lease land does not necessarily translate into action. 3. Loan Agreements need to be specific about the approaches to leasing and divestiture. 4. Articulation of an agriculture sector strategy would increase the value of SAL reforms. 5. In restructuring the agriculture sector, privatization (without fundamental changes in ownership/management, private business environment, and assistance to strengthen reforms including post-privatization enterprise support) may not fully achieve its expected benefits in terms of private sector growth and job creation. TAJ SAC (FY99) 1. The Bank needs to be very precise in specifying what is meant by privatization: what must happen for a tranche condition to be met. 2. In pursuing privatization it is better to err on the side of comprehensiveness than to take a gradualist, piecemeal approach; partly because the longer privatization is delayed the greater the depreciation of public enterprise assets. 56 Annex 6 Rural Finance Lessons Country Project Name Specific Lessons Cited HUN Agro-processing 1. The dangers of targeting of credit...need for open lines of credit available to a broad spectrum Modernization of users. 2. Greater reliance on appraisal capabilities of financial intermediaries would reduce (FY89) Bank inputs and aid institutional development. POL Ag Development 1. At the outset it is essential to agree upon a strategy for the overall development of the rural (FY91) finance sector. 2. Strong and sustained government commitment is needed to ensure timely improvement of the enabling environment. 3. Credit lines are inappropriate where financial intermediaries are weak and there is high inflation. ROM Private Farmer & 1. When carefully managed and focused to support private initiative a credit project can succeed, Enterprise Support even if the economy is distorted. 2. Credit lines should not be targeted at particular sub-sectors or (FY92) export activities. 3. Choice of the apex institution is critical; if a government body is used it may introduce a public spending control culture that is not compatible with the commercial thrust of the project. ALB Rural Poverty Village solidarity groups can effectively deliver small loans. Alleviation Pilot (FY93) ALB Ag Sector It is undesirable to use a state-owned bank to provide rural credit. Adjustment (FY93) BUL Ag Development 1. Financial intermediary lending is likely to fail if implemented before basic macroeconomic and (FY94) structural reforms are in place. 2. A credit-line operation is not an effective way to redress the lack of financially strong, well managed and supervised banks. 3. Successful credit line operations entail early contracting of a large number of sound intermediaries. 4. If credit line operations are executed in the absence of an experienced apex institution, the Bank's project supervision load will be substantial. 5. Central banks are usually unsuitable as apex institutions. ES Once it is clear that the environment is not right for on-lending, the best action is to close the loan rather than trying to make on-lending more attractive. LAT Ag Development 1. Credit lines in transition economies must be supported by land titling, and by better market (FY94) information. 2. Projects should have multiple private financial intermediaries to create competition and broaden credit access. 3. When projects combine loan and grant funds it is important that the latter not undermine repayment discipline. 4. Mobile credit offices are a worthwhile innovation. 5. Non-bank financial institutions can be effective stopgaps in transition economies. ALB Rural 1. The trade-off between channeling resources to poor rural households and developing Development sustainable financial institutions in the economically most viable villages needs to be explicitly (FY95) recognized. 2. Rural microcredit programs can sustain extraordinary repayment performance, even in very poor and remote villages, providing borrowers have been involved in the design of the delivery mechanism, delivery is adapted to local conditions, loan decisions are made in the villages, and peer pressure substitutes for traditional collateral. ALB Agro-processing The Bank should not provide lines of credit to be on-lent through state-owned banks, which are Development subject to too much political interference, and lack the incentives that private banks have to (FY96) appraise and supervise loans adequately. (This project repeated the same error of using the state-owned Rural Commercial Bank, even though its use was reported unsatisfactory in the FY93 Ag Sector Adjustment). LIT Private Ag 1. Successful macroeconomic adjustment is necessary for credit line projects to succeed. 2. Development Successful macroeconomic adjustment, along with a liberalized, open economy and fully (FY96) convertible exchange rates may obviate the need for credit line projects. MAC Private Farmer The formal banking system has difficulty in appraisal and supervision of agricultural loans, in part Support (FY96) because of the "cultural" bias within banks against agriculture and unique aspects of agriculture such as seasonality and weather vulnerability. KYR Rural Finance 1 1. Rural finance projects may be constrained by the lack of profitable opportunities for lending, (FY97) bank's lack of information about the clientele, and borrowers' lack of experience in preparing business plans. 2. The loan portfolio should be diversified by size, type of activity, client and region. 3. If it is to be sustainable group lending requires substantial technical assistance and training to communities. 4. Subsidized credit lines operated by monopoly parastatals hamper the development of sustainable rural financial intermediation. 5. Public credit agencies do not have the organizing skills needed to develop rural financial intermediation. 6. Technical assistance must be closely coordinated with stakeholders. 7. Investment lending, and lending to businesses, requires different skills and more effort than small-scale working capital lending; special institutions are needed. 57 Annex 6 Rural Finance Lessons (Continued) Country Project name Specific lessons cited (Year approved) KAZ Agricultural Post- 1. Macro-economic stabilization, market liberalization and privatization are pre-requisites for Privatization. effective financial interventions. 2. Rural credit projects often face difficulties because profitable Assistance (FY98) opportunities for lending are lacking, private financial intermediaries do not know the clientele, risks are high if lending is limited to agriculture or a small region, and the transaction costs of rural lending are high relative to other lending opportunities. MOL Rural Finance 1. Greater emphasis should be given to internal mobilization of funds by Savings and Credit (FY98) Associations (SCAs). 2. More than one commercial bank should participate. 3. Institutions like SCAs need to raise share capital. 4. Loan limits per SCA and per member should be adjusted according to the SCA's ability to mitigate risks. 5. If SCAs are to be sustainable, their elected officials, accountants and supervisory bodies need to be well trained. 6. In the long term, technical assistance to SCA's should be fee-based. 7. Sustainability of SCAs hinges on the capacity of the supervisory body. 8. When a subsidy is provided to a financial intermediary, there should be a phase-out plan. 9. There needs to be a balance between expanding the number of new SCAs and strengthening the existing ones. ALB Micro-credit 1. Borrowers should be involved in the design of a microcredit delivery mechanism, and credit (FY99) delivery should be piloted and adjusted to local conditions. 2. Because the integrity of the lending institution rests on peer pressure villagers desire to maintain their reputation as credit subjects, microfinance can withstand political and civil crises originating outside the village. 3. In past projects, earlier emphasis should have been placed on financial sustainability by establishing an automatic mechanism for interest rate adjustment, establishing the urban and rural credit departments as separate cost centers, and making salaries performance based. 4. While the Albanian Development Fund was a convenient institution for piloting the rural credit program, its multipurpose and soft budget environment impeded progress toward financial sustainability: private institutions should be used from the outset if possible. AZE Ag Development 1. Complementary actions, such as perfecting property rights in land and improving information (FY99) on input and output markets, are needed if rural finance is to be an effective development tool. 2. Rural financial intermediaries must operate in a conducive policy, institutional and legal framework. 3. Implementing agency needs to train local financial intermediaries in accounting, portfolio monitoring and management. 4. The target clientele for loans should be entrepreneurs with existing businesses from which loans can be repaid; providing people with credit does not necessarily make them entrepreneurial. 4. Credit targeting is ineffective and the intensive supervision that it calls for makes credit prohibitively expensive for the small entrepreneur. KYR Rural Finance 2 1. Rural finance projects may be constrained by the lack of profitable opportunities for lending, (FY99) bank's lack of information about the clientele, and borrowers' lack of experience in preparing business plans. 2. The loan portfolio should be diversified by size, type of activity, client and region. 3. If it is to be sustainable group lending requires substantial technical assistance and training to communities. 4. There should be multiple financial institutions involved in a credit line project to increase competition among banks and to reduce the cost of borrowing. 5. The high cost of making small rural loans can be reduced by group lending, mobile banks, and decentralized processing at local branches. 6. Private ownership of land facilitates lending by providing a source of collateral. LAT Rural 1. Macro-economic stabilization and market liberalization are prerequisites for sound financial Development intermediation. 2. Rural finance projects are more likely to work when there is legally-based (FY99) property ownership, a functioning legal system, functioning financial institutions, and effective farm and business advisory services. 3. The operation of financial markets is enhanced by the development of property registration, and ability to use moveable collateral. 4. Grant-loan hybrids should be used only as a one-time catalyst by new borrowers, making it clear that there will be no further government handouts. 5. Grant-loan hybrids can help establish new businesses in rural areas. 6. Projects should have multiple private financial intermediaries to create competition and broaden credit access. 7. Rural financial operations can be jump-started with subsidies to attract both borrowers and banks to participate, but this may undermine commercial lending in the long term. MOL Rural Investment & 1. Credit lines need to be complemented by capacity building investments and removal of Services (FY02) impediments to the rural inflow of funds. 2. Macro-economic stabilization, liberalization and privatization are prerequisites for effective financial interventions. 3. Rural credit projects need to address problems associated with the lack of profitable lending opportunities, intermediaries lack of information about potential clients, high risks involved in lending only to agriculture or to single regions, and high costs compared to loans made outside rural areas. 4. Savings and Credit Associations (SCAs) reduce the risk of rural lending without requiring collateral. 5. SCAs should aim to mobilize savings as well as extend credit. 6. Intensive training is needed to help loan officers assess risks and to aid potential clients prepare bankable projects. ROM Rural Finance 1. A precondition is to remove from the market poorly-run government agencies providing (FY02) directed credit at subsidized rates. 2. A framework for financial transactions secured with movable property will facilitate rural lending. 3. These sorts of projects require frequent monitoring of key performance indicators and rigorous supervision offering high level technical advice. 58 Annex 6 Institutional Reform: Agriculture Services Country Project Name Specific Lessons Cited POL Ag Sector Agriculture education and research entails long-term institution building with inputs from Adjustment specialists; adjustment lending is not a suitable vehicle. (FY93) KYR Sheep & Wool 1. Producers respond to incentives: internal price controls should be removed. 2. The roles of the Improvement public and private sector should be clearly defined. 3. Attention should be paid to livestock- (FY96) environment interactions. 4. The role of research in boosting productivity needs to be acknowledged by the government. 5. The focus should rest on family farms, not public livestock producing and processing enterprises. MAC Private Farmer Private farmers have demonstrated their willingness and ability to adopt new technology and Support (FY96) farming systems, provided these are economically viable. 2. Private farmers prefer private to social sector marketing channels. KAZ Ag Post- 1. Advisory services have no leverage over the manner in which enterprises are restructured and Privatization. their advice may be ignored by managers. 2. Because advisory center staff have no stake in the Assistance restructured enterprise their advice is often not practical or commercially oriented. (FY98) AZE Ag Development 1. Ag services rapidly need to prove their worth if budgetary commitment is to be sustained. 2. (FY98) Cost recovery should be the ultimate aim for services which have a private good character. 3. Farmers should be represented in governing bodies at local, regional and national level. 4. Programs should be regularly updated to incorporate changes in practices, technologies, and markets. GEO Ag Research, 1. Government must strongly support the principle of a Competitive Grant Scheme (CGS). 2. Extension & Design of the CGS needs to be based on close consultation with potential participants. 3. Early on, Training (FY00) applicants need assistance in preparing proposals. 4. Project quality gradually improves as participants begin to appreciate that well prepared proposals are more likely to be accepted. 5. Transparency is essential to remove suspicion that that grants are awarded only to favorites. 6. Work directly with farmers and agro-processors to encourage ownership of the initiative. 7. Disseminate information about improved environmental management. ROM Ag Support Same as Georgia above. CGS funding has proved effective in improving client orientation and the Services (FY00) productivity of research, as well as reforming supply-driven, centralized research management, prevalent in many transition economies. ALB Ag Services Decentralize the delivery of ag services by supporting the capacity of existing local associations Development and involving private suppliers. (FY01) MOL Rural Investment 1. If services are to work, the environment must be conducive to profitable farming. 2. Self- & Services financing of providers needs to be gradually phased in; there should be at least some element of (FY02) cost recovery. 3. The responsiveness of providers to client needs must be carefully monitored. 4. It takes considerable time to build up profitable businesses, capable of surviving without donor support. 5. Farmer organizations require intensive training. 59 Annex 7 Annex 7. Poverty Indicator Ratings Country, Project Name, FY Approval Good Good Good Indicators Poverty Poverty but No Indicators Indicators Poverty but No Data with Orientation Data Showing Results FY91-96* Albania, Rural Poverty Alleviation Pilot, FY93 X Latvia, Ag Development, FY94 X Albania, Rural Development, FY95 X Total 1 - 2 FY97-02** Azerbaijan, Farm Privatization, FY97 X/a Georgia, Agriculture, FY97 X/b Kyrgyz Republic, Rural Finance 1, FY97 X Armenia, Ag Reform Support, FY98 X Moldova, Rural Finance, FY98 X Albania, Micro-Credit, FY99 X Armenia, Title Registration, FY99 X Azerbaijan, Ag Development, FY99 X Latvia, Rural Development, FY99 X Georgia, Ag Research, Extension and Training, X FY00 Kyrgyz Republic, Land Registration, FY00 X Romania, Ag Support Services, FY00 X Albania, Ag Services, FY01 X Moldova, Rural Investment and Services, FY02 X/c Romania, Rural Finance, FY02 Xd Uzbekistan, Rural Enterprise Support, FY02 X Total 15 1 - * In FY91-96, 3 out of 23 projects in the evaluation cohort (13%) had good indicators. ** In FY97-02, 16 out of 39 projects in the cohort (41%) had good indicators. /a The project has a component focusing on vulnerable groups and cites "changes in income level of vulnerable groups" as an indicator; but no baseline or target is given making it impossible to assess progress. /b The project tracks the increase in the number of small farmers with access to credit (through membership of credit unions) but treats all credit union members as small farmers without stratifying by socio-economic profile. /c Poverty reduction is an explicit objective. A baseline survey was conducted. One of the indicators is "increased rural income in project areas as compared to the baseline survey". The baseline is US$270 per capita per year equivalent and the end-of-project target is for at least 5 percent growth in real terms. This project only became effective in August 2002 so insufficient time has elapsed to collect post-baseline income data. /d The development objective includes rural poverty reduction. The Quality Assurance Group's entry assessment assessed the baseline survey as very comprehensive and well done. But it rated arrangements for monitoring poverty as unsatisfactory; poverty or income data are not included among the key performance indicators. 60 Annex 8 Annex 8. Does ECSSD Lending Discriminate in Favor of Countries that Lead on Agriculture Policy Reform? 1. In the first half of the evaluation period (1991-96), commitments tended to favor countries that were performing well in terms of the 1997 ECSSD rating (Table A). It could be argued that the reform leaders in 1997 were so positioned because they had used Bank assistance to help them reform; their starting position on reform may not have influenced the Bank's decision whether or not to lend to them. Table A. Association of FY1991-96 Commitments With 1997 Reform Rating Commitments/Rural Gross National LAGGARDS LEADERS Income Agriculture policy reform rating Agriculture policy reform rating at or below 1997 median/1 above the 1997 median/1 LOW COVER Azerbaijan Czech Republic Commitments below the median/2 Belarus Georgia Croatia Slovak Republic Macedonia Slovenia Moldova Russia Tajikistan Turkmenistán Ukraine HIGH COVER Bulgaria Albania Commitments at or above the median/2 Kazakhstan Armenia Kyrgyz Republic Estonia Uzbekistan Hungary Latvia Lithuania Poland Romania /1 The median rating was 6.2 on a scale of 10, with 10 representing most advanced. /2 The median was 0.10 percent of rural GNI (gross national income times the rural share of the total population). 2. Therefore, a more compelling test of selectivity is to consider whether commitments made after 1997 bore any relation to 1997 reform status. This proves not to be so. In the later period (1997-02) lending favored reform laggards rather than leaders (Table B). Table B. Association of FY1997-02 Commitments With 1997 Reform Rating Commitments/Rural Gross National LAGGARDS LEADERS Income Agriculture policy reform rating Agriculture policy reform rating at or below 1997 median/1 above the 1997 median/1 LOW COVER Croatia Czech Republic Commitments below the median/2 Belarus Estonia Bulgaria Hungary Russia Latvia Turkmenistán Lithuania Poland Slovak Republic Slovenia HIGH COVER Azerbaijan Albania Commitments at or above the median/2 Kazakhstan Armenia Kyrgyz Republic Georgia Macedonia Romania Moldova Tajikistan Ukraine Uzbekistan /1 The median rating was 6.2 on a scale of 10, with 10 representing most advanced. /2 The median was 0.11 percent of rural GNI (gross national income times the rural share of the total population). 61 Annex 8 3. Another possibility is that 1997-02 commitments were influenced less by reform rank in 1997 than by the progress on reform between 1997 and 2001--the extent to which the reform rating improved. This also proves not to be the case: there is a total lack of association between commitments and reform progress (Table C). Table C. Association of FY1997-02 Commitments With Reform Progress, 1997-02 Commitments/Rural Gross National LAGGARDS LEADERS Income 1997-02 reform progress below 1997-02 reform progress at or above median/1 the median /1 LOW COVER Belarus Bulgaria Commitments below the median/2 Hungary Croatia Poland Czech Republic Russia Estonia Slovak Republic Latvia Slovenia Lithuania Turkmenistán HIGH COVER Armenia Albania Commitments at or above the Georgia Azerbaijan median/2 Kazakhstan Romania Kyrgyz Republic Tajikistan Macedonia Ukraine Moldova Uzbekistan /1 The median change was 11.1 perecnt. /2 The median was 0.11 percent of rural GNI (gross national income times the rural share of the total population). 4. A final possibility is that selectivity would be expected to manifest itself more clearly for investment lending: because ECA strategy indicates that investments should only flow to countries with an adequate policy framework. (To the extent that adjustment lending aims to move countries from weaker to stronger policy status, if adjustment loans are included in total commitments it will, by definition, be hard to demonstrate selectivity because weaker countries may be expected to be the target of adjustment lending). Paradoxically, laggards were somewhat more likely to receive investment loans than leaders (Table D). Table D. Association of FY1997-02 Investments With 1997 Reform Rating Commitments (investments only) LAGGARDS LEADERS /Rural Gross National Income Agriculture policy reform rating Agriculture policy reform rating at or below 1997 median/1 above the 1997 median/1 LOW COVER Czech Republic Commitments below the median/2 Belarus Estonia Bulgaria Hungary Russia Lithuania Turkmenistán Poland Ukraine Slovak Republic Slovenia HIGH COVER Azerbaijan Albania Commitments at or above the Croatia Armenia median/2 Kazakhstan Georgia Kyrgyz Republic Latvia Macedonia Romania Moldova Tajikistan Uzbekistan /1 The median rating was 6.2 on a scale of 10, with 10 representing most advanced. /2 The median was 0.08 percent of rural GNI (gross national income times the rural share of the total population). 62 Annex 8 5. Maybe the analysis is biased by the inclusion of "graduates"--countries that are so advanced with reforms that they do not qualify (or show much demand) for Bank assistance. Inclusion of these countries would clearly weaken the positive association between reform progress and Bank assistance. However, although excluding the graduates strengthens the association, it does so only slightly (Table E). Table E. Association of Commitments and Reform Status--The "Graduate" Effect All countries (N=25) All countries less "graduates" (N=16)/1 Correlation of FY1991-96 0.126 0.319 Commitments With 1997 Reform Rating Correlation of FY1997-02 -0.121 0.153 Commitments With 1997 Reform Rating Correlation of FY1997-02 0.025 0.063 Commitments With Reform Progress, 1997-02 Correlation of FY1997-02 -0.095 0.080 Investments With 1997 Reform Rating /1 "Graduates": Czech Republic, Hungary, Slovak Republic, Estonia, Latvia, Lithuania, Poland, Armenia. 6. An alternative way of measuring the linkage between lending and reform progress is to examine the number and distribution of dropped projects. (These are listed in Annex 3). To what extent does the decision to abandon a loan reflect a judgment about the country's lack of reform progress? Table F shows that countries rated as laggards were more likely than leaders to have projects dropped and that, after 1996, the Bank was much more aggressive in dropping projects, suggesting that it may have paid closer attention to reform progress. The country with the largest number of dropped projects (12) was Russia, very much a laggard: according to the ECSSD ratings Russia back-tracked on reform between 1997 and 2001. On the other hand, the country with the second-largest number of dropped projects (Poland, 7) was in the progressive camp. Some staff argue that this is to be expected: lending in this relatively advanced country should be left more to private investors. Table F. Project Drops and Reform Status Number of Projects Laggards Leaders Total Dropped 1991-96/a Belarus 1 Hungary 1 14 Bulgaria 1 Poland 2 Russia 7 Turkmenistán 1 Uzbekistán 1 1997-02/b Belarus 1 Albania 2 43 Kyrgyz Republic 2 Bulgaria 1 Macedonia 2 Croatia 1 Moldova 3 Czech Republic 1 Russia 5 Hungary 1 Tajikistan 2 Kazakhstan 1 Turkmenistán 3 Latvia 1 Poland 5 Romania 2 Total 34 23 57 /a Laggards and leaders are separated by the median rating on the FY97 ECSSD scale. /b Laggards and leaders are separated by the median rating on the FY01 ECSSD scale.