T H E W O R L D B A N K O P E R A T I O N S E V A L U A T I O N D E P A R T M E N T Sector Lessons Note OED's Seven Leading Rural Lessons Ridley Nelson Chris Gerrard John Heath Nalini Kumar Director-General, Operations Evaluation: Gregory K. Ingram Director: Ajay Chhibber Manager: Alain Barbu Task Manager: Ridley Nelson 2003 This paper is available upon request from OED. The World Bank Washington, D.C. ii ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION The Operations Evaluation Department (OED) is an independent unit within the World Bank; it reports directly to the Bank's Board of Executive Directors. OED assesses what works, and what does not; how a borrower plans to run and maintain a project; and the lasting contribution of the Bank to a country's overall development. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank's work, and to provide accountability in the achievement of its objectives. It also improves Bank work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. OED Working Papers are an informal series to disseminate the findings of work in progress to encourage the exchange of ideas about development effectiveness through evaluation. The findings, interpretations, and conclusions expressed here are those of the author(s) and do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent. The World Bank cannot guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply on the part of the World Bank any judgment of the legal status of any territory or the endorsement or acceptance of such boundaries. Contact: Operations Evaluation Department Partnerships & Knowledge Programs (OEDPK) e-mail: eline@worldbank.org Telephone: 202-458-4497 Facsimile: 202-522-3125 http:/www.worldbank.org/oed 1 Sector Lessons Note OED's Seven Leading Rural Lessons Purpose. The purpose of this brief, informal note is to outline what OED Rural Cluster staff consider the seven most important project lessons drawn, approximately, from two years of project evaluation results (2001-2002).1 Sector performance data is given for background, but the main focus is on project lessons. Although prescribing solutions is not OED's role, under each generic lesson, some suggestions are included on what might be done. The reader will notice that these lessons are largely project design lessons. However, OED did not set out only to look at lessons related to project design; all lessons were reviewed whatever their focus. This suggests that, although supervision of implementation can certainly redirect projects within limits, many implementation problems found by OED can still be traced back to faulty initial design ­ including weak specification of objectives, weak translation of objectives into outcome-focused design, and poor policy and institutional design fundamentals. Rural Sector Lending Volumes and Performance. With respect to lending volume, Agriculture lending has declined by about 50 percent over the FY 1985­2003 period. While all real sectors have declined since about 1993, including education, health, and urban, rural has declined more than all except energy and mining. The decline has been steeper than the decline in the aggregate share of agriculture in borrowers' GDP. This rural support decline has happened for nearly all donors. There is now a concerted attempt both within the Bank and globally to turn this around. With respect to performance, over the 10-year period 1991-2002, the Outcome performance of rural sector projects (defined as those projects mapped to the Rural Board) relative to all Bank projects went through phases of being worse than the Bank average, then similar to the Bank average, then worse again, and now in the most recent year, equal to or better than the Bank average (see Graph 1). Based on the old "agriculture plus three" classification (all agriculture plus natural resource management, rural water and sanitation, and rural roads), the story was similar except that over the most recent 5 years the trough and crest is somewhat less pronounced than for the new classification shown in the graph. 1. The sample reviewed covered all projects mapped to the rural family over the fiscal year 2001 and the majority of fiscal 2002 ­ those where Evaluation Summary reviews or PPARs were completed by end July 2002 (106 Evaluation Summaries and 31 Project Performance Assessments). However, in drawing from Project Performance Assessment lessons we have occasionally strayed outside this time range to draw upon some cases that seemed particularly relevant to an issue. 2 Graph 1. Outcome Ratings % Satisfactory or Above 90 84 80 76 72 73 71 83 Above 71 70 67 75 76 or 64 62 72 65 69 68 60 66 66 64 58 59 54 61 50 57 All Bank 40 "Satisfactory" 30 Rural 20 10 Percentage 0 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 Fiscal Year On Sustainability (Graph 2), rural performance has tracked the improving pattern of the all-Bank figures fairly closely but has nearly always been somewhat lower. However, again, it has caught up in the most recent year. Based on the old agriculture classification, the trend was similar. Graph 2. Sustainability Ratings % Likely or Above 100 90 80 71.64 Above 67.51 70 77 62.12 and 60 53.87 51.88 46.45 46.02 45.89 48.94 59 Rural 50 44.63 44.21 40.82 All Bank "Likely" 47 40 49 46 46 42 41 43 30 31 31 33 20 Percentage 10 0 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 Project End Year On Institutional Development Impact (Graph 3), rural projects had, before about 1998, tracked the all- Bank figures reasonably closely in trend, but with significant fluctuations above and below. However, more recently, rural has dropped more consistently below the all-Bank figure ­ by between about 5 percent and 8 percent. Based on the old agriculture classification, the trend in the most recent five years is higher than shown here, in fact more or less matching the all-Bank figure. 3 Graph 3. Institutional Development Impact Ratings % Substantial or Above 100 90 ovebA 80 or 70 ial" 60 nta 49.25 Rural 50 48 49.10 48.48 40 41 All Bank ubstS" 40 36 34.12 46 30.97 28.76 38.03 42 40 30 35.93 37.24 37.32 gea 33 24.38 31 26.59 20 26 21 21 10 Percent 0 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 Project End Year Looking at the more recent three to four years, one can summarize the findings as showing a period of improving outcomes but with quite persistent problems of sustainability and institutional development impact ­ below 50 percent likely sustainability until the jump in 2002 ­ and not much above 40 percent substantial IDI ratings. Clearly there is room for improvement. Recent QAG findings suggest some improvement in Quality at Entry but, of course, OED results will only be able to confirm that that has carried through to evaluated project performance once those projects have actually closed. Disaggregation of sector-wide data reveals a number of other findings worthy of note: · Africa has been particularly weak on IDI with only 29 percent of projects over the period FY 1991­2002 having a Substantial or above IDI rating, although, on this rating, MENA region was only marginally better. · Africa has been particularly weak on Sustainability with only 34 percent of projects over the same period having a Likely or higher rating. · However, there has been a fairly steady improvement over the 10 year-period in the Outcome ratings for Africa rising from about 40 percent Moderately Satisfactory or above at the start of the 1990s to about 60 percent in the early 2000s. · Average appraisal ERRs for rural projects have remained very stable over time, consistently between about 20 percent and 25 percent, somewhat lower than the all-Bank average. · Agricultural Extension and Research have generally and fairly consistently exhibited a low percentage of Likely Sustainability ratings, at about 22 percent. · Irrigation, Drainage, and Flood Protection have exhibited a steadily climbing percentage of Likely Sustainability from about 40 percent at the beginning of the 1990s to about 70 percent at the beginning of the 2000s. This may partly reflect a retreat from the larger schemes. 4 · Agricultural Adjustment exhibited very low Sustainability at an average 22 percent in the Likely category. · Agricultural Credit projects ended in FY95, thereafter whatever agricultural credit support was provided came through components of other projects or projects otherwise classified. · Agricultural Research and Extension fell from an average of about 10 projects per year in the mid-1990s to about 5 in the early 2000s. Most Highly Rated. The two most highly rated (by OED) closed rural projects over the last two years in Evaluation Summaries have been: Brazil Rural Poverty Alleviation ­ Ceara and China Xiaolangdi Multipurpose Project, both rated Highly Satisfactory on Outcome, Substantial on Institutional Development Impact, Likely on Sustainability, Highly Satisfactory on both Bank and Borrower Performance, and Exemplary on ICR Quality ­ the experience of these projects would be worth closer examination. Methodology. The methodology for this review of rural lessons was simple and relatively informal. Lessons from Project Performance Assessments and from Evaluation Summaries, many of which are drawn from lessons in Implementation Completion Reports, were grouped in loose categories (see Table 1) in order to understand the types of lessons and the approximate number of mentions in each category. This involved considerable judgment. OED Rural Cluster staff then consolidated and encapsulated these lessons into what seemed to be the most important seven, in some cases consolidating several lower-order lessons into a more generic higher-order lesson. These were then used as a basis to identify a limited number of specific project cases representative of those lessons. The process was iterative. No claim is made for any form of statistical validity in this exercise. In some cases, consolidated lessons as finally formulated have been extended somewhat beyond what was strictly contained in the component project lessons in an attempt to make them broadly relevant and actionable. Lessons were not set aside simply because they could apply to another sector. It was considered sufficient that they arose out of the rural projects reviewed. OED makes no apology that none are lessons that are entirely new. Many problems seem to be repeated quite often. In an attempt to be constructive, under each Lesson a Suggestions section is offered ­ with varying degrees of concreteness. OED does not claim to have solutions to all the identified weaknesses, nor, as an evaluator, is it OED's role to prescribe, but in some cases there seem fairly obvious directions worth considering. These are offered for what they are worth with no claim that they are the best or only directions to look for solutions. In the field and at the time, each case will be unique. Types of Lessons. While, as noted above, the classification of lessons is fraught with all sorts of problems related to type and size of category as well as placement, Table 1 shows that leading lesson areas include governance/ownership/incentives, project analysis/design, community participation, financial, poverty focus/gender, Bank or borrower performance or processes, institutions, and monitoring and evaluation. It should be noted that, since OED generally avoids listing more than about five lessons in either an Evaluation Summary or a Project Performance Assessment Report, lessons are prioritized and lack of a mention in a particular case does not indicate that a particular lesson area was not an issue, it simply means it was ranked somewhat lower than the selected lessons chosen for highlighting. As an example, monitoring and evaluation (M&E) is almost routinely weak but, as can be deduced from Table 1, by no means always features in the selected lessons. This may suggest that the real underlying problem often has not been identified. 5 Table 1. Tabulation of Lessons by Main Category, FY2001/ 02 (percentage of total lessons) Evaluation Performance Main Lesson Summaries Assessments Category (ESs) (PPARs) Governance / ownership / incentives 13 % 12 % 4, 5, Project analysis / design 16 % 5 % 1, 2, 3, 4, 5, 7 Community participation 10 % 8 % 2, 5, Financial 7 % 9 % 7, Poverty focus / gender 4 % 22 %* 2, 5 Bank/Borrower performance/processes 4 % 9 % 1, 2, 3, 6 Institutions 3 % 10 % 2, 4, 5 Monitoring & evaluation 9 % 2 % 3, Implementation / complexity / realism 8 % 3 % 4, 5, 7 Coordination / partnerships 4 % 5 % 6, Sustainability / environment 7 % 1 % 7, Relevance/ efficacy / efficiency 5 % 3 % Lending instrument 2 % 5 % Decentralization 4 % 4, 5, Liberalization / privatization 3 % 2 % 4, 5, Safeguards / procurement / miscellaneous 2 % 2 % Policy framework 2 % 1, 2, 4 Technology 2 % * This is influenced quite heavily by several poverty-related lessons in a package of five Ghanaian projects. 6 Lesson 1 When the rationale for a project and the linkages from the rationale to the design are not clearly spelled out, or when the rationale is not carefully maintained throughout the project cycle, sound judgments made during preparation and appraisal may be compromised and monitoring and evaluation may be hindered A project's rationale, and the linkages from that rationale to the design, need to be clearly spelled out in an outcome-focused design, and the trail needs to be maintained through preparation, appraisal, and supervision, in the files and project documents. In other words, the logical integrity within the evolving project rationale and design should be sustained and recorded for management and for evaluation throughout the project cycle. Without this discipline, many sound judgments during preparation and appraisal are lost, staff changes may obliterate earlier design logic, management may be functioning partly blind, and clear decision turning points in design evolution are not spotted. Cases. For example, in the Kenya Forestry Development Project there was a reversal of a key privatization plan at negotiations and the rationale for the decision was not provided or recorded. Recording such fundamental shifts in project rationale reduces the chances that sound design logic will be lost under late pressure, keeps management informed so decisions can be made, and makes subsequent evaluation easier because the nodal points in the decision trail are clear. In the Bangladesh Agricultural Research Management Project carefully analyzed and well- articulated institutional reforms were developed early in preparation in consultation with the borrower but were progressively dropped later without clearly articulated and recorded justification. Such situations tend to be associated with lack of initial ownership and increased pressure to lend as appraisal and negotiations approach. The Albania Rural Development Project was a more positive example of retaining a clear logic in design. The Bank and the borrower carried through with different kinds of institutional reforms for rural infrastructure and rural finance that had been piloted in the previous Rural Poverty Alleviation Pilot Project. For rural infrastructure (a local public good), the Albanian Development Fund (ADF) supported the decentralized provision of rural infrastructure in partnership with local governments, local communities, and the commercial private sector. For small-scale rural credit (a private good), the ADF and its successor, the Rural Finance Fund, carrying the logic through facilitated the establishment of sustainable village-level financial institutions that are providing non-subsidized credit at the local level. As a direct result of the collapse of the pyramid schemes in 1996­97, the Bank and the borrower put in place a new legal and institutional framework for rural finance, separating the credit department from the rest of the ADF, and transforming it into a new and independent apex organization ­ the Rural Finance Fund ­ and then preparing two follow-on projects. The design logic was also extended. The Community Works Project (approved December 1998) is now supporting the infrastructure function of the ADF in both rural and urban areas, while the Microcredit Project (approved May 1999) is now supporting micro-finance in both rural and urban areas. The logic throughout is mostly well documented in the files and reports. Suggestions. Project Task Managers should ensure that all significant design issues raised from the first draft Initiating Project Brief onwards up to the final PAD are separately identified and carried forward at each step and are addressed or disposed of rationally and explicitly in each superseding document. Managers should keep track of the trail of issues by referring back to earlier documents. 7 (For example, in the Bangladesh case fundamental reform elements were allowed to drop without searching re-analysis or a recorded decision point. Consequently, the project reverted largely to more of the same and, as predicted by the earlier analysis, performed poorly.) * * * 8 Lesson 2 Weakness in the analysis of who is poor and why they are poor can dilute the pro-poor focus of a project Project design needs to enhance the pro-poor focus through better analysis of who is poor and why they are poor and improved logical framework links to specific policy and investment actions targeted at addressing the poverty diagnosis made.2 Cases. In Ghana a combined assessment of five rural sector projects noted that there were opportunities for enhancing the pro-poor focus of rural interventions in a number of areas, including research, extension support for farmer-based organizations, and livestock breeding. The assessment of the National Agricultural Extension Project in Ghana notes that incentives are needed to encourage extension agents to work with the poor. Rigid target-based approaches to achieve rapid adoption tend to work against a pro-poor strategy. The assessment of the Agricultural Diversification Project in Ghana notes that though the project was not directed toward the poorest, minor modifications in strategy could have significantly sharpened the poverty focus. The outgrower programs for tree and horticultural crops could have contributed significantly to poverty reduction, especially among small and marginal farmers. The challenge in such cases is to develop the project logic and design so that it provides for increased participation of the poorer households at several complementary levels, through a group approach and through focused extension and through attention to access to inputs and marketing channels suitable for the poor. In China, the Bank has provided valuable support for significantly reducing poverty levels. However, tackling the remaining poverty in the country remains a concern. A combined OED assessment of eight projects in five sectors (agriculture, forestry, health, education, and transport) noted that sharpening the focus on poverty reduction would require adjustments to targeting. Improving support for China's poverty reduction strategy rests not only on increasing attention to the nationally identified poor counties but also to improved targeting at the township level. On the borrower's side, there is need for greater government coordination of all Bank poverty-related efforts, be it in the context of multi-sectoral poverty reduction projects or under sectoral projects like the Hebei and Henan Agricultural Development Projects. Strengthening such coordination would help to mainstream poverty reduction activities within sectoral operations. In Henan and Hebei provinces, the Bank could have made more effort to coordinate and link the activities of the assessed projects to maximize the poverty impact. Weak coordination between different levels of government in China, across central bodies, and between various departments and agencies particularly in relation to poverty needs improvement, and experience of the assessed projects shows that the Bank was unable to engage China on this issue. In Nepal, the assessments of the Hill Community Forestry and Second Forestry Project show that although the forest area has been regenerated the benefits to the poorest have been limited. In some cases, while the community as a whole may have benefited, there is evidence that the condition of the poor actually may have worsened. Similarly, in India, under the Andhra Pradesh Forestry Project, the poorer herders of small stock, and illegal forest encroachers, appear to have suffered due to lack of a more explicit poverty focus. 2. See OED's 2001 Report "Toward Sharpening the Focus on Rural Poverty: A Review of World Bank Experience" where poor performance in rural pro-poor design is identified. 9 Poverty targeting objectives need to be made clear and reconciled with objectives to increase productivity. This was unclear in the Chile Small Farmer Services Project. The 2003 OED ECA study "Agriculture Policy Reform in the ECA Transition Economies 1991­ 2002" found that of the 20 investment projects with good indicators only three included measures directly relevant to poverty reduction. One of these was the Moldova Rural Investment and Services Project (FY02), which included a baseline survey and an indicator to capture the change in rural incomes in project areas relative to the baseline. Suggestions. The poorest are generally the least organized and least likely to be able to articulate their needs.3 In each sub-sector, intervention-specific attention needs to be given to opportunities to meet the needs of the poorest. The experience of both Ghana and China shows the need to coordinate agriculture, water, rural electrification, communications, rural roads, rural health, markets, and agricultural extension around a consciously directed cross-sectoral poverty alleviation objective. The following are some examples. Design of the rural finance system needs to ensure that adequate attention is given to the formation of, for example, savings and loan groups among the poorer households and villages as has been done in the Pakistan Aga Khan Rural Support Program (recently evaluated by OED). In setting outgrower selection policy, for example, in Africa in rubber and oil palm, where public support is provided selection should accommodate some poorer households, to the extent possible without seriously compromising productivity (see OED Ghana PPARs). In crops like rubber, coffee, pineapple, and fish, where the export potential is high, increased participation of the poorer households may be possible through a group approach, as is being done in Bangladesh through NGOs such as BRAC. In the livestock sector, support for breeding of small ruminants in substantial volume will be important for improving the conditions of the poorest who rely on small stock. In demand-driven small rural infrastructure projects, some element of proactively seeking out the poor may be needed. In community forestry projects adequate attention to the interests of the poor in benefit-sharing arrangements and in the process of forming executive committees for user groups may avoid their marginalization (discussion of this in the 2002 OED India Andhra Pradesh Forestry Project PPAR). More specifically, consider having a clear section in the PAD covering poverty ­ linked to the logical framework, indicating: 1. Who are the main categories of poor people in the project area, how many are there, and what are the main reasons for their poverty? 2. Who are the likely gainers and losers from the proposed project interventions? 3. What elements of project area selection are pro-poor (for example, are these the poorest districts/zones/villages)? 4. To what extent is each main investment selectively pro-poor either directly or indirectly, or is the intervention predominantly pursuing growth with the assumption that the poor will share fairly soon in that growth? (Entirely valid provided that assumption is analytically supported.) 5. The extent to which each main policy reform is selectively pro-poor either directly or indirectly with an indication of time-frames for poverty benefits. 6. The extent to which any processes, in particular, community participation processes, are expected to be pro-poor. 7. Whether there are any grievance mechanisms accessible to the poor. 8. What indicators will suggest the achievement of any pro-poor objective? (It is surprising how often OED finds objectives stated but no matching indicators.) * * * 3. See OED 2001 ­ Toward Sharpening the Focus on Rural Poverty: A Review of World Bank Experience. 10 Lesson 3 Monitoring and evaluation will be much more effective when they are integrated into project design at the outset Monitoring and evaluation should be an organic part of project design, not an afterthought. It can only be built on a foundation of outcome-focused design. As soon as project objectives are initially formulated, even tentatively, consideration should be given to targets that are realistic and indicators linked to the objectives that are readily quantifiable and to the initiation of M&E design. Cases. The success of the Tunisia North-West Mountainous Areas Development Project rests partly on the incorporation from the outset of sound monitoring and evaluation arrangements. A baseline survey of 527 households in the project area was conducted in 1996, with follow-up surveys in 1998 and 2000. This made it possible to assess changes in incomes (comparing "with" and "without" project households) and to examine the impact of the participatory approach employed. The project entailed extensive M&E training at the central and the local level, including development of geographic information systems (GIS) and management information systems (MIS). The approach has been replicated in other regions. The ICR rates the performance of the implementing agency, ODESYPANO, as highly satisfactory, noting the high quality of the supervision reports it prepared. The ICR and Evaluation Summary for the Pakistan Punjab Forest Sector Development Project found that weakness in M&E was due to the failure to design M&E during preparation, inappropriate institutional location of the M&E function, and lack of incentives to report, partly due to the institutional location issue. As a result, there was no effective MIS system until far too late, which inhibited management responsiveness. For example, by the end of the project, due to both lack of M&E data and failure to do marketing studies, there were serious questions about possible over- supply of wood and marketing challenges that could have been answered and responded to through project action sooner if better identified. In the Nepal Hill Community Forestry Project, the failure to develop an efficient monitoring and evaluation system contributed to an inefficient use of resources. The Evaluation Summary for the Tunisia Agriculture Sector Investment Project notes the need to strengthen monitoring capacity through training in the measurement of outcomes, not just inputs and outputs. In the Mozambique Rural Rehabilitation Pilot Project, a planned three-year pilot, ended up as a seven-year rural development operation. There was almost complete lack of M&E, a particular weakness given the nature of the project. Effective M&E should have been in place in order to learn lessons for future, larger-scale operations, particularly with respect to the cost-effectiveness of decentralized and community-based delivery systems. Suggestions. 1. Get the objectives stated right. Many are either too high and abstracted or too low and detailed. Aim for outcome-focused design. Look at ECA Latvia Health Project or Poland Rural Environmental Protection for well-conceived story lines. The recent ECA attention to outcome-focused project design is worth looking at more broadly. 2. Start early with the logical framework (as done in the Tunisia case). 11 3. Re-read Logical Framework Guidelines or attend training. Make sure the logical framework prepared is truly logical. Note that the framework format itself is a receptacle for the results of logical thinking, but does not require logical thinking to be completed, although it encourages it. One option is to sketch out the input/output/outcome linkages first in the form of a logical tree before completing the chart. (As noted above, a good example of a clear and monitorable story line is given in the logical framework of the Latvia Health Project.) 4. Initiate plans for baseline surveys during early preparation. If possible, embed M&E in a permanent institution or link it to one. (Not done in the Pakistan case cited.) 5. Where possible, build sustainable evaluation capacity not just a one-off, project-specific M&E system. (OED has been working at different levels of intensity to support evaluation capacity development in Brazil, Egypt, Ghana, Honduras, Kyrgiz Republic, Malawi, Mexico, Mozambique, Poland, Sri Lanka, Tanzania, Uganda.) 6. Consider looking at the incentives for performance in M&E of different stakeholders. (In the India Uttar Pradesh Sodic Lands Reclamation Project, M&E was strong due to commitment at the project management level resulting in skilled committed staff and rapid feedback to sustain a challenging new program needing responsive management adjustment. In some other projects in India there was little management interest in M&E and M&E staff were mostly weak, temporary, and unmotivated.) Try to design the project to develop and link incentives to any results-based management-type process that is developing at higher national levels. 7. Try to ensure that M&E staff are recognized professionally and within the institution and/or civil service and not some temporary position from which they aim to escape as soon as possible. 8. Coordinate M&E methods and reporting formats with other donors. Some progress in this direction was made by the multi-donor supported Pakistan Aga Khan Rural Support Program, but the different demands remained a burden. The recent renewed attention to donor coordination may be an opportunity. * * * 12 Lesson 4 Designing not just the substance but the phased implementation actions involved in achieving policy and institutional reforms is essential to achieve both action and support for reforms Detailing the process by which policy and institutional reforms are to be achieved and being realistic about the amount of time required to implement them given the need for sequencing are as important as detailing the substance of the reforms. Ensuring action, not just rhetoric, is important for maintaining momentum. Cases. In its 2003 study "Agriculture Policy Reform in the ECA Transition Economies, 1991­2002: An Assessment of the World Bank's Approach," OED found that most practitioners believed the approach should be "to work simultaneously with several complementary instruments: using analytic work and regular conversation with diverse interest groups to build support for reform; employing technical assistance to work out the legal and regulatory details; and offering investment loans...as carrots to entice the skeptics.... Staff particularly emphasize the need for well supervised and well coordinated technical assistance: this is one of the lessons learned that is most frequently cited." In the Bulgaria Agriculture Sector Adjustment Loan, the Bank helped build and sustain a consensus for reform by conducting high-quality economic and sector work, publishing the findings in a series of "Agricultural Policy Notes," and extensively discussing these findings with the government and other domestic stakeholders in two sets of workshops. Once the broad outlines of the project were agreed, Bank staff continued to assist with the implementation of the agreed-upon policy conditions. While the government took longer than expected to pass certain pieces of legislation in order to meet these conditions, there has been little backtracking. The Minister of Agriculture himself played a major role in the reform process, articulating a vision for the agricultural sector, building consensus each step of the way, and engaging stakeholders in both the broad outlines and the implementation details. In the Ukraine Agriculture Sector Adjustment Loan, a less positive case, there was limited political consensus on the reforms and little domestic ownership of them. A major problem between the first and second tranches was the lack of a credible, representative forum in which the key domestic stakeholders could reach a workable consensus on each implementation step of the reform process. Eventually, the Inter-Ministerial Commission for Agrarian Reform, which included significant representation from nongovernmental stakeholders as well as all levels of government, came to play this role. In the Zambia Agricultural Sector Investment Project, the restructuring of the line agency took much longer than anticipated, resulting in a number of donors opting out, partly because known procedural steps for the restructuring had not been clearly set out in a projected timetable. More generally, OED's 2003 study "Agriculture Policy Reform in the ECA Transition Economies, 1991­2002: An Assessment of the World Bank's Approach" found the ECA Region to have a record of rich and varied sector work with a marked problem-solving, practical orientation. QAG rated the FY00 Belarus Farm Restructuring Study highly satisfactory. Suggestions. Reforms typically involve changes in laws and regulations (requiring legislative and Cabinet approval), new governance and administrative structures and procedures, financial arrangements, and capacity building (e.g., Bulgaria). Those who are managing reform processes day- to-day need political commitment and support from politicians and senior policy-makers. Reform managers, with the support of Bank staff, must be assisted to work constructively with the major stakeholders which, depending on the situation, can include all levels of government, civil society 13 organizations (including NGOs and community-based organizations), and the private sector. As in Bulgaria, good communications strategies are needed, and the reforms need to be approached in a logical sequence planned up front. Technical assistance can help. In ECA, technical assistance, notwithstanding doubts expressed about its effectiveness elsewhere, was rated as very important by staff. Such reforms should be expected to be a slow and tortuous process. In the ECA study a much- cited lesson learned was "that institutional development takes a long time ­ much longer than was originally expected." However, what OED often sees is not delays in sectoral reform implementation that were unpredictable. Often the time taken was largely predictable within a range but was not, in fact, predicted because the steps were not set out in detail and timed and phased realistically (e.g., Zambia ASIP). The tools are available. Some sort of scheduling, or computerized project management (CPM) program, could be used for such reform projections just as well as it can be (and is) used for project investment or construction scheduling. This would set out flow charts as a guide map showing the sequence, timing, and realistic length of each reform implementation step many of which are long-established government procedures of known periodicity. * * * 14 Lesson 5 If the incentives created for stakeholders by new policies and institutional arrangements are not thoroughly analyzed the desired impact may fail or take longer than expected to realize There should be a thorough analysis of the incentives that are created by any new "rules of the game" for the various individual and institutional stakeholders and the time needed for these to have impact. Creating capacity at all levels, especially at the community level, takes longer than is usually provided for. Cases. The two India Integrated Watershed Management Projects (Hills and Plains) addressed one of India's most serious environmental problems ­ watershed degradation in rain-fed agricultural areas. Achieving the two objectives of reversing environmental degradation and promoting sustainable production systems with quick realization of benefits by the poor for agricultural, livestock, and forest products required both an integrated approach and a participatory focus among project staff. Staff from different government departments (in this case, soil conservation, forestry, agriculture, horticulture, and animal husbandry) had to work together in an integrated way among themselves and in a participatory way with village-level project beneficiaries in both the planning and implementation of watershed investments. Two results of the participatory focus were a greater emphasis on managing the supply of water in each sub-watershed, and flexibly modifying the standard technical solutions where necessary to generate more immediate incentives for the beneficiaries (for example, harvestable vegetative soil and moisture conservation barriers suitable for livestock rather than simply effective erosion barriers but of limited fodder value). OED also concluded that the ongoing provision, management, and maintenance of public infrastructure investments and of commonly-held forests, plantations, and pastures required the establishment of more permanent institutional arrangements beyond the life of the two projects among line departments of state governments, local governments, local communities, and individual farmers which created incentives for each of these four groups to fulfill their respective roles and responsibilities. In these projects there have been differences of incentives for participation between men and women and between poor and less poor, not always fully resolved. Institutional arrangements for agriculture and rural development typically involve four sets of actors who all face different sets of incentives which are amenable to analysis: (1) central governments and agencies, (2) local governments, (3) community-level organizations, and (4) the commercial private sector, including individual farmers. Many incentive shifts within institutional reform programs take more time than expected. At the central government level, key functions include new laws and regulations, nationwide coordination, building capacity, and financial transfers to lower levels of government. In the Ukraine Agricultural Seed Development Project the lack of government commitment and ownership of the restructuring efforts in this area led to an unsatisfactory outcome. In the Madagascar Antananarivo Plain Development Project it was unrealistic to expect the far-reaching institutional reforms to take root in the limited period of one project. At local government level, decentralizing the provision of selected public services to that level involves two major institutional challenges. The first is to establish a national framework for decentralization involving an appropriate mixture of administrative, political, and fiscal decentralization. The second is to establish service-level institutional arrangements (among local governments, central agencies, local communities, and the private sector) for each of the services 15 assigned to local governments, be it water and sanitation, local roads and footpaths, basic education, basic health services, or agricultural extension. The Brazil CDD projects have spanned almost 20 years and 4 project cycles. It took time to get them right. Incentives at community level are needed to make rural service providers accountable to farmers. In the Indonesia Village Infrastructure Project, building capacity at village level was particularly important in making villagers more able to guide development and exert pressure on local governments. Even with the innovative, participatory, and farmer-centered methodological approach in the Sri Lanka Extension Project, there was still a lack of incentives for service providers to perform. At the community level, creating and sustaining incentives and stabilizing expectations takes time. There appear to be no projects where community organizations were established from scratch that have reached sustainability within the life of a single Bank project. NGOs do not seem to find it easier. The Aga Khan Rural Support Program in Pakistan4 took about 18 years to build up to close to 100 percent village coverage in the Northern Areas and even then with questionable sustainability in a number of the less capable villages. As found in the Bangladesh Forest Resources Management Project, setting up grass-roots organizations is just the first step. More time and resources are needed to strengthen and consolidate gains so that their sustainability over the long run is assured. An OED assessment found that the Pakistan Northern Resources Management Project closed with limited support follow-up, just as community organizations were getting established. The Evaluation Summary of the Madagascar Second Irrigation Project concluded: (1) that if a participatory approach is to be taken seriously, both the government and the Bank must be willing to be more flexible about project duration and not focus simply on short-term targets rather than longer-term sustainability; (2) that global knowledge about participatory processes is evolving and the Bank should not rely on the executing agencies to keep up with the latest developments ­ a Knowledge Bank plea. Finally, related particularly to the community level, having carried out a recent 2002 Social Funds Study, OED is now embarking on a Community Driven Development (CDD) Study to probe Bank performance in this area further. With respect to the private sector, successful liberalization and privatization of commercial activities may require capacity building and transitional arrangements as circumstances, capacity, and incentives evolve. In the Yemen Fourth Fisheries Development Project, new fish handling facilities were successfully privatized in stages first by renting the facilities to local fish cooperatives and then by selling them after the cooperatives had demonstrated their financial and technical capacity to operate and manage them. In the Ukraine Agriculture Sector Adjustment Loan, the lack of a credible government commitment to maintaining the policy reforms inhibited the private sector development of agricultural input marketing. In the case of Chile Small Farmer Services Project, incentives for farmer graduation were simply not there. Interestingly, in this case it was found that Bank staff themselves were not able to agree about the feasibility of graduating small farmers from public to private extension. Suggestions. Step 1 is for the Bank, the borrower, and other donors to agree upon an appropriate institutional reform strategy, given the existing degree of political and economic development in the country and a careful analysis of the public good elements. Fully understanding the extent of the public good element is important. (OED finds this is often weakly addressed in Agricultural Research Projects.) Different strategic approaches tend to be associated with different sub-sectoral areas. There is an approximate typology. Liberalization and privatization tend to be associated with agricultural 4. See OED Report: The Next Ascent ­ An Evaluation of the Aga Khan Rural Support Program, Pakistan. 16 marketing, rural finance, and land markets. Community-based approaches tend to be associated with managing renewable natural resources such as pastures, forests, and fisheries. Decentralization tends to be associated with rural infrastructure. Restructuring central government agencies tend to be associated with research, extension, and livestock services. Step 2 is to involve the participation of all legitimate stakeholder groups in the reform process in order to establish the credibility of the government with respect to the proposed reforms, to help assemble relevant information, and help instill ownership in the reforms (done well in the Bulgaria case). Working effectively with stakeholder groups is an art and not a science. One should work particularly closely with the potential losers who have influence to undermine the reform process, since even reforms that are win-win in the long run generally have losers in the short term. A revival in the use of farm budget analysis to assess minimum private financial incentives for income-earning interventions is needed to better understand farm level incentives. To understand incentives and issues within existing institutions, an Institutional Survey may be useful (see one example, including an Annex of survey questions and analytical tabulation, in OED's 2002 Pakistan Aga Khan Rural Support Program Evaluation). A quite useful reference, including an Annex of references, is UNDP's Technical Advisory Paper No. 3, "Capacity Assessment and Development ­ In a Systems and Strategic Management Context" - available on the Internet (magnet.undp.org). There are others. * * * 17 Lesson 6 Effective donor coordination depends on a project or program design that is based on a clear understanding of respective roles and responsibilities Project design should be based on a clear understanding of the roles, responsibilities, and procedural constraints of the various partners; without this understanding the leverage and broader development experience that may be obtained from effective donor coordination will never be realized. Cases. In Nepal forestry, OED concluded that the various donor's agendas, rather than a coordinated and well-planned strategic approach, had been the deciding factor in forestry interventions. Individual donors have supported different aspects of the community forestry program in Nepal depending on their individual priorities. In addition, individual donors have also preferred working in particular districts. Both these aspects have put a tremendous coordination burden on the government and stretched its limited institutional capacity. Lack of donor coordination has also meant that available resources have not been used as efficiently as they could have been. Diverse implementation strategies in different districts according to specific donor requirements have often created confusion, especially among field staff who are frequently transferred between districts. The uncoordinated approach has also resulted in some important institutional issues receiving inadequate attention ­ for example, despite the presence of numerous donors, support for research to address specific problems emerging from field implementation has fallen between the cracks. In the Benin Rural Water Supply and Sanitation Project, the satisfactory outcome of the program was partly a result of well-coordinated support by a coalition of external assistance agencies. In countries where the government's capacity is weak, the lack of donor coordination may be seriously undermining development progress. For example, in Lao PDR, donors have tended to set up enclave operations, such as those in research and extension, but there has been no concerted effort to build a national research and extension service capable of serving the long-term needs of the agriculture and forestry sectors. In a number of countries donors have been observed to be pushing their own currently fashionable extension techniques in enclave projects, as did the Bank with train and visit (T&V). A positive example of donor coordination was in the Malawi Fisheries Development Project, where partnerships with bilateral development agencies, who had a comparative advantage in many areas, was successful because of a clear definition of roles and carefully structured procurement responsibility. Procurement practice differences are a frequent issue. This should be addressed to the extent possible. In Ghana, there was evidence in a five-project OED assessment of weakness in the rural elements of the CDF process with a number of donors supporting various rural interventions but not fully in agreement with the strategic elements. For example, many donors agreed on the importance of extension but differed on the right extension strategy. In the Zambia Agricultural Sector Investment Project donors appear to have been inadequately convinced of the sector-wide approach and walked away very soon after effectiveness. In this case, notwithstanding agreement on broad sector objectives, there appear to have been uncertainties about what was expected of each donor and what precisely was meant by "implementation within the institutional framework of the ministry," by "supporting and cofinancing," and by "standardization of procedures." Initially, procurement and disbursement procedures were to be substantially harmonized, but by appraisal this had become "to the extent possible," and by the first year of implementation it was clear that nearly all donors took the view that harmonization was fine as long as their own procedures remained intact! The Laos 2001 Review of Production Forestry co-sponsored by the Bank with two partners is a good example of the potential for sound analytical work in partnership providing an excellent platform for 18 dialogue. In the Yemen Fourth Fisheries Development Project, effective donor coordination was crucial to project survival (when the executing ministry replaced the entire staff of the project implementation unit) and eventual success. In this case, the Bank, EU, and IFAD developed a common approach to policy matters and implementation that enhanced donor leverage on supervision, especially since EU funds were grant financed. Suggestions. The following appear to be some important ingredients: 1. Willingness to really listen to donors and cultivate close relationships but without ganging up on government. (This was done well in Malawi Fisheries.) What OED observes often is that what is said formally by donors at public meetings may not always reflect quite what is felt in private. (This seemed to be the case in Zambia ASIP.) Personal relationships at the country level with the country office have often been observed to be important but there is also evidence that this needs to be tempered with the knowledge that major policies and decisions come from the donor headquarters. 2. Plenty of time to build trust if the coordination attempt is new. OED has seen a number of cases where donors felt rushed to agree or to commit. This has been observed even in cases where project preparation took a long time. If donors feel rushed, they are rushed. This was clearly articulated to OED by other donors in relation to the Zambia ASIP mentioned above and it resulted in many donors pulling out very quickly when things started to go wrong. 3. Clear definition of what elements are being coordinated since each may present different problems for different donors (for example, coordination in the areas of overall strategy, procurement, disbursement, extension or research approach, phasing, follow-on commitment, community processes, financial management, reporting, subsidies or incentive payments) and realism about the limits to donor's procedural, especially procurement, flexibility. (This was done quite well in Benin Rural Water and Sanitation.) 4. Willingness to let other donors chair coordination efforts particularly where this may reduce risk of perceived Bank heavy-handedness or improve ability to read donor attitudes. OED is currently engaged in an Aid Coordination Study that should throw further light on this issue and will look at aid coordination also within the framework of the CDD study mentioned above. * * * 19 Lesson 7 Financial projections in project design can help ensure the financial sustainability of a project Project design needs to incorporate certain specific analysis and actions to enhance financial sustainability. Cases. The Staff Appraisal Report for the Bangladesh Agricultural Research Management Project, despite two earlier projects with serious financial sustainability issues, was found to have no analysis of financial sustainability. The SAR simply refers briefly to the risk of inadequate counterpart funding and notes that government agreed at negotiations that adequate funds would be made available for the project ­ the largely boiler-plate statement found in many SARs/PADs. The post-project scenario for funding was not discussed or analyzed in relation to past budgetary allocations. In five Ghana projects, counterpart funding was a persistent problem but had not been adequately covered during project preparation and appraisal, nor had the anticipated total budgetary demands of all Bank and donor projects together been factored in. The eventual local funding problem was foreseeable. Also in Bangladesh, in the Bangladesh Second Small Scale Flood Control, Drainage and Irrigation Project, counterpart funding became such a problem that the Bank recommended curtailing the project scope and threatened cancellation. With this, government scraped together some more funds, but since the project had not been able to address the broader operation and maintenance issue and cost recovery, this apparent improvement in counterpart funding probably simply pulled funds from some other important area. It seemed clear that the scale of the lending program in that sector as a whole had enormously over-stretched the government's capacity and that this had not been adequately projected during preparation. In the Niger Small Rural Operations Project, beneficiaries had to make annual contributions, when a small rural operation was financed, to a capital replacement account to maintain and replace machinery and equipment and a community development fund to finance collective projects for the benefit of the whole community. But these accounts had no legal status and therefore no legally binding mechanism to ensure that these obligations would continue to be fulfilled, especially after project completion when civil servants quit monitoring the state of the accounts. Suggestions. A PAD should provide at least the following: (1) a table projecting local funding needs for the project period and at least five years beyond and including other significant incremental projects in the sector ­ either Bank or other donors (partly done in the case of Punjab Forestry); (2) a table of the past five years of budgetary allocation to that sector or sub-sector (hardly ever attempted); (3) analysis of any gap between these two ­ that is, between the demand for funds and a reasonable expectation of the supply of funds (not possible in the absence of 2 and 3); (5) supporting evidence of changed government priorities for sectoral budgetary allocations if budget demand is shown to go significantly beyond past actuals (sometimes one or two qualitative statements); (6) a substantive discussion on the implications of the data for financial sustainability and an outline of how it has affected project design, for example, toward reduced scale, increased cost recovery, phasing, retrenchment, etc. (sometimes covered in Social Funds and Water Supply and Sanitation Projects shifting to privatization and cost recovery). There are no projects evaluated over the most recent two years in the period considered that fully meet all the above. The Pakistan Punjab Forest Sector Development Project appraisal report achieved a portion of what is needed and, perhaps significantly, the eventual ICR also did an above average job on plans for sustaining the operational phase after project closing, usually barely addressed at all in ICRs. Projects moving toward privatization of services such as Water Supply and Sanitation Projects sometimes cover financial sustainability better 20 than most since this is the heart of the project reform. The Jordan Agricultural Sector Adjustment Loan also did substantial financial analysis due to the national water cost recovery issue. However, while designing the project to be within the financial capacity of the borrower is important, simplistic budget-cutting solutions such as line agency staff reductions in order to release funds for operating costs need to be thought through carefully in the context of overall national budget management and civil service reform. In the Kenya forestry projects reviewed by OED, both the Bank and the line agency naively assumed that if substantial staff reductions were made more funds would be available for operating costs. But since line agency budgetary allocations by the Ministry of Finance were largely still related to staff numbers, after staff reductions were implemented under the project the operating cost budget was simply proportionally cut, giving no increase in the percentage of operating costs. * * *