ICRR 12708 Report Number : ICRR12708 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 08/20/2007 PROJ ID : P081773 Appraisal Actual Project Name : Emergency Drought US$M ): Project Costs (US$M): 61.7 60.3 Recovery Project Country : Ethiopia Loan /Credit (US$M): Loan/ US$M ): 60.0 60.0 Sector Board : RDV Cofinancing (US$M ): US$M): 0 0 Sector (s): General agriculture fishing and forestry sector (40%) Other social services (20%) Agricultural marketing and trade (20%) General water sanitation and flood protection sector (15%) General education sector (5%) Theme (s): Natural disaster management (25% - P) Nutrition and food security (25% - P) Social safety nets (24% - P) Vulnerability assessment and monitoring (13% - S) Rural services and infrastructure (13% - S) L/C Number : CH028 Board Approval Date : 03/27/2003 Partners involved : None Closing Date : 12/31/2006 12/31/2006 Evaluator : Panel Reviewer : Group Manager : Group : Silke Heuser Ronald S. Parker Alain A. Barbu IEGSG 2. Project Objectives and Components: a. Objectives: According to the Development Grant Agreement (Schedule 2), the objectives of the project were (i) to help affected families survive the emergency, retain productive assets, and develop sustainable livelihoods; (ii) to put in place infrastructure and support mechanisms to mitigate the medium -term effects of current and future drought emergencies on vulnerable groups; and (iii) to stabilize the macroeconomy. The MOP did not include the second of the three objectives. Since the ICR bases its analysis on the reduced list of objectives of the MOP, some activities undertaken by the project do not seem to correspond to objectives . Therefore, this review (with the Group Manager's approval) bases its analysis on the more comprehensive list of objectives as presented in the Development Grant Agreement. The project was prepared in the context of a Joint Government – UN Appeal launched in December 2002 to avert the humanitarian crisis brought about by a gradually deepening drought in the historically vulnerable areas of Ethiopia . 11.3 million people were estimated to be suffering from severe food insecurity , with another 3 million regarded as vulnerable if the February-March 2003 rains were to fail. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): The project consisted of three components : (i) Quick Disbursing Assistance (estimated cost US$43.5 million; actual cost US$43.5 million): This component was to finance essential imports from a defined list by providing additional foreign exchange . According to the Technical Annex, the critical imports were expected to support recovery of the economy and the agriculture sector in particular, while the provision of a foreign exchange grant would help keep both the trade and fiscal deficits down . (ii) Community Interventions (estimated cost US$16.5 million; actual cost US$15.00 million): This component was to finance three kinds of activities : (a) public works programs to improve coping capacities of affected populations and to create/rehabilitate productive assets (community assets and government owned /managed infrastructure); (b) targeted nutrition (specifically, inclusion of supplementary enriched food aimed at small children and pregnant and lactating mothers); and (c) school attendance (additional food allowance to targeted girl pupils attending school ). (iii) Project Management (estimated cost US$1.7 million; actual cost US$1.80 million). d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Delays occurred in the beginning of the project with respect to community -driven implementation arrangements. In addition, at project closure (on December 31, 2006), 114 sub-projects had been initiated but not yet completed . The Government has assured the Bank that these sub -projects will be completed utilizing resources from other sources as appropriate. However, according to the ICR, information on the current status of these sub -projects including actual availability of resources and the expected timeline for their completion is not available . Project records suggest that 99.93 % of the funds were disbursed . Undisbursed funds amount to US$ 47,609.81, which will be cancelled. 3. Relevance of Objectives & Design: The relevance of the project objectives is substantial, since the Bank's FY 2006-2007 Interim Country Assistance Strategy (ICAS) emphasizes broad-based growth and managing vulnerabilities . Project objectives were also in line with the Bank’s Country Assistance Strategy (CAS) for Ethiopia for 2003-06, which was endorsed two weeks after the EDRP was approved by the Board. In particular, the EDRP was consistent with the third pillar of the CAS, which explicitly referred to the need to provide safety nets to vulnerable households with a view to helping them sustain and build assets. The relevance of project design is substantial, since it stressed not just the need to deal with food insecurity but also to improve longer-term coping capacities, especially through the build -up of productive assets and sustainable livelihoods. According to the ICR, the project included the following innovative design features : (i) a blend of various kinds of support from macro-stabilization to targeted household -level assistance, to help in a broad -based recovery from the crisis; (ii) provision of cash for work scheme (as opposed to only food rations ) for protecting private assets; (iii) cash allowances in community sub -projects for the purchase of complementary inputs beyond community labor (e.g., materials and technical expertise ), in order to improve the relevance and effectiveness of public works carried out; and (iv) involvement of the communities in selection, implementation and management of sub -projects. Several of these features and lessons were subsequently utilized to inform the design of the much larger ongoing national Productive Safety Net Program (PSNP). The ICR rightly mentions flaws in the design and implementation of the "results framework": There was a gap between objectives and their indicators (including the fact that the indicators were not sufficiently specific or measurable and did not have identified pre - and post-project levels which could be used to calibrate implementation success). 4. Achievement of Objectives (Efficacy): (i) To help affected families survive the emergency, retain productive assets, and develop sustainable livelihoods : rated Modest . In the few instances where the targeted nutrition program (about 3.6% of the total community interventions component) was implemented, it reportedly took the form of cash transfers to help selected families purchase additional food. Overall, the expected implementation mechanism, under which Disaster Prevention and Preparedness Commission would receive supplementary food supplies at the federal level and distribute it at the local level along with normal food deliveries, did not work . Similarly, the school attendance program, meant to use food or cash transfers to help maintain school attendance of girl pupils during the drought, was implemented in only a few districts. The project generated direct employment for over 900,000 individuals (who, according to an end-user survey, earned an average of 340 Birr (US$40.00) from working on a sub-project). This was no doubt an important source of economic support, in part because it likely allowed the families to maintain (or reduce destocking of) their livestock. However, although the project created employment, it did not have a dedicated intervention for livelihood development. Therefore, many of the sub-projects with respect to soil and water conservation (described below) are likely to produce more general environmental benefits and strengthen future drought resilience rather than directly enhance livelihood prospects . (ii) To put in place infrastructure and support mechanisms to mitigate the medium term effects of current and future drought emergencies on vulnerable groups : rated Substantial . On the community assets side, there was a large range and number of physical achievements . Achievements in water development include 327 ponds, 127 micro-irrigation systems, 3 pump irrigation systems, 332 springs (development and maintenance), 119 hand dug wells, and 22 roof water harvesting structures . Achievements in soil and water harvesting included construction of bunds and terraces, check dams, cut -off drains and waterways as well as range management and 13.4 million of tree seedlings planted. In addition, this component financed construction and maintenance of 2,940 km of roads, and rehabilitated 368 schools and 88 health posts. Consistent with the community-driven nature of these sub -projects, it had not been feasible to define ex -ante specific end-of-project target values; hence, that yardstick is unavailable to measure project success . As explained above, because of the community-driven implementation arrangements, delays occurred in the beginning of the project . The flip side of the implementation delay was a process of learning that has allowed a more participatory and community -driven implementation process to be followed . Some available evidence – which is regrettably patchy and unsystematic owing to weaknesses in M&E – suggests that this led to improvements in sub -project identification and formulation (these are more locally relevant ), greater learning and ownership during project implementation, and emergence of various O&M arrangements which show greater concern for sustainability . (iv) To contribute to the stabilization of the macroeconomy : rated Modest . Combined donor support during project implementation almost certainly contributed to short -term macroeconomic stabilization. The inflation rate declined from 15.1% in 2002/03 to 8.6% in 2003/04, with an even more marked fall in the food inflation rate from 24.8% in 2002/03 to 11.8% in 2003/04. However, as the ICR rightly mentions, it is not possible to isolate and assess the impact of the quick disbursing component of this project on macroeconomic stabilization. This project's resources were part of a larger stabilization support offered by donors and, while helpful, may not have been decisive in terms of determining macroeconomic stability . 5. Efficiency (not applicable to DPLs): N/a ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Through the quick-disbursing component, the project helped GOE meet the livelihood -related needs of the drought-affected rural population. The community sub-projects not only created employment for more than 900,000 individuals, but the additional income also prevented families from selling off their livestock . In addition, community sub-projects in water supply and soil preservation helped mitigate the medium -term effects of current and future drought emergencies on vulnerable groups . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: In accordance with the ICR, this review assesses the risk to development outcome as moderate for the following reasons: The rural social services (schools, clinics) rehabilitated during the project should not experience significant problems of sustainability, as these are managed by government institutions . Community water supply schemes have generated significant positive welfare effects and hence there are likely to be a strong incentives at the community level to maintain and manage them . The public works executed are by and large technically simple in nature . Hence their operations and maintenance is unlikely to require skills beyond the beneficiary community . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: The Bank reacted with commendable speed to the Joint Government – UN appeal, and helped steer this operation from design stage to Board approval in about five months . Project design built on earlier lessons, provided timely and large scale non -food assistance, and sought to build on activities already underway . The project also introduced sensible innovations in design to better address the underlying causes of chronic food insecurity and vulnerability. However, as explained under 3. and 10. of this review, the M&E system was flawed and Bank supervision did not pay much attention to M&E during project implementation . at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Unsatisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: Overall, against the backdrop of an ongoing emergency, the Government put together a highly relevant project which achieved its outcomes moderately satisfactorily . Although the Government rightly pressed for greater devolution in implementation arrangements, the coordination arrangements at the center (especially the PMU) appear not to have been vested with sufficient oversight authority to guide project implementation effectively . Similarly, the national steering committee, formed to help coordinate interagency actions with respect to the targeted nutrition and school attendance sub -components, seemed unable to provide the overall coordination and guidance that was required . The PMU was not fully staffed all the time, and dedicated project coordinators for the six regions were appointed only around MTR . At the district level, there were significant time lags between approval and funding of projects, weaknesses and delays in financial reporting and auditing, and the de facto non -implementation of the agreed M&E system which, in turn, has precluded a more careful assessment of the actual project impact on the ground . In addition, while recognizing that this appears to be a systemic problem, the failure to adhere to environmental safeguards screening procedures is a matter of concern . Taken together, these shortcomings compromised the quality of program implementation. a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Moderately Unsatisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: The M&E system of this project showed weaknesses in design and implementation, adversely affecting the ability to evaluate and learn from the project experience . At the design level, there were weaknesses with the “results framework� in the sense of objectives defined in broad, non -operational terms, lack of sufficient congruence between project components/activities and the development objectives, and a gap between objectives and their indicators (including the fact that the indicators were not sufficiently specific or measurable and did not have identified pre- and post-project levels which could be used to calibrate implementation success ). During implementation, there was no collection of relevant process, cost, and output /impact data at regular intervals. Consequently, it was not easy to calibrate implementation progress and make mid -course corrections in light of the perceived problems and difficulties . The project, however, did conduct an end -user evaluation survey in October 2006, which currently is the main source of information about project impacts on the ground . a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): The ERP triggered the safeguard OP /BP 4.01 on environment. The project has been classified as EA Category B as the community grant component envisaged the construction of small rural infrastructure . A Resettlement Policy Framework (RPF) and an Environmental and Social Management Framework (ESMF) were put in place in May 2004, one year after project effectiveness . These included detailed procedures and community sub -project screening forms. The PMU further provided training sessions on the use of these forms and the RPF and ESMF procedures to regional and district level officials involved in EDRP implementation . The PMU also recruited an engineer whose tasks included the provision of safeguards monitoring and support . However, it appears that the safeguards screening forms were utilized only in some areas by district officials and communities when designing and appraising community project proposals. This appears to be a systematic problem with respect to district -level activities, and has also been identified as a concern in other similar World Bank supported projects in Ethiopia . In addition, the ESMF for the EDRP indicates that “under EDRP, there will be no funding for the construction or rehabilitation of dams.� However, three small masonry dams were constructed in the East Belessa districts of the Amhara Region at a cost exceeding US$ 100,000 for each. Relevant documentation concerning construction of these dams (e.g. documents showing utilization of generic dam safety measures designed by qualified engineers in line with OP 4.37, environmental impact assessment, or indeed financial /economic analysis of these sub -projects) were not available. 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Innovation in emergency projects : It is essential to ensure that implementation arrangements for emergency operations allow for a quick start -up. If innovative elements in design or implementation are introduced, the option of phasing them in should be seriously considered . Project Development Objectives : A clear “results framework� (scope and specificity of objectives, congruence between project interventions and objectives, links between objectives and indicators, and the measurability of indicators) is critical to facilitate effective project implementation, monitoring, and evaluation . Community-driven implementation approach : This has helped in significantly building up local community capacities for project identification, planning, implementation (including procurement and financial management), as well as ex-post operation and maintenance . It has also enhanced the relationship between communities and local authorities. Use of cash transfers in workfare programs : Project experience confirms that the provision of cash transfers (as opposed to only food rations ) in workfare schemes is helpful for building /protecting private assets . Further, allowing for cash to be used to purchase complementary inputs (e.g. materials and technical expertise ) helps improve the quality, relevance, and effectiveness of public works carried out . 14. Assessment Recommended? Yes No Why? This project implemented a great number of community -driven water supply and soil preservation projects . An assessment is recommended, especially in the context of the upcoming water study . A future PPAR should pay special attention to dam safety and inspect the three dams paying special attention to potential negative environmental impacts. 15. Comments on Quality of ICR: The ICR complies with the applicable guidelines and is internally consistent . The author's choice to go with the project objectives stated in the MOP made the task of explaining achievements of objectives more difficult than had he chosen to go with the legally binding objectives in the Development Grant Agreement . Therefore, as explained in section 2a, this review chose the Development Grant Agreement as a baseline for its assessment . a.Quality of ICR Rating : Satisfactory