ICRR 14267 Report Number : ICRR14267 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 06/12/2014 Country : Madagascar Project ID : P051922 Appraisal Actual Project Name : Rural Development US$M ): Project Costs (US$M): 142.19 114.03 Support Project L/C Number : C3524; C4525 Loan /Credit (US$M): Loan/ US$M ): 119.05 108.63 Sector Board : Agriculture and Rural US$M): Cofinancing (US$M ): Development Cofinanciers : Board Approval Date : 06/19/2001 Closing Date : 06/30/2007 12/31/2012 Sector (s): Other social services (27%); Agro-industry (21%); Irrigation and drainage (21%); Central government administration (18%); Agricultural extension and research (13%) Theme (s): Other rural development (40% - P); Improving labor markets (40% - P); Participation and civic engagement (20% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Ebru Karamete Alain A. Barbu Christopher David IEGPS1 Nelson 2. Project Objectives and Components: a. Objectives: The project development objectives stated in the Development Credit Agreement (p. 13) was " to contribute to the implementation of the Borrower’s Rural Development Action Plan(PADR) and selected elements of the Borrower ’s Regional Rural Development Programs (PRDR) by fostering development of revenue increasing activities, thus increasing incomes and reducing poverty in rural areas, while preserving the natural resources base . The Project Appraisal Document statement of project development objectives (p. 2) was to "to increase incomes and reduce poverty in rural areas, while preserving the natural resource base . The project is part of the Rural Development Action Plan (Plan d'Action pour le Developpement Rural - PADR), a broad-based program approved by the Government in 1999 to promote sustainable growth in agricultural production, foster food security and enhance access to basic services in the rural areas . The project would support demand -driven activities in agricultural production and technology transfer, and strengthen capacity at national, regional and community levels ". This Review uses the Development Credit Agreement version . b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 10/21/2008 c. Components: 1. Productive Investments (appraisal estimate US$ 62.33 million, actual US$ 68.09 million; additional financing estimate US$ 28.20 million and actual additional financing US$ 19.23 million) This component aimed to finance about 1,000 matching grants per year for demand driven small scale productive infrastructure, agricultural production and off -farm investments in rural areas. Additional financing was designed to scale up project activities and investments on small scale socio -economic infrastructure and services, with the goal of increasing the impact of the operation . Additional financing targeted an additional 70,000 families on top of the estimated target of 180,000 families of the original project. 2. Support Services (appraisal estimate US$ 11.60 million, actual US$ 2.30 million; additional financing estimate US$ 1.50 million and actual additional financing US$ 0.22 million) Component 2 aimed to finance extension services, technical support and training to build capacity of farmers and producers' organizations on implementation of demand driven investments . Also, it aimed to finance agricultural research through a competitive grant program as well as research . This component was integrated with Component 1 as a result of 2002 project restructuring. Outreach efforts and sponsored research activities were consolidated under this component. 3. Community Development (appraisal estimate US$ 6.00 million, actual US$0 million) This component originally aimed to finance preparation of business plans, development of organizational and managerial capacities of producer organizations and events for promoting beneficiaries' participation . However, in 2002 restructuring, this Component was also folded into Component 1. 4. Capacity Building and Policy Development (appraisal estimate US$ 5.03 million, actual US$ 2.52 million; additional financing estimate US$ 2.10 million and actual additional financing US$ 0.55 million) This component aimed to finance technical assistance and training in order to build capacity in implementing agencies to reinforce Rural development Action Plan process at national and regional levels, to formulate and analyze rural policy and to establish agricultural statistical systems . 5. Project Administration and Monitoring (appraisal estimate US$ 10.71 million, actual US$ 17.48 million; additional financing estimate US$ 4.30 million and actual additional financing US$ 3.90 million) This component aimed to finance project administration and coordination activities including supervision, monitoring and impact evaluation. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Costs : Total costs increased from the appraisal estimate of US$ 106.09 million to US$ 114.3 million because the project was scaled up to include more beneficiaries . and cancelation of US$ 6.5 million was approved on November 27, 2012. Financing : The original Loan for US$ 89.05 million was disbursed 95 %. Additional financing of US$ 30.0 million was approved on October 21, 2008 and out of which US$ 21.3 million (71 %) was disbursed at project closing . It was expected at appraisal that local communities would provide US$ 9.04 million under the original loan and US$ 6.1 million under the additional financing and the actual contribution of local communities was US$ 9.17 million under the original loan and US$ 2.34 under the additional financing . Borrower Contribution : It was expected at appraisal that the Borrower would provide US$ 8.00 million and actual contribution was US$ 8.97 million. No borrower contribution was planned under the additional financing . Dates : Closing date of the original project was extended 5 times during project's life span, reflecting political turbulence and natural shocks. Following the change in government on March 17, 2009, Bank management triggered OP/BP 7.30 (Dealing with de facto Governments) , leading to freeze on disbursement . On May 6, 2010 exceptional authorization was granted retroactively to extend the closing date . Other closing date extensions were (i) on June 6, 2007 for 18 months to permit finalization of all sub -projects; (ii) on December 17, 2008 for 4 months to allow for completion of on-going sub-projects; (iii) on June 4, 2010 for 2.5 years to coincide with additional financing closing date and (iv) on June 30, 2011 for 18 months to avoid loss of unspent IDA funds and to have it coincide with the closing date of the additional financing. On February 8, 2011, closing date of additional financing loan was extended by 18 months to compensate for the delayed effectiveness . The project finally closed on December 31, 2012. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Substantial Project development objectives were and still remains substantially relevant to the country strategies and priorities but balanced by this are quite broad and ambitious . At the time of appraisal, Madagascar was one of the poorest countries in the world, with an overall poverty rate of 70 %. Poverty was even higher in rural areas with 80 % of poverty rate and 60 % of rural poor were classified as extremely poor (ICR p. 1). Agriculture provided a livelihood for 75 % of the country's population; however, the sector had significant issues . Major constrains to the development of the sector included lack of rural infrastructure, poorly functioning rural institutions, limited access to credit, burdensome regulatory limitations, and high transportation costs . In addition, agriculture practices were poorly developed, suffered from a lack of support services, and mostly were carried out in an environmentally unfriendly manner. The need for sustained growth in agriculture as a means to alleviate rural poverty had been emphasized in the Interim Poverty Reduction Strategy Paper finalized by the Government of Madagascar at the end of 2000. In addition, at the time of project preparation the government had adopted a rural development strategy and the Rural Development Action Plan. The project became an integral part of the Rural development Action Plan by supporting the first three pillar of the Government ’s rural strategy, namely sustainable growth in agricultural production, promotion of partnerships in rural development, and institutional and regulatory reforms . The project development objectives are also relevant to the last Country Assistance Strategy (FY 07-11), Pillar 1, 'removing the bottlenecks to investment and growth in rural and urban areas ' and specifically the goals of rural development and green revolution . However, the objectives of increasing incomes and reducing poverty were also quite broad and ambitious . ICR also criticized this aspect by stating (p. 9): "It was indeed unrealistic to expect that the project by itself would increase income and reduce poverty in rural areas ". Therefore, it could have been more reasonable if the objective included " to contribute to increasing incomes and reducing poverty in rural areas ". b. Relevance of Design: Substantial The project tackled a major constraint to development and poverty alleviation with clear components that links project activities and project development objective . The selection of a demand driven approach for financing small scale productive infrastructure, agricultural production and off -farm investments in rural areas was relevant . The design also included activities for community development and support services including extension and training for the beneficiaries as well as capacity development activities for the implementing agencies at the national and regional levels; all of which were also very relevant . The initial design led to fragmentation of activities among several components such as separation of farmers' organizations capacity building from the related investments as well as separation of research and research outreach activities led to higher transaction costs and decreased effectiveness of the activities. Nevertheless, this shortcoming was resolved through the project restructuring in 2002 by combining related components. Another shortcoming was the Regional Rural Development Working Groups (GTDRs) was chosen as the primary clearing houses for identifying regional priority investments, ensuring regional participation and inter-sectoral coordination, and eventually approving sub -project proposals; however, later it was seen that their capacity to perform this function was over -estimated. During the Additional Financing phase, this role was assigned to the newly created Regional Executive Directorates of the Ministry of Agriculture . However, as also mentioned by the ICR (p. 10), during the project period the country's institutions were going through a continuous change, making it difficult to predict which institution would remain in place; therefore it was wise to have a robust project implementing unit to continue providing project implementation at the local level in such an unstable environment . 4. Achievement of Objectives (Efficacy): The project development objective is 'contributing to the implementation of the Borrower ’s Rural Development Action Plan (PADR) and selected elements of the Borrower ’s Regional Rural Development Programs (PRDR) by fostering development of revenue increasing activities, thus increasing incomes and reducing poverty in rural areas, while preserving the natural resources base' . Therefore, it comprises 3 main sub-objectives: (i) increasing incomes; (ii) reducing poverty; (iii) preserving the natural resources base in rural areas . Outputs : • The Project financed a total 9,510 sub-projects, exceeding the target of 6,200 sub-projects and benefiting around 238,000 families (without repetition) or 1,190,000 persons (average 5 persons/family) implying that the Project reached more than 10 percent of the total rural population, and 50 percent of population living in remote rural areas. 34 % of the beneficiaries were female, exceeding the 29 % of female beneficiary target. • About 85 % of sub-projects supported were on productive agricultural activities (development techniques for rice, maize and peanuts cultivation, with positive increase in production ) category, with productive infrastructure (irrigation and drainage infrastructure ) accounting for 12 % and productive non-agricultural activities (crafts) for 3 % of total sub-projects. In terms of cost, infrastructure development represented more than 44 % of total component costs, agricultural activities 53 % and non-agriculture activities 2 %. • Additional area with irrigation and drainage services was developed, which exceeded the target (30,069 ha vs. target of 14,300). • Water user associations that are operational were set up, which exceeded the target (657 vs. target of 643). • Infrastructure built or rehabilitated : 657 dams (vs. target of 657), 205 processing units (vs. target of 190), 169 storage facilities (vs. target of 165) and 28 slaughterhouses (vs. target of 30). • The area cultivated with improved technologies and /or inputs provided through the sub -project increased by 42,927 ha, exceeding the target of 39,150 ha. • Increase in rice, maize and peanut production exceeded their targets; • Savings schemes established and functioning exceeded the target (6,467 vs. the target 5,982). • Technical assistance was provided for about 9,537 sub-projects. According to a 2012 survey, only half of producer organizations were satisfied with the technical assistance provided due to (i) insufficient support for the elaboration of sub-project profiles, (ii) lack of educational materials available to the beneficiaries, (iii) lack of consideration of the different crops ’ seasonality (iv) inadequate and/or insufficient supervision, and (v) lack of transparency and insufficient communication . • Thematic research activities on 8 themes to address medium or long-term constraints to the intensification, diversification and/or sustainability of production systems were carried out . The dissemination of research results was done through the research institutions ’ own extension services and partner NGOs . • The Competitive Agricultural Research Fund financed 26 research projects. Eighty seven percent of producers adopted improved agricultural technologies developed under the project . • The National Rural Development Program was developed and approved; Regional Rural Development Programs were prepared; The Rural Development Policy Letter was developed; The agriculture census was completed and disseminated; The agricultural statistics system became operational, Regional Agricultural Statistics Units were expanded and strengthened; Planning for the elaboration of the Agriculture Sector Plan was developed, with regional workshops carried out in all 22 regions; Agriculture Support Centers were partially strengthened (21 vs. a target of 40); Regional Grant Approval Committees were established (12 with project financing and 4 with other donor funds for a total of 16 as per the target). Outcomes : According to the ICR (p. 18), the project benefitted more than 10 % of rural population and was active in approximately 70 % of rural communes, half of which located in remote areas, more than doubling the appraisal target of beneficiary groups . The ICR presented some evidence (see below) that the project had a positive effect on income and quality of life, production, and farmers ’ organizational skills. Furthermore, the project became the de facto national agricultural extension service and an important interface between farmers, agribusiness, and sector institutions. These were important achievements, however, as also stated in the ICR (p. 19): "As a result of its national presence, the project was pulled in different directions to address the challenges arising from the external shocks, rather than becoming an instrument for long -term development assistance..... the project and the thousands of sub-projects it financed did not reach a critical mass nor had a strong catalytic effect on the rural space, with added sustainability concerns ". The outcome results were based on Impact Evaluation of Productive Investment Sub -projects under the Rural Development Support Project, December 2012; and Economic Evaluation of Productive Investment Sub - projects, May 2012, which focused on a representative sample of sub -projects financed in 2007 and 2008 under the original project. Although the ICR did not provide any information on the baseline survey, the project team subsequently stated that an2005 evaluation that was conducted right before the start of sub -projects that were supported by the project, calculated a baseline for these sub -projects. The survey randomly selected 120 sub-projects (representing 20 % of the existing sub-projects). The survey conducted in 2012 included 408 sub-projects located in 10 regions. The project team stated that the sampling strategy ensured representativeness for the overall population of sub-projects, however due to a series shocks (political, economic and environmental ) in the country, conducting a statistically rigorous survey and evaluation was not possible . This affected particularly measuring of the poverty reduction outcome, which is susceptible to many other factors, making it difficult to attribute poverty related achievements solely to the project . (i) Income Increase, rated substantial . It was reported by the project team that the criteria used to select sub-projects under productive investments component favored activities that were expected to have immediate impact on raising incomes and food production . The ICR reported the following outcomes : • Average beneficiaries’ revenues increased by 38 % (between 2005-2012), compared to stagnation in revenues for non-beneficiaries for the same time; • More than 80% of beneficiaries reported an increase in income of more than 30%, with beneficiaries engaging in agriculture activities registering a higher increase, followed by those engaging in productive infrastructure; • Increase in income was primarily supported by increase in production, particularly for households with access to irrigated land and improved technologies for rice, corn and peanut production . The vast majority (88 percent) of beneficiaries report an increase in production compared to the situation before the project; • Overall, more than 80 percent of beneficiaries report an improvement in their livelihoods (in terms of better housing, access to basic social services (education, health) and agricultural inputs ) and 92 percent consider that sub-projects have responded to their priority needs; • Higher scale of production and social cohesion facilitated the farmer organizations link to institutions of micro-finance. The majority of producer organizations are now members of financing institutions, although they use them mostly for saving purposes, and 6,467 saving schemes have been established . • The project development indicator, 'infrastructure functioning one year after completion and mechanism in place to finance operations and maintenance for 85% of financed infrastructure' was partially achieved : 90 % of financed infrastructure was still functional one year after completion, was in very good condition, and was built according to specifications . However, the arrangements for operation, management and maintenance remained unclear and not well understood by most beneficiaries . ii) Poverty Reduction, rated modest due to attribution problems as described above . The ICR reported that (p.22) (ii) there was difficulty of directly linking the project specific activities to poverty reduction in rural areas . Although it can be inferred that project achievements on income increase should have contributed to poverty reduction, the project did not track poverty indicators, and figures on this outcome are not reliable . The ICR reported poverty reduction among project beneficiaries without any non -beneficiary comparisons. Accordingly, the 2012 beneficiaries’ survey reported that the percentage of poor beneficiaries decreased by 33 %. The ICR stated that this is due to the large number of agricultural production investments, improved irrigation infrastructure and access to micro - finance services positioned many rural families for a higher level of productive activity . iii ) Preserving Natural Resource Base, rated susbtantial . The project provided evidence mainly on the (iii) implementation of the Environmental Action Plan (EAP). Accordingly, field assessments conducted in December 2012 as part of the project final evaluation estimated that 75% of sub-projects adopted an EAP during implementation, while 87% of beneficiaries stated that they complied with such Plan . The final sub- project impact evaluation reported that 91 % of producer organizations applied the environmental measures recommended by the EAP, with reforestation being the measure adopted by most of beneficiaries . The ICR also stated that the project went beyond solely screening sub -projects and the specification of mitigation measures, and also actively promoted sound environmental management by : (i) developing alternative agricultural practices to slash -and-burn production systems; (ii) launching efforts to include environmental considerations into commune development plans; (iii) exploring revenue-generating potential of non-timber forest products; and (iv) facilitating forest- management transfer contracts with the communes. 5. Efficiency: Efficiency is rated modest, due to lack of information on sampling and main assumptions regarding the economic analysis as well as administrative and operational inefficiencies . At project appraisal an illustrative economic analysis focusing on most relevant six project activities was carried out (coffee, clover, rice processing, irrigated rice rehabilitation, poultry production ). Information contained in the models were based on data provided by the National Center for Applied Research for Rural Development . The farm models were based on conservative estimates of attainable changes in family households, taking into account the subsistence nature and risk minimizing strategy, typical of the targeted groups of population . The long- term opportunity cost of capital in Madagascar was taken to be 12%. The Internal Rate of Return results for the six models varied from 29 % to 46 %. The ICR also carried out an ex-post economic and financial analysis that reviewed 4 types of productive sub-projects, instead of a whole project evaluation due to the lack of a complete set of quantitative data . The following shortcomings were observed in terms of economic analysis : (i) Lack of clarity on how the sub -projects were selected to be included in the economic analysis . • The 4 sub-projects represented about 64 % of the most common projects; but it is unknown how much they represented in terms of project costs . • Furthermore, it is not clear why some major sub -project types were not included in the analysis; for example, among sub-projects there were productive infrastructure projects (e.g. small dams) representing about 44 % of project allocations (ICR p. 32), for which the ICR did not provide any economic analysis calculation . Agricultural activities represented about 54 % of project allocations; the ICR only provided calculations for the 4 types of sub-projects under this category : beef, dairy, rice and poultry. There were also other sub-project types, such as maize, peanuts, fisheries, bee keeping, sheep /goat, etc. for which no analysis was presented . Hence, it is not clear why only these 4 types of sub-projects were included in the analysis . • The ICR mentioned (p. 38) that lack of complete set of quantitative data made efficiency analysis difficult, but did not give details of this limitation, and how it affected the analysis . (ii) Methodological assumptions for economic analysis were missing : Using cash flows from four types of sub-projects (rice, beef, dairy and poultry) financial and economic indicators were calculated to evaluate the overall project profitability. However, the ICR did not present any assumptions for these calculations, i .e. how the benefit and cost streams were calculated and what data -source was used to come up with the assumptions and benefit cost streams. It is also not clear if PAD’s assumptions and methodology was followed; indeed PAD economic analysis presented more detailed information on these aspects (how farm models were used, duration of the model, benefit stream periods, etc). Therefore, it is not clear if PAD analysis results are comparable with the ICR analysis (including rice intensification). Internal Rate of Returns ranged from highs of 38% and 29% (significantly higher than the opportunity cost of capital ) for dairy and rice respectively, and lows of 11% for beef (less than the opportunity cost of capital ). Net Present Values were also high and positive, except for the beef model . Findings suggested that financial returns were high for all sub-project types except for beef . For the dairy model, the sub-project was financially viable (Internal rate of Return of 38% and positive Net Present Value ) but in terms of economic efficiency (after labor market adjustment) the dairy project was not attractive with a negative economic Net Present Value and an Economic Rate of Return less than the Opportunity Cost of Capital (NPV of US$ -578,000 and ERR of 9.2 %). The project closing date was extended many times for a total of 5.5 years from June 30, 2007 until December 31, 2012. The extensions were mainly due to external factors such as political turbulences in 2002 and 2009 and natural shocks as well as implementation of additional financing . Project investments were negatively impacted by several major cyclones that strongly hit Madagascar especially in 2004 and 2005 and many sub-projects had to be rebuilt. The civil disturbance in 2009 and the following triggering of OP/BP 7.30 (de facto governments) resulted in 18 month disbursement freeze. However, there were also some administrative and operational shortcomings that created delays. For example frequent turnovers in Ministry of Agriculture and Project Implementation Unit affected the performance and made it difficult to establish an effective communication between the host Ministry and the projec t. If the economic analysis was robust it would capture these delays in the ERR rates . However, since there are doubts about the validity of ERR calculations due to lack of information on sampling, assumptions and methodology, it is not possible to conclude that administrative and operational delays do not affect the overall efficiency of the project . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re-estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: In light of the' substantial' relevance of objectives and design; ‘substantial’ achievement of two sub- objectives – income increase and preserving natural resource base - but 'modest' achievement of poverty reduction sub -objective as well as efficiency, the overall outcome is 'moderately satisfactory' in accordance with IEG Guidelines . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: This Review concurs with the ICR's 'significant' risk rating . The ICR analyzed sustainability of sub - projects (financial viability, the maintenance of organizational structures for implementation and the continued adoption of improved techniques, the existence of markets and compliance with environmental action plans ) in order to determine the risk to development outcome . Accordingly, the percentage of sub -projects that remained functional by 2012 was high for the Additional Financing (90 %), but lower for the Original Project (80 %) and the economic analysis of selected productive sub -projects suggested varied degrees of sustainability . The ability of producer organizations to negotiate with financial partners and establish savings mechanisms had grown, but their management capacity needed further strengthening, and their financial viability and access to credit remained weak . Networks of technical partners were being put in place, but their development remained precarious due to isolation of production sites and difficulties of communication for disseminating both technical and commercial information . Access to new markets remained a constraint, particularly due to isolation, lack of communication, and insufficient marketing activities, The socio -organizational capacity of producer organizations had improved but needed further strengthening. Operations and Maintenance practices were still weak but the probability of survival of producer associations that were in charge of operation and maintenance was promising due to follow up support of local authorities. Two out of three producer organizations adopted the recommended technologies in the implementation of their sub-projects; and since such adoption had a positive impact on productivity and income, prospects for broader dissemination were good . Sound environmental management was increasingly taking place which was an indicator of enhanced awareness of the importance of sustainable production . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: Lessons drawn from several previous operations in the rural sector across the country and elsewhere since the early nineties were used in project design . ICR reported that (p. 8): "The following main lessons from previous similar operations were incorporated : (a) the need to adopt an integrated sectoral focus to address rural growth and development nationwide; (b) the effectiveness of decentralization in reducing bureaucracy and reinforcing accountability; (c) the importance of using a demand -driven approach and encouraging beneficiary participation at all stages of the investment cycle for building ownership and ensuring sustainability; (d) the importance of building partnerships with all sector stakeholders and in particular the private sector for efficient provision of technical services; and (e) the need to link research with on the ground extension services and community investments. The considerable amount of background analysis and consultation carried out during the rural strategy formulation provided a solid justification for Bank support . A Bank sector review mission carried out in 2000 identified the challenges and opportunities for alleviation of rural poverty in Madagascar and developed strategic options for policy and programmatic action ". According to ICR (p. 11) risks were estimated during appraisal based on previous experiences with similar projects, but there were omissions both in the types of risks that were identified and in the planned risk minimization measures. For example, after identifying the risk associated with political interference in the selection of sub-projects, the Project Appraisal Document might have called for the establishment early on of a strong monitoring and evaluation system . As mentioned in Section 10, the overall design of the M&E system had shortcomings (primarily focused on the measurement of inputs and outputs, rather than results and impacts ). Also, the pervasive lack of adequate technical assistance in rural areas should have been recognized, and a mitigation framework designed and implemented earlier on, especially in light of the project ’s productive goals and consequent need for sustained, specialized support . The ICR identified a couple of shortcomings in project design (p. 9)- mainly the project’s operational strategy and content under the project was not matched by institutional capacity on the ground in key areas . However, ICR reported that these shortcomings were corrected as part the 2002 overall portfolio restructuring. at -Entry Rating : Quality -at- Moderately Unsatisfactory b. Quality of supervision: Overall, during supervision, Bank teams were conscientious in working with the government and other partners in facilitating the implementation of the project . They clearly and accurately identified in a timely manner the operational bottlenecks and the appropriate remedial responses to be undertaken . Supervision was characterized by: (i) regular, well-documented missions, with randomized field visits to communities and sub-projects, and considerable follow -up with Task team Leaders based in the field; (ii) a focus on fiduciary capacity and performance, with consistent, intensive supervision of procurement in the post -crisis period (2004-05); (iii) clear identification of capacity needs; (iv) strong Mid-Term Review including comprehensive Action Plans, and sustained follow-up with government's rural strategy, with some weaknesses though on the Monitoring and Evaluation aspects; and (v) intense, pragmatic, rigorous efforts to salvage the operation post -crises; (vi) quick diagnose of implementation problems and strong proactivity in seeking restructurings, especially during difficult times. Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The ICR reported that the Government was fully-committed to the project methodology, objectives and activities and it collaborated closely with the Bank to improve key design elements and resolve problems at each stage. This commitment flowed through into the implementation phase, as reflected for example in the swift response to the 2004 rice crisis that temporarily derailed the project . Following the 2004 rice crisis, the government placed intense political pressure on the project to finance a rice intensification campaign and support cyclone repair works outside of the approved work plan . This pressure resulted in significant over - commitment of credit proceeds and loss of financial and monitoring control by the project team . Strong remedial measures had to be taken at that time, resulting in a declaration of mis -procurement and ineligible expenditures, major cancellations of sub-projects. After these remedial measures, the political interference stopped, and the project regained its institutional reputation as well as its perceived integrity . The successive governments showed a consistent commitment to rural development and to protecting and supporting the project as a major instrument for the implementation of their rural development policy . Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: According to the ICR (p. 26), the Ministry of Agriculture showed consistently strong, committed effort, even with formidable obstacles from 2009 onwards, to collaborate with the Bank team in moving the project forward . It was very responsive in restoring enabling conditions for effective project implementation following political and internal crises. However, implementation of component 2 and 3 (under the direct responsibility of the Ministry ) was somewhat weak and characterized by poor accountability in the use of funds and lack of clarity about incremental benefits generated . The Project Implementation Unit had a wide range of responsibilities, and was successful in gradually establishing an adequate regional field presence for the processing of a large number of sub-projects all over the country . However, it focused too much on quantitative targets at the expense of qualitative ones, namely provision of technical assistance and screening of sub -projects. Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The M&E system was designed to reflect the scope of the project and adopt an M&E approach that would allow for full participation of beneficiary rural communities through the establishment of rural management committees . Additional elements of the system were an internal monitoring conducted by M&E staff at both the central and local level, and periodic supervision and evaluation of project performance and impacts . The overall design of the M&E system had some weaknesses, as it was primarily focused on the measurement of inputs and outputs, rather than results and impacts. The Results Framework reflected the old -style project Logframe approach and lacked intermediate outcome indicators that would have allowed project performance to be tracked during the early years of implementation. The linkage between project objective and the single key outcome indicator was not very clear and key outcome indicator did not capture the meaning of the development objective; i .e. more precise measurement was needed to gauge the actual increase in income and reduction in poverty . Also there was no indicator measuring 'preserving the natural resources base' sub -objective. Absence of intermediate results indicators was another shortcoming. Some targets established for the OP were overly ambitious, representing an unrealistic benchmark to adequately measure performance, and they were not always in line with capacity on the ground . Revised and additional performance indicators were subsequently included, and targets were reduced to levels that the project could realistically achieve. However, the opportunity to focus the Project Development Objective and make it more measurable was never taken up . b. M&E Implementation: During the first year of project implementation, the system suffered from considerable deficiencies, and as a result it did not permit an adequate assessment of project performance . These shortcomings were subsequently rectified, and the M&E system was strengthened through integration with the overall project Management Information System . Following these improvements, the project M&E unit was able to collect and report on performance indicators . However, data collection and reporting procedures were often informal and ad hoc, and performance indicators remained heavily focused on inputs and outputs . Processing and timely dissemination of M&E data was not supported by a well-defined and transparent system, thus limiting the usefulness of the data and making it difficult to communicate the achievements of the project to larger audiences . In order to enhance M&E capacity to measure project performance, and considering the difficulties of assessing the cumulative impact of a myriad of small -scale sub-projects, the project financed periodic one -off impact assessment studies to measure progress achieved against performance indicators. As stated in Section 4, it is not clear if there was a baseline survey . The impact evaluation methodologies were also not clear, making it difficult to attribute the outcomes to the project . c. M&E Utilization: No information was provided. M&E Quality Rating : Modest 11. Other Issues a. Safeguards: The project's environmental safeguards category was Category B with an Environmental Management Plan and a Pest Management Plan. During implementation, questions about the applicability of OP 4.37 (Safety of Dams) were raised with regard to a number of proposals relating to the rehabilitation or development of small -scale irrigation schemes. Considering that safety risks associated with irrigation systems being financed under the Project were minimal and that the affected sub -projects were at an advanced stage of design or implementation and could not be stopped without risking timely achievement of the project development objective, the policy was not triggered . However, as a precautionary measure and to make sure that the spirit of the policy was respected, the project was asked to prepare guidelines for the management of small dams and applied these guidelines systematically to all irrigation schemes financed by the project . Implementation of mitigation measures was done by the beneficiaries themselves, with the support of local service providers. Project technical staff were responsible for monitoring implementation and for ensuring that no adverse environmental impacts were occurring . Activities carried out to ensure compliance with applicable environmental safeguards policies were in general satisfactory, and environmental themes were incorporated in training courses . Bank supervision mission identified minor implementation issues of some mitigation measures, which were promptly corrected. On the whole, mitigation measures associated with individual sub -projects were implemented reasonably well by sub- project beneficiaries. Although the implementation of environmental safeguards policies was generally satisfactory throughout the life of the project, implementation performance declined following the disbursement freeze instituted in March 2009, as the support being provided by local service providers to assist beneficiaries in the implementation of mitigation measures stopped . Similarly, monitoring by project technical staff of the rate of adoption of mitigation measures and assessment of the environmental impacts of project -financed activities ceased . Environmental monitoring and support activities were resumed and strengthened under the Additional Financing through the establishment of technical pools . According to the project team, the environmental safeguard compliance was' moderately satisfactory'. b. Fiduciary Compliance: The financial management of the project was handled by project staff directly contracted by the Project Implementation Unit, assisted by fiduciary agents (accounting firms) that were contracted over specific periods of the project to provide support and capacity building . Overall, the project had appropriately qualified and experienced personnel in charge of financial management . The internal controls were spelt out in a Procedures Manual, amendments to which were made on a periodic basis to take into account changes in the project ’s operations. In general, the quarterly financial reports and annual audit reports were submitted to the Bank on a timely basis, and recommendations made subsequent to implementation support missions were appropriately implemented . At the time of preparation of the ICR, the project had not yet submitted the documentation needed to determine the outstanding balances in the Designated Accounts . The amount that needed to be accounted for USD 7.1 million and the project team was working to ensure the submission of the related statements of expenditure before the end of the grace period (30 June 2013). The financial management of the project was rated as moderately satisfactory at the project closing date. In terms of procurement, as the project commenced, the procurement load became heavy due to the number, diversity, type and technical complexity of the sub -projects, and the project experienced delays requiring continuous need for specialist consultants to ensure compliance with procurement procedures . In view of the problem faced by the project in 2005 as a result of the financing of cyclones repairs and the rice intensification campaign outside of the approved work plan, the Bank launched an independent procurement audit which revealed significant irregularities with mini-tractors procured under direct contracts by the Project Implementation Unit, as well as suspicion of fraudulent pricing. After a thorough internal review, the Bank officially declared misprocurement in June 2006 on the purchase of 750 mini-tractors for a total commitment of US$1.6 million, requested reimbursement of US$ 435,024 and canceled US$ 718,327 in credit proceeds. Following the extensive corrective measures taken in 2005, the procurement function improved significantly with a well -established and functioning procurement unit . During the 2009 political crisis, procurement staff were placed on unpaid leave and the procurement function lapsed . The Additional Financing made significant efforts to rebuild procurement capacity, including the hiring of a Fiduciary Firm responsible for supervising all procurement processes under the project . However, minor weaknesses in the procurement system remained, mostly due to low capacity, which delayed implementation of the Additional Financing. c. Unintended Impacts (positive or negative): d. Other: Gender : The ICR reported that (p. 23), women’s participation in productive sub -projects exceeded the 29% target; i.e. over 34% of productive sub-projects were executed by associations led by women ( The project benefitted a total of 1,190,295 beneficiaries, 34 % of those being women). More than 30 % of the members of farmer organizations supported by the project were women . Where the women were strongly represented (e.g., rice husking, livestock, sewing, handicrafts), the organizations had generally higher le vels of savings and income than those dominated by men. 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Significant Significant Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR offers a number of lessons of which the following are the most important (with some reformulation of language): Innovative Community Driven Development operations in the rural sector, especially when operating in a fragile environment, require an extended horizon, a high degree of flexibility, continuous support for progressive capacity -building and repeat financing to maximize impact, learning and sustainability . The time required for such projects exceeds Bank averages, especially in fragile conditions, such as political instability and frequent natural disasters, where projects need to constantly adapt to changing external circumstances. The important elements are (i) a high degree of flexibility to respond to external shocks and adapt in terms of the type of support they provide; (ii) unique outreach and implementation arrangements for channeling support where it is needed; and (iii) experienced, multi-skilled, cross-sector Bank and Borrower project teams with continuity of leadership . Community Driven Development Projects should differentiate between target groups and sub -project objectives . The goal of ensuring food security would be suitable for the poorest segments of the rural population, who are usually characterized by very low capacity . On the other hand the goal would be to ensure financial sustainability over time for the farmers with proven farming experience and an entrepreneurial approach to production and marketing . Financing criteria and the level of subsidy provided through matching grants should be adjusted to the degree of vulnerability of the targeted group, geographical location, and the nature of the sub-project. The sourcing and quality of technical assistance needs focused attention from the earliest stages of project development . Allocating a portion of sub-project cost for technical assistance cannot resolve the fundamental problem of the scarcity and quality of such services in rural areas . Community Driven Development projects need to support an intense effort to build up such services over time, with a focus on : (i) building a sustained, long-term support for producer organizations; (ii) responding to the specific needs of potential entrepreneurs; (iii) tailoring the support to the unique technical, institutional and financial endowments of each region; (iv) ensuring adequate presence in rural areas through massive decentralization of staff and providers; (v) grouping service providers by area of specialty and introducing a black list of non -performers; and (vi) supporting "training of trainers." 14. Assessment Recommended? Yes No Why? The project used a demand driven approach for financing small scale productive infrastructure, agricultural production and off-farm investments in rural areas and reached a significant number of beneficiaries . IEG's ratings on achievement of outcomes differed from ICR's due to attribution problems . A closer look at project outcomes will help to verify project achievements and related ratings . 15. Comments on Quality of ICR: The ICR is clear and well argued with a frank description of implementation challenges and issues . The sections are comprehensive and the material and analysis presented is consistent with the OPCS guidelines . On the other hand there was a lack of information regarding the main assumptions of the Economic Analysis, and details of the M&E system, including survey methodologies, were missing . a.Quality of ICR Rating : Satisfactory