ICRR 12796 Report Number : ICRR12796 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 11/16/2007 PROJ ID : P068808 Appraisal Actual Project Name : Social Development US$M ): Project Costs (US$M): 28.66 30.02 Fund 2 (apl #2) (sdf2) Country : Romania Loan /Credit (US$M Loan/ ): US$M): 20.0 19.87 Sector Board : SP US$M ): Cofinancing (US$M): 0.25 0.25 Sector (s): Other social services (29%) Roads and highways (25%) Water supply (25%) Agro-industry (21%) Theme (s): Participation and civic engagement (29% - P) Improving labor markets (29% - P) Rural services and infrastructure (28% - P) Small and medium enterprise support (14% - S) L/C Number : L4645 Board Approval Date : 12/20/2001 Partners involved : DFID Closing Date : 08/31/2006 02/28/2007 Evaluator : Panel Reviewer : Group Manager : Group : Kris Hallberg Roy Gilbert Alain A. Barbu IEGSG 2. Project Objectives and Components: a. Objectives: The Romania Second Social Development Fund Project (SDF II) was the second phase of a two -phase program supported by the Bank. The objectives of SDF II were: "The development objectives of the second phase of the Program (SDF II, hereafter called the Project ) are similar to the objectives of the first phase : to contribute to poverty alleviation and community -driven development through: (a) improving the livelihood of Beneficiaries from poor rural communities and disadvantaged groups, and (b) promoting social capital enhancement and civic engagement among project Beneficiaries ." (PAD p. 3) The first phase of the program, SDF I, was approved by the Board in January 1999, and was implemented during the period March 1999 through December 2001. Thus, SDF I was under implementation when SDF II was appraised in September 2001. SDF I was supported by a $10.0 million loan from the Bank, of which $9.68 million was disbursed. IEG rated the outcome of SDF I as satisfactory (compared to the ICR's rating of Highly Satisfactory ), noting that convincing evidence of impact was lacking . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): (a) SDF Sub -Project Development (estimated cost at appraisal $ 25.86 million, 90% of total project cost; actual cost $27.22 million, 91% of total cost): grants to sub-projects prepared by recipients in poor rural communities and by disadvantaged groups, in four categories : (i) investments to rehabilitate, upgrade, build or equip rural roads, water supply, small bridges, community centers, small-scale flood defenses, etc . within a ceiling of $75,000 per sub-project grant. (ii) community-based social services directed toward disadvantaged children, the homeless, disadvantaged elderly, and youth at risk, with a $ 20,000 ceiling per sub-project grant. (iii) income-generating activities, with a ceiling of $ 20,000 per sub-project grant. (iv) follow-up activities to fix small problems in sub -projects already financed by the implementing agency, with a ceiling of $10,000 per grant, and second grants for innovative sub -projects in communities that had already successfully completed a sub -project, with a ceiling of $45,000 per sub-project grant. (b) Knowledge Sharing and Partnership Building (estimated cost at appraisal $ 0.26 million, 1% of total cost; actual cost $0.09 million, 0.3% of total cost): information, education, and communication (IEC) activities; creating networks of "graduating" community-based organizations (CBOs) and facilitating continued partnerships between CBOs and local authorities; and building linkages with other national poverty reduction strategies and projects . (c) Institutional strengthening (estimated cost at appraisal $ 2.54 million, 9% of total cost; actual cost $ 2.71 million, 9% of total cost), with two sub-components: institutional support to the implementing agency, and monitoring and evaluation. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: In 2004, the grant ceiling per income-generating activity sub-project was increased from $20,000 to $40,000; the ICR reports that the ceiling was raised in view of the European standards and rules for running a business . In addition, after four years of implementation, the Loan Agreement was amended to reallocate more resources to sub-project supervision. The closing date was extended by six months to accommodate delays in some sub -projects which were caused by the 2005 floods and landslides. 3. Relevance of Objectives & Design: Project objectives were and are substantially relevant to Romania's development needs . The post-Communist collapse resulted in a doubling of the share of the population living below the national poverty line, from 20 to 41 percent. In 2001, 60 percent of farmers were poor, 70 percent of households headed by the unemployed were poor; and 90 percent of Roma-headed households were poor . Building social capital was important to help communities escape passivity, engage collectively, and adapt to changes . The SDF program addressed two of the priorities identified in the 2001 CAS: greater access to opportunity (through, inter alia, rural development, outreach to ethnic groups, and promotion of social capital through community development ), and institution-building to strengthen the rule of law. The relevance of project design was modest. On the positive side, project design benefitted from the experience accumulated during the first phase of the program, including the results of a beneficiary assessment and an evaluation and monitoring report, both conducted in 2000. Drawing upon the current view that vulnerability is a critical dimension of poverty, SDF II added "youth at risk" to the target population. In line with CDD good practice, project design also focused on participatory processes for choosing community investments . The Knowledge Sharing and Partnership Building component was appropriately designed to support the objective of increasing local capacity for organization and self -help. However, there were significant shortcomings in project design . For example, since the majority of sub -projects were expected to be for small rural infrastructure, more technical expertise should have been utilized in the design phase of the project. In addition, although a lesson from the first phase of the SDF program was to improve monitoring and evaluation, and SDF II included a sub -component to evaluate the impact of the project on final outcomes, in fact the design of the monitoring and evaluation framework was weak (Section 10). Key performance indicators were not included to measure the project's impact on poverty, nor its success in targeting disadvantaged groups . 4. Achievement of Objectives (Efficacy): (i) Contributing to poverty alleviation and community driven development : The ICR contains no information on the extent to which poverty was alleviated . There is insufficient evidence on community driven development and the methodology is not presented . (ii) ii) Improving the livelihood of Beneficiaries from poor rural communities and disadvantaged groups : modest . The ICR provides no empirical evidence on the actual achievement of the intended outcome of improved livelihoods . Nor does the ICR provide evidence that the project succeeded in reaching the target population of the disadvantaged (defined in the PAD as the poor elderly with no family support, the homeless, women victims of domestic violence, poor women, poor parents with dependent children, street children, poor pregnant teenagers, and youth at risk ). In terms of outputs, the project financed 436 sub-projects in small rural infrastructure (285 sub-projects), community-based social services (70), income-generating activities (65), and follow-up activities (16). An M&E unit was established in the implementing agency, and three external impact evaluations were carried out . However, these were not rigorous evaluations of welfare impact; rather, they were qualitative beneficiary assessments . The beneficiary assessment carried out in 2006 found that the beneficiaries of community -based social services projects were more satisfied with their living standards than the average Romanian citizen, that their lives were better than they were a year earlier, and that they expected to be living better a year from now . The ICR notes that beneficiaries reported better access to health care, education, and food stores, that job opportunities had increased, and that infrastructure had improved . A 2007 assessment of works financed by the implementing agency found that the quality of works was satisfactory . iii ) Promoting social capital enhancement and civic engagement among project Beneficiaries : modest . Under (iii) the Knowledge Sharing and Partnership Building component, 8 networks between grantee ("graduating") and non-grantee CBOs were created, and 6 were still functioning at the time of project closure . Two professional associations and one association for administering a water network were created . An indicator of social capital, the level of trust reported by community members, is reported to have increased by 30 percent of respondents in the beneficiary assessment. 300 community-based organizations participated in project interventions, and 60 community-based social services were established . The ICR reports that the 2006 beneficiary assessment revealed improvements in social solidarity, social relations, people's trust in various local actors, the willingness of beneficiaries to get involved in community projects, their level of involvement in local governance, and the social inclusion of disadvantaged categories . Almost half of the beneficiaries' families and over 70 percent of the leaders contributed work, materials or money to the implementation of sub -projects. The ICR does not discuss the methodology used to gather this data, nor does it compare project achievements to a counterfactual . 5. Efficiency (not applicable to DPLs): No ERRs were estimated at appraisal due to the demand -driven nature of the sub-projects. At completion, an IRR was calculated for a sample of sub -projects and found to be 30 percent. However, the sample included only 11 sub-projects out of a total of 436 sub-projects, or 2.5 percent of the total. Moreover, the sub-projects were of the third type only (income-generating activities). Similar calculations should have been done for samples of the other three types of sub-projects. There is some evidence that sub -projects were more cost-effective than similar investments financed by other programs (the Rural Development Project financed by the Bank, and the European Union's Special Accession Programme for Agriculture and Rural Development ). However, at some points during implementation (in 2002 and again in 2004), the Bank team was concerned that SDF II was becoming overly bureaucratic, making it difficult for beneficiaries to use the fund . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate Yes 30% 2.5% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Despite the project's attainment of outputs, there is little evidence on the achievement of the objectives of alleviating poverty and improving livelihoods, and no evidence on the project's success in targeting disadvantaged groups. There were shortcomings in the relevance of project design, and the project's efficiency is difficult to evaluate because ERRs were not calculated for a sufficiently large and representative sample of sub -projects. On balance, the project's outcome is rated moderately satisfactory . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The ICR reports that there is significant support for the SDF approach, and that there is a good possibility of EU support. With respect to infrastructure sub -projects, there is no information in the ICR on funding for continued operation and maintenance. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: Quality at entry was moderately unsatisfactory . The QAG Quality of Supervision Assessment of November 2006 rated quality at entry as satisfactory, noting an "innovative, strong approach to social capital buidling " but with somewhat "limited attention to technical aspects of local infrastructure design and quality assurance ". The ICR notes that it would have been advisable to include engineering expertise in the Bank task team to ensure appropriate design and sustainability of infrastructure investments . In addition, there were significant shortcomings in project design (see Section 3), in particular the failure to design an M&E system to measure the achievement of outcomes. Quality of supervision was moderately satisfactory . The project was supervised by a task team largely based in-country. However, the ICR notes that it would have been appropriate to have used the services of an environmental specialist during supervision to facilitate proactive attention to environmental issues and verify compliance. In accordance with evaluation guidelines, overall Bank Performance is rated moderately satisfactory since the project's outcome rating is above the line . at -Entry :Moderately Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: The Government was highly committed to the SDF program . There were some minor shortcomings in Government performance: it took four months for the loan to become effective, due to the lengthy approval process in Romania; and inadequate counterpart funds delayed project implementation in 2002 and 2005. The performance of the implementing agency was satisfactory, despite a tendency to make procedures too bureaucratic. a. Government Performance :Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Satisfactory 10. M&E Design, Implementation, & Utilization: The quality of M&E design was modest. Following on a recommendation from the first phase of the program, SDF II attempted to pay greater attention to monitoring and evaluation . The project's monitoring and evaluation sub-component established an M&E unit within the implementing agency to generate and analyze data and commission independent studies . Included was a program of impact evaluation, including beneficiary and impact assessments. A survey instrument (called Social Capital, Poverty, and Social Inclusion, or SCAPSI ) was developed to measure social capital, to be administered to selected informants at each sub -project site, and repeated over time to assess changes. However, there were significant shortcomings in M&E design . Key performance indicators did not directly measure the achievement of the project objectives of improved livelihoods, reducing poverty, or reaching disadvantaged groups. Instead, the indicators were often weak proxies these outcomes . For example, the indicator for to measure "improved livelihoods" was "beneficiaries' level of satisfaction with sub -projects", confusing perceptions of results with actual results . The ICR does not discuss the methodology used for these indicators . Implementation of M&E was modest. The baseline survey envisaged in the PAD was not implemented until late 2003, a year and a half after loan effectiveness . The impact assessments did not evaluate the project's contribution to poverty alleviation or improvement in livelihoods . Instead, as noted above, beneficiary assessments mostly gathered information on the satisfaction of beneficiaries with sub -projects. M&E utilization was modest. The QAG Quality of Supervision Assessment noted the availability of good M&E information but criticized the task team for not summarizing this information in supervision reporting . However, the ICR notes that M&E information did serve the purpose of helping the implementing agency assess and improve its own performance. a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): The environmental checklist was used to assess the environmental impact of sub -projects, and the Bank task team did spot checks to ensure that environmental issues were appropriately addressed . No significant environmental impacts were reported in the ICR, but as noted in Section 9, there was a lack of environmental expertise in supervision. Resettlement safeguards were not triggered, as land acquisition was prohibited for sub -projects. Procurement arrangements were relatively simple, and the implementing agency had already developed procurement capacity during the first phase . All project audits were unqualified. 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Highly Satisfactory Moderately Despite the project's attainment of Satisfactory outputs, there is little evidence on the achievement of the objectives of improving livelihoods and alleviating poverty, and no evidence on the project's success in targeting disadvantaged groups. Risk to Development Negligible to Low Moderate The ICR contains little information on Outcome : the probability of adequate funding for maintenance of investments. There is a possibility of continued EU funding for the SDF program, but has not yet been secured. Bank Performance : Satisfactory Moderately The M&E system was inadequate. Satisfactory Borrower Performance : Satisfactory Satisfactory Quality of ICR : Unsatisfactory NOTES NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Task teams designing and implementing CDD projects should include technical expertise in infrastructure and environmental issues. Social fund projects are typically appraised and supervised by Bank staff from the Human Development and Social Development networks, but it would be useful to include, at a minimum, a civil engineer on the task team . Excessively bureaucratic procedures should be avoided . There may be a trade-off between simplicity and flexibility on the one hand, and rigor and control on the other hand . While governance concerns are valid, excessively bureaucratic procedures can drive away potential clients from CDD projects . 14. Assessment Recommended? Yes No Why? A PPAR for SDF I and SDF II is recommended to provide evidence on the SDF program's impact on poverty, the livelihoods of beneficiaries, and the extent to which targeted groups were reached . 15. Comments on Quality of ICR: ICR Quality is rated unsatisfactory due to a number of significant shortcomings : The ICR presents little evidence on the achievement of intended outcomes, particularly on the change in livelihoods and poverty resulting from the project . It also does not evaluate the project's targeting mechanisms, thus failing to assess the degree to which target beneficiaries were reached. There are examples of poor (or absent) analysis, where hypotheses are presented as conclusions despite the lack of supporting evidence . For example, the ICR (p. 9) notes that "The causal linkages between outputs and outcomes were well established in the Project's logframe ...the logic is that improving access to infrastructure, social services, and opportunities for income generation and employment ...improves livelihoods". However, no evidence on actual improvements in livelihoods is provided . Poorly constructed syntax at times shrouds meaning (for example, ICR p. 12 third bullet). (The Region, in their comments on the draft ICR Review, pointed out that this paragraph is a quote from one of the beneficiary assessments). Finally, the ICR does not report on all of the indicators identified in the PAD . (The Region has provided this data in their comments on the draft ICR Review .) a.Quality of ICR Rating : Unsatisfactory