IEG Report Number: ICRR14917 ICR Review Independent Evaluation Group 1. Project Data: Date Posted: 05/18/2016 Country: Sri Lanka Project ID: P087145 Appraisal Actual Project Name: Second Community Project Costs (US$M): 105.00 97.07 Development And Livelihood Improvement Proj L/C Number: Loan/Credit (US$M): 75.00 74.26 Sector Board: Agriculture and Rural Cofinancing (US$M): Development Cofinanciers: Board Approval Date : 09/10/2009 Closing Date: 03/31/2014 09/30/2015 Sector(s): Other social services (50%); General agriculture; fishing and forestry sector (20%); Sub-national government administration (10%); Water supply (10%); Agro-industry; marketing; and trade (10%) Theme(s): Rural services and infrastructure (28%); Participation and civic engagement (25%); Rural policies and institutions (17%); Rural non-farm income generation (16%); Gender (14%) Prepared by: Reviewed by: ICR Review Group: Coordinator: Ridley Nelson Lauren Kelly Christopher David IEGPS1 Nelson 2. Project Objectives and Components: a. Objectives: The project was the second phase of a three phase Adjustable Program Loan (APL) that was changed to a Sector Investment Loan at restructuring. The final phase of the APL was not pursued. The Financing Agreement stated that the objective of the APL was: To enhance incomes and quality of life of poor households in the poorest divisions in the territory of the Recipient. Three means were given: (i) empowering the poor and developing and strengthening institutions of the poor; (ii) improving access of the poor to basic infrastructure and social services and providing support for productive activities; and, (iii) developing policies, rules, systems, procedures and institutional arrangements to enable the Recipient to transfer funds directly to communities and local governments. At the time of restructuring, the project development objective was changed to the following: To enhance incomes and quality of life of the poor households in the poorest divisions in the country while building capacity of government agencies, local governments and community organizations for service delivery and overall project implementation. There was no difference between the pre- and post-restructuring objectives in the outcome level objectives, both called for the enhancement of incomes and quality of life. The main difference post-restructuring was that there was greater focus on government capacity building. The change at restructuring became necessary following a government decision to shift the focus away from Village Organizations towards local government by means of (ICR page 45, Annex 3) “a graduation of autonomy through capacity building and training from village to local government level with the Pradeshya Sabha.” b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval: 05/13/2013 c. Components: Original Components: Component A: Intra-Village Development. (Appraisal US$59 .40 million; Actual US$58.71 million). This was intended to build institutional capacity of local institutions and fund village development including livelihoods investments. There were two sub-components: (a) the development and strengthening of Village Organizations (VOs) and Estate Communities; this included sub-project planning and implementation; (b) Village Development Fund through which funds were transferred to VOs to finance capacity building, community infrastructure, social services and livelihood support subprojects. The original intention was that about 60% of the funds would be used for community infrastructure, about 10% for capacity building and about 30% for livelihoods support subprojects through the Village Savings and Credit Organizations (VSCO). Component B: Intra-village Connectivity Development . (Appraisal US$30.00 million; Actual US$16.53 million). This fund was to promote intra-village development to consolidate and sustain investments generated at the village level. There were three sub-components: (a) strengthening of Pradeshiya Sabha Institutions to enable them to plan, implement and manage village and cluster development investments; (b) establishing the Pradeshiya Sabha (PS) Inter-village Connectivity Fund to finance inter- village infrastructure and services and for PS capacity building. About 95% of the fund was to be utilized for infrastructure. (c) developing inter-village federation and community resource centers creating a vertical federation structure including financing livelihood and productive infrastructure activities through the federated institutions. Component C: Public, People and Private Sector Partnerships . (Appraisal US$4 .40 million; Actual US$3.64 million). This was intended to support and strengthen project teams, public sector agencies and other stakeholders in operationalizing the community-driven development approach and to provide support for strengthening the capacity of the proposed federations to market products and services and to link with the private sector and to support innovative ideas for scaling up through experimentation and incubation. Component D: Project Management and Monitoring . (Appraisal US$7 .80 million; Actual US$11.22 million). This was to support overall coordination, planning, implementation, and monitoring and learning for the project at national, provincial, and district levels. Component D: Convergence and Policy Support . (Appraisal US$3.40 million; Actual US$6.97 million). This was to provide technical assistance to the government’s Gamaneguma and Samurdhi programs to adopt the community-driven development approach. It was also to support a pro-poor orientation and to finance the Village Development Plans in demonstration villages. At the Level 1 restructuring in May 2013, the components were revised with the following main changes: Component A: Intra-village Development. (Estimate at restructuring US$59.83 million). Three changes were made: (a) VOs were restructured to focus more on enhanced livelihood and market linkages for producer groups; (b) there was greater leveraging of additional resources through formal bank linkages; (c) the number of village infrastructure subprojects was reduced as was the community contribution. A limit of four subprojects per VO was set and contributions were reduced to 20% for the first subproject and 10% for the subsequent ones. Contribution by estate communities was reduced to 5% including labor, cash and materials. Component B: Inter-village Connectivity Development . (Estimate at restructuring US$18 .20 million). There were three changes: (a) the number of inter-village infrastructure subprojects was reduced to only those that could be completed before project closing. Community cash contribution was changed to an in-kind contribution; (b) funding was provided for the establishment of Operations and Maintenance Units for the PSs with some changes in procurement procedures and direct contracting; (c) support for village federations, community resource center development, and financing through the federated institutions was dropped because federations were discontinued as a result of a government policy change. Component C: Public, Private and People Sector Partnerships . (Estimate at restructuring US$3.71 million). There were three main changes. First, activities supporting the operationalization of the community-driven development approach were moved to Component D. Second, activities supporting Producer Groups were scaled up as a partial substitute for the discontinued federated entities. Third, funds were provided for productive partnerships with public, private and small and medium enterprise sectors engaged in agricultural research, extension services, input supply, processing, value chains, and exports. These activities were to be strengthened through financial partnerships with commercial banks. Component D: Project Management and Monitoring . (Estimate at restructuring US$11.33 million). This was refocused to support a strategic unit for Project Management and strengthening professional accountability at district level to support the facilitation at village organization level. Component E: Convergence and Policy Support . (Estimate at restructuring US$7.28 million). For this there was transfer of activities from Component C to facilitate community access and support from line agencies and also support for staff capacity strengthening. Support was added for mainstreaming the project into the government established Divineguma and Samurdhi programs and for strengthening VOs for sustainability activities. There was also assistance for linking VOs to government agencies to achieve convergence with government programs for public service support following the policy changes. At the time of the first restructuring in 2013, because some of the APL performance triggers were no longer attainable, the lending instrument of an APL was dropped in favor of a traditional SIL (Sector Investment Loan). One of the main APL phase performance triggers had been the establishment of federations linked to the private sector. However with the new government policy not in favor of federations there was a focus on a more centralized government-based institutional mechanism. At about this time, the Gemi Diriya Foundation that had managed project implementation was dissolved and replaced with a Project Management Unit under the Ministry of Economic Development. Some additional safeguards were added related to natural habitats, pest management, and involuntary resettlement, due to risks in sub-projects supporting inter-village connectivity. Credit proceeds were reallocated to accommodate the changing needs. The percentage funding support was revised upwards to 80% and 70% from 75% and 65% for development and operating cost respectively. This was to substitute for a reduction in community contribution. There were also major changes in 2013 arising from changes in the 1996 Company Act that effectively abolished most Village Organizations because they were unable to comply with the financial and technical requirements to be registered as companies. Finally, the PDO was revised along with some indicators. A second, Level 2, restructuring in February 2014 resulted in other changes, increasing the funding ratio because government discontinued counterpart funding, scaling back activities to adjust for a more realistic implementation timeline, and adding a six-month project extension. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost Total Project Cost at appraisal was US$105.00 million. Actual was US$97.07 million. Financing The credit was US$75 million equivalent. Borrower Contribution The government contributed an actual US$14.84 million, a little less than planned. There was a local community contribution of US$7.97 million. Dates The project was effective on 12/11/2009 and closed on 09/30/2014. The two restructurings were on 05/13/2013 (the most substantive one) and 02/14/2014. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Original Objectives Rated High . Revised Objectives Rated Substantial Rated High for the original objective but only Substantial for the revised objective because the latter gave somewhat less emphasis to the empowerment element that was embodied in the Theory of Change for this project. The ICR notes (page 16) that the objectives were consistent with the government's poverty reduction strategy and the focus on marginalized communities as reflected in the 2013 Divineguma Act. It reports that a national policy framework is planned and areas identified as priorities include support to smallholder farmers, for vulnerable groups, for youth, and for the development and regulation of micro-finance. The project at closing was consistent with the Bank’s Country Partnership Strategy (2012 to 2016) which included increased support for livelihoods especially among disadvantaged groups and a focus on agricultural productivity. b. Relevance of Design: Original Rated Substantial . Revised Rated Substantial . The original design reflected the state-of-the-art on community driven development projects. It drew from earlier experience in Sri Lanka and also in India (Andhra Pradesh), Indonesia (Kecamatan Development Projects), and Brazil. The design Incorporated core project principles including transparency, participation, social accountability, and direct fund transfer to communities. It adopted an implementation agency that was a registered foundation, the Gemi Diriya Foundation, operating largely outside the government line agency structure. This had advantages and disadvantages. There was an advantage in ease of implementation, the ability to get close to communities, and flexibility. It also built on "what worked" as determined by a high quality independent assessment commissioned under the project's first phase. At the same time, there were also disadvantages in terms of longer term sustainability due to the sidelining of government agencies. The strengthening of village organizations and the proposal in the design to move towards federations to give those organizations support was sound on the basis of experience, drawing from experience under the prior project as well as from global experience, but it supported an institutional arrangement that was inconsistent with emerging government strategy. There are questions about whether the original design sufficiently addressed the known changing policy environment which included the abandoning of the VOs along with their proposed federations. This is discussed further under Quality at Entry. The original results framework and associated logical chain was generally sound with a focus on both outcomes and supporting outputs but it did not sufficiently address the significant institutional changes ahead. The revised design endeavored to respond more to the government policy shifts that had been signalled earlier. There are questions about whether the design, which moved away from VOs and Federations to a more centralized top-down approach through government agencies, is substantially relevant. However, the Bank was endeavoring to make the best of the strategy shift. Village Organizations were being abandoned except, in due course, for the few that had sufficiently high capacity to meet standards for company registration. There were sustainability arguments for the new strategy. But the revised design at restructuring offered an uncertain alternative to the empowerment roles the VOs were playing. The original notion of Federations supporting VOs had been sound but under the new strategy, federations were no longer relevant since there was to be a phase-out of VOs. It remains to be seen whether a more centralized top-down approach can achieve what was previously approached through a bottom-up approach. The revised results framework was generally sound in terms of responding to the objectives and the new policies. However, the logical chain from inputs to outputs to outcomes was weakened in the absence of VOs and federations. To sum up, with the benefit of hindsight, while the original design insufficiently addressed the fundamental institutional changes already planned, it was based on state of the art community development practices and "what worked" during the project's first phase. The revised design attempted to address convergence with line agencies to accommodate the policy reality, but it struggled to offer an institutional structure and associated processes that adequately addressed the core empowerment and poverty reduction challenge. 4. Achievement of Objectives (Efficacy): Original Objectives Rated Modest. The original objective was to enhance incomes and quality of life of poor households in the poorest divisions of the territory of the Recipient. A number of means were indicated including: empowerment of the poor, the development and strengthening of institutions, improved access to infrastructure and social services including support for productive activities and development of policies, institutional arrangements and procedures for the transfer of funds directly to communities and local governments. Note that, in applying a split rating for the pre- and post-restructuring periods, it is sometimes difficult to separate the pre-restructuring outputs and outcomes from the post-restructuring outputs and outcomes. There is considerable overlap. Outputs  ·All Village Savings and Credit Organizations (VSCOs) established a loan security fund for protection against default risks due to unforeseen circumstances or death.  By the time of restructuring, 19% of VO members had obtained credit through a linkage with the Bank of Ceylon.  ·Linkages were established between agricultural producers in public and private sector organizations including dairying, maize production, passionfruit, seed paddy production, tomatoes, ornamental fish export, and mechanized paddy production associated with a private business. Outcomes  Some of the assessment of the income impact lower down in this section under the heading Revised Objective is relevant here also to the pre-restructuring achievements. The income study did not, and probably could not, separate the pre- and post restructuring achievements on income changes but only some of those post-restructuring outcomes are attributable to the pre-restructuring project objectives and design. The Impact Assessment (IA) found a 39.9% incremental increase in income against the base year for 50% of the targeted households (about double the revised target of a 20% income increase and more than 25% above the original target which had been revised downwards). However, the income data did not directly differentiate the poverty cohorts so it is not clear what share the poor achieved. (The survey had a sample coverage of 2,080 treatment households and 440 control households in all nine districts. The same sample villages were used in the baseline but it is not clear whether this was panel data using the same surveyed households in the end survey.)  The project ensured protection against default of the VSCO security fund. In support of the rating above of the Original Objective, in the following paragraphs the means to the achievement of the income and welfare objective are assessed with respect to the three elements of empowerment, infrastructure and policies/institutional arrangements. The following were the achievements predominantly related to empowerment: Outputs  1,010 VOs were formed. 1,034 VOs from Phase I were given continued support.  The APL program as a whole supported 1,990 VSCOs covering 38,700 small groups. Phase II achieved 918 VSCOs with 19,200 small groups, 75% were women, there were US$1 million savings, and loans of US$7.5 million.  Some 6,600 of the most vulnerable community members were provided grants.  60% of VO’s met their operational costs mainly through fees and profits.  VSCOs provided US$5 million to paddy and maize producers in Producer Groups yielding US$21 million output.  By the time of restructuring, 50% of the VOs were benefiting either from government programs or from private-sector linkages achieving 100% of the target.  Linkages were established with the Bank of Ceylon and some 95 staff of that bank were placed in bank branches to focus on project clients.  Federations became inconsistent with government policy, so the pre-restructuring project supported the formation of 144 Producer Groups, linking 112 to line agency support.  Community Professional Learning and Training Centers (CPLTCs) were established to support VO’s through the training of Community Professionals (CPs). However, the CPLTCs were unsustainable and were discontinued. By restructuring 1,890 CPUs had been trained. The CPs were increasingly sustained by salaries provided from the VO Revolving Fund activities raising questions about sustainability. Outcomes or Intermediate Outcomes  42% of households participated in VSCO activities and 80% had taken loans for agricultural purposes.  The VSCOs had savings of US$3 .2 million and took loans of US$38 million.  By the time of restructuring, 40% of VO Federations and VSCO Federations created under the project were rated A or B for functional performance, achieving 80% of the target. But this left at the time of restructuring 60% that were C or D rated suggesting a significant number of poor performers. Later the performance level improved. The ICR notes (page 19) that federations did not fully evolve into critical bridges between communities and lower government tiers. They were seen by government as competing for resources that might have gone to local authorities. This was one of the government's main reasons for abandoning them. Another related reason appears to have been that heads of federations with access to resources might have been elected and might not necessarily have been members of the party. IEG makes the assumption that there is a sustainability implication in empowerment. The policy shift that has sidelined VOs in favor of a top-down approach through line agencies appears less likely to create sustained empowerment of the poor. Looking to future sustainability, government employed Rural Development Officers will need substantial training and, due to rotation, may not be in post long enough to establish sufficient connections at village level. It is not clear what impact this is going to have on empowerment. The following were the achievements predominantly related to access to infrastructure and social services. Outputs  There was identification and prioritization of intra-village infrastructure based on decisions by members. Under Phase II, 3,554 infrastructure subprojects were initiated and 1,513 completed (42% of planned). A further 676 were in the final stages of construction and awaiting handover (together with those completed this would bring completions to 61% of planned). However, again the pre-restructuring attribution is difficult to separate from the post-restructuring.  The ICR reports (page 21) successful capacity building of Pradeshiya Sabhas (PS) including refurbishing and rearranging building space, training of staff and allocation of staff for zonal coordination to work alongside communities on interconnectivity subprojects”. PS’s were supported to evolve into service centers for their communities. In some, websites, MIS systems and e-libraries were established.  20 Producer Groups had business plans funded. Outcomes or Intermediate Outcomes  MIS data showed that 43% of targeted households were utilizing community infrastructure, a somewhat modest achievement against the 61% target. At restructuring further development of infrastructure was continuing.  The project supported organized operations and maintenance systems. 1,381 O&M accounts were set up by VOs and, according to the ICR, work was progressing to cover all VOs before the end of the project. However, it is not clear how well O&M was actually implemented by VOs and the future of O&M is now uncertain with the policy changes underway.  A total of four Pradeshiya Sabhas (PSs) had started 23 subprojects and completed 18 that were testing out an interconnectivity component that supported connective infrastructure between villages as opposed to within villages. Some delays were experienced in implementation. However, a further 6 PSs provided support for 46 subprojects. By the time of restructuring, the achievement was 51% of target. This is a modest achievement given that an important aim in the second phase was inter-connectivity. The following are the achievements predominantly related to development of policies, institutional arrangements and procedures for the transfer of funds directly to communities and local governments. Outputs, Intermediate Outcomes and Outcomes The achievement of this objective was affected by the evolving policy shift. Even at the time of project design, the Samurdhi Authority (SA) and the Federation of VOs were expected to be the implementation vehicle following the phaseout of the Foundation. However, at project effectiveness this policy was in the early stages of implementation. Prior to restructuring, the SA was implementing the project approach in a number of pilot and Samurdhi Villages to “study the dynamics in facilitation by government officials of the planning units at the district level and the officials of the SA”. At this stage there were a number of quite serious transitional and governance relationship problems with government officials handling additional functions, lack of incentives, and administrative difficulties. Collaboration with project staff at that time is reported as being superficial. Project staff were reported to be doing much of the implementation work. By the time of restructuring, the SA had not integrated community development approaches into the modus operandi. By this stage, 36% of PSs had implemented at least one inter- village subproject, 40% of the VOs were implementing networked activities, 36% of PSs were rated positively for community performance, and 54% had established MIS systems and were operating websites. Finally, as noted by the ICR (p.22), "... at the time of restructuring, (the project) had only partially succeeded in its horizontal and vertical expansions. 36% of PSs had managed to implement at least one inter-village sub-project; 40% of VOs had formed network of federations ...: 36% of PSs had received positive scores in community performance assessment; and 54% of PSs had established MIS systems. ... The remaining triggers (referring to the proposed further phase triggers) related to the operation of Federations of VOs and adoption of the CDD model at the national level could not materialize as originally envisioned." · Revised Objective The post-restructuring revised objective can be broken down into its two elements: (i) the enhancement of incomes and quality of life of poor households (similar to the before restructuring objective); and (ii) building capacity of government agencies, local governments and community organizations for service delivery and overall project implementation. Since the first is the more outcome oriented we give this more weight in aggregating the Revised Objective Efficacy rating. Enhancement of incomes and quality-of-life of poor households. Rated Substantial. Outputs VSCO loans supported 80,391 households for livelihood activities. 83% were women and 81% from the poor and poorest categories. Some 60% of loans were agriculture activities and 28% were for self-employment. These were used for agriculture and business activities, mainly agriculture (70% crop agriculture), such as paddy growing, onions, bananas, dairying, also skills development such as driver training, small businesses, marketing, and small industries (21%). The IA found that the loans contributed to a diversification of livelihoods, lowered interest rates, lower transaction costs, and increased returns to income generating activities. Outcomes or Intermediate Outcomes  There is evidence that the objective of enhancing the incomes of the poor was achieved but the evidence does not come through evidence on the first indicator (#1. "At least average 30% incremental increase in income against base year for 50% of the targeted households by the end of project." (ICR p. vii )). This indicator does not ask for the target 50% to be the poor households. Moreover, in implementation, the survey tool used to collect income data had a sample that, as far as we know, was not stratified across different income levels. The survey included a randomly selected sample of less than 1% (2,080 hh) of the total households reached across the 9 districts with a comparator of 440 for a control. As indicated in the ICR, targeted households could be categorized into four levels: 1. the poor and vulnerable; 2. members of estate communities; 3. Small and Marginal Farmers that had formed Producer Groups; and, (iv) Pradeshiya Sabha and some federated institution members support for whom had ended in 2013. The ICR does not give a clear breakdown of these different categories by cohort in terms of poverty. However, in different parts of the ICR there are indications of the number of households that may have been part of each cohort. On page 19 we learn that about 6,600 most vulnerable members (poorest of the poor) had benefitted from one-time grants and 5,700 had initiated income generating activities. We also know that of the 220,000 households, 80,391 received VSCO loans. This was 37% of all households of which between 61,901 and 64,313 were from the poor and poorest categories, which would mean between 28% - 29% of total households.(Impact Assessment 2014 and ICR p. 23). According to the IA, 88% of all loans went towards agricultural activities and self-employment. The rest were used for consumption smoothing. Based on these figures and using a best case scenario, 88% of the poor and poorest that took a loan were invested in agricultural or self-employment activities and we assume a positive private rate of return here for which there is some evidence in the Efficiency section of the ICR. So of the 64,313 poor or poorest hh we can extrapolate that 56,595 had increased incomes fhrough taking VSCO loans, representing 26% of the treated hh. There are weaknesses in this estimate because, in order to show income gains the ICR would hav e needed to show what share made a profit, broke even or made losses, for example due to household shocks.  There is also some poverty-related evidence (ICR p. 59) that per capita income for those sampled in the Phase II VOs rose by 133% from the baseline whereas the change in the control sample was 106% although whether these percentage changes were drawn from real, i.e. deflated, or nominal income data is not clear. There is also evidence presented that the Head Count Index for the Impact Study beneficiary sample fell from 43% to 26% using income data while the control area showed a much more modest reduction from 48% to only 42%. The Head Count Index based on expenditure data rather than income data in the control areas suggested some increase in those below the poverty line for the control households. The Head Count differences here triangulate to offer some supporting evidence that there was an incremental project impact.  Looking at the Quality of Life element of the objective, there was support through the extension of the VSCOs and the provision of community infrastructure creating greater access. There were 4,526 community sub-projects completed that were being accessed by 76% of all targeted hh. Many of these were public assets such as rural roads facilitating access to markets, fields, schools, health centers and input supply shops. These are very likely to have had an economic and social impact but the ICR did not measure these although there is some indirect measurement evidence within the assumptions for the ERR calculations. Building capacity of government agencies, local governments and community organizations for service delivery and overall project implementation. Rated Modest. Outputs  There was capacity development through 380 training programs related to the Community Operational Manual for a range of staff including community office bearers. The project provided training to 14,000 Economic Development Offices on the CDD approach.  All PSs were funded to implement inter-connectivity subprojects. 100% were implemented exceeding the target of 70%.  Based on MIS data, Producer Groups representing 58% of villages benefited from both public and private sector support exceeding the target set at restructuring of 50%.  In collaboration with private companies, the project supported the introduction of farm equipment including paddy transplanters, milking machines, and choppers for compost preparation.  Four memoranda of understanding were signed to improve convergence with line agencies including the Department of Agriculture, the Department of Export Agriculture, the Department of Animal Production and Health, and the Tea Small-holding Authority. Some 45,000 households were estimated to have benefited from these linkages. They covered commodities such as rice, maize, pepper, cinnamon, dairy, and tea. Outcomes or Intermediate Outcomes  The ICR reports limited success in strengthening the VOs. The project supported the building of VOs into sustainable units and, in collaboration with line agencies, to help extend the contribution of the government Divineguma Program. However, the MIS data showed only 66% of VOs graded A or B against a target of 75% and, based on IA data, only 59% covered their operating costs. As noted, VOs were later to be phased out.  According to the IA, about 70% of VSCOs had an on-time repayment rate of 95%, falling only somewhat short of the 80% target.  The ICR reports (page 26) that productivity enhancement programs implemented with government and private sector support showed “great impact on productivity and farmer incomes”. Household sample data is not provided but the ICR reports evidence of one experiment with maize farmers giving an average income increase of about Rs85,000 per farmer.  The ICR reports (page 25) that funds supporting the PSs, including funds for software accounting packages, revenue collection, water billing etc. led to an increase of 45% in revenue collection and a 30% reduction in unpaid water tariffs.  All 11 PSs are now receiving higher ratings in the government performance grading system. The project team reported that the computerized billing systems established improved turnaround time in following-up unpaid bills and improved accuracy and repayment. Grievance systems for each PS improved the addressing of complaints. Results were posted on websites. It is reported that technical service providers, although sometimes lacking experience, enhanced capacity to provide services to communities.  MIS data indicated that at least 83% of villages were utilizing inter-connectivity infrastructure for improved market access and access to schools and hospitals exceeding the 60% target.  As noted, there was establishment of an O&M fund and unit in each PS with an average monthly contribution of about US$20 at the start -- a modest scale of contribution. But by the end of the project this had increased to an average 13% of the PS budget. How much this 13% of budget represents in relation to the optimal O&M needs on the ground is not clear.  Participating PSs started collecting community proposals for budget preparation and from 2013 onwards allocated a substantial share of financial resources for proposals submitted by communities. According to a thematic study by the PMU, at least 40% of community proposals were included in annual budgets.  Based on the MIS system, 72% of PSs developed participatory processes and accountability mechanisms somewhat exceeding the 70% target at restructuring. A community scorecard system provided evidence of improved service delivery at PS level and the accountability mechanism helped to identify governance and corruption risks. However, the policy shift away from a bottom-up CDD approach through VOs and their supporting federations to the proposed top-down approach through government line agencies raises significant sustainability and empowerment questions. This is discussed further under the Risks section. There is limited evidence so far on the extent to which local government management performance changed beyond some improvement in cost recovery. Finally, the second phase of the program overall did not achieve the performance triggers set for the originally proposed Phase III of the Adjustable Program loan which was not pursued. 5. Efficiency: No ex ante rate of return analysis was done for the PAD based on the argument that the investments could not be projected due to their demand-driven nature and that empowerment was difficult to quantify. While the latter is a valid argument, it is not clear why some attempt could not have been made on the demand-driven activities since this was a second phase with some experience of what might be priority community choices. In the ex post ICR rate of return calculation, the benefit streams were calculated for inter-village development (roads, bridges, new irrigation infrastructure), intra-village development (roads, bridges, irrigation, culverts, water distribution, sanitation improvement, the Livelihoods Support Fund, the Skills Development Fund, and VSCO loans for productive purposes), and for producer group partnerships. Cost streams were based on overall project costs linked to disbursements and component expenditure on an annual basis. Assumptions were made about the number of beneficiaries per period based on the disbursement schedule. It is not entirely clear why MIS data of actual numbers by activity and year was not used for this. Data sources included project accounts, the MIS database, thematic studies, the Impact Assessment and the borrowers Completion Report. However, the ICR notes (page 47) that the impact study did not provide detailed analysis of ground-level interventions or impacts, the focus was on statistical analysis of a questionnaire. The ICR notes that the impact study data “could not be disaggregated or attributed to sub-component investments and was largely unusable for cost-benefit analysis”. For prices, economic adjustments were made using parity prices for internationally traded goods and a Standard Conversion Factor for non-tradables of 0.8 was applied. The project had direct and indirect beneficiaries and, in the analysis, the assumption was that indirect beneficiaries, for example of roads, would receive 50% of the benefits of direct beneficiaries. Benefits were calculated based on estimates of crop models and estimated (not, it appears, measured) changes in planted area arising from improved connectivity, reduced post-harvest losses and an effective increase in farm gate price. Irrigation benefits applied a four crop model with an assumption (again not measured) of a 30% increase in yield. Drinking water and sanitation sub-project benefits are based largely on the WHO Daily Adjusted Life Years (DALY) methodology using already estimated Sri Lanka diarrheal disease data, thus well validated. But this was national level data not project area data. Cost savings from the project were claimed due to improved management, reduced investment supervision costs, and the use of competitively priced community labor and in-kind contributions. There was an assumption of a 20% cost saving over conventional construction. (Other estimates of cost savings reported in the ICR from other sources range from 9% to about 30%.) VSCO loans (at 18%) were assumed to give benefits in agriculture (70% of the total), animal husbandry, small industries and other categories. Again, for VSCO loans a four crop model was estimated. Based on data from a thematic study on micro-finance by the PMU and loan members responses an estimated increase in productivity of 20% was applied. Assumptions were made about the size of crop enterprises that appear realistic. For the One-Time Grant (OTG) program, designed to provide a financial start for poorer individuals, assumptions had to be made about revenue gains. Similarly, under the Skill Development Loans for activities such as dressmaking, driving and IT skills an assumption of an increase of Rupees 50 per day was made. Under the Producer Group analysis, five value chains were assumed and data from case studies, MIS, and ICR field assessment was used. However there were limitations in evidence since PG activities did not start until mid-2012. Based on this quite detailed financial and economic analysis, albeit with a mix of data and assumptions and more of the latter than the former, the overall economic rate of return was estimated at 22.4% and for the financial rate of return 23.7%. No sensitivity analysis was attempted but the ICR argues with some justification (page 54) that the assumptions are conservative partly because no further incremental benefit increases attributable to the project over the estimated end of project 2014 levels are assumed. As noted by the ICR (page 29), in terms of cost effectiveness and particularly operational efficiency the significant midterm disruptions of the policy changes, the shift from the Foundation to the government PMU management, resulted in some efficiency losses as the change process was protracted. It probably also reduced the potential impact for the beneficiaries by the end of the project due to this hiatus. As noted earlier, there are questions about sustainability under the new policy modality. In terms of efficiency, it seems likely that a shorter-term loss of benefits and momentum may be an unavoidable cost for the benefit of an anticipated but uncertain longer-term institutional sustainability benefits through the handling of both empowerment and support through government agencies within a still evolving institutional structure. It remains to be seen whether that will be realized. If it is not then there will be some losses in the benefit streams projected. The ICR efficiency analysis was a generally thorough analysis based on a range of data, substantial coverage, and what seem reasonably conservative assumptions. On balance, despite the uncertainties, Efficiency is rated Substantial. a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate Yes 22.4% 80% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Relevance of Objectives for the original pre-restructuring objective is rated High and for the revised objective Substantial. Although the latter was less ambitious on empowerment, it did reflect the strategy change and the sustainability rationale for that change. Relevance of Design is rated, on balance, Substantial for both the pre-restructuring and post-restructuring periods. Achievement of the stated objective prior to restructuring was, on balance, Modest. After restructuring, of the two elements of the objective, one is rated Substantial (incomes/welfare) and one Modest (capacity building). In the post-restructuring period IEG gives a higher weighting to the first element of the objective since it is more outcome oriented. Efficiency is rated Substantial despite the risks to a sustained benefit stream from the new institutional arrangements. This is partly justified by the fact that in the economic analysis, quite conservatively, no further incremental gains in yields were assumed after project closing. From these before and after restructuring outcome ratings we get an aggregate Moderately Unsatisfactory rating for the pre-restructuring period and a Satisfactory rating for the post-restructuring period. Weighting by the 69.5% disbursed before restructuring and 30.5% after restructuring (mathematically a rating value of 3 multiplied by 0.695 plus a rating value of 5 multiplied by 0.305 = a weighted score of 3.6 which rounds up to 4) gives an aggregate Outcome rating of Moderately Satisfactory. a. Outcome Rating: Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: As evolving at the time of the ICR, the new Divineguma Act of 2013 had come into force but implementation was still being worked out. There was uncertainty about the role and modalities of participation. As noted by the ICR, “this … could jeopardize the key principles of the project and adversely affect community activities”. As noted by the ICR (page 32), under the Act some 15,000 graduate students were employed as Economic Development Officers to support community development managed from the center. However, an IEG Project Performance Assessment of the earlier Gemi Diriya Project that closed in 2010 found that training of these officers was inadequate, the pay was low, and their expected movement in the service to other jobs had potential to undermine strong community-based development support. This remains a significant risk. Another risk is with the regulation of the VSCOs. The Divineguma Act took away the supervision of financial services under the act from the Monetary Board of the Central Bank. This removes an important supervision mechanism over these organizations. At the time of the ICR, the operation of the proposed Divineguma Community-Based Banks and the regulatory plans were still unclear. The new policies have taken away the legal status of Village Organizations except those few (14 in total) that are strong enough to be registered and supervised as companies. This effectively kills the majority of VOs and leaves these organizations’ use of funds, or of their remaining unutilized funds, with uncertainty and without adequate supervision. It appears that VOs may be folded into the proposed Divineguma Community-Based Organizations. However, the modalities of this remain unclear. Finally, as is often the case with such projects, there is the question of operations and maintenance. While this project has done more than many to attempt to address this by setting up an O&M fund at VO level, the continuation of maintenance of project financed investments remains a concern for the future, given the shift away from VOs. The significant risks noted here raise the question of whether the risk is so great as to seriously jeopardize the achievement of the sub-objectives some of which have at least an implication of sustainability. In particular, establishing empowerment that does not last can hardly be defined as empowerment. (In the post-restructuring objectives formulation, the language “… while building capacity of … community organizations for service delivery …” implies an element of sustainability.) Overall, IEG considers that, until the design of the Divineguma institutional linkages, responsibilities, and funding have become more clear, the policy changes put empowerment at significant risk. One of the striking signs of the core problem in the institutional and policy changes is that it is reported that the many staff who had developed skills in community development under the two projects have not been able to find employment. a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: At preparation and appraisal, the Bank took into account the lessons of Phase I and the global and regional lessons of experience and provided an adequate mix of staffing. In many respects, the project was well prepared. The results framework was generally outcome oriented and followed a logical chain. However, as noted above, the government policy shift towards a more top-down approach of implementing rural development programs through the government system which entailed phasing out the foundation and dropping VOs was known at appraisal. The project team reported that, during the first phase, a lot of effort had gone into managing the political and bureacratic changes at national and local level. From 2007 onwards there was an active dialogue and there was a Convergence Report prepared which was reflected in the design of the Phase 2. However, despite this, as noted by the ICR (page 34) there was no transition plan or exit strategy and limited discussion of the trade-offs and risks to community empowerment in this shift. Right from the time of appraisal, the change in government strategy to the more top-down approach presented the project with challenges that made the implementation of the foundation-managed VO model difficult to justify unless the project aim was purely to cushion a difficult transition or unless it had the potential to demonstrate quickly weaknesses in the new approach that might lead to adjustments. Given the awkwardly timed policy shift, the project was a risky design seeking a transition between two quite fundamentally different approaches, the newer approach arguably being premature beyond a pilot scale. The ICR also notes that there should have been a communication strategy to ensure engagement with government officials that would have made the transition easier and more open and might have preempted some of the concerns the Bank had. It is worth noting that the IEG Project Performance Assessment Report (Report No.92788 dated March 13, 2015) for the pre-cursor Phase 1 project also pointed out the need for a strategy for communicating the tested merits of the new model. Quality-at-Entry Rating: Moderately Satisfactory b. Quality of supervision: The records show that a good balance of staffing skills and adequate budget was applied to supervision. The supervision documents suggest generally strong identification of issues and response to those issues. However, partly arising from the concern noted above on the Quality of Entry, as reported by the ICR (page 34), the borrowers Project Completion Report noted that the Bank did not pay sufficient attention to an exit strategy for Village Organizations. The ICR responds to this by noting that this was an issue quite extensively discussed in supervision reports in terms of recommendations but that there was insufficient follow-up on those recommendations. This was a continuation of the design-stage issues noted above. Another weakness was that the Task Team Leader changed four times over the project period. However, this may have had only a modest negative impact since all of them had been team members. Finally, as noted in the ICR, the restructuring that eventually came in May 2013 could have been done earlier which would have given more time for the preparation of the significant transition of strategy and institutional responsibilities. Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: Appraisal took place at the end of April 2009 with effectiveness late January 2010. However, the changes in policy were already being discussed during the later stages of the Phase 1 project. The election that changed the political environment took place in May 2010 only four months after effectiveness. From this point government commitment declined, significantly affecting implementation. The lack of commitment finally resulted in a government suspension of counterpart funding. The Bank chose to pick up the difference by bringing funding up to 100%. Even with this, the government delayed approval of the project budget. There was a delayed process also in the dissolution of the implementing agency, the Gemi Diriya Foundation. The government poorly handled the VOs. The Bank had made requests for legal recognition of VOs but nothing was done. Then the Divineguma Act was passed but the status of VOs remained in limbo without guidance and necessary bylaws. As a result, up until the time of the ICR, VOs have had no legal status other than those with strong capacity that were able to register as companies. As noted earlier, the Bank project team did not sufficiently follow through in the development of a communication strategy. This contributed to a negative perception of the project at the higher levels of government. Government Performance Rating Unsatisfactory b. Implementing Agency Performance: There were effectively two implementing agencies over the period of the project, the GDF and then later, following the dissolution of the GDF, the Project Management Unit under the Ministry of Economic Development. The ICR reports some retention of staff from the first phase of the project who were familiar with the project model. The ICR offers little information about the performance of GDF, perhaps because it did not see GDF as the implementing agency but rather the overseeing government unit. However, GDF appear to have performed well with a range of skilled staff particularly in empowerment at community level. During the transition, GDF appears to have handled the changing responsibilities with dedication despite their pending closure. The PMU is reported by the ICR to have had generally dedicated and qualified staff and the district project management units coming under the PMU and the Pradeshya Sabha structure implementation units are reported to have been committed and motivated. In the first year the Project Director changed three times. In fact, the project never had a full-time project director since he was director of three other donor projects. This made it difficult to get timely decisions. There was a misuse of funds towards the end of the project arising from weak internal controls and resulting in a downgrading from the Moderately Satisfactory rating for most of the project to a Moderately Unsatisfactory rating. Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: Design A reasonably effective results based monitoring, evaluation and learning system had been established under the previous phase. This was continued with some changes. There was a web-based management information system that allowed the monitoring of results at village, divisional and district levels. The results-based framework from the PAD was generally sound and was the basis for the design of the monitoring system. There were some changes made to outcome and intermediate indicators at the time of project restructuring. b. M&E Implementation: The data at the village level was collected by community mobilizers who entered it into the web-based system. There was a village monitoring matrix covering all the indicators that was updated monthly. This was used for village grading that covered indicators including the share of women and the poor in decision-making positions, data on the one-time grants, capacity building, livestock support to beneficiaries, youth employment, skills development, infrastructure progress, and the performance of the village savings and credit organization. From the database, the PMU generated annual reports, progress reports, and supervision mission reports. It is not clear from the ICR how the implementation of the M&E system changed with the shift from GDF to the PMU. c. M&E Utilization: The ICR reports (page 13) that the data collected was used for decision-making regarding project activities. It notes that, while the data collection was detailed, it was not “fully optimized for decision-making”. Of particular value was the village grading system arising from the monitoring which is reported to have been time-saving, reducing the need for on-site visits to some 2,000 villages. M&E Quality Rating: Substantial 11. Other Issues a. Safeguards: Environmental and social safeguards compliance was rated by the ICR as Satisfactory. The project was rated category B. The following were the safeguards to be complied with: Natural Habitats OP 4.0, Environmental Assessment OP 4.01, and Pest Management OP 4.09. According to the ICR (page 14) the Environmental Management Plan is reported in the ICR to have been well prepared and the implementation levels were tracked in the MIS system. However, from the evidence presented it is difficult to assess the extent to which all the triggered safeguards were adhered to. For example, no evidence is presented on Pest Management or Natural Habitats. The land used for small-scale infrastructure was mostly land obtained from government agencies. The ICR notes that the project adopted a “voluntary land donation” system. This was backed up by a grievance address mechanism from the village level to the project level. It is reported that independent reviewers found the grievance mechanism to be functioning well at village level. This suggests that the resettlement policies and plans were adhered to although in voluntary land donations it is always difficult to assess the community pressures on land donors. The ICR reports (p. 14) that "there were no serious adverse social impacts or violations of safeguards". b. Fiduciary Compliance: Fiduciary Overall fiduciary compliance was rated by the ICR as Moderately Satisfactory over the full period of the project. There was a financial management information system and reporting. Internal controls and audits were reported by the ICR to be “on track” with a few identified issues. Audit reports were unqualified until FY 11. The remainder of the audits had some qualified opinions but according to the ICR the qualifications were generally not major. The main weaknesses reported when the rating was downgraded to Moderately Satisfactory were in record-keeping, reporting and internal controls. These weaknesses were not improved after this and by the final supervision mission the rating had been lowered further to Moderately Unsatisfactory. This was largely due to an allegation of misuse of funds that, when followed up, was proven. Procurement Due to lack of knowledge and experience there were some lapses in the following of procurement procedures. Most of these were at the sub-national level. No misprocurement issues were identified during supervision reviews. c. Unintended Impacts (positive or negative): d. Other: 12. Ratings: ICR IEG Review Reason for Disagreement/Comments Outcome: Moderately Moderately Satisfactory Satisfactory Risk to Development Significant Significant Outcome: Bank Performance: Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Government commitment was weak Satisfactory Unsatisfactory and variable and at one point had declined to the point of not providing counterpart funding. Quality of ICR: Satisfactory NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The following lessons are taken largely from the ICR with some modification, 1. Effective implementation of a project expected to be integrated into a government structure requires the design of a clear transition plan from the outset covering the responsibilities of all institutions playing a role in the transition. In this case, despite the planned dissolution of the Gemi Diriya Foundation there was no clear implementation plan setting out processes, stages, responsibilities, and milestones. 2. In a project managed by a foundation or other entity outside government , it is particularly important to have a strategy for communicating with senior responsible government officials in line agencies and more so if there is to be a transition towards greater responsibility of those line agencies . In this case, the Bank and Foundation communication with senior government officials was weak and the project was not "sold" well. There was insufficient understanding in government of the potential for results. 3. One of the most important elements in the design of a community driven development project that is aimed at greater empowerment of the poor is to ensure responsiveness and flexibility while also involving government line agencies that , although typically less nimble , have resources and skills important for longer -term sustainability. In this case, there were tensions between the project and government agencies because the management of the necessary links and complementary arrangements were not well designed or implemented. 4. The mix of public and private goods in CDD operations calls for community organization linkages to local authorities for public goods and a federation of village level societies to achieve scale for private goods. In this case, the intent had been to use the Samurdhi Authority for public goods and Federations of VOs for private. However, both needed to work with VOs. In the absence of registered and supervised VO’s, the voice and empowerment to demand public and private goods may now be lost. There are a number of other thoughtful lessons in the ICR that are worth referring to for those designing CDD operations. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR is analytical and thorough and is open about weaknesses. The lessons are thoughtful and drawn from the project experience at both the broader project level and the more specific CDD activity level. The economic analysis is detailed although it would have benefited from a sensitivity analysis. The ICR is generally outcome oriented in its analysis of efficacy. However, it would have benefited from a few improvements: (i) a more probing analysis of the extent to which the new top-down policy, that so far seems unfavorable to empowerment, affected the efficacy rating; (ii) it is not entirely clear from the ICR how the shift from the GDF to the PMU impacted M&E; and, the biggest weakness, (iii) it analyzed efficacy against the key indicators not the objectives (with, where relevant, the key indicators in support). Projects should be rated against objectives not key indicators which may provide supporting data but may not fully reflect the objectives. a.Quality of ICR Rating : Satisfactory