Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LK: N&E Local Services Improvement (P113036) Report Number : ICRR0021324 1. Project Data Project ID Project Name P113036 LK: N&E Local Services Improvement Country Practice Area(Lead) Additional Financing Social, Urban, Rural and Resilience P152623,P158048,P158290 Sri Lanka Global Practice L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-47280,IDA-56880,TF-13787 31-Dec-2015 90,729,152.75 Bank Approval Date Closing Date (Actual) 13-May-2010 31-Aug-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 50,000,000.00 20,300,000.00 Revised Commitment 69,818,118.81 20,299,131.11 Actual 70,430,021.64 20,299,131.11 Prepared by Reviewed by ICR Review Coordinator Group Cynthia Nunez-Ollero Vibecke Dixon Christopher David Nelson IEGSD (Unit 4) PHPROJECTDATATBL Project ID Project Name North East Local Services P150140 Improvement Pr ( P150140 ) L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) 0.00 Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LK: N&E Local Services Improvement (P113036) Bank Approval Date Closing Date (Actual) 29-Apr-2014 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 0.00 Revised Commitment 0.00 0.00 Actual 0.00 0.00 2. Project Objectives and Components a. Objectives The Financing Agreement (FA) stated that the Project Development Objective (PDO) was "to support local government authorities in the Northern and Eastern provinces of Sri Lanka to deliver services and local infrastructure in a responsive and accountable manner." (FA, Annex 1, p. 16). The PDO cited in the PAD was slightly different - "to improve the delivery of local infrastructure services by local authorities in the Northern and Eastern Provinces of Sri Lanka in an accountable and responsive manner," (PAD, p. 14). The PDO in the Additional Financing (AF) agreement in 2014 expanded the project area "to support local authorities in the Northern and Eastern Provinces and Adjoining Provinces to deliver services and local infrastructure in a responsive and accountable manner" (AF, 2014, letter to the Borrower) but was otherwise identical to the original PDO. This ICR will assess the PDO in the 2014 AF: • "to support local authorities in the Northern and Eastern Provinces and Adjoining Provinces to deliver services and local infrastructure in a responsive manner." • "to support local authorities in the Northern and Eastern Provinces and Adjoining Provinces to deliver services and local infrastructure in an accountable manner." b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LK: N&E Local Services Improvement (P113036) 1: Infrastructure Service Delivery (US$ 76.0 million at appraisal of which US$ 41.0 million was IDA. The 2014 AF introduced two changes to this component - first, the IDA allocation for this component was reduced by US$ 4.5 million to US$ 36.5 million; and second, US$ 18.3 million of resources from the Australian Department of Foreign Affairs and Trade (DFAT) were added for a total allocation for this component of US$ 89.8 million, US$ 91.9 million actual). This component financed eligible subprojects in participating provinces including roads, water supply, rural electrification, markets or weekly fairs, drainage systems, public parks and playgrounds, and waste collection facilities. 2: Institutionalizing Accountabilities (US$ 2.0 million in IDA resources at appraisal. The 2014 AF introduced two changes to this component - first, the IDA allocation was reduced by US$ 1.5 million to US$ 0.5 million; and second, US$ 0.5 million in DFAT resources were added for a total allocation of US$ 1.0 million, US$ 1.0 million actual). This component financed independent annual financial audits of Local Authorities (LAs), social and technical audits of LAs' public expenditures, systems and processes in support of greater transparency of LA activities. This component also financed community driven development (CDD) initiatives such as participatory planning, grievance redress, and Social Audit Committees (SACs) to ensure community engagement and LA accountability. In addition, this component financed an Information Education and Communication campaign. 3: Building Capacities (US$ 4.0 million in IDA resources at appraisal. The 2014 AF added US$ 7.5 million in IDA resurces and another US$ 0.5 million in DFAT resources for a total allocation of US$ 12.0 million, US$ 14.0 million actual). This component financed improvements in efficiency of LA systems and procedures related to planning, budgeting, financial management, revenue management, and procurement; improvements in efficiency, timeliness and follow up of the internal and external audits of LAs, training elected LA representatives and staff, day to day support to LAs, and a long-term LA capacity building strategy. This component was increased at the request of the Government to accommodate the reconstruction of LA office buildings damaged during the conflict. 4: Assessments and Evaluation (US$ 1.0 million in IDA resources at appraisal. The 2014 AF introduced two changes to this component - first, IDA resources were reduced by US$ 0.5 million to US$ 0.5 million; and second, US$ 0.5 million in DFAT resources were added for a total allocation of US$ 1.0 million, US$ 1.0 million actual). This component financed the establishment of a comprehensive monitoring system, evaluating technical and social audits, and preparation of citizen score cards and other evaluative analyses. 5: Project Management (US$ 3.0 million at appraisal, of which US$ 2.0 million were IDA resources. The 2014 AF introduced two changes to this component - first, the IDA allocation was reduced by US$ 1.0 million to US$ 1.0 million; and second, US$ 0.5 million in DFAT resources were added leading to a total allocation of US$ 2.5 million, US$ 2.5 million actual). This component financed consultants, equipment, and operational support to key participating agencies at the central, provincial, and local levels in managing project implementation. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The total project cost reached US$ 126.30 million and disbursed a total of US$ 106.73 million or 84.5% of original project cost. There was a shortfall in the central Government's counterpart financing (see below). Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LK: N&E Local Services Improvement (P113036) Financing: The International Development Association (IDA) credit financed this project with a US$ 50.0 million Specific Investment Loan. The Australian Department of Foreign Affairs and Trade (DFAT) co- financed the project with an additional US$ 20.3 million in grants to expand the geographic scope of the project and include three Adjoining Provinces. IDA provided an additional US$ 20.0 million credit to address the shortfall in Government counterpart financing (see below). The Government provided counterpart financing of US$ 34.0 million from the Central Government and US$ 2.0 million from the local authorities. Borrower Contribution: The Borrower committed US$ 34.0 million in counterpart financing but disbursed only US$ 14.0 million because of budgetary constraints (ICR, paragraph 13). Local communities also committed and disbursed another US$ 2 million in counterpart financing. Dates: The project was approved on May 13, 2010 and effective on December 15, 2010. The Mid Term Review was conducted on March 12, 2013. The original closing date was December 31, 2015 and the actual closing date was August 31, 2017, i.e., a 20 month's extension. There were three restructurings: • a Level 1 restructuring on April 29, 2014 to supplement the project with a grant of US$ 20.3 million from the Australian DFAT as co-financing and expand the geographic scope of the project to include three Adjoining Provinces and change the financial allocations among the project components. • a Level 2 restructuring on August 17, 2015 to provide US$ 20.0 million in additional IDA credit to compensate for the shortfall in Government counterpart financing and extend the closing date to December 31, 2016. The Government moved its Project Coordination Unit (PCU) to its newly constituted Ministry of Provincial Council and Local Government (MPCLG). • a Level 2 restructuring on November 13, 2016 to extend the project closing date to August 31, 2017 in order to complete 50 subprojects financed under the AF and reallocate resources among disbursement categories. 3. Relevance of Objectives Rationale The objectives remained relevant to the country's development plan. Its strategy envisaged a globally competitive, private sector led economy with an emphasis on jobs and inclusion. In its Vision 2025, under Governance and Accountability, the Government indicated that it will strengthen its policies of good governance and strengthen reconciliation efforts, which form the backdrop of the PDOs. In addition, the Government pledged to improve participation in governance, similar to efforts addressed by the project's objectives. The objectives also remained relevant to the World Bank's Country Partnership Framework (CPF) for FY 17- 20. One of the three pillars of this partnership was promoting inclusion and opportunities for all. To support this pillar, interventions would address inclusion challenges and economic opportunities through reforms and transition to a more effective model for service delivery and local governance (CPF, paragraph 62.2). In addition, improved governance was also noted as a cross cutting objective of the CPF. The objective of this project supported improved governance through accountability measures of participating Local Authorities Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LK: N&E Local Services Improvement (P113036) (LAs). The emphasis of the objective under the AF on gender equality, job creation, and livelihoods also indicated relevance to the CPF. Women were integrated into advisory roles and the project supported increased female participation. However, while there was a clear alignment between the PDOs and the country and WB strategies, the relevance of the objectives did not adequately address a potential solution to a development problem. While acknowledging the difficulty of the operational environment, the PDOs should have been directed at a level where it could be attributed to the improvements that the local infrastructure services would lead to, whether it be improved living standards, improved health outcomes, improved production, and increased income or other similar factors favorably affecting the community's livelihoods. These may be longer term targets but tracking them and identifying them were important aspects of a successful development operation. Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective to support local authorities (LAs) in the Northern and Eastern Provinces and Adjoining Provinces to deliver services and local infrastructure in a responsive manner Rationale The original PDO was modified by the 2014 AF to expand the geographic scope of the project and include three Adjoining Provinces. The target values of the key outcome indicators were revised to accommodate this expansion but the overall objective was not changed. The objective was expressed in input terms (support to LAs), did not focus on the expected outcomes of the services and local infrastructure delivered.There were nine key outcome indicators and 10 intermediate outcome indicators, which were all at output levels. OUTPUTS: The following outputs for delivery of services in a responsive manner were delivered as targeted or exceeded: • 99% of capital grants were released against allocation (baseline 0%, original target 70%, exceeded) and financed 1,214 eligible subprojects in 12 provinces. These predictable and transparent grants and the subprojects it financed were in response to the consultative processes included in the project design. For example, subproject identification required community engagement and prioritization and approval at the district level required the participation of elected officials as well as village level constituents in response to defined public needs. • A total of 23,962 complaints and 9,244 requests or suggestions were reported to the Assistant Commissioner of Local Government, 98.7% of registered grievances against projects delivered were Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LK: N&E Local Services Improvement (P113036) resolved within a quarter (baseline 0, original target 80%, exceeded) indicating an increased level of participation and responsiveness to issues raised. • 96.4% of subprojects established community engagement post project and Operations and Maintenance (O&M) provisions (baseline, 0, original target 80%, exceeded) in response to heightened constituent participation. • 282 training programs were delivered (no target reported) training 9,941 officials (no target reported). The training included Right to Information, training on public participation methods to guide how to deliver services in a responsive manner. The following target outputs for delivery of local infrastructure in a responsive manner were achieved or exceeded: • 2,275 new piped household water connections (baseline 7, original target 600, exceeded) were installed. These targets reflected village and district level consultative processes implemented under the project. • 11 km of constructed and rehabilitated rural electrification (baseline 0, original target 10 km, exceeded) were completed n response to target setting consultative processes at the village and district levels designed under the project. The following target outputs for delivery of local infrastructure in a responsive manner were not achieved or almost achieved in response to target setting activities during the village and district level consultative processes under the project: • 155 markets and maternity homes were constructed and rehabilitated (baseline 0, original target 200) • 49,898 meters of drainage system were constructed (baseline 0, original target 245,000 meters) • 84 playgrounds and nurseries were constructed (baseline 0, original target 205) • 790 km of rehabilitated roads (baseline 0, original target 1,000 km) • 98.4% of subprojects delivered community contributions (baseline 0, target 100%) • 1 waste collection facility constructed and rehabilitated (baseline 0, original target 100) OUTCOMES: • The Government estimated that 2.5 million people benefited from the project, of which 1.3 million were female. DFAT estimated that the project resulted in creating 3,900 new jobs annually and generated US$ 241.8 million in infrastructure related economic activities as well as improved the capacity of 200 local contractors. • The responsiveness of service delivery in the LAs improved with training received under the project. The training staff covered topics such as participatory methodologies, financial and procurement management, good governance, social auditing, and information and communications. As a result of the training received, participatory local development plans were prepared. According to perception surveys conducted after project close, 95% of staff who attended the training were satisfied with the training Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LK: N&E Local Services Improvement (P113036) received although evidence of usefulness, particularly in procurement management proved inadequate. All the project extensions were attributed to lack of capacity in procurement management. • The Government adopted governance requirements introduced under the project indicating that these improvements have been mainstreamed. • Additional assessments of the impact of the project were carried out after completion using a combination of beneficiary perception surveys and geo spatial analysis. These beneficiary surveys indicated citizen participation and responsiveness of the LAs. The Government conducted a survey of beneficiaries to measure their satisfaction with the project at each district level. These perception surveys randomly surveyed 100 residents in each of the 101 LAs and found 100% satisfaction with reduced travel times in the roads funded under the project as well as in the travel comfort provided by these same roads. 95% were satisfied with the reduced accident rates from improved road services. Those surveyed also expressed 100% satisfaction with the increased land values, reduced vehicle maintenance, and increased access to employment and education. Drainage projects improved road conditions, reduced mosquito levels, and lowered levels of incidence of floods. 98% of surveyed beneficiaries reported satisfaction that the project investments reflected their expressed needs (target 1.9 million, achieved 98% - there was a mismatch of nomenclature between target (number) and achievement (in % share) in the PDO indicators but not in the intermediate indicators - see ICR, footnote 15). • At project close, five year Local Authority Participatory Development Plans were in place and annually revised according to post project interviews and that participating LAs planned to continue the practice. A new operation is under preparation that will build on these plans. • The Social Audit Committees (SACs) provided quality control and ensured timely delivery of inputs. The SACs provided an informal social space for the villagers to air their concerns, exchange information with the authorities, and mediate disputes. The Ministry of Provincial Councils and Local Government drafted a policy paper mandating these committees for all local service delivery for cabinet review. • Pilot geospatial analysis conducted by the Bank after project close indicated improved accessibility to employment, education, and social services. The analysis used 982 (of the 1,214 subprojects) and 12 in- depth case studies of these subprojects estimated the number of subproject beneficiaries using satellite data and isochrome maps to measure improved cooectivity to local assets such as markets and schools. These assets and residential units within a specified radius were mappesd to establish the likely travel patterns and assessed site suitability of these assets. The analysis indicated reduced travel times and improved overall accessibility. With moderate shortcomings, particularly in the choice of indicators, the outcomes showed heightened responsiveness by way of improved community engagement and enhanced inter-institutional coordination (ICR, paragraph 21). The grant system, for example, required the establishment of a standing committee for citizen participation and Annex 4 of the ICR noted that projects were identified in response to residents' needs. However, evidence of sustaining this responsiveness to delivering services would require more time. Rating Modest PHREVDELTBL Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LK: N&E Local Services Improvement (P113036) PHEFFICACYTBL Objective 2 Objective to support local authorities (LAs) in the Northern and Eastern Provinces and Adjoining Provinces to deliver services and local infrastructure in an accountable manner Rationale The original PDO was modified by the 2014 AF to expand the geographic scope of the project and include three Adjoining Provinces. The target values of the key outcome indicators were revised to accommodate this expansion but the overall objective was not changed. OUTPUTS: • The fiscal transfer system was strengthened as evidenced by predictable and transparent capital grants transferred directly to participating LAs. Capital grants were allocated to all LAs after meeting requirements such as adoption of participatory budgeting process, having a working grievance redress mechanism in place, completing annual financial statements, and installing mechanisms for post project O&M. • One time performance grants were awarded based on criteria related to planning, budgeting, service delivery, financial management, community participation, grievance redress, and sustainability. Assessments evaluated accountability ensuring that monthly progress review meetings were held, that implementation instructions were followed, and that LAs increased maintenance budgets to ensure subproject sustainability. • Good Governance Resource Centers were established within the 12 districts to ensure that participatory planning, procurement administration, financial management, social auditing, and safeguard mechanisms were sustained evidenced by their adoption for use in the proposed follow on project, the Local Area Development Support Project. • At the provincial level, the Commissioner of Local Government organized a Local Government Division to guide and monitor the 12 Good Governance Resource Centers. • 1,228 Social Audit Committees were formed (no target reported) with 6,610 members (no target reported) who were elected at community meetings and trained on social auditing. These committees served as liaison between the implementing entities, local officials, and the community. These committees reflected a level of accountability in delivering infrastructure services. • A Financial Analysis Cell (FAC) was piloted in the Office of the Deputy Chief Secretary in the Northern Province to monitor LAs financial reports on internal and external audit findings and were given annual performance scores based on criteria such as collection efficiency of own source revenue, efficiency in total expenditures, and managing surplus. While initial scores ranged between 25-58 points, by 2017, scores improved to between 46 and 96, with 28 of 34 LAs out of a universe of 101 LAs scoring over 60 points. FAC also identified new revenue sources for the Department of Provincial Revenue and the LAs implemented performance based monitoring frameworks for budgeting and expenditures, which implemented the Provincial Management Information Systems. Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LK: N&E Local Services Improvement (P113036) • After project close, the FAC conducted a workshop to assist the Northern Province's LAs with preparing their 2018 budgets. OUTCOMES: • The project replaced the former accounting system of the LAs with a simplified Sri Lanka Public Sector Accounting Standards first introduced in 2012. In January 2018, the Government's cabinet approved the preparation and implementation of a common accounting system for all local government institutions based on what the project adopted for its participant LAs. The local accounting and the procurement systems introduced under the project were now required for all LA operations regardless of funding source. • Additional assessments of the impact of the project were carried out after completion using a combination of beneficiary surveys, geo spatial analysis, and completion reports from each participating LA. The beneficiary surveys indicated citizen participation and improved accountability of the LAs as evidenced by five-year Local Authority Participatory Development Plans in place by project close, to be annually revised and with participant LAs planning to continue the practice, according to post project interviews. A new operation was under preparation that will build on these plans. • The Social Audit Committees (SACs) provided an informal social space for the villagers to air their concerns, exchange information with the authorities, and mediate disputes. The Ministry of Provincial Councils and Local Government drafted a policy paper mandating these committees for all local service delivery for cabinet review, reflecting the adoption of accountability measures in delivering services. • LAs achieved their counterpart commitments which were measured in kind (i.e., labor). The outcomes of this objective showed a positive direction of meeting accountability in delivering services. A framework was installed including systems and procedures to assess transparent and predictable capital grants. Simplified procedures in grant transfers were instituted. Rating Substantial PHREVDELTBL PHOVRLEFFRATTBL Rationale The Theory of Change for this project represented by its Results Framework lacked an explicit articulation of the expected outcomes of the PDO. The PDO itself was at a relatively low level, aiming for support to local authorities (a project input) and did not express higher level outcomes that the activities could lead to. These included those outcomes that were noted in the project surveys such as improved travel time to reach markets, reduced accident rates from improved roads, lowered incidence of floods with improved drainage, improved earnings from economic activities facilitated by the improved markets. There was a lack of indicators that could have been monitored over the life of the project to support these outcomes. These outcomes would have been useful to document the project's contribution to the higher level objective of inclusion and strengthened local Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LK: N&E Local Services Improvement (P113036) government institutions. The subprojects were demand driven based on community needs. However, seven of eight key outcome indicators in the Results Framework were outputs. Four intermediate outcome indicators in support of the outcome were also outputs. A few of the target output indicators were not achieved but most were exceeded. Perception surveys expressed high satisfaction rates with the project outputs and processes and geospatial analysis showed improved access to employment education, and social services. The Government adopted a common accounting system first introduced by the project, and citizens were given voice as a result of their active engagement in the project's Social Audit Committees. The Government submitted policy papers to the Cabinet to consider institutionalizing the bodies and processes first introduced under the project. Some of the institutional arrangements introduced under the project were also being considered under the proposed follow on Local Area Development Support Project. Overall Efficacy Rating Substantial PHREVISEDTBL 5. Efficiency Economic Efficiency: There was no economic analysis of subproject investments during preparation because these were designed to be demand driven and identified during implementation. The PAD noted that there was a lack of basic service delivery data. The project investments were directed at supporting institutions and a conducive environment to better delivery of basic services (PAD, Annex 9, paragraph 4). During the 2014 AF, randomly selected subprojects showed satisfactory economic internal rates of return (EIRRs) for drinking water supply (22%), drainage (23%), roads (22%), and rural electricity (22%). At project close, these EIRRs were not used as comparators due to lack of documentation to support these earlier estimates. Instead, a sample of subprojects with similar geographic and sector breakdown were evaluated. For roads, the least cost method alternative method was used to determine economic efficiency. Revenue generating subprojects used its ability to recover operations and maintenance (O&M) costs. Economic utilities used willingness to pay. Other services used qualitative outcomes of beneficiary surveys. Analysis showed positive economic returns for water supply (5% - 29%), electricity (55%), markets (9%-36%) and drainage (9%). Against investment amounts, the weighted average returns were: markets (21%), water supply (12%), power supply (55%) and drainage (9%). Based on least cost method, the roads subprojects had an EIRR of around 20%. Administrative and Operational Efficiency: The project was originally a 5 year project. It was extended twice for a total of 20 months to accommodate two AFs, the first, in 2014 to expand the geographic scope of the project to include Adjoining Provinces, and the second, in 2015 in response to the shortfall in Borrower counterpart financing. The PCU was moved to the new Ministry of Provincial Council and Local Government and there was no PCU staff between January and May 2015. These actions, including procurement delays, staffing shortages in some LAs due to the conflict, and Government policies of transferring staff leading to unfilled vacancies, led to implementation delays. Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LK: N&E Local Services Improvement (P113036) Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 65.00 ICR Estimate  20.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of objective was substantial. The efficacy of the project objective regarding responsiveness was rated modest primarily because of inadequate outcome indicators but the efficacy of the second objective regarding accountability was substantial because of the positive direction of instituted changes to access transparent and predictable capital grants and simplified processes for transfers. The overall efficacy of the project was substantial. The project efficiency was modest. The scope of the project expanded with the AFs and a split rating was not warranted. There were moderate shortcomings in the project's efficiency as evidenced by procurement delays, staff shortages, as well as shortfall in counterpart financing which negatively affected operational efficiency. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The following posed risks to development outcome: • Government Ownership, Governance, Political, and Institutional Support: Government ownership of the project outcomes were reflected in the submission of policy papers to the Cabinet to institutionalize aspects of the project leading to the sustainability of its outcome. However, there were no declared budgetary commitments to support the continued use by the Assistant Commissioner of Local Government offices in each district who were mandated to regularly supervise the accounting and auditing by LAs. In some cases, like the Good Governance Resource Centers established within the 12 project districts to sustain participatory planning, procurement administration, financial management, social auditing, and safeguards mechanism, were noted to have assigned relevant staff to allow them to operate. To mitigate the risk that institutional Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LK: N&E Local Services Improvement (P113036) support may not continue due to budgetary constraints, the proposed follow on project, the Local Area Development Support Project is expected to continue to use these centers to encourage regional coordination and encourage subproject clustering. • Social and Other Stakeholder Support: The institutions introduced in the project that allowed for voice, community participation, and grievance redress require follow through by the relevant authorities. These include the institutionalization of the consultative five year development planning by LAs, and the various consultative bodies engaged by this project. To mitigate this risk, a follow on operation under preparation, would include strengthening institutional support for intergovernmental fiscal transfers, and expand the capital and performance based grant system. In addition, this operation would maintain the accounting systems and the performance indicators pioneered by this project. • Technical and Financial: Some subprojects were not revenue generating and may need budgetary support for its O&M needs. In addition, the geospatial reference mapping used in M&E would require technical updates and substantial budgetary support to ensure that the latest technology are adopted. There is a risk that there will be another shortfall of financing, similar to that experienced in this project. To mitigate this risk, the project team has started talks with the European Union to explore financial support. 8. Assessment of Bank Performance a. Quality-at-Entry This project was the first reengagement by the Bank following a thirty year internal conflict in Sri Lanka. There was extensive physical damage to the infrastructure and splintered community links. Nearly all residents in the project area had poor access to the labor market and economic opportunities. With these issues, project preparation included a Governance and Accountability Action Plan and a Conflict and Reconciliation Filter to build trust between the Government and its citizens. Relevant lessons from other global operations (Uganda and Indonesia) and in the region (Bangladesh) were considered in project design - such as (I) community participation at all stages of project planning, investment and implementation, (ii) rapid provision of basic infrastructure using local resources, (iii) sufficient capable staff in place, (iv) wide information and communication to inform citizens about design, plans, and grievance redress mechanism. The Bank team proactively consulted stakeholders in project design and ensured technical buy in. Implementation arrangement was adequate. Reputational, social, and operational risks were adequately identified. Mitigation included a continuous social impact assessment to foster public trust in the state. Other mitigation efforts included accountability and effective local governance activities such as technical and social audits of investment projects. Building institutional capacity was the priority focus of design and included in preparatory advisory and capacity building program. However, the M&E system did not establish baselines for beneficiary survey indicators. The PDO was low level and the indicators in the Results Framework were mostly output indicators that did not capture evidence to support the project outcomes. The cause and effect framework was rather limited to inputs and outputs. Safeguard documents were delayed. Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LK: N&E Local Services Improvement (P113036) Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The Task Team Leader and team specialists were based in Colombo, Sri Lanka, which facilitated frequent field level supervision and monitoring and timely assistance to the implementing unit. In 2014, the Australia DFAT added resources to expand the geographic scope of the project to the Adjoining Provinces. This move resulted in a total of 101 LAs, up from the original 79. The fund transfer arrangement was also modified when disbursements were noticeably delayed. War and conflict also affected implementation in affected LAs and the Bank team responded with additional training and support. However, M&E implementation could have benefited from more robust institutional PDO target indicators and more rigorous cross checking of quarterly and annual reports. The lack of addressing the weaknesses in the indicators used in the Results Framework contributed to the project's failure to measure and document results at the outcome level. There was an effort in the 2015 AF to introduce indicators that were monitored but not part of the Results Framework and others removed due to duplication or refined to improve relevance, but output indicators remained. The team continued to conduct field based reviews and LA based training even as the Government did not have counterpart staff in place January to May 2015. The team continued to support and coordinate with Project Appraisal Teams and Social Audit Committees to ensure timely implementation of subprojects. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design A computerized Management Information System (MIS) was developed to track LAs expenditures against measurable outputs. The project's Results Framework reflected how the activities and outputs would lead to the project outcome but was too rigid to allow for subprojects that were identified during implementation. Specific service delivery targets were not included in the PAD. Targets were only set formally in the 2014 AF and revised in the 2015 AF. However, the 2015 AF targets reverted to the original informal targets that were replaced by the 2014 AF targets. The Results Framework did not include outcome indicators that could have better captured the impact of the many institutional tools of local governance and community driven operations under the project. Women participation in the project were not adequately tracked. Only one intermediate outcome indicator, rather than a key outcome indicator referred to improved institutional accountability. Since the projects were demand driven and prioritized during implementation, the Results Framework did not include adequate Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LK: N&E Local Services Improvement (P113036) indicators that could have captured results on the ground. There were errors in the unit of measure and in the phrasing of indicators, particularly with indicators associated with beneficiaries, such as expressing beneficiary satisfaction in numbers rather than percent share of surveyed respondents, and outcome indicators expressed in units of measure of outputs. b. M&E Implementation The implementing agency, first, the Ministry of Economic Development and then replaced in 2015 by the Ministry of Provincial Council and Local Government implemented the M&E system. LAs collected and tracked financial audits and data on infrastructure investments. LAs were trained to collect and maintain regular data and subproject related information. All 101 LAs completed progress reports for their subprojects which included information related to subproject selection. LAs also submitted subproject data to offices of the Commissioner of Local Government, who then prepared quarterly and annual progress reports to the implementing agency and the Provincial Department of Census and Statistics' Finance Department. The project fully achieved the tracking of the timely submission of quarterly reports as an intermediate indicator. A baseline survey was completed in 2012 but noted that it was not consistent with the Results Framework (ICR, paragraph 40). The M&E system was not in place unitl 2016 to track real-time subproject progress (ICR, paragraph 57). The M&E system did not establish baseline for beneficiary survey indicators (ICR, paragraph 83). c. M&E Utilization The LAs and the implementing agency generated reports that were used to monitor and evaluate the project. In 2016, DFAT supported efforts to develop a matrix to track sub projects at risk of not being completed by project close, which led to full disbursement of project funds and completion of all subprojects. Direct beneficiary satisfaction values reflected in the Results Framework were drawn from a onetime survey and supported by the project completion reports, the completion report prepared by the Government, and the Impact Assessment Study. A post project geo referencing analysis confirmed the number of beneficiaries for individual subprojects using high remote sensing data to map human settlements. Population information was validated using 2012 census and cross checked with subproject completion reports. Overall, the Results Framework was not as effective in informing the progress of reaching key project development outcomes or warning on problem areas. For example, there was no reported use of the score cards identified as a key outcome indicator. There was a lack of dedicated M&E staff and no supporting M&E staff at the provincial or District levels. There were opportunities during the 2014 and 2015 AFs to refine targets based on priority subprojects but service delivery indicators did not reflect investments made under the project. Even with the positive features of the M&E system, particularly with regard to local level institutional development and regular monitoring of accounting and procurement process, there were important shortcomings that did not capture important results achieved by the project. These shortcomings affected the project's ability to measure and document achievement of results at outcome level (efficacy), efficiency, and implementation effectiveness. Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LK: N&E Local Services Improvement (P113036) M&E Quality Rating Modest 10. Other Issues a. Safeguards Environmental Safeguards: The project was assigned a category B and triggered Safeguard Policy OP/BP 4.01 which required a partial Environmental Assessment. The Environmental and Social Management Framework (ESMF) was included in the PAD and revised in the 2014 AF. The following safeguards were triggered, including OP/BP 4.01 Environmental Assessment, OP/BP 4.04 Natural Habitats, OP/BP 4.10 Physical Cultural Resources, OP/BP 4.11 Indigenous Peoples, and OP/BP 4.12 Involuntary Resettlement. Projects were not identified during preparation but were to be identified based on demand during implementation. All site selections were to be based on social and environmental sustainability criteria and subprojects were required to have site specific Environmental Management Plan and in line with the Government's own National Environmental Act, the Land Acquisition Act, and the National Involuntary Resettlement Policy. When there is new construction, an Environmental Assessment with mitigating measures along the lines of the ESMF are prepared. Four annual training programs were conducted to build the safeguards capacity of implementing agencies. In 2016, a consultant assisted the Environmental Officer to strengthen field level monitoring in project implementation. Environmental Officers were appointed at the level of the Assistant Commissioner of Local Government and included officers from the Central Environmental Authority. An independent audit of environmental compliance in June 2017 found no significant safeguard issues. Social Safeguards: Project sites were not available during preparation but the PAD noted that the ESMF provided a negative list of characteristics that would make a subproject ineligible for support. A Framework for Land Acquisition and Involuntary Resettlement consistent with the requirements of the National Involuntary Resettlement Policy and OP/BP 4.12. No subprojects affecting more than 200 people would be allowed. And if resettlement and land acquisition were unavoidable, these were to be carried out according to the guidelines. The ICR reported that the implementing agency implemented the Information Education and Communication activity, which included social safeguards. The Gender Responsive Mechanism was published in 2016. LA staff were trained to prepare Social Impact Assessments and development Social Management Plans for critical subprojects to address adverse social impacts. The independent audit noted above indicated compliance. b. Fiduciary Compliance Financial Management: This was rated satisfactory even though there were staffing issues in the first quarter of 2015 due to changes in implementing arrangements and was accompanied by delays in financial reporting, submission of external audit reports, and weaknesses in internal audit. However, there were no significant issues in external audit reports and overall, the project submitted accurate and timely interim financial reports. The Assistant Commissioner of Local Government closely supervised LAs who managed Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LK: N&E Local Services Improvement (P113036) newly devolved project contract accounting function. Lack of capacity marked initial difficulties and the Bank responded with training. Before this project, LAs accounting capacity was limited to transfers from the Central and Provincial Governments with little oversight of local expenditures. The project established internal audits at the implementing and LA levels. An external audit recommended the use of a single accounting system, which the project installed. Procurement: There was a high volume of procurement for subprojects funded under the project. The Procurement Manual was translated into Sinhala and Tamil and adopted by LAs. There were numerous trainings on preparing and managing procurement plans. The Bank team sent fiduciary missions to the field to ensure timely disbursements, particularly when subproject contractors continued to claim lump sum final payments rather than progress payments. Procurement delays occurred because (i) technical staff were unavailable to prepare supporting procurement documents, (ii) there were seasonal delays and mismatch in timing of the procurement, (iii) there was a lack of contractor capacity and their access to credit, (iv) there was a lack of availability of materials, and (v) contract management, proper closeout and contract completion were inadequate. One procurement case was reported to the World Bank Integrity Vice Presidency, which was subsequently resolved by the Government and the World Bank. Districts that implemented subprojects have prepared procurement plans and monitored performance against set targets for all projects, not just those funded under the project. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory --- Satisfactory Moderately Bank Performance Moderately Satisfactory --- Satisfactory Quality of M&E Modest Modest --- Quality of ICR Substantial --- 12. Lessons Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LK: N&E Local Services Improvement (P113036) The Government prepared a comprehensive document highlighting seven key lessons from this operation and presented in various languages for public consumption. The ICR incorporated these lessons and presented 8 in the ICR. Some of them are slightly reworded below: • Participatory planning, transparent project prioritization, and introduction of voice, are simple design components that could be effectively used to support local government authorities to foster social cohesion, strengthen local institutions while delivering infrastructure services, and enhance accountability. These mechanisms, including participatory monitoring, establishing social accountability committees, and grievance redress mechanisms could be used to reduce the risk to sustaining project outcome and can serve as take off points for future engagement in the area of local area development support. • Another institutional support mechanism that could be promoted is the use of a body such as an independent audit agency to verify subproject outputs and their findings as part of the eligibility criteria for fiscal transfers to local authorities. • A low capacity starting point for participating local authorities unfamiliar with Bank policies and processes requires a robust just-in-time training program in areas critical to project implementation such as Financial Management, Procurement, and Contract Administration. Based on the project's experience with low contract management capacity, clustering similar subprojects may be useful during implementation. • The use of geospatial data may be an effective way to visualize the impact of a project (before and after) especially when there are numerous subproject local government investments involved. There needs to be an adequate system in place to control for attributing the changes in visuals directly to the project. In this case, after project close, the Bank analyzed 982 (of 1,214) sub-projects and conducted in depth analysis of 12 sub-project cases studies to estimate the number of sub-project beneficiaries. The studies used satellite date and isochrome maps to measure improved connectivity to local assets (e.g., schools, markets). Assets and residential units within the radius were mapped to establish likely travel patterns and assessed site suitability. Results depicted reduced travel time and improved overall accessibility. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was concise and provided a detailed overview of the project, following the guidelines. There was a logical link and integration of various parts. However, the PDO was at a relatively low level and the theory of change did not adequately support the ratings. The ratings were not well supported by the narrative that should have been more results-focused, as evident from the lack of indicators in the Results Framework that would have better reflected project outcomes. The narrative did provide the challenges posed by introducing accountability following a 30-year civil war. This was the backdrop for aiming to deliver locally determined services using a participatory method and in the process address social cohesion, reconciliation, build local government capacity, and strengthen accountability. The quality of evidence and analysis, while aligned with Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LK: N&E Local Services Improvement (P113036) the objectives, had significant shortcomings in results monitoring, particularly with regard to indicators that could have provided robust data on the outcome with regard to impact on economic activity, livelihood fostered, improved health, promoting women leadership, governance participation, and their economic empowerment as well as participation by returnees, people with disabilities and marginalized groups. Lessons were clear, useful, and based on specific experience and findings from the operations. For example, the challenges posed by conventional top down approach practice in local development and accountability, or engaging Social Audit Committees in post project implementation to better institutionalize accountability, or ensuring that O&M budgets form part of subproject plans. a. Quality of ICR Rating Substantial Page 18 of 18