Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID PAMSIMAS Support Trust Fund (P116236) Report Number : ICRR0021296 1. Project Data Project ID Project Name P116236 ID PAMSIMAS Support Trust Fund Country Practice Area(Lead) Additional Financing Indonesia Water P147365 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-94792 31-Dec-2012 66,292,144.96 Bank Approval Date Closing Date (Actual) 26-May-2009 30-Sep-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 74,383,614.00 74,383,614.00 Revised Commitment 66,292,144.96 66,292,144.96 Actual 66,292,144.96 66,292,144.96 Prepared by Reviewed by ICR Review Coordinator Group Vibecke Dixon John R. Eriksson Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives The ICR covers a grant which is part of a larger parent project. The Implementation Completion Report (ICR) was done for administrative reasons as the grant had to be closed for the donor (The Australian Department of Foreign Affairs and Trade, DFAT) to be able to release new funding. The grant’s implementation period does thus not comply with the implementation period of the parent project. The ICR covers activities financed by the grant for the period of September 2009 to September 2017. A final ICR covering the parent project, including the grant, will be completed after closing of the parent project, which is expected on June 30th, 2021. Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID PAMSIMAS Support Trust Fund (P116236) The Project Development Objective (PDO) in the Original grant agreement was to "increase the number of low income rural and peri-urban populations accessing improved water and sanitation facilities and practicing improved hygiene behaviors as part of the Recipient's efforts to achieve Water Supply and Sanitation Millennium Development Goals (WSS-MDG), through programmatic mainstreaming and scaling-up of a nationwide community-driven approach" (Original grant agreement, Article I, paragraph 2.01., page 3). In March 2013, the Borrower submitted a request for restructuring of the parent project including a request for an adjustment of the PDO, which was approved by the Bank on October 30th, 2013, to; "increase the number of under-served and low income rural and peri-urban populations accessing improved and sustained water and sanitation services and practicing improved hygiene behaviors as part of the Recipient’s efforts to achieve WSS-MDG Targets, through programmatic mainstreaming and scaling-up of a nationwide community-driven approach." Thus, the revised PDO was almost identical to the original one, only adding a couple of words ("under-served", "sustained" and "targets"). Moreover, the PDO for the grant is the same as the one for the parent project. Therefore, this ICR review adopts the revised PDO. The review parses the PDO as follows: Objective 1: to increase the number of under-served and low income rural and peri-urban populations accessing improved and sustained water supply services; Objective 2: to increase the number of under-served and low income rural and peri-urban populations accessing improved and sustained sanitation services; and Objective 3: to increase the number of under-served and low income rural and peri-urban populations practicing improved hygiene behaviors. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components Component 1: Community Empowerment and Local Institutional Development (Total grant allocation, including amendments: US$26.34 million; Actual: US$18.51 million) This component was to fund a program-specific training to 990 community facilitators in the field; to support interventions at community, district and provincial levels on Community Driven Development (CDD) planning; and management of Water Supply and Sanitation (WSS) facilities. The component was Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID PAMSIMAS Support Trust Fund (P116236) also to provide technical training to support the training of trainers in the "Methodology for Participatory Assessment/Participatory Health and Sanitation Transformation (MPA/PHAST) approach and the Community Led Total Sanitation (STBM) program. Component 2: Village Incentive Grants: (Total grant allocation, including amendments US$54.79 million; Actual: US$43.96 million) This component was to provide village grants to finance community water improvement subprojects based on a menu of possible options. Water supply and communal sanitation infrastructure provision was based on Community Action Plans (CAP) that were prepared using the CDD approach. Village incentive grants were also allocated to villages that demonstrate good practice and exceeded the hygiene improvement, community mobilization, and WSS access targets identified in the CAP. Component 3: Implementation Support and Program Management (Total grant allocation, including amendments: US$9.27 million; Actual: US$3.82 million) This component provided technical assistance and capacity building for quality assurance of PAMSIMAS including an implementation adviser, a procurement and safeguards adviser; a monitoring and evaluation baseline survey and impact evaluation consultant, a community hygiene and sanitation behavior specialist; a management information systems/knowledge management specialist, an MPA/PHAST specialist, three Community Led Total Sanitation (CLTS) program specialists and community facilitators. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost The total grant allocated from DFAT was US$90.4 million, of which US$46.4 million was provided in August 2009. As part of the preparation of the PAMSIMAS II, DFAT provided an additional grant in the amount of US$44.0 million. US$74.4 million is the sum of the amounts in the signed grants between the Bank and the Government as outlined in the first agreement in May 2009, and then in the five extensions of the agreement. Financing The grant was a TF financed by DFAT and administered by the World Bank. There was no borrower contribution to this grant. Dates The grant was approved on May 26, 2009 and effective on September 10, 2009. The original closing date was December 21, 2012 and the actual closing date was September 30, 2017, i.e. the grant experienced a 4 year and 9 months extension. The grant underwent five Level II restructurings; 1) on November 25, Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID PAMSIMAS Support Trust Fund (P116236) 2010; 2) on June 13, 2012; 3) on October 31, 2012; 4) on April 23, 2014; and 5) on November 10, 2015. The reason for the restructurings were additional funds from DFAT (the grant was disbursed little by little), some minor re-allocations between categories and extension of the grant period. 3. Relevance of Objectives Rationale The Community Based Water and Sanitation Project (PAMSIMAS) was the third World Bank supported Project for Water Supply and Sanitation for Low Income Communities in Indonesia. PAMSIMAS is referred to as the "parent project" in the ICR and in this ICR review. The ICR of the DFAT PAMSIMAS Support Trust Fund grant is the subject of this review. Building on previous World Bank-supported Water and Sanitation projects in Indonesia, PAMSIMAS was a vehicle to help scale up the operationalization of the national sector Policy Framework for community-based water supply and sanitation and to contribute to the achievement of the Millennium Development Goal for Water and Sanitation (MDG) 7c, to halve the proportion of people without access to safe drinking water and basic sanitation by 2015, as was outlined in the National Medium-Term Development Plan (RPJMN 2010- 2014). The Project Development Objective (PDO) is well aligned with national priorities outlined in the National Medium-Term Development Plan (RPJMN 2014-2019), which targets the achievement of universal access to water and sanitation by the year 2019. The National Plan also promotes the community-led total sanitation approach. The PDO is furthermore well aligned with the World Bank’s Indonesia Country Partnership Framework (CPF) 2016-202 in two key engagement areas: The delivery of national infrastructure (Engagement Area 1), and the delivery of local infrastructure and services (Engagement Area 4). The gender and governance approach under PAMSIMAS are well aligned with the Bank’s cross cutting engagements of shared prosperity, equality and inclusion. However, while there is clear alignment between the project’s development objectives and the country- and WB strategies, the relevance of the objectives (increased access to water and sanitation facilities) is not pitched at a sufficiently high level to suggest a potential solution to the development problem that the project is intended to address.　 In other words, the objective was not defined such that its achievements would be plausibly traceable to improvements envisioned to arise from increased access to water supply and sanitation services, such as improvements in health outcomes, production and income or other factors affecting community livelihoods (which are the expected high level outcomes as stated in the ICR’s Theory of Change on page 9). These may be longer term targets but tracking them and identifying them is an important aspect of a successful development operation. Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID PAMSIMAS Support Trust Fund (P116236) Rating Modest 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Objective 1: to increase the number of under-served and low income rural and peri-urban populations accessing improved and sustained water supply services Rationale Theory of Change for Objective 1: The inputs/activities were training of community facilitators, sanitation trainers and stakeholder committees; technical assistance and capacity building in M&E, management information systems and Community Driven Development; technical support for community action plans; financial support in design, construction and supervision of water supply infrastructures; and provision of grants for villages. Outputs were expected to be Community action plans; improved water supply systems; and improved water supply in schools. These outputs were expected to lead to outcomes such as an increased number of under-served and low income rural and peri-urban populations accessing improved and sustained water supply services. This was in turn expected to lead to higher level outcomes such as improved health condition of the rural and low-income population and to increased human development and earning capacity. Outputs: All listed outputs are the same for all three objectives, as they may equally relate to water supply services, sanitation services, and hygiene behaviors: • 2,251 villages developed Community Action Plans, exceeded both the original target of 460 and met the revised target of 2,215. • 387,660 people participated in village plena meetings for the preparation of Community Action Plans (this indicator had no original nor revised target). • 95 percent of the districts had a project monitoring structure and tools that provided regular information on project implementation quality, exceeding both the original and revised targets, which both were 90 percent. According to the ICR (page 38), the central government built a strong MIS system that facilitated the flow of information. • 20 percent of the total project cost was actual community contribution (in cash and kind), meeting both the original and revised targets of 20 percent. Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID PAMSIMAS Support Trust Fund (P116236) • 1,669 villages and districts exceeded the project performance criteria and received supplementary grants, exceeding both the original target of 500 and the revised target of 1.649 villages. • 92 percent of all registered grievances related to project activities were addressed, almost reaching the original and revised targets of 95 percent. • 87 percent of target schools had improved water supply and sanitation facilities and hygiene programs exceeded the original target of 75 percent but did not achieve the revised target of 95 percent. The following outputs were reported in the ICR but without information on what the targets were: • 1,187 community facilitators trained on preparation of community action plans to improve water and sanitation in the community. • 365 district stakeholder committees trained on selection and monitoring of eligible villages. Outcomes: Outcome indicator #1: Number of additional people with sustainable access to improved water facilities, differentiated by socioeconomic status. • 1,870,067 additional people had sustainable access to improved water facilities, exceeding both the original target of 550,000 people and the revised target of 1,772,000 people. • Of the total number of beneficiaries, 1,002,712 beneficiaries were women, exceeding both the original target of 275,000 women and the revised target of 886,000 women. • About 44 percent of the beneficiaries (corresponding to 882,830 beneficiaries) were poor • 83 percent of the villages with improved water supply systems functioned to the satisfaction of the majority of the target community, not achieving the original or revised targets of 90 percent. There are no outcome indicators to measure outcomes beyond the number of beneficiaries and the satisfaction of the target community. The ICR (figure on p. 9 and para. 16 on p. 11) states that improved health conditions and earning capability were intended to be high level outcomes of this project while also making it clear that these outcomes were expected to be long term and outside the scope of the project. To be able to measure a reduction of disease within the project’s time frame would perhaps be too much to ask, but the impact on intermediate outcomes, such as measures of direct improvements in water quality (e.g. foreign matter, toxins, harmful bacteria, etc.) would have been possible and beneficial. Furthermore, the village satisfaction with improved water supply was below both the original and revised target. Rating Modest Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID PAMSIMAS Support Trust Fund (P116236) PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Objective 2: to increase the number of under-served and low income rural and peri-urban populations accessing improved and sustained sanitation services Rationale Theory of Change for Objective 2: The inputs/activities were training of community facilitators, sanitation trainers and stakeholder committees; technical assistance and capacity building in M&E, management information systems and Community Driven Development; technical support for community action plans; financial support in design, construction and supervision of sanitation infrastructures; and provision of grants for villages. Outputs were expected to be Community action plans; improved sanitation systems; and improved sanitation in schools. These outputs were expected to lead to outcomes such as an increased number of under-served and low income rural and peri-urban populations accessing improved and sustained sanitation services. This was in turn expected to lead to higher level outcomes such as improved health condition of the rural and low-income population and to increased human development and earning capacity. Outputs: All listed outputs are the same for all three objectives, as they may equally relate to water supply services, sanitation services, and hygiene behaviors: • 2,251 villages developed Community Action Plans, exceeding both the original target of 460 and the revised target of 2,215. • 387,660 people participated in village plena meetings for the preparation of Community Action Plans (this indicator had no original nor revised target). • 95 percent of the districts had project monitoring structure and tools that provided regular information on project implementation quality, exceeding both the original and revised targets, which both were 90 percent. According to the ICR (page 38), the central government built a strong MIS system that facilitated the flow of information. • 20 percent of the total project cost was actual community contribution (in cash and kind), meeting both the original and revised targets of 20 percent. • 1,669 villages and districts exceeded the project performance criteria and received supplementary grants, exceeding both the original target of 500 and the revised target of 1.649 villages. • 92 percent of all registered grievances related to project activities were addressed, almost reaching the original and revised targets of 95 percent. Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID PAMSIMAS Support Trust Fund (P116236) • 87 percent of target schools had improved water supply and sanitation facilities and hygiene programs, exceeding the original target of 75 percent but not achieving the revised target of 95 percent. The following outputs were reported in the ICR but without information on what the targets were: • 1,187 community facilitators trained on preparation of community action plans to improve water and sanitation in the community. • 365 district stakeholder committees trained on selection and monitoring of eligible villages. Outcomes: Outcome indicator #2: Number of additional people with sustainable access to sanitation facilities, differentiated by socioeconomic status. • 1,825,904 additional people with sustainable access to improved sanitation facilities, exceeding both the original target of 160,000 people and the revised target of 775,000 people. • Of the total beneficiaries, 861,150 beneficiaries were women, exceeding both the original target of 80,000 women and the revised target of 378,000 women. • About 44 percent of the beneficiaries (corresponding to 803,400 beneficiaries) were poor. According to the ICR, this overachievement was made possible because the target had originally been estimated at sub-village level. During implementation it became clear that it would be possible to expand project activities to village level and this resulted in the over-achievement. •56 percent of the target communities were free of open defecation, not achieving the original target of 60 percent but exceeding the revised target of 50 percent. There are no outcome indicators to measure outcomes beyond the number of beneficiaries and reduction in open defecation. Rating Substantial PHREVDELTBL PHEFFICACYTBL Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID PAMSIMAS Support Trust Fund (P116236) Objective 3 Objective Objective 3: to increase the number of under-served and low income rural and peri-urban populations practicing improved hygiene behaviors Rationale Theory of Change for Objective 3: The inputs/activities were training of community facilitators, sanitation trainers and stakeholder committees; technical assistance in M&E and Community Driven Development; and technical support for community action plans. Outputs were expected to be Community action plans; improved sanitation systems; improved sanitation in schools; communities free of open defecation; and hygiene programs in schools. These outputs were expected to lead to outcomes such as an increased number of under-served and low income rural and peri-urban populations with improved hygiene and behavior change. This was in turn expected to lead to higher level outcomes such as improved health condition of the rural and low-income population and to increased human development and earning capacity. Outputs: All listed outputs are the same for all three objectives, as they may equally relate to water supply services, sanitation services, and hygiene behaviors: • 2,251 villages developed Community Action Plans, exceeding both the original target of 460 and the revised target of 2,215. • 387,660 people participated in village plena meetings for the preparation of Community Action Plans (this indicator had no original nor revised target). • 95 percent of the districts had project monitoring structure and tools that provided regular information on project implementation quality, exceeding both the original and revised targets, which both were 90 percent. According to the ICR (page 38), the central government built a strong MIS system that facilitated the flow of information. • 20 percent of the total project cost was actual community contribution (in cash and kind), meeting both the original and revised targets of 20 percent. • 1,669 villages and districts exceeded the project performance criteria and received supplementary grants, exceeding both the original target of 500 and the revised target of 1.649 villages. • 92 percent of all registered grievances related to project activities were addressed, almost reaching the original and revised targets of 95 percent. • 87 percent of target schools had improved water supply and sanitation facilities and hygiene programs, exceeding the original target of 75 percent but not achieving the revised target of 95 percent. Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID PAMSIMAS Support Trust Fund (P116236) The following outputs were reported in the ICR but without information on what the targets were: • 1,187 community facilitators trained on preparation of community action plans to improve water and sanitation in the community. • 365 district stakeholder committees trained on selection and monitoring of eligible villages. Outcomes: •56 percent of the target communities were free of open defecation, not achieving the original target of 60 percent but exceeding the revised target of 50 percent. Apart from the indicator on reduction in open defecation, there were no outcome indicators in the results framework covering the improvement of other kinds of hygiene behaviors. However, the ICR text and tables contain references to handwashing with soap as an outcome (see e.g. para 12 p. 10 and para 51 p. 20). The ICR reports (para 49 p. 20) that “According to the MIS, of the targeted communities by the Project, 70% (target 60%) had adopted hand washing programs. However, the Rapid Beneficiary Survey conducted in December 2017 found a higher result, that 84% of households interviewed practice regular hand-washing. Of the targeted communities financed by the Grant, about 57% have reached ODF status exceeding the target of 50%, and more than 87.5% (75% target) of targeted schools have improved sanitation facilities and hygiene programs. Although the results framework did not have a relevant indicator to measure the improvements of other hygiene behaviors than reduction in open defecation, the ICR demonstrates achievement beyond targets for the practice of regular handwashing with soap. The achievement of objective 3 is therefore rated Substantial. Rating Substantial PHREVDELTBL PHOVRLEFFRATTBL Rationale The overall efficacy rating is Substantial as two of three objectives were substantially achieved. Overall Efficacy Rating Substantial PHREVISEDTBL Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID PAMSIMAS Support Trust Fund (P116236) 5. Efficiency No economic analysis was undertaken of the grant funds. An economic analysis was done for the overall project in 2015, which the ICR is referring to and using to assess the efficiency of the grant, as "the type of interventions is exactly the same" (ICR para 55 page 21). The ICR, based on an ex post economic analysis undertaken in 2015, concluded that the Economic Rate of Return EIRR) for PAMSIMAS was estimated at 44 percent. The EIRR for the adjusted benefit and cost streams was estimated at 36.1 percent, taking into consideration partly functioning and non-functioning sub-projects and comparable to EIRRs observed for piped water supply systems financed by other Community Driven Development programs in Indonesia. As the cost per capita of water supply interventions decreased and the benefits remained the same, the ICR states that the EIRR was expected to be the same if not better at the time of the ICR. The average unit cost for a piped water connection under this project was less than 35 percent of the average cost of a connection provided by a water utility or by the central government. For administrative efficiency, the co-financing arrangements allowed for efficiency gains, according to the ICR (para 58, page 22) because the existing institutional arrangements for the overall project were used also for the grant. The total cost of the Implementation Support and Project Management component dropped from 19 percent of the total project cost to 12 percent. The lack of an economic analysis of the grant funds is an important shortcoming. Furthermore, there were some administrative inefficiencies (delays and unused funds). IEG finds it plausible, however, that the ICR’s assertion that the 2015 economic analysis also applies to this grant project. This is the main reason for the Substantial rating of Efficiency. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID PAMSIMAS Support Trust Fund (P116236) 6. Outcome Although the project’s objectives are well aligned with both country and Bank strategies, the PDO is pitched at a level that does not adequately represent a development outcome in terms of impact on beneficiary well-being, and Relevance of Objectives is therefore rated Modest. Overall efficacy was rated substantial, with that of the first objective rated modest, but efficacy of the second and third objectives rated substantial. Efficiency is rated Substantial despite some inefficiencies, as IEG is prepared to accept the assertion that the 2015 economic analysis of the parent project also applies to this grant. Thus, the overall Efficacy rating is Satisfactory. a. Outcome Rating Satisfactory 7. Risk to Development Outcome Technical risk: According to the ICR (page 33), the technical risk is moderate, as the investments for the physical infrastructure for both water supply and sanitation were relatively inexpensive and not technically complex. Despite mitigation measures like the forming of dedicated management units and community groups, 24 percent of the PAMSIMAS-financed water supply systems were functioning at less than 80 percent of their design capacity, partly because of design errors or lack of technical capacity to repair the systems. Financial risk: Despite the expectation that the Drinking Water Supply System (BP-SPAM) projects are to implement full cost-recovery tariffs, only 75 percent of the projects collected tariffs. But only 42 percent of those collected tariffs at a level sufficient to cover operations and maintenance costs. (ICR, pp.33) The financial risk is therefore moderate to substantial. Political risk: The political risk is considered to be low as the central government and most local governments have shown a strong and long-term commitment to PAMSIMAS. 8. Assessment of Bank Performance a. Quality-at-Entry The design of the grant was based on the design of the parent project which took lessons from previous projects into account in its design, notably the need to establish better inter-sectoral collaboration and the need for incentives for local government ownership. The project and grant design included provision of simple and affordable options to keep cost-recovery tariffs low. Implementation of the social and Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID PAMSIMAS Support Trust Fund (P116236) environmental management framework was built into the project oversight structures, guidelines and tools. Technical, financial and economic aspects followed the parent project and poverty and gender aspects were taken into account in the design. The results framework had some weaknesses in that very few indicators at outcome level were identified, making it challenging to document and report on achievements at outcome level. Quality at entry is rated Satisfactory. Quality-at-Entry Rating Satisfactory b. Quality of supervision The Bank team conducted 11 supervision mission during the eight-year grant implementation period and addressed key issues that adversely affected the project at an early stage. These were delays in budget execution and revision, consultant mobilization and compliance with safeguards policies. The Bank team also conducted technical missions in between the supervision missions, visited about 150 villages per year, supported pilot programs and mobilized technical expertise. All team members were located at the country office which allowed for close and continuous supervision, and the team collaborated closely with the government. One shortcoming was that the results framework was not adequately revised in any of the many restructurings to incorporate more and more relevant outcome indicators so as to allow for the documentation and reporting on project achievements at the outcome level. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The M&E design of the grant was fully based on the one for the parent project. The results framework is sound, but with some shortcomings, especially in the relatively low level of the PDO as it does not offer a solution to a development problem, and the lack of appropriate indicators at outcome level beyond the number of beneficiaries is also a weakness in the M&E design. The institutional arrangements were sound and followed those of the parent project. The Central Management Advisory Consultants were to help the implementing agency to establish and maintain a comprehensive Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID PAMSIMAS Support Trust Fund (P116236) computerized management information system (MIS) to monitor achievement of each identified indicator. b. M&E Implementation According to the ICR (para 84 page 28), the monitoring of the grant was well implemented through the MIS, spot-checking by consultants and by the complaints handling system. Grant funds were used to upgrade the parent project’s M&E from a manual paper-based system to a web-based platform that was later integrated into a mobile system. The funds from the grant also allowed for gradual and sustained evolution of the MIS. MIS data collection and independent verification was systematic and institutionalized. User satisfaction was measured through surveys. One shortcoming was the failure to address the lack of adequate outcome indicators in the results framework. c. M&E Utilization M&E data was used to identify and remedy quality problems and incorporate better controls as the project progressed, to fine-tune the targeting of resources and to ensure transparency and openness. According to the ICR (para 88 page 29), the MIS was an essential tool used beyond the project implementation by the GoI to monitor progress and as a tool for decision making. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The Grant was classified Category B as it was regarded unlikely to lead to significant social and environmental impacts. The safeguards assessment was done for the parent project PAMSIMAS and it applied to the grant as well. The safeguards triggered were Environmental Assessment (PO/BP 4.01) and Indigenous Peoples (OP/BP 4.20). Given the small size of the subprojects (not exceeding US$30,000 equivalent), it was expected that any adverse social or environmental impact would be small and negligible. The safeguard frameworks documents were disclosed in both Bahasa Indonesia and English on March 27, 2013. The Indigenous Peoples’ Plan was disclosed in country on December 19, 2012 and in the Infoshop on December 28, 2012. The parent project put in place a comprehensive safeguards framework, according to the ICR (para 90 page 29), which was monitored by the Central Project Management Unit (CPMU) and the Bank. The grant Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID PAMSIMAS Support Trust Fund (P116236) component 3 financed a procurement and safeguards adviser who provided technical assistance and capacity building for quality assurance of PAMSIMAS. The ICR concludes (para 94 page 30) that the grant had been generally complying with the safeguards requirements except for the minor deficiencies in the recording of Voluntary Land Donations (VLD). b. Fiduciary Compliance Financial Management: The grant experienced delays in issuance of budget documents and challenges with the introduction of a new public financial management system, and these problems were exacerbated by difficulties in finding qualified facilitators. The project received unqualified opinions from an external auditor for each year from 2009 until 2016. The Implementing agency ensured that provincial and district governments followed up on audit findings. Procurement: The grant generally complied with the Bank’s procurement guidelines. Procurement was rated moderately satisfactory throughout project implementation mainly due to delays in the recruitment of consultants. These delays affected the quality of the grant implementation, according to the ICR (para 96 page 31), and included delayed mobilization of consultants and facilitators, salary payments, distribution of socialization and training materials, and delays in training of facilitators and consultants. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Bank Performance Satisfactory Satisfactory --- Quality of M&E Substantial Substantial --- Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID PAMSIMAS Support Trust Fund (P116236) Quality of ICR Substantial --- 12. Lessons The lessons are taken from the ICR (pages 34-35) with some adjustment to language: • Partnership and collaboration between stakeholders are important for successful implementation of a large-scale program. The instruments developed under the program to boost partnership and collaboration were the following: institutional (collaborative task force), planning (single plans for all), single information systems (web-based MIS), and support systems (facilitators). • Use of ICT helps maintain a high level of transparency in all aspects of the project cycle. All information about the project was made available on the project's website, including payment advice from the　 State Treasury Office and audit findings. The policy of transparency and openness aimed at creating public trust and confidence in the project. 　 • Investments in sanitation infrastructure　are important to　secure　benefits of hygiene and sanitation behavior programs in CDD projects. People in the villages were not always able or willing to invest in private sanitation facilities. The project therefore provided communal sanitation facilities in schools and public locations so everybody would have access to sanitary facilities. 　 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provides a lot of information and it generally follows the guidelines. The project’s theory of change is clearly presented (p.9). Producing an ICR of a small grant while the parent project is still ongoing is not an easy task, especially since the grant’s administration, implementation and M&E system are part of the overall project. The presentation of the grant and the project is relatively unclear and messy in the ICR. It does, however, clearly present the evidence of the achievements of the project according to the identified indicators. The ICR lacks information on the five restructurings of the grant and it would have been useful for the ICR to delve into the limitations of the project's limited outcome indicators which do not sufficiently cover the complexity of the activities and the project intentions. a. Quality of ICR Rating Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ID PAMSIMAS Support Trust Fund (P116236) Substantial Page 17 of 17