Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Water Resources and Irr Mgmt Program 2 (P114348) Report Number : ICRR0021483 1. Project Data Project ID Project Name P114348 Water Resources and Irr Mgmt Program 2 Country Practice Area(Lead) Indonesia Water L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-80270 30-Nov-2016 119,287,947.05 Bank Approval Date Closing Date (Actual) 22-Mar-2011 31-May-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 150,000,000.00 0.00 Revised Commitment 150,000,000.00 0.00 Actual 119,287,947.05 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ranga Rajan John R. Eriksson Christopher David Nelson IEGSD (Unit 4) Krishnamani 2. Project Objectives and Components a. Objectives The Project Development Objective (PDO) as stated in the Loan Agreement (Schedule 1, page 5) and in the Project Appraisal Document (PDO, page 5): " To assist the Borrower to improve its capacity for basin water resource and irrigation management and increase irrigated agriculture productivity in the Project Area" Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Water Resources and Irr Mgmt Program 2 (P114348) This review is based on the three sub-PDOs: (1) To improve the capacity for basin water resource management; (2) To improve the capacity for irrigation management and (3) To increase irrigated agricultural productivity in the project area. b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components This project was the second and final phase of the Water Resources Irrigation Sector Management Program, aimed at increasing the capacity for water resources management of river basins and irrigation and improving agricultural productivity in the project area. The first phase of the program was completed in December 2010. The second phase project covered a total of 18 river basins, 14 provinces and 101 districts. There were four components (PAD, pages 6-7). The estimated costs at appraisal and the actual costs at closure by component are given below, as recorded in the ICR (page 39). One. Basin water resources management improvement. Appraisal estimate US$38.72 million. Actual cost at closure US$24.98 million. The main reason for the 35 percent decline in actual cost was the significant devaluation of the Indonesian Rupiah relative to the US$ during implementation. This component aimed at improving the operational effectiveness and performance of basin management institutions in selected national and provincial river basins and piloting community-based catchment management programs. Activities included: (i) preparing operational plans for river basins management, implementing network/systems and staff training; (ii) providing office facilities and equipment to the Secretariat of the basin coordination council, establishing data base and management information systems and financing study tours to more developed basins; (iii) providing operational support to the National Hydrology Management Program and operationalizing the National Water Resources Information System; and (iv) rehabilitation of check weirs, river bank protection, dykes, water allocation structures, small retention ponds and related works. Two. Improvement of participatory irrigation management. Appraisal estimate US$84.71 million. Actual cost at closure US$82.66 million. This component aimed at improving the delivery of irrigation services through participatory irrigation management (PIM) institutions in participating provinces and local governments (Kabupaten). Activities included, capacity building activities to local government irrigation agencies, establishing irrigation commissions, Water Users Associations (WUAs) and Water User Association Federations (WUAFs), financing capacity building workshops, light and moderate rehabilitation of irrigation system infrastructure and implementing climate change adaption measures (including support to farmers for adopting new technology and developing agribusiness management skills and encouraging public-private partnerships for farming and marketing in community subprojects). Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Water Resources and Irr Mgmt Program 2 (P114348) Three. Jatiluhur Irrigation management improvement. Appraisal estimate US$62.49 million. Actual cost at closure US$37.47 million. The 40 percent drop in actual cost at closure was due to the significant depreciation of the Indonesian Rupiah relative to the US$ during implementation. This component aimed at financing the first stage of a program to modernize the Jatiluhur irrigation system, the largest irrigation system in Indonesia. Activities included: (i) rehabilitation of irrigation, drainage infrastructure and secondary irrigation systems: (ii) strengthening PIM institutions, developing inter-agency management arrangements and a funding mechanism for managing the system: and, (iii) technical assistance for implementing the system and preparing institutional and technical design for future modernization of canals and rehabilitation of additional secondary irrigation systems. Four. Project management and implementation support. Appraisal estimate US$16.64 million. Actual cost at closure US$15.82 million. This component financed project implementation support through advisory support to entities at central, provincial and local government levels in the areas of fiduciary management and monitoring and evaluation. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. Appraisal estimate of total cost (inclusive of costs associated with front-end fees and interest cost during construction) was US$202.56 million. Actual cost was US$160.93 million (79 percent of the appraisal estimate). The difference between the actual cost and appraisal estimate was mainly due to the devaluation of the Indonesian Rupiah relative to the US$ during project implementation. Project financing. The project was financed by an IBRD loan of US$150.00 million. US$119.29 million of the loan was disbursed (about 80 percent of the loan). Most of the underspent amount and loan cancellation was due to devaluation of the Indonesian Rupiah relative to the US$ during implementation. Borrower contribution. Appraisal estimate was US$52.56 million (including funding from the central and local governments). Actual contribution was US$41.64 million. The ICR provides does not provide any reason for the reduced borrower contribution during implementation. Dates. The project was approved on March 22, 2011, became effective on November 14, 2011 and scheduled to close on November 30, 2016. The following changes were made through a Level 2 restructuring on December 21, 2015: (1) Given the savings from competitive bidding and exchange rate changes, some activities (light and moderate rehabilitation of irrigation infrastructure in all project districts) were added to the scope of the project; (2) Core indicators were added to the results framework (discussed in section 9b) and targets were adjusted to reflect the results of the additional activities; (3) The list of available procurement methods was expanded to accommodate the procurement methods for small goods and works; and (4) The closing date was extended from November 30, 2016, to May 31,2018, to complete ongoing activities that had been subject to delays in the first two years of the project due to a combination of factors including, delays in using the on-granting mechanism (the new mechanism of funds Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Water Resources and Irr Mgmt Program 2 (P114348) flow from the national level to provincial and local levels), delay in the bidding process for construction works under component three and delays associated with mobilization of consultants at the national level. The project closed 18 months behind schedule on May 31, 2018. 3. Relevance of Objectives Rationale The PDO’s were realistic in the country context and with respect to the government strategy. Effective management of river basins for securing irrigation supplies and flood protection was essential in matters pertaining to food and water security in Indonesia, given that the bulk of the grain crops were grown in irrigated land. And while Indonesia is a water-rich country, spatial and seasonal variations in availability of water resources posed high risks of water insecurity. The PDOs were well-aligned with the government strategy. The government strategy aimed at adopting a basin-based integrated water resources management approach and improving governance for effective service delivery of bulk water supply, irrigation and flood management in the different basins. In the irrigation sector, the strategy aimed at promoting participatory management through improving management institutions, empowering water users through WUAs and WUAF's and linking them to irrigation and agricultural services through multi-stakeholder irrigation committees at the district level for better accountability of service delivery. The Law on water resources (Law number seven) promulgated in 2004 provided the legal foundation for a new institutional framework aimed at decentralization of the sector and establishing participatory management of river basins and irrigation systems. The government's medium-term development plan for the 2009-2014 period highlighted the need for improving agricultural productivity and water security for food security, sustaining economic growth and poverty reduction (PAD, pages 4-5). The government's midterm development plan for the 2015-2019 period reiterated the need for enhancing the reliability and effectiveness of existing irrigation systems and called for increasing rice production by 15 percent (10 million tons). The PDOs continue to be well-aligned with the Bank strategy. The Country Partnership Strategy (CPS) for the 2009-2014 period highlighted the importance of decentralization of public services, improvement of water resources and governance of irrigation management. The PDOs are relevant for the Country Partnership Framework (CPF) for the 2016-2020 period. Irrigation and dams are included in the CPF's Engagement Area One (Infrastructure Platforms at the national level). This area highlighted the need for enhanced engagement in issues pertaining to water and sanitation and irrigation and dams. Rating High 4. Achievement of Objectives (Efficacy) Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Water Resources and Irr Mgmt Program 2 (P114348) PHEFFICACYTBL Objective 1 Objective To improve the capacity for basin water resource management Rationale Theory of Change. Better management of basin water resources was to be achieved through project-supported capacity building activities focused on operationalization of basin water allocation plans, implementation of operation and management plans for river basins, improving the organizational capacity for hydrology, asset and information management and training members of the provincial and basin water councils on integrated water resources management and enhancing the Ministry of Public Works knowledge base for water resources management. These activities in conjunction with activities of light and moderate rehabilitation of river infrastructure (such as of check weirs, river bank protection, dykes, water allocation structures, small retention ponds and related works), were expected to improve the capacity for basin water resource management in the selected national and provincial river basin. Outputs (ICR, page 27 and pages 29-31). • 14 basin water allocation plans were operational at project closure. This exceeded the target of 12. There was no baseline for this indicator. • 15 river infrastructure and operation and maintenance plans were formulated and implemented at project closure as compared to none at the baseline. This exceeded the target of 12 plans. • 93 percent of river infrastructure established under the auspices of the project was fully functional at project closure. This was short of the target of 100 percent. There was no baseline for this indicator. • Of the 18 river basin organizations in the project, all had Management Information Systems, 16 had water councils and seventeen were considered to have adequate budget. There were no targets for this indicator. • 53 percent of training packages (such as through twinning arrangements and on-the-job training) planned during project preparation were conducted during implementation. This was short of the target of 100 percent. • 18 national and provincial river basin agencies were fully functional at project closure. This exceeded the target of 12. There was no baseline for this indicator. The ICR (page 14) notes that functionality of national and provincial basin agencies was based on a scoring system for 12 indicators. These indicators were: (1) operationalization of basin water allocation plans; (2) formulation and implementation of river infrastructure and operational and maintenance plans; (3) percentage of river infrastructure established under the project that were functional at project closure; (4) establishment of the basin council: There was no target. (5) adequacy of human resources: (6) coordination and cooperation between national and provincial governments through a Memorandum of Understanding; (6) preparation of the agreed annual work plan amongst national and provincial governments in the project basins; (7) completion of a strategic plan of the Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Water Resources and Irr Mgmt Program 2 (P114348) basin (Pola) and a basin management plan (Rencana); (8) adequacy of the operational budget: (9) hydrology management; (10) establishment of the Management Information System; (11) water quality; and (12) flood management. The ICR (page 14) notes that the river basin organizations were considered to be fully functional if all twelve indicators achieved a score of 75 percent or above. These twelve scores, measured as a percentage was averaged into a total score. Eighteen national and provincial river basins, were as per this methodology, considered to be fully functional at project closure, as compared to the target of 12. The ICR (page 14) notes that according to the implementing unit survey assessment, five basins achieved a total score of over 90 percent, 11 river basins had a total score of 80-90 percent and two basins a total score of 75 to 80 percent. Outcomes (page 27). The indicators such as operationalization of basin water allocation and basin water operation and maintenance plans, were output-oriented. The theory of change linking these improvements to project interventions was less than robust and the linkages between outputs and outcomes either in terms of functionality or built capacity, were not clear. In the absence of evidence such as through a beneficiary survey, the efficacy of the objective of improving the capacity for basin water resource management is rated as modest. Rating Modest PHREVDELTBL PHEFFICACYTBL Objective 2 Objective To improve the capacity for irrigation management Rationale Theory of Change. Better management of irrigation services was to be achieved through project-supported capacity building activities focused on strengthening, provincial and district irrigation service providers, participatory irrigation management institutions by empowering farmer user communities through establishing WUA's and WUAF's and linking them to irrigation and agricultural services through multi-stakeholder irrigation committees to address cost recovery for irrigation schemes through joint management and sharing of operation and maintenance responsibilities. These activities in conjunction with rehabilitation and upgrading of Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Water Resources and Irr Mgmt Program 2 (P114348) infrastructure including in selected parts of the Jatiluhur were expected to improve the capacity for irrigation management in the project areas. Outputs (ICR page 16 and page 30). • 63 Irrigation Commissions (ICs) were "fully functional" at project closure. This exceeded the target of 45 IC's. The ICR (page 15) reports that "fully functional" was based on the extent to which an IC: (i) had a functioning secretariat with necessary staff and facilities: (ii) had obtained timely provision of funding for IC operations: and, (iii) had fulfilled its key functions, such as formulating plans for maintaining and improving irrigation systems, endorsing cropping plans prepared by relevant agencies, with consideration to water availability and crop suitability and formulating Operation and Maintenance (O&M) and rehabilitation plans for local irrigation schemes. • 395,450 hectares of infrastructure in the target area were rehabilitated. This exceeded the revised and original targets of 381,000 and 360,000 hectares respectively. There was no baseline for this indicator. The ICR (page 16) reports that works mainly comprised schemes below 3,000 hectares, where the improvement of irrigated infrastructure consisted of light and medium rehabilitation. • Regarding the Jatiluhur irrigation system, 54,936 hectares were rehabilitated. This exceeded the original target of 30,000 but was short of the revised target of 56,649 hectares. The ICR (page 16) notes that a state- owned company was to operate the Jatiluhur irrigation scheme through delegation by the central government. Regarding the Jatiluhur irrigation management, the operational and management mechanism was not functional at project closure, as the rehabilitation work on the system has not yet been completed to date by the contractor. • The web-based project management information system was established in the project management and implementation units as targeted. Outcomes. • 3,112 Water User Association Federations were "duly constituted" at project closure as compared to 1175 at the baseline. This exceeded both the original and the revised targets of 2030 and 2995 respectively. The ICR (page 67) reports when the WUAF's were considered to be "duly constituted;" they could enter into contracts with third parties for any activity related to their business (such as entering into contracts with the private sector to supply bulk fertilizer, seeds or seedlings). Given that the outputs were largely realized and the outcome with respect to establishment of duly constituted Water User Association Federations was exceeded, the achievement of this objective is rated as substantial. Rating Substantial PHREVDELTBL Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Water Resources and Irr Mgmt Program 2 (P114348) PHEFFICACYTBL Objective 3 Objective To increase irrigated agricultural productivity in the project area Rationale Theory of Change. The activities described above in relation to the first two objectives in conjunction with capacity building activities aimed at providing support to farmers in relation to adoption of new technology and development of agribusiness was expected to contribute to agricultural productivity in the project area. The combination of these activities was aimed at the long-term development objective of improving food and water security and thereby sustaining economic growth. Outputs. In addition to the outputs described above for Objective 2, which were also relevant to this objective, the following outputs were relevant to this objective. • 54936 hectares of the command area were provided with improved irrigation infrastructure and services. This amounted to a command area that contributed to at least a 15 percent increase in paddy production. This exceeded the original target of 30000 hectares but was slightly short of the revised target of 59649 hectares. There was no baseline for this indicator. The ICR (page 17) notes that the total increase in production in much of the project area was estimated to be 30-31 percent, with some areas, especially off- Java, seeing a much higher production increase. The areas in local and provincial systems production increased by at least 15 percent and the percentage of production increase in the rehabilitated systems varied from 16.6 percent to 34.8 percent. There was no target for this indicator. • More than 5,000 agricultural training packages were conducted in organization (administration and financial), agribusiness, system rice intensification, climate adaptation, integrated pest control and management and agriculture intensification. The project also provided Agribusiness Packages to the WUAF's through competition and 508 exchange visits amongst WUAF's. Outcomes. The average increases in production (tons) before and after rehabilitation in the sample areas were as follows. In the Component two command area, the average increase in production was 31 percent and 33 percent in 2015 and 2016, respectively. The average increase in production however dropped to 31 percent in 2017. In the Component three command area (Jatiluhur), the average increase in production was 27 percent in 2015. It however dropped to 23 percent and 8 percent in 2016 and 2017 respectively. There was no baseline for this indicator. The ICR (page 17) notes that the key factor affecting productivity was excessive rainfall and pest attacks in Jatiluhur in 2016, which reduced 2017 yields in that command area. Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Water Resources and Irr Mgmt Program 2 (P114348) Given that irrigation is a necessary even if not a sufficient condition for increasing agricultural productivity, it is reasonable to conclude that the improved irrigation management, planning, water allocation and cropping patterns that resulted from the project made a substantial contribution to increasing irrigated agricultural productivity in the project area. Rating Substantial PHREVDELTBL PHOVRLEFFRATTBL Rationale Efficacy of the second and third sub-objectives - to improve the capacity for irrigation management and to increase irrigated agricultural productivity in the project area- is rated as substantial. Efficacy of the first sub- objective, to improve capacity for basin water management, is rated as Modest, in view of lack of adequate evidence. With two of the three sub-objectives rated as substantial, efficacy is rated as substantial. Overall Efficacy Rating Substantial 5. Efficiency Economic analysis. An economic analysis was conducted for the first three project components, which accounted for 92 percent of the appraisal estimate and 90 percent of the actual total cost. The methodology followed for the economic analysis is as follows: First, to gauge the total effects of the soft (capacity building activities) and hardware (river infrastructure investments) project interventions, before and after analysis of the project was based on comparison of baseline and end of project situation of sample schemes; second, the effects of rehabilitation interventions were isolated using the double difference method of impact evaluation, using data collected on/before/after and with/without the project. The quantifiable benefits of the project were assumed to come from, increased agricultural productivity resulting from project interventions in irrigation infrastructure improvement, water management and agronomic practices, avoiding losses due to improved capacity for flood management and benefits from industrial and drinking water supply due to investment in upgrading basin water resources management and multi-purpose river infrastructure (PAD, pages 11-12). The other project benefits not factored into the economic analysis included savings from timely maintenance of river and irrigation infrastructure, flood damage reduction and efficiency gains from institutional development and capacity building. The ICR (page 18) notes that the ex post Economic Internal Rate of Return for the entire project at 12 percent discount rate was estimated at 16.2 percent, with ex post EIRRs of 12 percent, 18 percent and 16.2 percent for project components one, two and three respectively as compared to the average ex-ante EIRR of 19 percent (with ex ante EIRRs of 22 percent, 21 percent and 23 percent for project components one, two and three respectively) (PAD, pages 11-12). Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Water Resources and Irr Mgmt Program 2 (P114348) The ICR (page 18) notes that the estimated project cost per beneficiary and per hectare at project closure was significantly lower than estimated at appraisal. At appraisal, the project cost per beneficiary was estimated to be US$47 per person, while at completion it was US$32 per beneficiary. Similarly, the estimated cost per hectare at closure was US$357 per hectare as compared to the estimated cost of US$493 per hectare at appraisal. The project costs per hectare and per beneficiary were lower at closure due to the increases in the number of beneficiaries and increase in irrigation areas at closure. The number of people that benefitted from improved irrigation from the project at project closure was estimated at 5.1 million in 2017 as compared to the estimate of 500,000 at appraisal. The ICR (page 49) notes that the results of household income analysis showed an increase in farmers' income from increased crop production. Overall, the average increase in farm household for all regions of project intervention was 41.5 percent. As compared to the unaffected farmers (control group), the farmers' income in the project area, on average, was 18 percent higher than the control group. Administrative and operational efficiencies. There were delays in implementation in the first two years of the project due to a combination of factors including, delay in the use of the on-granting mechanism (the new mechanism of funds flow from national to local levels) due to lack of understanding on the part of the implementing agency of how the on-grant mechanism would work, delays in hiring consultants, early exit of the Ministry of Home Affairs, which contributed to less clarity on how schemes would be funded, high turnover in consultants and the cancellation of the Law no 7/2004 (This law was to be the basis for regulations for water resources and irrigation.) Key consequences of the Law’s annulment contributed to lack of clarity on: (i) division of authority between central, provincial and district governments regarding irrigation management; (ii) charging user fees: and, (iii) uncertainty regarding the relationships between the institutions that managed irrigation services. These issues in the initial years were rectified and most of the activities were completed, albeit with the extended closing date. The project management and implementation cost (Component four) activity at project closure at 9.83 percent of the total project cost was more than anticipated at appraisal at 6 percent. The ICR (page 19) notes that a total of 79 percent of project funds in US$ terms were utilized (not including cash and contributions in kind from beneficiaries). The reduced utilization was due to a combination of factors including devaluation of the Indonesian Rupiah relative to the US$ during implementation and savings through competitive bidding which enabled completion of additional activities that were added during implementation. Taking all these factors into account, efficiency is rated as substantial. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Point value (%) *Coverage/Scope (%) Available? Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Water Resources and Irr Mgmt Program 2 (P114348) 92.00 Appraisal  19.00 Not Applicable 90.00 ICR Estimate  16.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the PDO to the government and the Bank strategy is rated as High. Efficacy of the two objectives – to improve capacity for irrigation management and to increase irrigated agricultural productivity in the project area – is rated as Substantial as the outcomes were for the most part realized. Efficacy of the objective – to improve capacity for basin water resource management is rated as Modest, in view of lack of relevant evidence. Efficiency is rated as substantial, based on significant cost economy in delivery of irrigation services (in terms of lower costs per hectare and per beneficiary,) and robust economic viability. Overall outcome is rated as moderately satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Financial risk. The financial risk is rated as substantial. Although the WUA's and WUAFs created under the auspices of this project were carrying out operation and maintenance of projects that they have been involved in, the annulment of Law Number 7/2004 has made the legal basis for charging end users for irrigation services unclear. The ICR (page 25) notes that therefore the WUAF's are relying on government subsidies and voluntary contributions (in kind and financial) from farmers and that the farmers are providing maintenance support for schemes and financing this with own resources. The ICR notes that WUAF's are also receiving financial support from local governments to implement operation and maintenance and that these "partnership" agreements with local government are informal and are to be formalized through service agreements that are currently being developed under an ongoing Bank-financed project (Strategic Irrigation Modernization and Urgent Rehabilitation Project). Government risk. The ICR (page 25) notes that with the cancellation of the7/2004 Law, the government should clarify any remaining uncertainty on division of responsibilities for provincial and local schemes in terms of implementation and funds and should find ways to enable the WUA's to charge users for irrigation schemes so that the schemes are sustainable. According to the information provided by the team, the Ministry of Public Works and Housing prepared to date a set of ministerial decrees that basically reinstated the principles and implementation of Law 7/2004 and that the institutional paragraphs were again strengthened by the promulgation of Law 23/2015 on Local Government in which the water resources articles very much resembled the principles of the revoked law. The government risk is rated as Moderate. Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Water Resources and Irr Mgmt Program 2 (P114348) 8. Assessment of Bank Performance a. Quality-at-Entry This project was prepared based on the Bank supported analytical work and investments in Indonesia and elsewhere as highlighted in the Independent Evaluation Group's studies on water resources and participatory irrigation management and the Asian Development Bank (ADB) and Japanese International Cooperation Agency's (JICA) supported participatory irrigation management operations in Indonesia. Lessons incorporated at design included, a participatory approach for management of river basins and irrigation to enhance ownership by the relevant stakeholders, integrating irrigation sector reform interventions with agricultural production activities to secure active participation of beneficiaries and outsourcing of certain activities to WUA's in view of lack of capacity in human resources, both at the central government level and at the level of local governments (PAD, page 8). Several risks were identified at appraisal including risks associated with vulnerability to fraud and corruption, inadequate capacity at local and provincial levels for procurement and financial management and risks associated with lack of due diligence in implementing safeguards and environmental policies in sub-projects (PAD, page 10). Mitigation measures included inclusion of a better governance plan in the project management manual, development of procurement and financial management manuals and hands-on training to the Project Implementation Staff, before and after implementation (PAD, page 11), The implementation arrangements for the project, with three ministries, the Ministry of Public Works and Housing, the Ministry of Home Affairs and the Ministry of Agriculture responsible for implementing the project, was similar to the arrangements for the first phase of the program. Project management units at the national and at provincial and district levels as part of the local government structure, were established at appraisal (PAD, page 9). The arrangements made at appraisal for safeguards and fiduciary compliance were appropriate (discussed in section 10). There were however some moderate shortcomings in design. The mitigation measures associated with implementing a new on-granting mechanism were inadequate. This contributed to delays during implementation as the implementing agencies were not clear about the mechanism. The project relied on contracts at the national level with multiple consulting fees and this contributed to procurement delays during implementation. There were shortcomings in M&E design (discussed in section 9a). Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Twelve Implementation Status Results (ISR) Reports were filed over a seven-year period (implying supervision missions of approximately twice a year). The supervision team included specialists in financial management, procurement, institutional development, economics and project evaluation. According to the ICR (page 24), supervision missions included frequent field visits to construction sites, resettlement sites and other project areas, and this aided in ensuring effective mechanisms for quality control and compliance Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Water Resources and Irr Mgmt Program 2 (P114348) with Bank requirements. Advice provided by the team helped in overcoming delays and expediting administrative steps to keep up with the implementation schedule. For instance, the advice provided by the team helped in improving and speeding up implementation of the on-granting mechanism (ICR, page 24). The support provided by the team helped in compliance with safeguards and in resolving fiduciary issues (discussed in section 10). Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project indicators were to be tracked and reported through a web-based project management information system managed by the National Project Management Unit. The impact monitoring and evaluation unit in the National Steering Committee Secretariat was to be in charge of monitoring project performance (PAD, page 9). There were three key outcome indicators for monitoring project performance (PAD, page 5). This included the number of national and provincial river basin management agencies and the number of irrigation commissions that were fully operational at project closure and the increase in irrigated areas with over 15 percent increase in agriculture production. Given that the indicators for improving the capacity for management of river basins was output-oriented and since there was no provision for a beneficiary survey, there was no way of measuring the extent to which this project activity contributed to realizing the PDO. b. M&E Implementation The two core indicators that were added to the Results framework at project restructuring as part of the core sector indicators initiative, were appropriate. These were:(1) area provided with new/improved irrigation of drainage services and (2) the additional operational WUAs that were created during implementation. The ICR (page 22) notes that although the monitoring and evaluation system was set up at the start of the project, it took some time to be well-established. However, this was rectified after implementation of the recommendations of the Mid-Term Review to have the National Project Management Unit and the Project Management Unit coordinate monitoring and evaluation of the National Project Implementation Unit and to cross check indicator data through sampling. c. M&E Utilization Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Water Resources and Irr Mgmt Program 2 (P114348) According to the ICR (page 22), the data collected by the M&E system was used to identify ad remedy problems, monitor project performance and used as a tool for decision making. The ICR also notes that data entered into the Project Management Information System was used for monitoring and evaluation purposes. The ICR (page 22) notes that the project management information system used for managing dispersed sub- projects was retained beyond project closing by the Ministry of Public Works and Housing. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as Category B for purposes of environmental assessment. In addition to Environmental Assessment (OP/BP 4.01), four safeguard policies were triggered: Pest Management (OP 4.09): Indigenous Peoples (OP/BP 4.10); Involuntary Resettlement (OP/BP 4.12; and Safety of Dams (OP/BP 4.37). (PAD, pages 14-15). Environmental assessment. The PAD (page 14) notes that the small-scale physical project investments in the river and irrigation components in Components one and two could have temporary localized adverse environmental impacts, and the small to medium scale activities under Component three that supported rehabilitation of irrigation and drainage structures (including canal lining, desilting and cross structure repairs) could have negative environmental impacts (such as due to increased erosion, temporary deterioration of water quality, scouring of canals and other construction related impacts). An Environmental and Social Safeguards Framework (ESSF) was prepared and publicly-disclosed at appraisal (PAD, page 14). The ICR (page 22) reports that there were no environmental issues during implementation. Pest management. The safeguards on pest management were triggered because the improved irrigation management could result in increased use of pesticides. The PAD notes that the project did not finance activities associated with procurement or application of pesticides. An Integrated Pest Management Plan was developed as part of the project management to address pest management issues at appraisal (PAD, page 14). The ICR (page 71) notes that there were no pest management issues during implementation. Indigenous peoples. The PAD (page 14) reports that some indigenous people lived in several participating provinces. Although no adverse impacts on indigenous peoples were anticipated at appraisal, given that the activities and sites for year two were yet to be identified, it was possible that project activities could affect indigenous groups at subproject sites. where indigenous people were present. An Indigenous Peoples Policy Framework, that was acceptable to the Bank, was developed and included in the ESSF. The ICR (page 71) notes that indigenous peoples were not identified in the project area during implementation. Involuntary Resettlement. The PAD (page 14) reports that no significant negative impacts were expected, beyond very small-scale land acquisition that may be required for the construction of works. The PAD also Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Water Resources and Irr Mgmt Program 2 (P114348) notes that if land acquisition or resettlement was needed, a land acquisition and resettlement action plan was to be developed and implemented. The ICR (page 22) notes that there were no social issues during implementation. Safety of dams. This policy was triggered because the Jatiluhur irrigation system is partially fed by the Jatiluhur dam/reservoir. The PAD (page 15) notes that the Jatiluhur dam was part of an ongoing Bank- financed Dam Operational Improvement and Safety Project (DOISP) and the necessary actions required by Bank policy were included under DOISP. b. Fiduciary Compliance Financial management. The financial management arrangements (in terms of budgeting, flow of funds and auditing mechanism) for the project were similar to the first phase of the program (PAD, page 13). The ICR (page 23) notes that in general, the financial management covenants were met and the project's annual financial statements were submitted in a timely fashion and the audits were unqualified. Procurement. An assessment of the project implementing units conducted at appraisal, concluded that the units had the required basic capacity to address procurement issues (PAD, page 13). The ICR (page 23) notes that there was an investigation by the Integrity Vice Presidency to follow up on complaints related to several Bank-financed civil works contracts that substantiated collusion among several bidders, as well as unauthorized subcontracting of several contracts to some losing bidders. The ICR notes mitigation action was taken by the Bank for the remaining packages under the project, by requesting the National Project Implementation Unit and the Project Implementation Unit to group several packages with similar nature of works to attract a different category of firms and qualified contractors to break the collusive practices of smaller firms. The ICR however provides no details on the extent to which the mitigation measures were successful in breaking the collusive practices of smaller firms. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome --- Satisfactory Satisfactory Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Water Resources and Irr Mgmt Program 2 (P114348) Moderately Moderately Bank Performance --- Satisfactory Satisfactory Quality of M&E Substantial Substantial --- Quality of ICR Substantial --- 12. Lessons The ICR (page 26) draws the following three lessons from the experience of implementing this project, with some adaptation of language. (1) Participatory irrigation management by enabling farmers to participate in sector development, can help in increasing the sense of ownership by WUAF's. The participatory irrigation management in this project, by involving farmers in regulatory development, resulted in improved allocation of water to farmers. The success of this approach is attested to by the fact that the government is extending it to national level irrigation schemes. (2) Clarity of the legal framework is required for the sustainability of the participatory irrigation management approach. The uncertainties created by the annulment of the law no 7/2004, were partly mitigated by temporary decrees. It is important that legislation be enacted to replace Law no 7. (3) On-granting mechanisms can help in improving the capacity of districts for project implementation. Although delays were experienced in view of the lack of clarity of the mechanism, these were resolved and such mechanisms, by providing more independence to the districts in implementing their annual work program, can aid in improving their capacity for project implementation. IEG draws the following lesson. A robust theory of change requires a clear chain between project activities and expected outcomes, as well as indicators which enable progress in meeting objectives to be measured. In this case, there were no outcome indicators for monitoring performance with respect to the objectives of improving the capacity for basin water resource management. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is reasonably clear and candidly discusses the delays that arose in the initial years of the project. It provides a clear description of the issues associated with the new on-granting mechanism and the issues that arose following the annulment of the Law no 7/2004. The ICR is, for the most part consistent with the OPCS guidelines, and draws good lessons from the experience of implementing this project. The ICR could have been more concise and at 26 pages is more than the recommended length of 25 pages. Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Water Resources and Irr Mgmt Program 2 (P114348) a. Quality of ICR Rating Substantial Page 17 of 17