Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) Report Number: ICRR0022776 1. Project Data Project ID Project Name P129640 BO PPCR Phase 2 - Basin Management Country Practice Area(Lead) Bolivia Environment, Natural Resources & the Blue Economy L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-16083,TF-18119 30-Jun-2020 36,786,063.91 Bank Approval Date Closing Date (Actual) 25-Jul-2014 31-Dec-2020 IBRD/IDA (USD) Grants (USD) Original Commitment 45,500,000.00 45,500,000.00 Revised Commitment 45,500,000.00 36,786,063.91 Actual 36,878,925.37 36,786,063.91 Prepared by Reviewed by ICR Review Coordinator Group Richard Anson J. W. van Holst Christopher David Nelson IEGSD (Unit 4) Pellekaan 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The project development objective (PDO), as stated in the Financing Agreement (Loan Agreement and Grant Agreement, 2014) and the Project Appraisal Document (PAD, 2014), was to: “support the implementation of the Strategic Program for Climate Resilience in Bolivia by: (a) strengthening the institutional capacity to define the new integrated river basin management approach to climate change adaptation; and (b) supporting its implementation in three pilot sub-basins in the Rio Grande river basin. Page 1 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) For the purpose of assessing the achievements of this project in Section 4 of this review the PDO has been parsed into two separate objectives, namely Objectives 1 and 2 as follows Objective 1: to support the implementation of the Strategic Program for Climate Resilience in Bolivia by strengthening the institutional capacity to define the new integrated river basin management approach to Climate Change Adaptation; and Objective 2: to support the implementation of the Strategic Program for Climate Resilience in Bolivia by supporting its implementation in three pilot sub-basins in the Rio Grande river basin. This approach is consistent with that adopted in the ICR. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? No d. Components A. Strengthening national capacity for climate change adaptation: (Original allocation: US$5.25 million; Actual: US$4.60 million). This component included 4 inter-connected sub-components: (1) the design and establishment of a National Climate and Water Information System in relevant government agencies; (2) support for national hydro-meteorological and climate change related studies; (3) updating national guidelines on river basin planning and management, and on water sector related pre-investment studies, and integration of climate change adaptation aspects in its National Development Plan; (4) provision of training on the use of the developed National Climate and Water Information System and updated guidelines (ICR,para. 11); and (5) support to the Project Implementation Units (PIU) in terms of coordinating, monitoring and evaluating the project, and disseminating related findings and lessons learned. B. Strengthening capacity for adaptation to climate change in the Rio Grande River Basin: (Original allocation: US$5.15 million; Actual: US$1.15 million). This component included 5 sub-components: (1) strengthening institutional capacity of the SEARPI and the Departmental River Basin Service (Servicio Departamental de Cuencas - SDC) of Cochabamba for an integrated, participatory, basin-scale, climate resilient planning and management in their respective Pilot Sub-basins; (2) establishing mechanisms to facilitate stakeholders participation in river basin planning; and (3) formulating integrated, multi-sectorial, participatory, climate resilient River Basin Master Plans (Planes Directores de Cuenca - PDCs) in the three Pilot Sub-basins; (4) strengthening the SNICA in the pilot sub-basins by updating the hydro-meteorological observation networks; and (5) establishing and/or strengthening the data processing centers and early warning systems for flood and droughts. C. Design and implementation of subprojects that improve climate resilience in the Rio Grande River Basin: (Original allocation: US$61 million; Actual: US$31.13 million). This component included: (1) Page 2 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) the design and implementation of pre-investment studies (including the environmental and social aspects) for Infrastructure and Integrated River Basin Management (IRBM) Subprojects; (2) based on said studies, implementation of those subprojects; (3) provision of training to eligible beneficiaries for the operation and maintenance (O&M) of the sub-projects; (4) for the infrastructure subprojects, the project would finance the operating costs for the implementation, supervision and auditing of the activities, led by the National Fund for Productive and Social Investment (Fondo Nacional de Inversión Productiva y Social - FPS); and (5) For the IRBM Subprojects, the project would provide support to the SEARPI and the SDC for the implementation and supervision of the activities under their respective territorial jurisdiction. The ICR and other available project documentation (Aide Memoires/ISRs do not show the proportion of project funds allocated to the sub-project’s implementation. The above summary description of the three components reflects the relevant adjustments made during the project’s two restructurings, as presented in the ICR. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates (a) Project Cost: The total project cost at approval was US$71.4 million (including an assumed counterpart contribution from Government of US$25.9 million). The actual project cost at closing was US$36.88 million (or 51.65% of estimated total project cost at appraisal). The difference between the original and actual costs was due to zero counterpart funds and delayed implementation. (ICR, Annex 3). (b) Financing: At approval, two trust funds executed by the World Bank provided a total US$45.5 million (Climate Investment Fund/CIF – Credit and Grant Agreements, for US$36.0 million and US$9.5 million, respectively). At appraisal the Government committed counterpart funding of US$25.9 million. Due to fiscal constraints, as part of the project’s re-structuring in 2018 and the Government’s severe fiscal constraints, the Government withdrew its commitment to provide counterpart funding. By the project’s closing date, disbursements from the Bank totaled US$36.88 million (or 51.65% of the total original costs and financing). (c) Borrower/Recipient Contribution: At approval, the local counterpart contribution from the Government was US$25.9 million. During implementation, as noted above this contribution was decreased to zero (TBC), due to the Government’s severe fiscal constraints. (d) Dates: The project was approved on July 25, 2014, became effective on May 7, 2015. A mid-term review was carried out in November, 2018. The original closing date was June 30, 2020, with the actual closing date being December 31, 2020 (i.e., an extension of 6 months). (e) Restructurings and Significant Changes During Implementation: The project had two Level II restructurings. While the PDO was not revised, there were some revisions to components 1 - 3, several result indicators and some of their targets, and to counterpart funding contributions. The main revisions and rationales for the two restructurings were as follows (ICR, paras. 15 – 24): Restructuring 1: (approved on December 7, 2017): This restructuring included changes in counterpart financing, one PDO indicator/target and adjustments to components. By the latter part of 2017, the project had: disbursed only 6% of the loan proceeds (US$2.27 million) and 9% of the grant proceeds (US$0.83 million; and had been rated Moderately Unsatisfactory in ISRs for poor implementation progress and for slow progress towards achievement of the PDO. Restructuring 1 included the following adjustments. Page 3 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) (i) Excluding the Pirai sub-basin, in response to the Department of Santa Cruz decision to withdraw from the project (ii) Adding an alternative sub-basin – Arque- Tapacari, using the funds reallocated from the Pirai sub-basin (iii) Revising downwards one of the PDO indicator target: direct project beneficiaries, was reduced from an original end target of 3,000 to 1,875 individuals (iv) Adding a sub-component to Component A: Following the severe drought of 2016, the scope of the project was expanded to include the establishment of a national drought monitor and strengthening forecasting capabilities at the national level. Accordingly, US$1.2 million of the loan proceeds, originally allocated to project activities in Santa Cruz, were reallocated to this sub-component (under Component A) (v) reducing the Government counterpart funds from US$25.9 million to US$6.74 million, and by project closing, counterpart funds became zero, due to severe fiscal constraints (vi) introducing flexibility by reducing the requirements for counterpart funding to accelerate project implementation, including: a lower counterpart funding requirement of 50 percent and 30 percent for infrastructure subprojects and IBRM sub-projects, respectively, to be borne by each municipality; with the fiscal crisis in 2017, there was a further lowering of counterpart funds to 20 percent for infrastructure sub- projects and 15 percent for IRBM subprojects, and allowing further flexibility of counterpart funds in 2018 and 2019 (ICR, para. 21 for details) (vii) with the withdrawal of the Department of Santa Cruz from the Project, no investments in IRBM subprojects were financed in Santa Cruz sub-basin, while financing infrastructure sub-projects in the three sub-basins, through their corresponding municipal governments (ICR, para. 22). Restructuring 2: (approved on June 1, 2020): This restructuring included: (i) An extension of the project closing date by six months (until December 31, 2020), mainly due to the various implementation delays (ii) Changes to several result indicators. By mid-2020, the project had disbursed 43% of the loan proceeds (US$15.5 million) and 51% of the grant proceeds ($4.8 million). In the ISR (mid-2020), the overall implementation progress and progress toward achievement of the PDO were rated Moderately Satisfactory, while retaining this improved rating until project closure. The main reason for the extension was to facilitate the achievement of the PDO, given the delays arising from the disruptions due to COVID-19. There were 3 PDO indicators which were changed to capture more realistic and clearer results: (a) “Number of tools developed by the PPCR for CC adaptation, used by Government institutions”, with a new end target of 4 tools (vis-à-vis original target of 2 tools, including: irrigation guides/trainings; IRBP methodologies; forest baselines; ABRO software operationalized); (b) A change in the unit of measure of project beneficiaries, from individual to family, with new end target of 3,500 families (vis-à-vis original target of 3,000 beneficiaries), due to using the same unit used to monitor beneficiaries by key government entities in the same project area (ICR, para. 18), although it is not clear in Page 4 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) the ICR why this change was made 6 months prior to project closure; and (c) The female beneficiary indicator (of 50%) was eliminated due to change of unit of measure of beneficiaries, from the individual to family. In summary, the main changes during the two restructurings included (ICR, paras. 15-24): (i) reduction in counterpart funding requirements; (ii) reduction in target number of direct beneficiaries, followed by a subsequent increase in the target, and change of the unit of measurement, from individuals to families; (iii) substitution in one of the sub-basins for sub-projects; (iv) adding a national drought monitoring system; (v) the number of climate change adaptation tools were increased; (vi) project closing was extended one time, for only six months. Accordingly, because these changes did not involve changes in the PDOs, but involved increases in the performance targets, which were achieved/exceeded, there was no need to undertake a split rating of objectives. 3. Relevance of Objectives Rationale The project objectives were highly relevant to addressing the country’s key developmental challenges and priorities, as well as contributing to the country’s developmental strategies, especially the implementation of the Strategic Program for Climate Resilience (SPCR). Context and Government Strategies: The project contributed to various climate change resilience commitments and initiatives involving the Government’s strategies and commitments, including: (a) Bolivia’s first Nationally Determined Contribution (NDC) under the Paris Climate Agreement, to include specific project actions which was intended to promote adaptation to climate change; (b) Government’s higher objective of reducing the country’s vulnerability to climate change as stated in its Patriotic Agenda of 2025 (especially Pillar 9, “Environmental Sovereignty with Integral Development, Respecting the Right of Mother Earth”); and (c) The National Basin Plan, which would: incorporate the improved Integrated River Basin Management (IRBM) methodology developed by this project; and use the experiences and lessons from this project to share with farmers and authorities from other river basins in the country, including basins which are experiencing downstream contamination from Bolivia’s vast mining operations. Bank Strategy: The project also was and remained highly relevant and aligned with several Bank framework documents and initiatives, including (based on the ICR, paras. 26-28): (a) World Bank’s Country Partnership Framework (2016 – 2020); Page 5 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) (b) Performance and Learning Review (PLR) completed in 2018, including contributing to: PLR Objective 5 (“Strengthen Capacity to Manage Climate Change and Reduce Vulnerability to Natural Disasters”); and to Pillar 2: Support Environmental Sustainability and Resilience to Climate Change”; contributing to reduced poverty by reducing vulnerability to climate change induced economic shocks; (c) The on-going (to be completed in FY22) Systematic Country Diagnostic (SCD) for Bolivia, which is placing emphasis on climate change, and taking an integrated and multi-sectoral approach similar to the IRBM approaches promoted by this project and the SPCR; and (d) The on-going Bank-wide initiative, “Water Matters: Resilient, Inclusive and Green Growth through Water Security in Latin America”. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To support the implementation of the Strategic Program for Climate Resilience in Bolivia by strengthening the institutional capacity to define the new Integrated River Basin Management (IRBM) approach to climate change adaptation. Rationale Theory of Change. While the project’s original design included a results framework in the Project Appraisal Document (PAD), it did not include a theory of change/TOC (and associated results chain) because it was not required at the time the PAD was written. The ICR reconstructed a ToC for the project which highlighted the rationale and strategy for addressing the main constraints to, and core elements of, an integrated river basin management (IRBM) approach to promoting adaptation to climate change (ICR, paras. 31 – 40). The ToC in Figure 1 presents a broad framework based on overall strategies, reflected in associated prioritized project-financed activities/inputs, which would generate outputs/intermediate results leading to the achievement of Objective 1, namely strengthened institutional capacity; in summary, this logic aims to define an enhanced IRBM approach to adaptation to climate change. The prioritized project activities/inputs included: (a) developing a national climate and water information system, supported with the needed studies; (b) integrating adaptation to climate change into the National Development Plan (NDP) and National Guidelines; (c) strengthening capacity/training activities for basin- scale climate resilient planning and management; (d) developing climate resilient river basin management plans; (e) establishing and strengthening the early warning systems for floods and droughts; and (f) updating the hydro-meteorological observation networks. These inputs would generate corresponding outputs and intermediate outcomes, namely: (a) National Climate and Water Information System; (b) NDP, which reflects Page 6 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) the coherent integration of adaptation strategies to climate change and prioritized interventions; and (c) completed climate resilient river basin management plans. These outputs and intermediate results were expected to generate the core outcome of strengthened institutional capacity to define and operationalize the new IRBM approach to adaptation to climate change, which would contribute to reduced social, economic and environmental vulnerability to climate change. Outputs/Intermediate Outcomes (ICR, Annex 1: original and revised targets during restructurings are outlined below):  Key institutions (MPD, MMAyA and SDC) demonstrate a better capacity to understand and take into account climate change impacts, through a capacity assessment: Base Line/BL: none; original target: yes; actual: yes; actual as % of target: 100%;  Installed capacities for the operation of the National Climate Change and Water Information System and of National Drought Monitoring System: baseline: none; original target: yes; actual: yes; actual as % of target: 100%;  Number of protocols that establish and define the coordination and continuous interchange of hydro- meteorological information between the identified relevant institutions: baseline: 0; original target: 3; actual: 5; actual as % of target: 167%;  Website for dissemination of centralized hydro-meteorological data is operational: baseline: none; original target: yes; actual: no; actual as % of target: 0%; due to procurement delays;  Number of IRBM participation mechanisms established: baseline: 0; original target: 2; actual: 2; actual as % of target: 100%;  Number of new rehabilitated hydro-meteorological monitoring stations: baseline: 0; original target: 30; revised target (2020): 45; actual: 55; actual as % of revised target: 122%;  Number of rehabilitated hydro-meteorological monitoring stations: baseline: 0; original target: 20; revised target (2020): 5; actual: 5; actual as % of revised target: 100%;  Number of Operating Drought and Flood Early Warning Systems: baseline: 0; original target: 2; revised target (2020)*: yes; actual *: yes; actual as % of revised target: 100%+;  *Updated the Hydro-meteorological Monitoring System of Reservoirs of Bolivia, the Drought Monitoring System, and the System of Seasonal Climate Change Forecasts;  No. of integrated river basin management plans elaborated with integrated river basin planning methodology adopted: baseline: 0; original target: 2; actual: 2; actual as % target: 100%; Outcomes (ICR, Annex 1: original and revised targets during restructurings are below):  Adoption by Government (Ministerial Resolution) of Integrated River Basin planning methodology that considers climate change scenarios: BL: no; original target: yes; actual: yes; actual as % of target: 100%;  Number of tools developed by the Pilot Program for Climate Resistance (PPCR) for climate change adaptation, used by Government Institutions: baseline: 0; original target: 2; revised target (2020); 4; actual: 4; actual as % of revised target:100%;  Availability and adequacy of timely and reliable hydrometeorological data, forecasts and climate change related studies (measured the increase in target users’ satisfaction: baseline: 0; original target: 30; actual: 20.8; actual as % of target: 69%. Page 7 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) Conclusion: The efficacy with which Objective 1 was achieved is rated Substantial because the targets of the key outputs (and intermediate outcomes) and outcomes were achieved or exceeded, in most of the target indicators (i.e., 67% of the outcomes; 89% of the key outputs). Rating Substantial OBJECTIVE 2 Objective To support the implementation of the Strategic Program for Climate Resilience in Bolivia by supporting its implementation in three pilot subbasins in the Rio Grande river basin. Rationale Theory of Change. Similar to Objective 1, the rationale for Objective 2 is reflected in the project’s original design and in the Theory of Change/TOC reconstructed in the ICR (Figure 1). The ToC presents a broad framework based on overall strategies, reflected in associated project-financed activities, to generate the outputs/intermediate results, leading to the above objective 2. The prioritized activities/inputs would include: (a) designing and implementing infrastructure sub-projects in the 3 targeted sub-basins; and (b) designing and implementing watershed management sub-projects in the 3 target sub-basins. The resulting outputs (and intermediate outcomes) were expected to include: (a) implemented and completed infrastructure sub-projects in the target sub-basins; and (b) implemented and completed watershed management sub-projects in the 3 target sub-basins. The resulting outcome was the IRBM approach adopted and implemented in three pilot sub-basins, which contributed to reduced social, economic and environmental vulnerability to climate change. Outputs/Intermediate Outcomes (ICR, Annex 1: original and revised targets during restructurings are below):  Area provided with irrigation and drainage services (ha.): baseline: 0; original target: 3,000; actual: 3,581; actual as % of target: 119%;  Additional area protected from erosion (ha): baseline: 0; original target: 20,000; revised target (2020): 2,300; actual: 2,238; actual as % of revised target: 97%;  Length of waterways (meters) equipped with new or rehabilitated defensive flood protection infrastructure or natural bank stabilization (meters): baseline: 0; original target: 50,000; 1st revised target:15,000; 2nd revised target: 1,700; actual: 28,697; actual as % of 2nd revised target: +1,688 %;  Number of water basin management sub-projects within the pilot river basins financed by PPCR: baseline: 0; original target: 40; revised target (2020): 17; Actual: 17; actual as % of revised target: 100%; Outcomes (ICR, Annex 1):  Number of pilot sub-basins where an Integrated River Basin Management system focused on improving climate resilience is operational: baseline: 0; original target: 2; actual: 2; actual as % of target: 100% + (also, other development partners applied the new IRBM methodology in other sub- basins); Page 8 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640)  Direct project beneficiaries: baseline: 0; original target: 3,000 (individuals); 1st restructuring target: 1,875 (families); 2nd restructuring target: 3,500 (families); actual: 15,041 (families); actual as % of 2nd revised target: +430%. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The overall efficacy of the extent to which the original objectives (which remained the same), and their original and revised targets (arising from the 2 restructurings, including actual estimated direct beneficiaries) were achieved is rated Substantial, for each objective and hence for the project overall. The targets for the main outputs (and associated intermediate outcomes) and outcomes for both objectives were met or exceeded in almost all cases (i.e., 4 of the 5 PDO/outcome indicators and 11 of the 13 output/intermediate result indicators). The ICR provides additional narrative that supports a conclusion of substantial progress toward meeting each of the two objectives with respect to the implementation of the Strategic Program for Climate Resilience, in terms of strengthening institutional capacities of key entities, and achieving climate resilience in three pilot sub-basins. At the same time, there were some significant changes in the definition of the number of direct beneficiaries (3,000 individuals vs. 15,041 families), due largely to the original conservative estimates and the estimated number of direct beneficiaries arising from the expanded irrigation works, which were located near highly populated peri-urban areas. Overall Efficacy Rating Substantial 5. Efficiency Overall, the project’s implementation performance and results demonstrated an efficiency rating of Modest, based on the following factors/evidence involving economic and administrative efficiency. Economic Efficiency: Although there was no ex-ante financial and economic analyses at project appraisal, there was a reasonable ex-post internal economic rate of return (IERR) of 12.4% (with IERRs ranging from 12.26 to 13.01 for the 3 components) based on an unclear methodology and without a sensitivity analysis. According to the ICR the average EIRR compared favorably with a national discount rate of 8.4% (ICR, Annex 4). It also states that the rate is comparable with “other satisfactory projects in the sector” but without evidence (para. 55). The ICR noted four institutional-related “complex dynamics” contributed to implementation delays and lower use of available funds, namely: (a) multiple Project Implementation Units (PIUs) involving 4 national and various regional entities, which added coordination challenges; (b) initial counterpart fund requirements, which added administrative and procurement procedures, thereby contributing to completion delays of sub-projects, Page 9 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) and the resulting need to the extend project closing date; (c) high turnover of project staff, resulting from national and subnational elections and shifts in government priorities and decisions; and (d) local regulations and restrictions arising from COVID-19, thereby delaying various procurement/bidding activities, and contributing to a low disbursement rate (US$36.9 million out of US$45.5 million, or 81%). Administrative Efficiency: According to the ICR, notwithstanding the above-cited implementation challenges, project management (with Bank support) enabled the project to adapt with “administrative efficiency” (para. 53). Key adaptive interventions mentioned in the ICR included: (a) with the withdrawal of the Department of Santa Cruz, the project reallocated funds to support a strategic need (i.e., establishing a national drought monitoring and forecasting system) and substituted the target Pirai sub-basin with the Arque-Tapacari sub- basin; (b) reducing substantially the counterpart financing requirements and contributions from Government, due to unforeseen fiscal constraints affecting the participating municipalities and the national government; (c) managing project activities in the midst of disruptions caused by the pandemic; and (d) a modest extension of the project, of only an additional 6 months. Nevertheless, the ICR did not provide quantitative evidence of achieving “administrative efficiency” (e.g., proportion of project cost used for administrative overhead to implement the project) in the ICR's analysis of efficiency. To the contrary, as noted above, there were significant inefficiencies in financial management (including delays in counterpart funding) and in procurement, which together resulted in implementation delays; in addition, a number of activities were not completed before the project closed (ICR, paras 52, 53 and 54).. Given these shortcomings, but in particular the complete absence of any information on the project's administrative costs in the ICR, the project's efficiency is rated modest.. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate  12.36  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The overall outcome rating is based on the assessment of the 3 dimensions as summarized above, namely: (a) High Rating for Relevance of Objectives: This rating is reflected by the project’s strong alignment (at design and during implementation) with: (a) Government’s national and regional policies and strategies to promote climate resilience and adaptation in specific ways, especially through its strategic sub-basins, which Page 10 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) are also contributing to reducing poverty, especially of the rural-based populations relying on the sustainability of their natural resources; and with the World Bank’s (and other development partners) country partnership strategies and priorities to address Bolivia’s complex and intensifying climate change challenges; (b) Substantial Rating for Efficacy: This rating reflects the achievement of most of the performance targets of the project’s two complementary PDOs, with many targets being exceeded. The ICR provided additional evidence of other associated benefits; and (c) Modest Rating for Efficiency: This “modest” rating reflects the various implementation challenges and resulting delays, and a lack of evidence on the administrative efficiency of the project. To summarize, in light of the project's minor shortcoming in efficacy, but moderate shortcomings in efficiency, the project has moderate shortcomings and its overall outcome is therefore rated moderately satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Overall, there is a “Moderate” risk to sustaining the project’s outcomes, based on the following assessment presented in the ICR (paras. 112 – 114): (a) the main risk depends largely on the likelihood of sustaining the strengthened capacities of the relevant entities (national, regional and municipal) and beneficiary community groups. This sustainability will depend on various factors which are largely within the control of the Government entities and relevant stakeholders, and which can be “nourished”/reinforced with modest follow-up activities, including: sustaining the results of the new climate-informed River Basin Plans on drought and flood protection; continued entity support for implementing and updating periodically the new IRBM approach, and relevant enhanced technologies for monitoring, predicting and responding to future extreme climate change events; (b) the ICR correctly highlights the need for sustained actions by relevant entities over the “longer term”, including the need for: refining/updating periodically the IRBM guidelines; ensuring relevant entities (at various levels) and their staff provide the needed training, funding and technical services, together with involving community beneficiary groups, to provide adequate O&M and periodic rehabilitation of: the infrastructure and watershed management subprojects; hydrometeorological stations; and (c) encouraging other development partners to continue to adopt, apply and support the new IRBM approach and PPCR tools, which can provide reinforcing messages and support to the relevant entities and beneficiary groups. In summary, based on evidence from the ICR and the follow-up discussion with the Bank’s TTL/ICR author, it appears that the likelihood of sustaining the project achievements is “moderate”. Page 11 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) 8. Assessment of Bank Performance a. Quality-at-Entry The quality-at-entry of this project is rated “Moderately Satisfactory”, based on assessing the following six factors and relevant good practices for this type of climate change projects (ICR, paras. 102 – 106): (i) The Bank identified climate change as a significant strategic challenge Bolivia’s developmental priorities, and placed strategic importance on enhancing climate resilience through supporting the implementation of the Government’s Strategic Program for Climate Resilience (SPCR) (ICR, paras. 3-5); (ii) The Bank team, working closely with the Government team, adopted a sound approach to project design, including the following good practice principles: integrated, multi-sectoral, participatory basin- scale approach including strong multi-stakeholder engagement and coordination. However, these arrangements were complex, and their achievement contributed to project implementation delays; (iii) The Project’s Results Framework was strongly aligned with the relevant developmental objectives, appropriate indicators and targets. During implementation, a few indicators and targets were revised, generally upwards, to become clearer and more realistic, including dropping the gender indicator (to reflect “family” as the relevant beneficiary measure), (ICR, para. 67); (iv) The Bank identified substantial risks for project implementation, and together with Government, included adequate measures to mitigate these risks. Despite these mitigation actions, as mentioned in Section 4 above, project implementation was slow and delayed for various reasons explained in the ICR (para. 69); (v) The design of the project's M&E system was poor, for reasons stated in Section 9, and where the Bank could have played a more positive role; and (vi) The project's readiness to undertake effective and timely financial management and procurement activities was weak (with further details cited under Section 10 (b)). Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The quality of supervision was rated Moderately Satisfactory, based on candid evidence presented in the ICR (paras. 107 – 110), other relevant project documentation (including ISRs, MTR report, project evaluation study, and IEG evaluator’s discussion with the Project’s TTL/ICR main author. There are 4 aspects to highlight: (i) According to the ICR the World Bank project team provided adequate supervision inputs for the achievement of development outcomes and stable implementation performance, including: (a) appropriate and timely advice to project counterparts, addressing issues involving compliance with the Bank’s procurement and financial management requirements, associated delays with disbursements and Page 12 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) strengthening the project’s M&E system (paragraphs 54, 108 – 110); (b) guidance during two project restructurings, to enable achievement of project outcomes and improved performance, including actions to improve disbursements. At the same time, Bank supervision cost was lower than the preparation cost, suggesting that the World Bank team could have spent more time providing relevant implementation advice/assistance (ICR, para. 107). (ii) According to the ICR, the World Bank team provided adequate support to counterparts on compliance/fiduciary issues during project implementation, including compliance with World Bank policies on procurement and financial management (ICR, para. 88); candid project ratings, including moderately unsatisfactory” for almost two years; provision of adequate details and agreed recommended actions outlined in the regular mission Aide Memoires; engagement of Bank management and senior level Government officials, as needed (ICR, para. 66). (iii) The World Bank team conducted regular supervision mission about every six months, prior to travel restrictions introduced with COVID-19. After early 2020 virtual meetings were conducted on a bi-weekly and monthly basis to ensure timely assessments and implementation assistance by multi-disciplinary Bank teams, to provide appropriate, quality and timely implementation support, and project documentation. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The PPCR Program Coordinating Unit (UCP-PPCR) was designated in the PAD to have overall responsibility for project monitoring and evaluation (para 40). The M&E system experienced various challenges because of design issues: The project had five PDO indicators. Three for institutional strengthening to define the new IRBM approach and two for implementation of the new IRBM approach in pilot sub-basins. The ICR comments that the results framework provided no baselines or zero baselines for these indicators, reaching the conclusion that this would not allow for "an attributable evaluation" of the project's achievements, However for most of the PDO indicators no baseline or a zero baseline was appropriate. For example, for river basins where the IRBM approach had not been introduced the baseline was zero. For most of the PDO indicators there were output and intermediate outcome indicators which in line with the theory of change established an expected results chain toward the achievement of the PDO as measured by the PDO indicators. For one of the PDO indicators ("direct project beneficiaries") the initial definition of beneficiaries (i.e. individuals) was not suitable for practical monitoring, requiring a change in the unit of measure. The Page 13 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) change from “count of individuals” to “count of families” was introduced at the first restructuring which also meant that the sub-indicator "count of women beneficiaries" had to be dropped. Five additional Pilot Program for Climate Resistance (PPCR) indicators were included at the project design stage. Two of them, namely "Degree of integration of climate change into national planning" and the "Evidence of strengthened government capacity and coordination mechanism to mainstream climate resilience" enabled the measurement of progressive improvement in the inclusion of climate change in national planning and strengthening of government capacities and coordination mechanisms, to achieve enhanced resilience to climate change. According to the ICR additional indicators assessed the emerging impacts of the implementation of the PPCR project based on the review of the project, monitoring reports, and interviews conducted with key stakeholders. This additional information on the project's results supplemented the PDO indicators. b. M&E Implementation The ICR asserted that the Project’s monitoring data were collected and analyzed in a sound manner (para 85). The UCP-PPCR, responsible for data collection, mostly followed the methodology and the frequency of data collection as specified in the PAD, but altered reporting frequency for several indicators. When the monitoring methodology was not followed, it affected the project’s ability to compare results indicators during implementation. This reflected a weakness in the design and implementation of the project's M&E system. c. M&E Utilization The monitoring data on performance and results progress was used to inform project management and decision making. Using the Project's monitoring network, the validation methods used for collected data as well as the feedback mechanisms, project management could identify the major issues, including: (i) the initially slow roll-out of IRBM subprojects reflected in low disbursement rates; (ii) replacement of the Piraí sub-basin due to the exit of Santa Cruz; (iii) changes in the project implementation timeline reflected in the six months extension of the closing date, due largely to the COVID-19 pandemic; (iv) alignment of the targets of the result indicators; almost all of them were achieved upon completion at or above the original targets. Overall, the PDO and intermediate result indicators were changed to more realistic target levels, although the basis for revising the direct beneficiary target was not clear in the ICR, nor was it clearly explained by the Bank project team in an exchange with IEG. M&E Quality Rating Modest 10. Other Issues a. Safeguards Page 14 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) The project was classified as an environmental category “B” project (partial assessment), and triggered the following Bank safeguard policies (ICR, paras. 95 – 101; information provided below where available): (i) Environmental Assessment (OP/BP 4.01): Component 3 included small-scale infrastructure subprojects that improved climate resilience in the Rio Grande River Basin with potential short-term, limited, localized, manageable, and reversible negative environmental impacts, such as soil erosion, water and soil pollution, loss of vegetation, generation of waste (plastic), pesticide poisoning, and pesticide residues in the food chain; (ii) Natural Habitats (OP/BP 4.04); (iii) Pest Management (OP/BP 4.09); (iv) Indigenous Peoples (OP 4.10): It was triggered because about 40% of the 1.7 million people living in the sub-basins of Mizque and Pirai identify themselves as indigenous. Accordingly, an Indigenous People Planning Framework was prepared. No sub-projects were implemented in indigenous territories, mainly because interventions were prioritized in basins where there were several degraded areas. Therefore, there was no need to prepare specific Indigenous People Plans. (v) Physical Cultural Resources (OP/BP 4.11); (vi) The Involuntary Resettlement operational policy (OP/BP 4.12) WAS (??) was triggered because some of the subprojects of Component 3 may require the use of land, but no physical displacement was expected due to the nature and very limited scope of these subprojects; (vii) Forest (OP/BP 4.36); (viii) Safety of Dams (OP/BP 4.37); (ix) The International Waterways Safeguard (O.P. 7.50) was triggered because the Project was going to finance irrigation and flood protection infrastructure subprojects in tributary basins of the Mamoré River, an international watercourse that flows from Bolivia to Brazil. However, the notification of other riparian States was waived because the proposed subprojects would not adversely change the quality or quantity of water flows to other riparians, they would not cause harm to other riparians; and the water use by the other riparians would not be adversely affected. During project preparation, the implementing agency, UCP-PPCR, prepared an Environmental Management Framework (EMF), as the specific subprojects were not identified. The EMF defines and provides tools to manage potential environmental impacts in compliance with safeguards, and includes provisions for key implementation entities (SDC and FPS) to manage environmental aspects in sub-projects. The EMF was used to conduct local consultations, which were publicly disclosed (August 2013). During a mission in March 2019, the need to harmonize environmental management instruments between FPS and SDC was identified. In July 2020, the UCP-PPCR updated the FPS environmental management worksheets in the EMF that were implemented since 2017. The UCP-PPCR included an environmental specialist in the project team to coordinate and consolidate the work of and reports from the environmental specialists from SDS and FPS, who had one environmental specialist each. Page 15 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) The grievance and redress mechanism (GRM) was effective during the project lifetime. The Communal Book, a local open record of ongoing development activities, was the main mechanism used to receive complaints, suggestions, and questions from project-affected people in the implementation of infrastructure subprojects. In addition, the project implementing agencies provided their phone numbers to the communities, providing a good option during the COVID period. Overall, the ICR provides evidence that the project was in compliance with the Bank’s environmental safeguard policies/guidelines, and with the EMF (ICR, para. 99). b. Fiduciary Compliance (i) Financial Management: The Project demonstrated adequate financial management/FM arrangements, which provided reasonable assurance that grant and loan proceeds were used for the intended purposes and whereby FM requirements were fulfilled throughout most of the implementation period. The arrangements provided reliability in the financial statements, as well as the effectiveness of internal control by management, as well as internal and external audits. Other supporting evidence included: - there were two project agencies (MMAyA and FPS) which executed the financial management tasks/outputs, including their Integrated Financial Management System and reconciliation of the project accounts and individual audit reports; - FM qualification was mostly “moderately satisfactory” as reported in the Bank’s ISRs, and the FM risk was rated between modest and substantial during project implementation (for further details, see ICR, para. 90); - The high turnover of MMAyA’s financial management staff, the project’s complex institutional processes and the low capacity of executing agencies hindered the timely preparation of financial information. As a result, delays occurred in the process of decision making of the Project activities; - At the time of finalizing the Project’s ICR, the submission of one final audit report and the return to the Bank of non-executed funds remain pending. (ii) Procurement: Based on evidence presented in the ICR (and the project’s ISRs), procurement for the project was carried out in line with the provisions stipulated in the Loan and Grant Agreements, and the various procurement guidelines of the Bank. Overall, procurement responsibilities were handled in a satisfactory manner; during implementation, ISR ratings for procurement fluctuated from “satisfactory” to “moderately satisfactory”. Various procurement-related challenges arising during implementation included (ICR, para. 94): - There were some delays in procurement processes due to legal review requirements based on national procedures; - There was limited capacity in contract management. The Bank team made efforts to train the staff of the implementation agencies, with respect to strengthen their capacities to manage procurement procedures; Page 16 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) - The use of the Systematic Tracking of Exchanges in Procurement (STEP) improved over the last period of project implementation; - The implementation agencies’ procurement teams managed hard and digital document files; - The contract execution records were only partially included in the STEP; - During the implementation, Procurement Post Review was carried out, and no ineligible contracts nor serious issues were found that would be considered fraud or corruption; and - A major issue in Procurement during implementation was the high turnover of agencies’ staff. c. Unintended impacts (Positive or Negative) The ICR highlighted 3 main unintended positive impacts of the projects, as follows (ICR, paras. 60 – 62): (i) Enhanced coordination between local governments (para. 60): The project-based agreements with local governments helped to establish new coordination and exchange platforms to delegate the maintenance of the hydrometeorological stations. Additionally, the technical stakeholder platforms of each subbasin helped to generate buy-in and ownership across local governments for the new IRBM approach. This will enhance the prospects for ensuring adequate O&M of the various investments and project activities; (ii) Wide adoption of the IRBM approach and enhanced development partner coordination (para. 61): The participatory, climate-informed IRBM approach has been applied in other river basins in Bolivia, supported by development partners such as the German Corporation for International Cooperation (GIZ), the Swedish International Development Cooperation Agency (SIDA) and the Swiss Agency for Development and Cooperation (COSUDE). Further, Japan’s International Cooperation Agency (JICA) helped with activities related to water quality control in the Rocha Basin. Again, this support will contribute to enhanced sustainability; and (iii) Opportunities for synergies were utilized (para. 62): COSUDE, through HELVETAS, an independent Swiss development organization, designed the first version of ARI (Investment Resilience Analysis); the project, based on this model, developed a specific instrument for irrigation considering infrastructure and production aspects. Based on this improvement, HELVETAS in agreement with the Government, developed specific instruments for other sectors like transport. d. Other The ICR highlighted 3 other positive aspects contributed by the project: gender, institutional strengthening and coordination, and poverty reduction/shared prosperity. While recognizing some attribution challenges with respect to the precise role and contributions of this project to these broad topics, the nature/scope of these “other” benefits are summarized below, based on evidence presented in the ICR (paras. 57 – 62): (i) Gender (ICR, para. 57): A major conclusion is that rural women have benefitted from the project activities. Rural women rely more than rural men on ecosystem services for their energy needs, livelihoods, and food security, as they are often heavily involved in agricultural production and the management of Page 17 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) natural resources (based on various cross-country empirical studies, including from FAO sources). Climate change causes more frequent extreme weather events, such as floods and droughts, leading to crop failure and degradation of natural resources. Since the project activities improved adaptive capacity to climate change at the national and local level, an outcome is improved adaptive capacity of rural women. While the gender-specific beneficiary sub-indicator was removed due to monitoring constraints, the project continued to focus on women beneficiaries of the subprojects. Using the female proportion from 2012 census data (50.14 percent), FPS estimated 1,827 women beneficiaries of 62 completed irrigation and drainage projects. Likewise, SDC estimated 4,729 women beneficiaries of the IRBM subprojects in the Department of Cochabamba; (ii) Institutional Strengthening and Enhanced Coordination (ICR, para. 58): One of the main conclusions of the ICR is that the project strengthened institutional capacities of key entities to adapt to climate change. As defined in the PDO, the project strengthened institutional capacity to define and implement an integrated, climate-resilient, river basin management approach. The setup of inter-agency coordination mechanisms, the improvement of the hydrometeorological information system, and the use of PPCR tools and knowledge enhanced Bolivia’s institutional capacity to comprehend and appropriately respond to the increasingly relevant climate change challenges. For example, the project established five interoperability protocols between SENAMHI and relevant institutions, leading to strengthened coordination and continued exchange of hydro-meteorological data, which is critical for climate-informed river basin management. The ICR also adds that the country’s capacity to adapt to climate change was evaluated through specific project- supported workshops to capture information from stakeholder representatives. The IEG evaluator discussion with the Bank’s team confirmed the usefulness of these workshops, which provide reliable evidence, mostly of a qualitative nature (TBC in the forthcoming discussion); and (iii) Poverty Reduction and Shared Prosperity (ICR, para. 59): The project contributed to reducing extreme poverty and promoting shared prosperity for the following reasons: (a) the poor and vulnerable groups, such as women, children and the elderly are the most vulnerable to climate change impacts, which were addressed by the project activities; (b) project beneficiaries of the infrastructure and watershed management subprojects are primarily part of the rural population, which has almost twice the poverty rate than the urban population in Bolivia; and (c) the subprojects have helped the beneficiaries to develop adaptive capacities to the adverse impacts of climate change. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Modest 12. Lessons Page 18 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) The ICR presents five lessons arising from this project (ICR, paras. 115 – 119). This review highlights the two most relevant lessons, which could be applicable to other countries/similar type of projects. Lesson 1: Importance of establishing and strengthening strategic alliances among key stakeholders, especially for complex projects (ICR, para. 117): At the time of preparation and early stages of implementation, the project used only a few alliance-building instruments. During implementation, the project developed and used additional formal and informal alliances, with different stakeholders, which contributed to project performance. Para. 117 provides 3 positive examples of effective alliances, together with their tangible results. The lesson from this project is: "Work out and ensure strong alliances at the project design stage with relevant stakeholders (and not only Government actors), and to “nourish” these alliances during implementation, using various project activities". Lesson 2: Relevance of formulating and implementing a clear strategy to achieve sustainable project results (ICR, para. 118): This project, especially involving a strong capacity-building orientation, illustrates the importance of ensuring a clear and sound strategy for achieving the proposed strategic PDOs and project sustainability, from the project design stage, and throughout project implementation, notwithstanding some shortcomings in the PDOs for this project. The ICR shares a positive example of how the project-supported network of installed hydrometeorological stations was sustained through shared responsibilities among the relevant entities. In this instance, SENAMHI used its expertise in the design of the network and selection of stations, whereas the participating municipal governments committed to the maintenance of the network. These shared arrangements ensured that: (i) quality data are accessible and shared between participating entities; and (ii) the stations are in continuous operation. Accordingly, this project highlights the lesson of the importance of ensuring appropriate collaborative arrangements and supportive mechanisms for the sustainability of O&M requirements of relevant project activities (e.g., in this project, the arrangements for various monitoring activities and the extensive investments in infrastructure for IRBM sub-projects). 13. Assessment Recommended? No 14. Comments on Quality of ICR This review rates the quality of the ICR as Modest. The ICR is: reasonably concise; structured clearly, consistent with the ICR guidelines; and provides a reconstructed Theory of Change/ToC, albeit with some shortcomings. The ICR also included a detailed description of the institution building and the development of plans for improved river basin management. While the quality of evidence on the project's physical achievements appear to be sound, the ICR's basis for the estimated internal rate of return for the project in Annex 4 is not clear. In addition, the evidence on the number of beneficiaries from integrated river basin management in the three pilot river basins (arguably the project's most important outcome indicator) was Page 19 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) inconsistent in the ICR because the unit for the measurement of beneficiaries was changed substantially during project implementation, and with unclear explanation (see below). The following six aspects weakened the clarity and quality of the ICR: (i) The Theory of Change (ToC) did not include explicit outputs, showing only “intermediate results". Making a clear distinction between these two levels of results (as was done in Annex 1B of the ICR) would have improved and added clarity to the TOC (Figure 1). A concise narrative on the TOC, to highlight the key aspects and inter-linkages in the results chain would have been a useful complement to Figure 1; (ii) The derivation of the project’s estimated economic rate of return was unclear and without a sensitivity analysis. In addition, there was no quantitative evidence on how “the project was able to adapt with administrative efficiency” (ICR, para. 53 and Annex 4); (iii) The project made a significant change in one of the key PDO indicators regarding the number of “direct project beneficiaries” within six months of the project’s closing date. The change was from an original target of 3,000 individuals, to a revised target of 3,500 families, without providing a clear and convincing explanation for this late change (ICR, para. 18). In the event the final number of families was recorded as 15,041, a four-fold increase above the target set only six months before the project closed. Following questions from IEG, the Bank’s project team provided an explanation (cited in Section 9c of this review), but with a tenuous rationale; (iv) Inadequate explicit and clear information on the priority actions to address the project’s main risks (ICR, paras. 112-114), especially with respect to the lack of information on the Government’s commitments and funding for ensuring sustainability of project activities and benefits (e.g., the O&M of various monitoring instruments and of the infrastructure and IRMB sub-projects funded by this operation); (v) There was a lack of clarity in the articulation of lessons from the project that could provide a rationale for effective and sustainable scaling-up of the project’s activities in Bolivia for other river basins, or for similar projects in other countries (ICR, para. 115-119). The main shortcomings were that the text for many ”lessons” focused on problems rather than lessons on how to address those problems. (vi) Incomplete assessment of the project’s compliance with the various environmental and social safeguard policies triggered for this project. While the Environmental Management Framework and several other safeguards were in compliance, the compliance of another five safeguards was not assessed in the ICR (paras. 95 – 101). While some strengths in the ICR are recognized, in light of the shortcomings mentioned above this review rates the quality of the ICR as "modest". a. Quality of ICR Rating Modest Page 20 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO PPCR Phase 2 - Basin Management (P129640) Page 21 of 21