Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370) Report Number: ICRR0024150 1. Project Data Project ID Project Name P153370 SWIOFish2 Madagascar Country Practice Area(Lead) Southern Africa Environment, Natural Resources & the Blue Economy L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-59870,IDA-D1720,TF-A4588 29-Sep-2023 72,542,687.19 Bank Approval Date Closing Date (Actual) 28-Mar-2017 29-Sep-2023 IBRD/IDA (USD) Grants (USD) Original Commitment 74,000,000.00 2,730,000.00 Revised Commitment 74,047,138.44 47,138.44 Actual 67,390,052.04 47,138.44 Prepared by Reviewed by ICR Review Coordinator Group Katharina Ferl Vibecke Dixon Avjeet Singh IEGSD (Unit 4) P159562_TBL Project ID Project Name P159562 SWIOFish2 Madagascar ( P159562 ) L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-A4564 5200612.77 Bank Approval Date Closing Date (Actual) 28-Mar-2017 Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370) IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 6,422,018.00 Revised Commitment 0.00 6,422,018.00 Actual 0.00 5,200,612.77 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Project Appraisal Document (PAD) (p. 1) and the Financing Agreement of May 30, 2017 (p. 6) the objective of the project was “to improve the management of selected fisheries at regional, national, and community levels and to increase access by targeted fishers to alternative livelihood activities”. The objective will be parsed into four objectives for the purpose of this review: i. To improve the management of selected fisheries at regional level. ii. To improve the management of selected fisheries at national level. iii. To improve the management of selected fisheries at community level. iv. To increase access by targeted fishers to alternative livelihood activities. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components The project included four components: Component 1. Enhanced regional collaboration (appraisal estimate US$8.20 million, actual US$7.46 million): This component included three sub-components: Subcomponent 1.1: This sub-component was to promote the implementation of Indian Ocean Tuna Commission (IOTC) Resolutions in the South-West Indian Ocean (SWIO) countries enhancing their compliance with their international fisheries obligations. Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370) Sub-component 1.2: This sub-component was to provide a coordination mechanism between African and Indian Ocean Developing Island States (AIODIS) for collaborating and sharing their own experiences to address some of their specific challenges. Subcomponent 1.3: This sub-component was to strengthen the newly created Indian Ocean Federation of Artisanal Fishers (Fédération des Pêcheurs Artisans de l’Océan Indien, FPAOI). Component 2. Strengthening capacities for priority fisheries management (appraisal estimate US$21.62 million, actual US$22.62 million): This component included three sub-components: Sub-component 2.1: This sub-component was to promote fisheries public policy, including the improvement of the legal and institutional framework, the statistical system, the applied research, the central administration capacity, the inter‐ministerial coordination, and the sanitary certification system. Sub-component 2.2: This sub-component was to reinforce sustainable management of fisheries and conservation of resources, including the development of fisheries management plans, and their enforcement. Sub-component 2.3: This sub-component was to strengthen deconcentration and the co‐management system and the presence of the administration in the regions. Component 3. Strengthening capacities for engaging in alternative fishing practices and livelihoods activities (appraisal estimate US$47.53 million, actual US$37.16 million): This component included four sub-components: Sub-component 3.1: This sub-component was to finance short‐term compensatory measures through cash transfer mechanisms. Sub-component 3.2: This sub-component was to promote good practices and alternative activities, through in particular training and extension services, promotion of safety at sea, and financial education. Sub-component 3.3: This sub-component was to strengthen local participatory strategic planning. Sub-component 3.4: This sub-component was to finance infrastructure in support of community and sector development, including the Policy and Human Resources Development (PHRD) financing for off‐grid energy services. Component 4. Project management and coordination (appraisal estimate US$6.80 million, actual US$5.63 million): This component was to finance regional project coordination, implementation and management, country‐level implementation, and management, and monitoring and evaluation (M&E) at regional and country levels. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The project was estimated to cost US$84.15 million. The actual cost was US$72.86 million due to delays in the implementation of intended off-grid electrification activities as well as fishing and community infrastructure, the alternative livelihoods program, a surveillance mission, and a second Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370) satisfaction survey in Madagascar, which resulted in those activities not being implemented by the time the project closed. Furthermore, the actual government’s contribution was lower than originally estimated. Financing: The project was financed by an IDA credit in the amount of US$65.0 million (of which US$58.9 million disbursed), an IDA credit in the amount of US$9.0 million (of which US$8.4 million disbursed), a Trust Fund (TF-A4588) in the amount of US$2.7 million (of which US$47,130 disbursed), and a Trust Fund (TF-A4564) in the amount of US$6.4 million (of which US$5.2 million disbursed). Borrower Contribution: The Borrower was estimated to contribute US$1.0 million. The actual contribution was US$272,759. Dates: The project was approved on March 28, 2017, and became effective on August 31, 2017. The Mid- Term Review took place on September 21, 2020. The project closed on its original closing date of September 29, 2023. On May 8, 2020, the project was restructured to extend the closing date of the PHRD grant from June 30, 2020, to September 29, 2023, to allow for synchronization of activities of the different grants and closing of all credits/TFs September 29, 2023. On December 7, 2022, the project was restructured to; i) reduce the target of two intermediate outcome indicators and revise the phrasing of one intermediate outcome indicator; ii) reallocate funding between categories; iii) expand the project’s geographic scope to two additional regions (Menabe and Atsimo- Andrefana) and extend the support to decentralized administration offices to carry out interdiction of illegal, unreported and unregulated (IUU) fishing and safety at sea-related activities to the whole country; iv) shift the funding source of unfinished off-grid electrification activities from the PHRD grant to the IDA credit; and v) add construction of central administration buildings and purchase of drones for monitoring, control and surveillance to component 2. Even though the targets for two intermediate results indicators were reduced, it did not have implications for the PDO and the geographic scope was also expanded. Thus, a split evaluation is not required. 3. Relevance of Objectives Rationale Country and sector context. Fisheries was a key sector for the South West Indian Ocean (SWIO) coastal countries including Comoros, Kenya, Madagascar, Mauritius, Mozambique, Seychelles, Somalia, South Africa, and Tanzania, according to the PAD (para. 1). Together with Yemen, Maldives, and France, these countries were members of the South West Indian Ocean Fisheries Commission (SWIOFC), a regional fisheries body. Coastal populations in the SWIO region depended heavily on fisheries for their livelihoods and the fisheries sector was a major contributor to nutritional health and food security. At project appraisal, the fisheries sector formally accounted for up to 30 percent of Gross Domestic Product (GDP) in Seychelles, eight Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370) percent in the Comoros, about six percent in Mozambique, five percent in Madagascar, two percent in Tanzania, 1.5 percent in Mauritius, and 0.5 percent in Kenya and South Africa. However, pressure on fish stock reached a high level. At the time of appraisal, an estimated 28 percent of the regional fish stock was overexploited or depleted and a further 40 percent was fully exploited. Tuna fisheries were, by far, the most important in the region both in value and total catch, and as tuna resources are migratory, they required regional management to ensure stock productivity and optimum harvesting. Tuna fishing, processing, and marketing remained dominated by foreign operators, with modest benefits retained in the region. There was a strong need for efficient fisheries management required a coordinated approach from the regional to the local level and vice versa. At the time of appraisal, regional collaboration was being reinforced through SWIOFish1, where a wide range of issues were addressed between member countries, particularly with regard to improving the conditions for access to tuna fisheries, strengthening the regional fisheries body, SWIOFC, to ensure effective and sustainable regional collaboration across stakeholders and improving regional monitoring, control, and surveillance (MCS) activities. Alignment with the Government Strategy. The project's objective was in line with the government of Madagascar's efforts to achieve sustainable fisheries management and resource conservation, as demonstrated through the development of the 2012 Agriculture, Livestock and Fisheries Sector Program and the implementation of the 2015 Fisheries Law. Alignment with the World Bank Strategy. The objective of the project was in line with the World Bank’s latest Country Partnership Framework (FY17-21) for Madagascar, specifically with objective 2 “enhanced resilience of livelihoods of vulnerable households in rural and urban areas” (focus area 1) and objective 7 “rural productivity” (focus area 2). At the regional level, the objective of the project supported the World Bank 2008 Regional Integration Assistance Strategy for Sub-Saharan Africa which stressed the importance of management of shared natural resources and provision of regional public goods. Also, the objective was in line with the 2015 Africa Climate Business Plan and its objective to foster climate-smart ocean economies and boosting social protection. The World Bank has significant experience supporting fisheries in the SWIO region through analytical work and investment projects. Even though the project's objective was at a relatively intermediate outcome level, it was pitched at an appropriate level given the past experience, implementation capacity , and resources allocated. The relevance of the objective is rated as High. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370) OBJECTIVE 1 Objective To improve the management of selected fisheries at regional level. Rationale The selected fisheries were fisheries present in priority sites in five regions - Analanjirofo, Androy, Atsimo‐Atsinanana, Diana, and Melaky – which were considered strategic for fisheries management and/or were inadequately serviced by the fisheries administration. Theory of Change: The project’s theory of change stated that project activities/inputs such as conducting Indian Ocean Tuna Commission (IOTC) Compliance Support Missions (CSMs) in participating countries, holding an annual meeting of African and Indian Ocean Developing Islands States (AIODIS) as well as holding an annual Indian Ocean Federation of Artisanal Fishers (FPAOI) General Assembly meeting was to result in several outputs. These outputs were to include IOTC missions, studies and awareness programs being carried out, compliance guidelines being developed, AIODIs meetings and studies being conducted, and proposals being developed. These outputs were to result in the compliance rate of SWIO countries with Indian Ocean Tuna Commission resolutions management of selected fisheries at the regional level being improved. However, the theory of change has some logical gaps. It is not clear how holding annual meetings automatically would improve the compliance rate of SWIO countries or how this improvement would improve the management of selected fisheries at the regional level. Outputs:  16 IOTC Compliance Support Missions (CSMs) were conducted in participating countries, exceeding the target of five CSMs. Compliance action plans were adopted by individual countries, whose implementation was monitored by the IOTC Compliance Secretariat. However, it is not clear if the adoption of action plans can be attributed to the CSMs. For example, no compliance mission was conducted for South Africa. However, South Africa exceeded the target for PDO indicator 1 (“Compliance rate of SWIO countries with IOTC resolution”). Therefore, it is questionable if there was a direct correlation between the CMS and countries’ compliance rate.  The project financed the development and rollout of an e-MARIS application and an Electronic Record of Authorized Vessels (e-RAV) to facilitate compliance monitoring and reporting. This output lacked a target.  Five Annual high-level meeting of AIODIS were conducted, almost achieving the target of six meetings (due to COVID-19 travel restrictions). As a result of the meetings, action plans for the countries to combat marine plastic pollution were developed.  Six annual FPAOI General Assembly meetings were conducted, achieving the target of six meetings. Outcomes:  The overall compliance rate of SWIO countries with IOTC resolutions marginally increased from 72 percent in 2017 to 72.1 percent in 2023, not achieving the target of 77 percent. o For Comoros, the compliance rate with IOTC resolutions decreased from 75 percent in 2017 to 58 percent in 2023, not achieving the target of 80 percent. Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370) o For Kenya, the compliance rate with IOTC resolutions increased from 66 percent in 2017 to 76 percent in 2023, exceeding the target of 71 percent. o For Madagascar, the compliance rate with IOTC resolutions increased from 66 percent in 2017 to 77 percent in 2023, exceeding the target of 71 percent. o For the Maldives, the compliance rate with IOTC resolutions increased from 78 percent in 2017 to 83 percent in 2023, achieving the target of 83 percent. o For Mauritius, the compliance rate with IOTC resolutions increased from 81 percent in 2017 to 93 percent in 2023, exceeding the target of 86 percent. o For Mozambique, the compliance rate with IOTC resolutions decreased from 80 percent in 2017 to 70 percent in 2023, not achieving the target of 85 percent. o For the Seychelles, the compliance rate with IOTC resolutions increased from 73 percent in 2017 to 80 percent in 2023, exceeding the target of 78 percent. o For Somalia, the compliance rate with IOTC resolutions decreased from 71 percent in 2017 to 23 percent in 2023, not achieving the target of 76 percent. o For South Africa, the compliance rate with IOTC resolutions increased from 76 percent in 2017 to 88 percent in 2023, exceeding the target of 81 percent. o For Tanzania, the compliance rate with IOTC resolutions increased from 56 percent in 2017 top 73 percent in 2023, exceeding the target of 61 percent. Rationale: While the project delivered the outputs as planned, it was not able to achieve the outcome regarding the compliance rate of SWIO countries with Indian Ocean Tuna Commission resolutions regarding the management of selected fisheries at the regional level. Also, there were logical gaps in the theory of change under this objective. Overall, achievement under this objective is rated Modest. Rating Modest OBJECTIVE 2 Objective To improve the management of selected fisheries at national level. Rationale Theory of Change: The project’s theory of change stated that project inputs/activities such as training and deploying fisheries inspectors, purchasing inspection vehicles, speedboats and drones for the central and regional FMC offices, developing regulatory texts for the implementation of the 2015 Fisheries Law, conducting small-scale vessel registration and marking campaigns and implementing a National Fisheries Information System with a multi-module central database and a data collection system were to result in several outputs. These outputs were to include fisheries planning, implementing, monitoring, enforcement, and certification capacities being strengthened. These outputs were to result in the outcome of improved management of selected fisheries at the national level. The theory of change was sound and logical. Outputs: Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370)  47 legal texts (by-laws) for the implementation of the 2015 Fisheries Law were adopted by the Ministry of Aquatic Resources and Fisheries (MRHP), not achieving the original target of 120 legal texts or the revised target of 90 legal texts. The legal texts included revisions to the Fisheries law, seven decrees and 39 implementing texts on various issues including aquaculture, organization of marine and continental fisheries activities, lobster fishing, commercialization and valorization of fisheries and aquaculture products etc.  The number of small-scale fishing vessels being registered and externally marked in priority areas increased from 457 vessels in 2017 to 12,275 vessels in 2023, exceeding the target of 2,800 vessels.  Six regional branches (in Androy, Atsimo-Atsinanana, Melaky, Analanjirofo, Diana, and Atsimo- Andrefana) of the Fisheries Monitoring Center were established and are operational, exceeding the target of five regional branches. In addition, the following results at output level were reported in the ICR. They were not included in the RF and thus did not have any targets:  Five regions (Analanjirofo, Atsimo-Andrefana, Diana, Melaku, and Menabe) collected and transmitted statistical catch and effort data electronically at the national level to the MRHP, achieving the target of five regions. The project financed smartphones, data collection kits and materials, recruitment, and training of investigative officers on fisheries data collection measures and statistical analysis.  55 Fisheries inspectors and 20 observers deployed and trained.  58 inspection vehicles, eight speedboats, 74 motorcycles, two refrigerated trucks and drones for the central and regional FMC offices were purchased.  Small-scale vessel registration and marking campaigns in the 13 coastal regions conducted.  A National Fisheries Information System with a multi-module central database and a data collection system were developed.  The Madagascar’s 2023 National Frame Survey was conducted. Data were collected on the number of fishing vessels and gear for fisheries management planning and implementation purposes.  23 central and decentralized fisheries administration offices established.  13. 778 office staff trained on fisheries management subjects. Outcomes:  The share of small-scale fishing units in the priority areas using prohibited gear being observed during on-shore inspections decreased from 80 percent in 2017 to 46 percent in 2023, not achieving the target of 30 percent. Furthermore, this indicator was not appropriate for measuring the achievement of the objective. Rationale: While the project was able to increase the number of small-scale fishing vessels being registered and externally marked in priority areas, it was unable to improve the use of prohibited gear as defined by the law. The ICR notes that the reduction in the use of prohibited gear was due to the implementation of the findings of technical studies on the FMC’s surveillance strategy and function, which informed the establishment and operationalization of six FMC regional branches, the training and deployment of fisheries inspectors/observers, maintenance of FMC surveillance vessels, and the acquisition of inspection vehicles, speedboats, and drones for the central and regional FMC offices. These inputs reportedly increased the frequency of the surveillance missions and helped to deter the use of prohibited gear by small-scale fishers. Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370) However, there is no data available on the increase in the number of patrols or if and how the new gear – for example, drones – was deployed. Overall, the achievement under this objective is rated Modest. Rating Modest OBJECTIVE 3 Objective To improve the management of selected fisheries at community level. Rationale Local-level activities were to initially concentrate in three Ultra Priority Zones (Zones Ultra Prioritaires, ZUPs): Ambaro ‐ Ampasindava Bay in the Diana region, Antongil Bay in Analanjirofo region, and Melaky Coast in Melaky region - areas that contained important fishing grounds for which fisheries management plans had been prepared, gradually extending to the other regions as their fisheries plans were put in place. Theory of Change: The project’s theory of change stated that project inputs/activities such as implementing fisheries management plans, fishers receiving professional cards, fisher association members being trained, restoring mangroves and installing submerged acritical habitats were to result in several outputs. These outputs were to include fisheries management plans being implemented, official fishing licenses being granted, locally managed fishing/marine areas (LLMAs) being co-managed by local community structures and local administration (with the support of non-governmental organizations (NGOs)), and stock enhancement activities being conducted. These outputs were to result in the outcome of improved management of selected fisheries at the community level. The theory of change was logical. Outputs: · Three fisheries management plans were implemented, achieving the target of three plans. While implementation was not defined in the results framework, the ICR noted that implementation focused on key actions in the fisheries management plans, such as the creation of locally managed fishing/marine areas (LMMAs) co-managed by local community structures and the local administration with the support of non- government organizations (NGOs), local vessel and fisher registrations to feed into the national registration program, and implementation of stock enhancement activities, as relevant to each ZUP. · 13,986 fishers in priority areas received a professional card, exceeding the target of 7,200 fishers. In addition, the following results at output level were reported in the ICR. They were not included in the RF and hence not have any targets: · 976.76 hectares of LMMAs across 18 villages in the three ZUPs established and managed. · Two vessel and fisher registration campaigns conducted in the three ZUPs. Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370) · 2,000 hectares of mangroves were restored. · 4,000 m2 of coral reef were transplanted. · 12,000 submerged artificial habitats were installed. Outcomes: · The share of targeted community members with rating “Satisfied” or above on project interventions was 71 percent in 2023, exceeding the target of 25 percent. The ICR notes that this result also reflects satisfaction from value addition activities Rationale: The project was able to implement fisheries management plans and distribute professional cards to fishers. However. there was not sufficient details regarding the implementation of the fisheries management plans. Also, while 71 percent of community members were satisfied with the project activities, it is not clear if the objective was achieved since the outcome indicator was not appropriate for measuring the achievement of the objective. Also, the outcome indicator target of 25 percent of community members being satisfied with project interventions was not sufficiently ambitious. Therefore, the achievement of this objective is rated Modest. Rating Modest OBJECTIVE 4 Objective To increase access by targeted fishers to alternative livelihood activities. Rationale Theory of Change: The project’s theory of change stated that project inputs/activities such as constructing and installing infrastructure in three ZUPs, adopting local community electrification plans, providing fishing equipment, training school children in rice-fish integrated farming, and community saving groups being mobilized were to result in several outputs. These outputs were to include expanded livelihood skills of fishers, enhanced availability of local financial sources to fishers, and increased access to infrastructure and services for value addition and alternative livelihoods. These outputs were to result in the outcome of increased access by targeted fishers to alternative livelihood activities. The theory of change had few logical gaps. The livelihoods PDO-level Indicator did not measure the outcome, but rather an intermediate result (access to electricity) contributing to livelihood access. Outputs:  The project financed the construction and installation of 171 infrastructure units (62 fish centers, 17 ice rooms, 10 cold rooms/units, a landing ramp, 73 wells, and 8 fish smokehouse) in the three ZUPs. However, these assets had not been legally transferred to the communities by project closure. Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370) Therefore, the original target of 320 infrastructures and the revised target of 170 infrastructures being constructed and installed was not achieved.  20 local community electrification plans were adopted for target communities, achieving the target of 20 plans. In addition, the following results at output level were reported in the ICR but not included in the RF and hence did not have any targets:  Value addition and alternative fishing equipment (e.g. fish product valorization kits and non-destructive fishing equipment) provided to 769 trained community members, including 555 females. Also, the fishers participated in alternative (non-destructive) fishing practices and fish value addition training (on fish valorization techniques, seaweed farming, crab farming etc.).  Secondary school children in 87 rural secondary schools were trained in rice-fish integrated farming. According to the ICR (para. 34) an alternative livelihoods program for affected fishers was not launched due to planning delays. According to the Bank team (July 17, 2024) no data on the number of children trained was available.  352 community savings groups were mobilized in fishing communities within eight ZUP districts with a total active membership of 13,193 members. US$731,612 was mobilized in financing which helped community members to purchase fishing equipment, furniture, solar panels, conduct agricultural activities and other livelihood inputs. Outcomes:  38 additional villages in priority areas were able to access electricity for postharvest and value- addition activities, exceeding the target of 20 villages.  2,587 fishers participated in alternative fishing practices and livelihoods activities in the priority areas, lower than the target of 3,500 fishers. Rationale: The project was able increase fishers’ access to participating in alternative fishing practices and livelihoods activities though it was below the target. Also, the project was able to deliver a number of outputs. However, since those outputs lacked targets, it is not clear to what extent the delivery of these outputs was successful. Overall, the achievement under this objective was Modest. Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The achievement of all the objectives was Modest as discussed above. Therefore, the overall efficacy rating is Modest. Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370) Overall Efficacy Rating Primary Reason Modest Low achievement 5. Efficiency Economic efficiency: The PAD (p. 81) conducted a cost benefit analysis. The costs were the total project costs (US$84.15 million) and there were two categories of benefits attributed to improved governance: a) benefits from a recovery of co‐managed coastal fisheries in priority areas through improved regulatory, strategic planning, information and capacity building systems at local level; and b) increased rate of return and gross value added (GVA) from distant tuna vessels through the strengthening of statistics and the authority in charge of the quality and safety of food products. An additional third category of benefits was attributed to private sector activities: i) improved incomes for traditional and artisanal fishers; ii) increased value added to products along value chains; iii) benefits of improved or new infrastructure; and iv) benefits from improved ecosystem services. Regional activities were also to generate increased economic returns for Madagascar and costs savings for the region. A scenario for the baseline ‘without project’ and for the ‘with project” was developed for all project activities. Conservative assumptions at all stages led to an estimated Internal Rate of Return (IRR) after five years of 11.0 percent, which was slightly higher than prevailing discount rates (between 6 percent and 7 percent). It is questionable to what extent the benefits as stated above were sufficiently quantifiable such as benefits from a recovery of co‐managed coastal fisheries in priority areas through improved regulatory, strategic planning, information and capacity building systems at local level. The ICR did not conduct a traditional economic analysis due to the lack of data and quantifiable benefits to beneficiaries. Operational efficiency: According to the ICR (para. 44) the project experienced several delays. For example, as a result of the dismissal of the PIU in December 2015, delays in the preparation of environmental and social safeguard instruments due to the long procurement process and low-quality drafts, and extended discussions within the World Bank on the allocation of regional IDA resources and given the one-country focus of the project, project preparation took 23 months. Also, project implementation experienced several delays due to: i) the PIU missing key staff; ii) frequent staff changes and insufficient experience in World Bank procurement procedures as well as the MRHP steering committee’s participation in procurement; iii) lengthy financial procedures; iv) delays in construction activities due to the country’s complex land transfer procedures and insufficient collaboration between the relevant government entities; v) lengthy mobilization process of experienced Non-Governmental Organizations (NGOs) to support local activities due to the government’s resistance. Overall, the project’s efficiency is rated Modest. Efficiency Rating Modest Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370) a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the objective was High. The project partially achieved all its objectives, resulting in an overall Modest Efficacy rating. Efficiency was Modest. The overall outcome rating is, therefore, Moderately Unsatisfactory. a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome Government commitment: Government commitment was initially weak due to changes in ministers, changing priorities of successive ministries and ministers, and the government’s reluctance to invest in local communities to strengthen their organizing and co-managing capacity. There is a risk that political instability might continue to result in a lack of political commitment to move the fisheries reform process forward. Financing: While the project was able to establish government systems that are critical for the long-term management of fisheries sector in the country, the continued function of these systems will require additional external financing. According to the ICR (para. 74) while the recent and future fisheries partnership agreements between the European Union and Madagascar are expected to generate additional financing to support MCS activities, the availability of funds will depend on successful negotiations and follow-through by the government to allocate these resources accordingly. Therefore, there is still a risk that the negotiations will not succeed, and uncertainty remains as to the level of resources that the government will allocate. Technical: According to the ICR (para. 74) during project implementation, LMMAs were mostly transferred to the communities and the government assigned LMMA monitoring responsibilities to the local NGOs. However, the continuous engagement of the NGOs with communities to build capacity will be based on financing being available. Therefore, there is a risk of a lack of financing for the NGO’s continued engagement. Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370) 8. Assessment of Bank Performance a. Quality-at-Entry According to the ICR (para. 49) the project was built on lessons learned from the Southwest Indian Ocean Fisheries Governance and Shared Growth Project 1. These lessons included: i) the need for an extended time horizon, flexibility, and continuous support to foster social change; ii) the importance of livelihood activities to mobilize stakeholders around the sustainability agenda; iii) the importance of broad-based consultative processes to build consensus; and iv) the role regional fisheries entities can play in addressing concerns. The project addressed these lessons learned by allowing for a six-year project timeframe, finance cash transfers for alternative livelihoods, support local and regional activities. According to the PAD (para. 73) the World Bank team identified relevant risks to project implementation with the political and governance risks as well as stakeholder risk being rated High due to the country’s political instability and stakeholders potentially being resistant to change. Several risks were rated Substantial including: i) sector strategies and policies as achievement of the PDO was dependent on changes needed to fisheries’ policies, strategies, institutions, legal frameworks; ii) technical design due to the complexity of a regional project and multiregional interventions and difficulty in developing viable and functional production and value‐chain models and alternative livelihoods at different levels; and iii) institutional capacity due to the governance and capacities in the regions being inadequate. According to the ICR (para. 50) the World Bank implemented several mitigation measures such as promoting decentralized sector administration that was less affected by political instability, technical support to the government, engagement with local experienced NGOs, providing fiduciary training, and implementing a compensation mechanism for resistant communities. However, the mitigation measures were not sufficient, resulting in lengthy administrative procedures within the government and complex land transfers procedures, and weak procurement capacity resulting in implementation delays. Also, the World Bank team did not identify the risk of frequent minister and staff turnover within the PIU, and lengthy process to mobilize an experienced NGO due to government resistance. The World Bank took 23 months to prepare the project due to the dismissal of the PIU following the change of MRHP minister and a six-month suspension of activities until a new PIU was established. Also, the project experienced delays in the preparation of environmental and social instruments as a result of a long procurement process and capacity issues. Furthermore, the World Bank had lengthy internal discussions about the allocation of regional IDA resources given that the project focused on one country. The project was not ready for implementation when it became effective due to a lack of budgeted action plans for Ultra Priority Zones (ZUP) Fisheries Management Plans, a lack of analysis of actions needed to improve fishing methods, reduce postharvest losses, strengthen local capacities, and improve local administrative and fishing infrastructure. The Results Framework had several shortcomings regarding the selection of PDO indicators and some logical gaps in the Theory of Change (see section 9a for more details). Quality-at-Entry Rating Moderately Unsatisfactory Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370) b. Quality of supervision According to the ICR (para. 72) the World Bank team consisted of experts with the appropriate capacity to support project implementation. Some team members were based in the country, allowing for continuous collaboration with the government. The World Bank team conducted a total of 18 supervision missions to Madagascar and eight to the IOC. Aide memoires discussed progress, implementation issues, fiduciary and safeguards compliance as well as detailed action plans. However, the aide memoires could have benefited from more in-depth analysis of project results compared to targets. While Implementation Status Reports (ISRs) were candid and identified implementation bottlenecks, they did not always result in follow up actions by the World Bank team and the government. Due to the lack of readiness when the project implementation started, the project experienced implementation delays. The World Bank team restructured the project twice. In May 2020, the project was restructured to extend the loan closing date. In December 2022, more than two years after the Mid-Term Review and just nine months before the project closed, the project was restructured to expand the project’s geographic scope to two additional regions to increase disbursement rates and impact despite consistently weak implementation capacity. The restructuring took place late in project implementation as a result of lengthy discussions between the World Bank and the government due to a change in the ministry. However, the two restructurings did not sufficiently revise the project’s Results Framework to allow for an adequate measuring of project achievements. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project’s objective was clearly specified. According to the ICR (para. 62), the M&E design and arrangements were well-embedded institutionally. The Regional Integration Unit (RIU) and the PIU were responsible for the project’s M&E activities, and data submitted to them by executing entities informed semi-annual, mid-term, and completion reports. However, the Results Framework had several shortcomings. First, the theory of change and some of the links between how key activities and outputs were to lead to the intended outcomes were not logical. For example, there was no direct link between PDO indicator 1 (“compliance rate of SWIO countries with Indian Ocean Tuna Commission resolutions”) and measuring the improvement in the management of selected fisheries at the regional level. Second, PDO indicator 3 (“share of targeted community members with rating “Satisfied” or above on project interventions”) lacked objectivity to appropriately measure the achievement of the first aspect of the objective. Also, the target of 25 percent was not sufficiently ambitious. Third, PDO Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370) indicator 4 (“additional villages in priority areas accessing electricity for postharvest and value-addition activities”) was not appropriate to measure an “increase to alternative livelihood activities” since it measured an “increase in accessing electricity.” And fourth, the project included some key outputs that were not included in the Results Framework and lacked targets. b. M&E Implementation According to the ICR (para. 63) M&E data collection faced challenges since some stakeholders were reluctant to provide data to the PIU due to inter-personal differences. Also, M&E reporting was input- oriented without sufficient analysis. Therefore, when the project closed, there was a lack of data to measure the actual achievement of project outcomes. Furthermore, the project conducted a satisfaction survey in May 2022. However, the survey was not repeated when the project closed due to a lengthy procurement process that was not finished on time to allow for another survey to be conducted. In December 2022, the World Bank team only made minor changes to the project’s Results Framework by changing the phrasing of one intermediate outcome indicator and reducing the targets of two intermediate outcome indicators. c. M&E Utilization According to the ICR (para. 63) stated that within the IOC, the RIU collected and analyzed data in a methodologically sound and timely manner. Also, the RIU and PIU used M&E data to develop progress reports and submit them to the World Bank team, which used these reports to identify implementation challenges. However, the ICR (para. 64) stated that the government did not use M&E data consistently to inform budget planning resulting in continued implementation delays as work plans and budgets did not always prioritize delayed activities. Given the shortcomings in M&E design, implementation and utilization, the overall M&E quality is rated as Modest. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as category B and triggered the World Bank’s safeguard policies Environmental Assessment (OP/BP 4.01), Natural Habitats (OP/BP 4.04), Physical Cultural Resources (OP 4.11), and Involuntary Resettlement (OP/BP 4.12). The Government developed an Environmental and Social Management Framework (ESMF) and a Process Framework (PF). Environmental Social Management Plans (ESMPs), Resource Access Restriction Action Plans (RARAPs), and Livelihoods Restoration Plans (LRPs) were prepared during project implementation. According to the ICR (para. 66) the project prepared ESMPs for infrastructure construction. Safeguard compliance had several shortcomings. When the project Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370) closed, it was not clear if the ESMPs were properly implemented since construction works had not been finished, and the closing mission in September 2023 did not include any field visits. Also, three RARAPs and LRPs were prepared but were not implemented, as required, by the time the project closed. Furthermore, despite several written reminders, the government did not comply with the project's Environmental and Social (E&S) Safeguard requirements before starting the distribution of project-financed snorkeling kits. As a result, during the closing mission in September 2022, the World Bank and the government agreed on measures to manage E&S risks of project-funded investments after closing with the PIU submitting E&S performance reports. However, these reports were never submitted to the World Bank. When the project closed, the overall safeguard rating was Moderately Satisfactory. b. Fiduciary Compliance Financial Management: According to the ICR (para. 69) the project’s interim unaudited financial reports and audits were mostly unqualified, of acceptable quality and submitted in a timely manner. Financial management on the regional level was performed appropriately with unqualified audits, and unaudited financial reports which were submitted in a timely manner. However, overall, the project did not comply with the World Bank’s financial management procedures. The 2023 interim external audit report found that the distribution of some of the safety equipment and the snorkeling kits potentially did not comply with the grant agreement. Also, the independent consulting engineer who was hired by the World Bank reported that construction activities continued in several sites after project closing. Throughout implementation disbursements were delayed due to implementation delays and lengthy withdrawal processes. Between May 2021 and March 2022 no withdrawals were made due to changes in the implementing agency and the PIU. Furthermore, the project experienced a high turnover of financial management staff. The World Bank addressed the lack of financial management procedures by providing training. However, lack of understanding of these procedures persisted throughout implementation. When the project closed, financial management was rated Moderately Satisfactory. Procurement: According to the ICR (para. 68), while the project complied with the World Bank’s procurement procedures, it also encountered procurement-related implementation issues. Procurement was delayed during most of implementation. Also, procurement was not sufficiently rigorous in terms of planning, following up and documenting due to lack of understanding of World Bank procedures, high staff turnover, and ineffective communication between the PIU and technical directorate during bid evaluations. Furthermore, the PIU provided work authorizations to infrastructure contractors even though it was clear that works would not be completed before project closing. The ICR stated that regional procurement activities were implemented by the IOC in a satisfactory manner despite some start-up delays. When the project closed, procurement was rated Moderately Satisfactory. Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370) c. Unintended impacts (Positive or Negative) NA d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome Unsatisfactory Unsatisfactory Moderately Moderately Bank Performance Unsatisfactory Unsatisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The ICR (p. 26) included several lessons learned which were adapted by IEG:  Partnering with NGOs which are locally engaged and experienced can support local government entities especially with limited implementing capacity. In his project, three international but locally invested NGOs worked with local communities to improve their capacity to organize and co-manage. This allowed the regional branches of the fisheries department to focus on other activities that could not be delegated such as enforcement and registrations.  Failure to ensure long-term political commitment may jeaopardize the achievement of sustainable outcomes. In this project, changing priorities of successive ministries and ministers negatively impacted project implementation since complex fisheries reform processes require continued government commitment over time. Performance-based financing instruments and sequencing of activities might help to secure long-term commitment by the government.  Fisheries reform processes can be lengthy and require continuous World Bank support. A Multiphase Programmatic Approach could help achieve such continuous support. Also, requiring critical readiness conditions, such as the effectiveness of Monitoring, Control and Surveillance (MCS) or addressing the open access nature of the fisheries, and allowing sufficient time for countries to meet these conditions might positively impact the implementation and sustainability of the project. Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SWIOFish2 Madagascar (P153370) 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provided an adequate overview of project preparation and implementation. It was internally consistent and candid, sufficiently critical of implementation issues, and included useful lessons learned. However, due to the lack of data and quantifiable benefits to beneficiaries, the ICR did not include an economic analysis. Overall, the ICR quality is rated Substantial. a. Quality of ICR Rating Substantial Page 19 of 19