Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) Report Number: ICRR0022483 1. Project Data Project ID Project Name P132306 Niger Community Action Programm Phase 3 Country Practice Area(Lead) Niger Agriculture and Food L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-D2190,IDA-H8550,TF-A7627 22-Dec-2017 64,618,597.49 Bank Approval Date Closing Date (Actual) 24-May-2013 20-Jun-2020 IBRD/IDA (USD) Grants (USD) Original Commitment 40,000,000.00 2,250,000.00 Revised Commitment 63,050,000.00 2,162,607.44 Actual 60,529,514.97 2,162,607.44 Prepared by Reviewed by ICR Review Coordinator Group Katharina Ferl John R. Eriksson Christopher David Nelson IEGSD (Unit 4) P143079_TBL Project ID Project Name P143079 Integrated Ecosystems Mgmt (PAC 3) (PSG) ( P143079 ) L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) 4089082.52 Bank Approval Date Closing Date (Actual) 31-May-2013 Page 1 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 4,518,000.00 Revised Commitment 0.00 4,089,082.52 Actual 0.00 4,089,082.52 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Project Appraisal Document (PAD) (vii) and the Financing Agreement of June 7, 2013 (p. 5) the objective of the project was “to strengthen the recipient’s local development planning and implementation capacities, including the capacity to respond promptly and effectively to an eligible crisis or emergency, and to improve the access of the targeted population to socio-economic services”. The Financing Agreement for the Additional Financing (AF) of August 7, 2017 (p. 4) stated that the project’s objective was revised to “to strengthen the recipient’s local development planning and implementation capacities, to support the targeted population in improving agricultural productivity, and to respond promptly and effectively to an eligible crisis or emergency.” (see Section 2e below for AF, TF and GEF amounts) This project was the third phase of the Community Action Program (CAP). According to the ICR (p. 6) the CAP supported the government’s decentralization efforts by using a community-drivel development (CDD) approach to improve access to services for local communities and to address the issue of poverty and food insecurity. The Global Environment Facility (GEF) for the GEF financing was “to promote sustainable land and natural resource management and productive investments at the commune level in selected areas of Niger”. The project was implemented at the national level and in each of the eight regions. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 29-Jun-2017 c. Will a split evaluation be undertaken? Page 2 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) No d. Components Component A: Capacity building (appraisal estimate US$8.35 million, AF US$3.26 million, actual US$11.3 million): This component was to finance the following sub-components: Sub-component A.1: Capacity building for communes and communities and improving local governance: A variety of cross-cutting initiatives - built on achievements from earlier phases by deepening technical and capacity building for local stakeholders – was to enhance the governance of communes in five domains: i) participation of local stakeholders (including farmers groups, livestock keepers groups, women associations, etc.) in planning and implementing local development activities; ii) social accountability to enable local authorities to conduct local government activities in an inclusive and transparent manner; iii) equity at the level of local communities between different social categories of population in general, and between men and women in particular (by guaranteeing, for instance, that within the Annual Investment Plan of each Commune a quota of micro-projects specifically targeted to women’s priority needs such as e., micro- projects identified and managed by women’s organizations is set aside); iv) modernization and efficiency of public services at the level of communes and deconcentrated line departments; and v) sustainable use of natural resources and biodiversity conservation following a landscape approach. Sub-component A.2: Promoting inter-communal collaboration: Key activities were to include: i) carrying out, institutional studies and assessments of the legal, institutional administrative and financial conditions favoring horizontal cooperation between local governments including cooperation towards the sustainable management of inter-communal natural resources; and ii) preparation of a complete guide of inter- governmental cooperation at local level. These activities were to be designed and implemented together with other development partners (mainly United Nations agencies, European Union, and German, Belgian, French and Swiss cooperation). Sub-component A.3: Building capacities of Regional governments: This sub-component was to support the High Authority for the Modernization of the State (HCME), the Ministry of Planning, Territorial Management and Community Development (MPATDC), the Ministry of Interior, Public Security, Decentralization and Religious Affairs (MISPDAR) and the Niger Association of Regional Governments (ARENI), for the implementation of the following initiatives: i) conducting a diagnostic study of all regional governmental bodies, to assess their institutional, technical, administrative and financial capacities to assume their mandate for local development; ii) presenting results and recommendations of the diagnostic study at a national workshop in which representatives of the different regions, key sectoral ministries and national agencies, as well as development partners were to participate; iii) contributing to the design of a general framework for comprehensive regional development plans consistent with both the national Plan for Economic and Social Development (PDES) and Commune Development Plans (PDCs) that were to incorporate the landscape approach and the climate change dimension; iv) organizing two study tours to neighboring countries with significant regional development experience; and v) supporting institutional and capacity building initiatives for selected regional governments. All these initiatives were to be defined, supported and implemented in close synergy and collaboration with other development partners (mainly United Nations agencies, European Union, and German, Belgian, French and Swiss cooperation) supporting decentralization in Niger. Sub-component A.4: Building the capacities of the Agency for the Investment of Local Governments (ANFICT), sectoral line departments and non-state actors: This sub-component was to finance Specific Page 3 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) activities intended to address institutional capacity building needs of ANFICT, to enable it to manage funds to be allocated to communes for their respective investments. Under the AF this component was to focus on targeted training and agricultural advisory services needed by small farmers for the implementation of their micro-projects. There were to be an estimated 11,200 direct beneficiaries under this component. Component B: Local Investment Fund (appraisal estimate US$29.39 million, AF US$15.75 million, actual US$45.0 million): This component was to finance the following sub-components: Sub-component B.1. Implementing targeted investments of Commune Development Plans (PDCs): Direct support was to be provided for the implementation of PDCs and ‘Annual Investment Plans’ (PIA). Appropriate capacity-building activities and other institutional measures were to be identified and implemented to enable ANFICT to effectively exercise its role in the Local Investment Fund (FIL). An institutional evaluation was to assess the legal, financial and operational readiness of the ANFICT and a decision was to be made with respect to channeling funds through this mechanism. In order to avoid dispersion and optimize impact, communes were to focus on a number of eligible investments identified in their development plans and annual investment budgets, including the following: i) Sustainable and improved land and water management practices; ii) Diversification of income-generating activities; iii) Creation and maintenance of collective economic facilities and infrastructures; Sub-component B.2: Implementing targeted investments of ‘Inter-communal Development Initiatives: This sub-component was to focus on two areas: i) sustainable management of intercommunal natural resources; and ii) provision of inter-communal social services. Also, the sub-component was to define with affected Communes precise and adaptable “contingency plans”. Affected Communes were to be allowed to rapidly and efficiently revert to community-based emergency initiatives a portion or the totality of the funds that have already been allocated by the Communes to their respective ‘Annual Investment Plans’ (AIPs). Affected communities were defined in the PAD (p. 10) as communities that experienced a localized crisis – such as climate hazards (severe floods, droughts, climate-related epidemics, and the like). Under the AF this component was to support agriculture-related micro-projects in 125 communes already participating in CAP3. Overall, categories for eligible micro-projects were to include: a) Micro-projects related to improved land and water management, including to strengthen climate resilience; b) Income-generating activities related to agriculture; and c) Micro-projects for socio-economic infrastructures. Component C: Project Coordination, management, Monitoring and Evaluation (M&E) and Communication (appraisal estimate US$6.78 million, AF US$4.04 million; actual US$14.1 million): This component was to finance the following sub-components: Sub-component C.1: Coordination and Management: Activities were to include management of staff and equipment, financial management, procurement activities, management of the environmental and social safeguards aspects, preparation of annual work plans, and organization of supervision missions. Page 4 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) Sub-component C.2: Planning and M&E: The project was to adopt the same M&E system put in place by the second phase of the Community Action Program (CAP2), with the addition of adequate tools and indicators to track new outcomes on sustainable use of natural resources and biodiversity. Sub-component C.3: Communication: This sub-component was to finance knowledge management and sharing. A communication strategy and an action plan were to be designed and implemented in order to promote appropriate dissemination of all the information concerning approaches, processes, results and lessons learned by the program. AF was provided to all sub-components. Sub-component C2 was to finance key studies on agricultural sector development and for the preparation of an upcoming commercial agricultural project. Sub-component C4: Nutrition related activities. including M&E was added and was to support communication, sensitization, and training with regard to nutrition as well as M&E, related to the micro- projects in the five selected communes. Component D: Contingent Emergency Response (appraisal estimate US$0, actual US$0): A ‘Contingent Emergency Response Component’ (CERC) of CAP3 was to be used to providing immediate response to an Eligible Crisis or Emergency channeling resources from rapid restructuring of CAP3 portfolio itself, to finance emergency response expenditures and meet crises and emergency needs. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The project was estimated to cost US$44.52 million. Actual cost was US$70.4 million since the project received Additional Financing in the amount of US$22.85 million. US$849,596 was not disbursed. Financing: The project was financed by an IDA grant in the amount of US$40.0 million (of which US$37.88 million disbursed); a Trust Fund (TF) (TF-A7627) in the amount of US$2.2 million (of which US$2.16 million disbursed); an IDA grant in the amount of US$20.8 million (of which US$20.47 million disbursed) and a TF (TF-14700) by the GEF in the amount of US$4.51 million of which US$4.08 million disbursed. Borrower Contribution: It was not planned for the Borrower to make any contributions. Dates: The project was restructured twice:  On June 29, 2017 the project received Additional Financing (AF) in the amount of US$20.8 million to support the government in its struggle to achieve food security and improved nutrition in line with its action plan for accelerating the 3N initiative (see description in Section 3 below). The project was restructured to; i) change the PDO to “strengthen the Recipient’s local development planning and implementation capacities, to support the targeted population in improving agriculture productivity, and to respond promptly and effectively to an eligible crisis or emergency”. The change introduced with the AF made the project focus more on irrigation, agriculture productivity and nutrition by replacing the outcome “improved access of the targeted population to socio-economic services” by “increased agriculture productivity”. Also, the AF was to continue to be implemented at the national level and in each of the eight regions. It was to scale-up project activities to cover an additional 109,800 direct beneficiaries with a focus on small-scale farmers, small livestock producers, young Page 5 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) people, and women. The Results Framework was adapted to revise intermediate indicators and their targets. New indicators were added to measure new outcomes of the revised PDO. Also, new intermediate indicators for small-scale irrigation were added; ii) reallocate funds among categories, and iii) extend the original closing date by 24 months from December 20, 2017, to December 20, 2019.  On December 2019 the loan closing date was extended from December 20, 2019 to June 20, 2020 to ensure the successful completion of the IDA and TF financed project activities. 3. Relevance of Objectives Rationale According to the PAD (p. 1) in 2012, one year before project appraisal, two-thirds of the population in Niger lived below the poverty line. The 2012 Human Development Index ranked Niger last with an approximate Gross National Income (GNI) at purchasing power parity of US$701. Only 29 percent of the population was literate, and the child mortality rate was 198 deaths for 1,000 children. Also, over 50 percent of the population was affected by food and nutrition insecurity, with 22 percent of the population chronically suffering extreme food insecurity and 40 percent of children under five were underweight. In order to address these issues, the government developed the “Plan de Development Economique et Social” 2012-2015 (PDES) and its four focus areas: i) creation of conditions conducive to sustainable, equitable and inclusive development; ii) food security and sustainable agricultural development; iii) promotion of a competitive and diversified economy; and iv) promotion of social development. The objective of the project was in line with the updated PDES (2017-2021), especially with the “3 N Initiative” (Nigeriens feed Nigeriens), which aimed to protect communities from hunger and malnutrition and ensure communities with conditions that let them participate in production and income generating activities. In 2004, based on these policies and strategies, the Bank supported the decentralization and local development initiatives through a 12 year three-phase adaptable program loan, the Community Action Program (CAP). The CAP aimed to support the government’s decentralization efforts by using a community-driven development (CDD) approach to improve access to services for local communities and to overcome poverty and food insecurity. The original objective of the project was in line with the Bank’s Country Assistance Strategy (CAS) (FY13- 16) and its two strategic pillars: i) promoting resilient growth; and ii) reducing vulnerability. The revised objective was in line with the Bank’s Country Partnership Framework (FY18-FY22) (CPF) focus area 1 “increased rural productivity and incomes” and under focus area 3 “better governance for jobs, service delivery and growth” and its sub-section “strengthened public finance and human resource management for improved service delivery” since it included planning and implementation of investments. Overall, the project’s objective was realistic in terms of resources committed and highly relevant. Rating Relevance TBL Rating Page 6 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To strengthen the recipient’s local development planning and implementation capacities: Rationale Theory of Change: The project’s theory of change envisioned that project activities such as capacity building for increased local governance by communities, supporting intercommunal collaboration, capacity building for regional governments as well as capacity building for (ANFICT) were to result in outputs such as communes being trained in good governance practices, guidelines for intercommunal collaboration being developed, and local government (LG) representatives and communities’ leaders with capacity strengthened. These outputs were to result in outcomes such as improved governance in communes, better collaboration in local government entities as well as improved capacity for local governments and national institutions in local planning and implementation of development projects. These outcomes were to result in the project’s original objective as stated above. According to the ICR (p. 8), the achievement of the outcomes rested on two critical assumptions, namely effective collaboration between central and local government, and local capacity to operate and maintain project investments sustainable. The ICR did not discuss to what extent the achieved outcomes can solely be attributed to the project's activities. Outputs (no targets or baselines were provided for these outputs):  38 communal and two regional development plans were prepared/updated and 750 annual investment plans were supported. In addition, consultation meetings were conducted.  Technical assistance was provided to communes for financial reporting and the preparation of regulations and decrees was supported.  A grievance mechanism was established as part of the project’s compliance with social safeguards.  More than 15,000 local government officials were trained in good practices for project planning and management, annual budgeting, financial reporting, and grievance mechanisms.  Regional agencies received training, equipment, technical assistance to prepare and update regional development plans and support the preparation of regulations and decrees.  20 directorates/departments were equipped with computers, printers and office furniture. Outcomes: Page 7 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306)  In 2017, 87 percent of communes implemented good governance (participation, accountability, and equity) surpassing the original target of 85 percent. This indicator lacked a baseline.  The percentage of targeted communes delivering timely financial reports increased from six percent in 2013 to 76 percent in 2017, achieving the target of 75 percent.  The percentage of communes, which had working grievances mechanisms, increased from eight percent in 2013 to 75 percent in 2017 surpassing the original target of 65 percent.  The percentage of deconcentrated line departments which were strengthened, increased from zero in 2013 to 90 percent in 2017, achieving the original target of 90 percent.  In 2017, 85 percent of regional agencies’ planning capacities were strengthened, surpassing the original target of 60 percent. There is clear evidence that local development planning and implementation capacities were improved as demonstrated through the substantial increase in targeted communes delivering timely financial reports, the increase in communes which had working grievances mechanisms and the increase in deconcentrated line departments which were strengthened. Rating Substantial OBJECTIVE 1 REVISION 1 Revised Objective The objective was not revised but the project received Additional Financing and the targets were increased. Revised Rationale Outputs: Same outputs as above. Outcomes:  The percentage of communes delivering timely financial reports increased from 76 percent in 2017 to 98 percent in 2020, surpassing the revised target of 76 percent.  The percentage of communes, which had working grievance mechanisms increased from 75 percent in 2017 to 100 percent in 2020, surpassing the revised target of 85 percent.  In 2020, 98 percent of communes implemented good governance, achieving the revised target of 98 percent. This indicator lacked a baseline.  In 2020, 99 percent of deconcentrated line departments were strengthened, achieving the revised target of 99 percent.  In 2020, 87 percent of regional agencies’ planning capacities were strengthened, surpassing the revised target of 85 percent. There is also clear evidence that local development planning and implementation capacities were improved under the AF as especially demonstrated through the increase of communes delivering timely financial Page 8 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) reports and the increase of communes, which had working grievance mechanisms. Due to the lack of targets for achievement of outputs, the achievement of this objective was rated Substantial. Revised Rating Substantial OBJECTIVE 2 Objective To improve the access of the targeted population to socio-economic services: Rationale Theory of Change: The project’s theory of change envisioned that project activities such as supporting the implementation of communal development plans and annual investment plans and intercommunal activities as well as supporting the ANFIC to manage the Local Investment Fund were to result in outputs such as communal development plans being developed/updated, annual investment plans being supported, micro- projects and intercommunal investments developed and readiness assessment of the ANFIC being implemented. These outputs were to result in outcomes such as improved increased availability of socio- economic facilities and infrastructures at local level, improved operation and maintenance of facilities and infrastructure at the local level as well as improved land and water management. These outcomes were to result in the project’s original objective to improve the access of the targeted population to socio-economic services. The assumptions made for objective 2 were the same as under objective 1. Outputs (no targets or baselines were provided for these outputs):  Socio-economic infrastructure projects were implemented including: nine food and feed storage facilities, 39 warehouses, nine administration offices, 109 schools, 18 health posts and a memorandum of understanding was signed with 42 communities for radios to be used for the dissemination of information on project activities and on nutrition.  530 income-generating micro-projects such as animals (sheep, goats, and cattle) for sheep and bovine fattening and livestock replenishment, small scale irrigation, grain mills, fishponds, animal feed banks for livestock and cereal banks, and small agro-processing units were supported.  455 sustainable land and water management micro-projects were implemented.  Nutrition awareness campaigns and trainings were conducted. The ICR did not specify how many of each were conducted. Outcomes:  The percentage of communes that were able to sustain proper operation and maintenance of local development investments increased from nine percent in 2013 to 99 percent in 2017, achieving the target of 100 percent.  In 2017, 86 percent of communes used at least 80 percent of their investment allocations, surpassing the original target of 80 percent. Page 9 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306)  In 2017, the access to education was improved for 28 percent of people in newly targeted communes, not achieving the target of 90 percent. “Improved” in this instance refers to the number of additional students attending school as a consequence of the 109 school infrastructure projects. While this indicator was dropped during the first restructuring, it is an important outcome indicator for the overall project performance given the significant investment in improved education facilities.  In 2017, 23 percent of people in targeted communes had improved access to health and nutrition services, not achieving the target of 90 percent. As outlined above for education access, this refers to additional people attending facilities as a consequence of the investments in health facilities. This indicator was also dropped during the first restructuring due to the limited number of investments in health posts, but is important given the obvious alternative preference amongst communities for other CDD priorities. There are significant shortcomings in linking investments made under this project to improving access to socio-economic services, particularly as determined by the original project outcome indicators. The project fell well short of the original PDO indicators largely due to the limited number of investments directed towards these priorities amongst communities. Rating Modest OBJECTIVE 2 REVISION 1 Revised Objective The objective was not revised but the project received Additional Financing and the targets were increased. Revised Rationale Outputs: Same outputs as above. Outcomes:  The percentage of communes that were able to sustain proper operation and maintenance of local development investments increased from 99 percent in 2017 to 100 percent in 2020, achieving the target of 100 percent.  In 2020, 100 percent of communes used at least 80 percent of their investment allocations, surpassing the revised target of 86 percent.  In 2020, 50 percent of people in newly targeted communes experienced improved access to nutrition services, achieving the target of 50 percent. There are significant shortcomings in linking investments made under this project to improving access to socio-economic services under the AF. As stated above, there was an inadequacy or inability to measure project performance regarding education, health and nutrition due to the limited investments made in facilities directly responding to these indicators. Having removed these at restructuring, the outcome indicators Page 10 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) became largely focused on outputs with some information on nutrition improvements which align to the increased focus on agriculture. However, there is a deficit in evidence on the impact of the relevant CDD investments with little information on the outcomes associated with productive investments, income generating activities or agriculture improvements. Without this information, it is difficult to judge the level of improvement that might have been made. CDD projects are influenced by the choices of where community funds will be spent, but the impact of these funds needs to be tracked and recorded to confidently measure the degree of improvement. Revised Rating Modest OBJECTIVE 3 Objective There was no objective 3 until the project received AF and the project was restructured. Rationale N/A Rating Not Rated/Not Applicable OBJECTIVE 3 REVISION 1 Revised Objective To support the targeted population in improving agricultural productivity Revised Rationale Theory of Change: The project’s theory of change envisioned that project activities such as agriculture- related micro-projects related to improved land and water management, including to strengthen climate resilience, income-generating activities related to agriculture, and micro-projects for socio-economic infrastructures were to result in outputs such as development of land to increase the agro-sylvo-pastoral production on favorable soil and water conservation, livestock development, livestock fattening, livestock replenishment, storage and processing structures/equipment for processing agricultural outputs and small- scale irrigation for the production of food were to result in the above stated objective. Outputs:  201,974 farmers with agricultural assets or services (including farmers reached by natural regeneration activities, improved seed distribution, animal stocking activities, irrigation and PHRD grant) were reached, surpassing the target of 109,800 farmers. Of those farmers, 96,902 were female, surpassing the target of 76,900 female farmers being reached. Page 11 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306)  The area provided with new/improved irrigation increased or drainage service increased from 100 hectares in 2017 to 1,351 hectares in 2020, achieving the target of 1,351 hectares. No targets or baselines were provided for the following outputs:  Agricultural inputs such as seeds, fertilizers and pesticides were distributed.  1,708 irrigation motor pumps were provided.  1,701 drip irrigation kits were provided.  9,055 hectares of agro-forestry were supported through planting 245,224 trees. Outcomes:  Agricultural productivity of major crops (tomatoes) increased from 30 metric tons per hectare in 2017 to 42.5 metric tons per hectare in 2020, not achieving the target of 60 metric tons per hectare.  Agricultural productivity of major crops (onions) increased from 30 tons per hectare in 2017 to 43.0 tons per hectare in 2020, not achieving the target 50 tons per hectare.  Agricultural productivity of major crops (peppers) increased from 15 tons per hectare in 2017 to 28 metric tons per hectare in 2020, not achieving the target of 35 tons per hectare.  Agricultural productivity of major crops (cassavas) increased from 15 tons per hectare in 2017 to 34 tons per hectare in 2020, surpassing the target of 31 tons per hectare.  98 percent of communes approved micro-projects which integrated gender equity surpassing the target of 90 percent. According to the ICR (p. 21) the project enabled large numbers of women to increase their incomes mostly from income generating activities such as animal fattening and agro- processing. In total, the project benefitted 3,194,961 beneficiaries, surpassing the original target of 3,000,000 beneficiaries and the revised target of 3,110,142 beneficiaries. Due to a significant productivity improvement for the four target major crops (tomatoes, onions, peppers and cassavas) over a relatively short period of time, with achievements in 2020 ranging from 42 percent (tomatoes) to 127 percent (cassavas) above the 2017 respective baselines, the efficacy of objective 3 is rated Substantial. However, the rating is a relatively weak Substantial due to the fact that for three of the four major crops, achievements were below targets, ranging from 14 percent to 29 percent below their respective targets. Revised Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Page 12 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) Rationale The achievement of the first original and revised objective was Substantial. The achievement of the second original and revised objective was Modest. The achievement of the third objective was Substantial. Thus, the overall efficacy rating is Substantial with moderate shortcomings. Overall Efficacy Rating Substantial 5. Efficiency Economic Efficiency: The project did not conduct a traditional Economic analysis at appraisal. The PAD (p. 18) stated that intangible or qualitative capacity building initiative could not be quantified in monetary terms. Also, ex-ante cost-benefit analysis and rates of returns of demand-driven investments that had not been pre-identified could not be fully assessed at entry. In 2017, when the project received AF, an economic analysis was conducted, based on expected benefits from agricultural micro-investments. The AF paper (p. 58) stated that the financial and economic analysis consisted of comparing the situation “with project" with the situation “without a project". The AF was to support the following production/enterprise models: (a) irrigated production of onions, cassava, and sweet potato; (b) processing, conservation, and marketing of agro-sylvo-pastoral products; (c) conservation and commercialization of agro-sylvo-pastoral products; (d) livestock production; (e) production of millet, fonio (a local crop), peanut, and cowpea, intercropped with acacia senegalensis in the framework of soil and water conservation actions; and (f) production of improved seeds of millet. The gross margins due to the project (additional situation = situation with project - situation without project) were positive: 0.45 million FCFA; 1.05 million FCFA; 3.8 million FCFA; 0.071 million FCFA; 0.088 million FCFA, and 0.086 million FCFA, respectively, for: (i) production of onion, cassava, sweet potato with small irrigation; (ii) processing and marketing of agro-sylvo-pastoral products; (iii) conservation and marketing of agro-sylvo- pastoral products; (iv) fattening of sheep; (v) production of millet, fonio, cowpea, and groundnut in intercropping with acacia plants; (vi) improved seed production; production of gombo (a local crop), (vii) production of moringa, and fishing. The Internal Rate of Returns (IRR) due to the AF project were: 31.5 percent for onions with small-scale irrigation, 61.4 percent for processing and marketing of agro-sylvo-pastoral products, 46.5 percent for conservation and marketing of agro-sylvo-pastoral products, 83.2 percent for bovine fattening, 64.4 percent for production of millet, fonio, cowpea, and groundnut in intercropping in association with acacia plants, 98.7 percent for improved seed production, 73.8 for production of gombo, 70.9 for production of moringa (a local crop), and 41.6 for fish production in ponds. The Net Present Values (NPVs) corresponding to these IRRs are FCFA 0.71 million; FCFA 4.3 million; FCFA 10.7 million; FCFA 0.25 million; FCFA 0.4 million; FCFA 0.4 million CFA francs; FCFA 0.26 million; FCFA 3.09 million; and FCFA 2.0 million, respectively. As a result of the project, cumulative cash flows were negative for the first two years for all targeted operating models except for the seed production operating model, which was negative only in the first year. Cash flow Page 13 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) analysis showed that targeted farmers need liquidity in the first two years without which they cannot carry out their productive activities. The financial analysis confirmed the relevance of the grant mechanism provided to communes to finance micro-projects. The economic analysis was based on the following assumptions: i) the period considered was 15 years, corresponding to the lifetime of the hydro-irrigation infrastructure considered to be the longest-lived investment; ii) 100 percent of the cost of the disbursement categories "Works", "Equipment and materials", "Consultant services", "Training, workshop, seminars, study tour". The amount of the disbursement category "Grants or subsidies" was not considered because these resources were already included in the operating budget analyzed; iii) 90 percent of the financial prices corresponding to the economic price, the 10 percent difference between financial and economic producer prices were margins taken by the intermediaries and were not economic distortions and iv) a discount rate of 6 percent. When the project received AF the Net Present Value (NPV) (at country level) was estimated at approximately US$23.4 million. The Economic Rate of Return (ERR) for the entire project was estimated at 19.4 percent. The ICR (p. 19) conducted an Economic analysis following the methodology above and was based on empirical data and on investments made in the Local Investment Fund component of the project, which accounted for 71 percent of the total micro-project expenditures. The analysis calculated an NPV of US$108 million and an EIRR of 52 percent. However, the analysis found significant differences in the returns of different activities. For example, the analysis found that irrigated horticulture plots were extremely profitable while the livestock feed warehouses were not profitable. While during project appraisal it was estimated that 3.1 million beneficiaries were targeted with a total cost per beneficiary of US$23.1, at completion 3.2 million beneficiaries benefited from the project at a cost of US$21.6 per beneficiary, which was in line with similar investments in Niger. Operational efficiency: The project did not encounter any major implementation delays and only required a six months extension due to the delay in the approval of the Policy and Human Resource Development (PHRD) grant. However, the project exceeded its management budget by 30 percent. The ICR (p. 20) stated that this was a result of the broad scope of the project rather than implementation inefficiencies. Taking the economic and operational efficiency together, the project’s overall efficiency rating is Substantial. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable Page 14 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the original objective was High given its alignment with the Bank’s Country Assistance Strategy (CAS) (FY13-16). Relevance of the revised objective was High given its alignment with the Bank’s Country Partnership Framework (FY18-FY22) focus area 1 “increased rural productivity and incomes” and other focus areas emphasizing governance, and service delivery and strengthened public finance and human resource management. Achievement of the original and revised first objective is rated Substantial. The achievement of both the original and revised second objective is rated Modest given the lack of evidence of what improvements had been made to access to socio-economic services. Achievement of the third objective, which was added when the project received AF, is rated Substantial. Therefore, Overall Efficacy is rated Substantial. The Outcome of both the original and the revised objective are rated Moderately Satisfactory given the moderate shortcomings against objective two. Since the outcome ratings for the original and the revised objective are Moderately Satisfactory, the overall Outcome rating is also rated Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The project’s risks to development outcome can be classified into the following categories: Government commitment/financing: The government remains committed to the project objectives and will continue to collaborate with the Bank on a follow-on project- PIMELAN (Agricultural and Livestock Transformation - P164509, financing amount US$150 million) with the objective “to increase agriculture productivity and access to markets for small and medium farmers and agri-food small and medium enterprises in the participating project regions.” Also, according to the ICR (p. 23) the biocarbon initiative restored degraded lands through plantation of Acacia Senegal (Acacia gum) trees, which were eligible for carbon credits under the Kyoto protocol. The first payment in 2006, which benefitted 100,000 people, was in the amount of US$450,000. Under this initiative, the Bank was able to facilitate a deal in the amount of US$3.5 million with a new buyer of carbon credits up to the year 2035 resulting in communities continuing to receive payments. Security situation: During project implementation the security situation in Niger deteriorated. This resulted in the government needing to use more financial resources for security. Also, several project areas faced insecurity resulting in trainings by the government not being carried out. According to the ICR (p. 32) the Page 15 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) security situation in Niger remains fragile, which might result in the government continuing to put financial resources to security rather than local development. Institutional capacity: The project built institutional capacity of local governments and community organizations for local development planning and for the implementation and management of local development projects. Also, according to the Bank team (February 18, 2021) the project supported the formulation of a number of decrees such as Decree N°2016-75 and 2016-76 on the transfer of skills and resources to municipalities and regions. Also, the project supported the formulation of Decree N°2016- 301, important for the implementation of micro-projects, but beyond the scope of individual communes. However, further support might be needed to maintain and build on this capacity. The ICR (p. 31) stated that there might be a risk that communes and committees that joined the program more recently might not have built sufficient capacity. The support of the Niger Association of Reginal Governments (ARENI) for these communes and committees will be critical to ensure their sustainability. 8. Assessment of Bank Performance a. Quality-at-Entry According to the ICR (p. 23) the project was fully aligned with the government’s Plan for Economic and Social Development (2012-2015) and the project design built on the two previous projects of the series. Adequate implementation procedures and experienced staff were available when project implementation started only one month after approval. Furthermore, the project design took several lessons learned from previous phases of the CAP and other projects into account. These lessons included that a local government has comparative advantages in terms of knowledge of local socio-economic knowledge, constraints and potential, climate-related hazards and appropriate mitigation measures compared to central government. Also, the project increased women's participation in the investment decision-making process in order to provide a more gender balanced access to land and financing, and reduce gender- related vulnerabilities. The PAD (p.17) identified the following risks: i) lack of a fully adequate system of transferring human and financial resources to local governments and lack of concrete synergies between local authorities and deconcentrated technical services; ii) social insecurity at the local level; and iii) lack of collaboration and competing priorities among different sectoral ministries, departments, agencies, central and local authorities. To mitigate these risks, implementation mechanisms were designed with a view to be transparent and participatory for the beneficiaries and stakeholders. However, the Bank team did not identify the risk of changes in commune administration and the central government replacing more than 70 percent of municipal councils, resulting in new appointees requiring significant training. Also, the ICR (p. 29) stated that the project design had shortcomings. First, the design of the inter-communal component was incomplete and the likely higher demand was not fully assessed. Second, the project design included a large number of agencies, making implementation arrangements challenging. Third, the readiness and risk assessment for the National Agency for the Financing of Municipalities (ANFICT) was not adequate. In addition, there were obvious limitations in the structure and approach to the results framework and M&E. The outputs lacked substantive targets. This need only have been at the aggregate level given the Page 16 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) options available to communities, but there was a poor link between what was intended for community investments and what was delivered. In addition, the project’s Results Framework and design of the M&E system had several shortcomings (see section 9a for more details). Thus, the Quality at Entry rating is Moderately Unsatisfactory. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision According to the ICR (p. 30) the Bank’s supervision missions were conducted at irregular intervals and on average there was only one per year. Also, the project had three Task Team Leaders (TTLs), with only the one at appraisal being based in the country office, resulting in lack of continuity and delays in addressing issues. In addition, during the last 1.5 years the Bank team did not conduct any supervision mission even though it would have been critical for developing an exit strategy and ensuring the sustainability of outcomes of the final phase of a 16 years Adaptable Program Loan. Even though there were two supervision missions during the last six months of project implementation, the ICR (p. 30) stated that they were too late for having any impact on implementation since 99 percent of funds were disbursed. The Bank team restructured the project twice (with the second restructuring taking place just two weeks before the original closing date). When the project received AF the Bank team revised the PDO as requested by the government. The Bank team adapted the Results Framework accordingly. However, the revised Results Framework did not sufficiently reflect necessary details on improvements in nutrition. According to the ICR (p. 30) the Bank’s supervision missions assessed the project’s fiduciary and safeguards aspect appropriately. However, when the Bank did not conduct supervision missions the Implementation Status Reports (ISRs) did not assess safeguards and procurement aspects sufficiently until project closure. Given the lack of regular supervision missions, which resulted in issues not being addressed in a timely manner, Bank supervision is rated Moderately Unsatisfactory. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization Page 17 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) a. M&E Design According to the PAD (p. 15) the project’s M&E design was built on the system used in the previous phase. The project’s objective was clearly specified and its theory of change and how outputs were to lead to the intended outcomes was sound. However, the Results Framework had several shortcomings. First, the selected indicators in the Results Framework did not encompass all outcomes of the PDO statement. The Results Framework did not include any indicator to measure the part of the objective “improve the access of targeted population to socio-economic services”. Second, the ICR (p. 27) stated that the methodology for obtaining data of some indicators such as increasing institutional capacity, adopting messages, and improving services was not sufficiently defined in the M&E manual. Third, the indicators did not sufficiently explain the lack of baselines regarding access to services, nor information on productive investments. According to the PAD (p. 15) at the national level, the M&E team (NCU) was to lead all aspects of monitoring and evaluation and provide operational tools and instruments for data collection at the regional and local levels. Furthermore, the team was to assist local governments in monitoring the implementation of their respective communal development plans (PDCs). When the project received AF in 2017, the PDO was revised to focus more on irrigation, agriculture productivity and nutrition as requested by the government. However, the revised Results Framework did not sufficiently reflect and illustrate improvements in nutrition. b. M&E Implementation According to the ICR (p. 27) the Project Implementation Manual defined the project’s M&E arrangements, definitions and type of data collection for most indicators. The project used an electronic M&E system (DELTA M&E software). Furthermore, the project provided training to staff and partners on Geographic Information System (GIS) and remote sensing, as well as the use of the Ex-Ante-Carbon Balance Tool (EX-ACT) tool (which is an appraisal system developed by the Food and Agriculture Organization (FAO) providing ex-ante estimates of the impact of agriculture, forestry, and fishery development projects, programs and policies on the carbon-balance). The ICR (p. 27) further stated that the project was praised by the Bank for using the KOBO Toolbox (an application for the production of geo-localized data and information). The project conducted a total of five beneficiary surveys throughout the implementation period. At project closure a performance evaluation was conducted, which consisted of sample surveys of project results as well as beneficiary satisfaction and an evaluation of data from project reports. It was disappointing these could not have been more extensively used in recording progress against the project objectives. According to the Bank team (February 19, 2021) M&E functions are likely to be sustained for instance for the Bio-carbon initiative as it is instrumental for data collection for carbon sequestration and utilization of carbon credit payment by communities etc.. c. M&E Utilization According to the ICR (p. 27) the project implemented a communication strategy using different communication tools such as newsletter, briefs, reports on activities and community radio in local languages to report on the progress of project implementation. The ICR further stated (p. 28) that M&E Page 18 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) data on sustainable land management was used by the government to report on progress in regards to the Sahel Great Green Wall initiative. According to the Bank team (February 19, 2021) data collected under the project and documentary films and reports produced were crucial in convincing partners to make decisions in signing agreements to buy carbon credits resulting in a deal worth US$ 3.5 million with a new buyer of carbon credits up to the year 2035. As stated above the project’s M&E had significant shortcomings such as the selected indicators in the Results Framework not encompassing all outcomes of the PDO statement and the methodology for collecting data was not being sufficiently defined. Therefore, the project’s overall M&E rating is Modest. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as category B and triggered the Bank’s safeguard policies OP/BP 4.01 (Environmental Assessment), OP/BP 4.04 (Natural Habitats), OP/BP 4.09 (Pest Management), OP/BP 4.11 (Physical Cultural Resources), OP/BP 4.12 (Involuntary Resettlement), and OP/BP 4.36 (Forests). The project prepared an Environmental and Social Management Framework (ESMF), Pest Management Plan (PMP) and Resettlement Policy Framework (RPF). According to the ICR (p. 28) the Bank team reviewed the project’s safeguard compliance regularly on its supervision missions. Key issues were related to delays and inconsistencies in recording and reporting complaints and haphazard and incomplete documentation and filing of land gift and tenure cases. Due to the documentation issue the project’s safeguard rating was downgraded from Satisfactory to Moderately Satisfactory after the last supervision mission. b. Fiduciary Compliance Financial Management: According to the ICR (p. 28) the project prepared a financial management manual and shared a simplified version with communes to support the implementation of micro-projects. The Bank team reviewed the project’s financial management arrangements and procedures on a regular basis to ensure compliance with fiduciary requirements. During the first year of project implementation the Bank team identified financial management related issues such as lack of key staff in the regions and delays in financial reporting. The ICR did not state how the Bank team addressed these shortcomings. The ICR (p. 28) stated that the external auditor’s opinions were mostly unqualified. When the project closed, Financial Management performance was rated Satisfactory. Page 19 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) Procurement: According to the ICR (p. 29) the project prepared an implementation manual, which included procurement procedures. A simplified version was shared with communes for the implementation of micro-projects. At the beginning of project implementation, the Bank team identified some procurement related issues such as non-adherence to prior review requirements, non-publication of the procurement plan and contract awards, lack of proper filling of procurement files, and weakness in using the PROCUS system and later the Systematic Tracking of Exchanges in Procurement (STEP) system. c. Unintended impacts (Positive or Negative) NA d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment High relevance, Substantial Efficacy with moderate Outcome Satisfactory Moderately Satisfactory shortcomings and Substantial Efficiency. Not all risks identified, significant Moderately Moderately shortcomings in the Results Bank Performance Satisfactory Unsatisfactory Framework and lack of supervision missions. Significant shortcomings in the Quality of M&E Substantial Modest M&E Design. Quality of ICR --- Modest 12. Lessons The ICR (p. 32-33) included several lessons learned, which were adapted by IEG:  Conducting an institutional financial and economic assessment, especially for a lengthy Bank engagement, is critical for ensuring its sustainability. Niger, as the world’s poorest country, faces a very weak fiscal position, which will not allow the government to support the continuation of local development once project support ends. Therefore, Page 20 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) identifying future financing options before the 16 years long program closed would have been essential to ensure the sustainability of project outcomes.  When the project is aiming to provide opportunities for women it is critical, during project preparation, to assess project activities in terms of gender inclusion. While this project supported women with certain micro-projects, it did not invest in capacity building for women in critical areas such as financial management and budgeting. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provides an adequate overview of project preparation and implementation. It also includes an appropriate economic analysis. However, the ICR is not sufficiently outcome-driven and does not provide sufficient data to support the project’s outputs and outcomes. For example, the ICR states that the cash-for- work program provided a significant income boost for poor households but it does not provide data to support this statement. Overall, the ICR is rather vague and lacks specificity. For example, on page 28 in regard to FM issues it states that “actions were taken” but does not specify which actions. On page 22, it states that capacity was built within the Ministry of Agriculture and Livestock and the Ministry of Environment through the Bio Carbon Acacia Senegal activity but does not specify what this activity was and only does so one page later. Also, the ICR does not include a theory of change for the four major crop productivity outcomes of the revised PDO. Finally, the ICR does not explain the difference in data between table 4 and the Results Framework (e.g. while table 4 states that the increase in agricultural productivity of major crops – tomato- 42.5 tons per hectare was, the Results Framework states that it was 61 tons per hectare). The lessons learned were useful but could have benefitted from more details. Finally, the ICR does not sufficiently mention anything related to attribution. Given these significant shortcomings the overall quality rating is Modest. a. Quality of ICR Rating Modest Page 21 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Niger Community Action Programm Phase 3 (P132306) Page 22 of 22