Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432) Report Number: ICRR0022504 1. Project Data Project ID Project Name P119432 LS - Smallholder Agriculture Dev Project Country Practice Area(Lead) Lesotho Agriculture and Food L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) COFN-C1250,COFN-C1260,IDA- 31-Mar-2018 16,069,938.88 50170,IDA-61440 Bank Approval Date Closing Date (Actual) 10-Nov-2011 28-Feb-2020 IBRD/IDA (USD) Grants (USD) Original Commitment 10,000,000.00 0.00 Revised Commitment 17,270,019.49 0.00 Actual 16,071,364.85 0.00 Prepared by Reviewed by ICR Review Coordinator Group Katharina Ferl Vibecke Dixon Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Project Appraisal Document (PAD) (p.vi) and the Financing Agreement of November 16, 2011 (p. 5) the objective of the project was “to increase marketed output among project beneficiaries in Lesotho‘s smallholder agriculture sector”. According to the Financing Agreement of October 11, 2017 (p. 5 ) the objective of the project was changed to “increase in marketed output among project beneficiaries in Lesotho’s smallholder agriculture sector and, to Page 1 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432) improve the recipient’s capacity to respond promptly and effectively to an eligible crisis or emergency” when the project received Additional Financing (AF). b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 11-Oct-2017 c. Will a split evaluation be undertaken? Yes d. Components The project included four components: Component 1: Increasing Agricultural Market Opportunities (appraisal estimate US$4.07 million, actual US$11.66 million): This component was to include two sub-components: Sub-Component 1.1: Promotion of Innovative Agri-Business Initiatives: This sub-component was to finance, through a Competitive Grants 2 Program (CGP), the introduction, testing and demonstration of new business initiatives and technological innovations by small and medium agriculture-related and rural businesses, registered associations and cooperatives. Grants were to be awarded based on proposals received from applicants and chosen through a competitive selection process. Subcomponent 1.2: Market Linkage Development: This sub-component was to finance the development and improvement of links between agricultural producers and markets, reduction of market transaction costs, and alignment of production decisions with business and market opportunities. This were to be achieved by setting up market linkage mechanisms and providing for improved information flow and responsiveness between all actors in the market chain. The sub-component was to target commodity-based farmer associations, district and local apex associations, registered farmer cooperatives, informal farmer organizations or producer interest groups, market intermediaries, agri-businesses, input suppliers and other market participants. Supported activities were to include: a) an upgraded public market information system, to be managed by the Department of Marketing of the Ministry of Trade and Industry, Cooperatives and Marketing (MTICM), which were to seek to improve information quality and services by providing training, supporting sub-sector and commodity studies, and introducing IT-supported data and information exchange b) agricultural trade fairs at district level; c) round-table meetings of farmers/farmer groups and traders/processors; and d) a mentoring service providing direct field support to producer groups and associations helping them to understand and adopt technologies in accordance with market requirements. Page 2 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432) Component 2: Increasing Market-Oriented Smallholder Production (appraisal estimate US$14.61 million, actual US$5.54 million): This component was to finance two sub-components: Sub-component 2.1: Preparation and implementation of Agricultural Investment Plans (AIPs): This sub- component was to finance the preparation and implementation of AIPs. The AIPs were to identify promising agricultural activities, establish investment priorities, and indicate training that will be needed to ensure that the activities can be taken up successfully. The AIPs will target three main groups of beneficiaries: (a) existing producer groups that want to improve the production and productivity of their crops, improve their market integration, increase their membership or join forces with other groups; (b) broader community- based groups that manage resources or facilities which are important for market-oriented production; and (c) poorer farmers who have an interest in joining a group or committed farmers with a common interest wishing to form new groups. Sub-component 2.2: Technology packages for smallholders: This sub-component was to finance the provision of contracts to locally-based Non-Governmental Organizations (NGOs) and private operators to support the introduction, adaptation and dissemination of new and improved technologies, coupled with training and support. Topics were to be drawn from common themes emerging during the preparation of the AIPs and requiring innovative solutions not necessarily obvious to the local communities to be prioritized at annual technology forums attended by district and national staff and experts. Examples of technologies likely to be supported through the project were to include: conservation agriculture, water harvesting, improved homestead gardening, mushroom production, micro-scale irrigation systems, use of open- pollinated varieties, livestock feeding and improvement, and small-scale processing technologies. Component 3: Project Management (appraisal estimate US$4.18 million, actual US$4.34 million): This component was to finance the project’s management. A Project Management Unit (PMU) was to be established in the Ministry of Agriculture and Food Security (MAF). Also, an effective project management and administrative system were to be set up to ensure coordination between the project, other initiatives and national institutions working in the sector. During the restructuring in 2016 the following component was added: Component 4: Contingent Emergency Response Component (CERC) (appraisal estimate US$ zero, actual US$1.36 million): The component with zero-dollar allocation was added in 2016 as a response to the El Nino drought and was to allow for rapid reallocation of project proceeds in the event of a natural or man-made disaster or crisis that has caused or is likely to imminently cause a major adverse economic and/or social impact. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The project was estimated to cost US$34.46 million. Actual cost was US$30.81 million. Financing: The project was financed by a US$10.0 million credit by the Bank of which US$8.75 million was disbursed. The project also received Additional Financing of US$10.0 million of which US$7,32 million was disbursed. The project also received financing in the amount of US$10.0 million by the International Fund for Agriculture Development of which US$8.06 million was disbursed. Page 3 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432) Borrower Contribution: The Borrower was to contribute US$3.48 million which materialized. Also, local farmer organizations were to contribute US$980,00. The actual contribution was US$3.2 million. Dates: The project was restructured twice:  On July 26, 2016 the project was restructured (level 1) to: i) change the development objectives to include the Contingent Emergency Response Component (CERC) to finance emergency expenditures in response to the El Nino related drought. The new PDO included “..and, in the event of an eligible crisis or emergency, to provide immediate and effective response to said eligible crisis”; ii) revise the Results Framework accordingly; iii) reallocate financing between disbursement categories; iv) change institutional arrangements; and iv) modify procurement.  In October 2017, the project was restructured (level 1) to: i) receive AF in the amount of US$10.0 million to scale up the project interventions from four to seven districts; ii) revise the PDO to reflect the newly introduced language of the CERC objective to “To increase marketed output among project beneficiaries in Lesotho’s smallholder agricultural sector and, to improve the recipient’s capacity to respond promptly and effectively to an eligible crisis or emergency”; and iii) revise target values of PDO indicators to reflect the AF scope; and iv) the project’s closing date was extended from March 31, 2018 to February 28, 2020.  On October 18, 2019 the project was restructured (level 2) to: i) modify an error in the share of IDA financing for disbursement category 3 for eligible emergency expenditures (which was increased from 50 percent to 100 percent); and ii) revise the PDO indicator on direct beneficiaries accordingly (upwards) 3. Relevance of Objectives Rationale According to the PAD (p. 1) between 1980 and 2009 Lesotho’s economy moved away from depending on subsistence agriculture and remittances from mine workers based outside the country and experienced considerable diversification with a more balanced mix of industry, services and exports. During this time real Gross Domestic Product (GDP) increased by 3.3 percent per capita annually. However, despite these positive developments, poverty rates remained high, particularly in rural areas. 76 percent of Lesotho’s population lived in rural areas earning significantly lower incomes and facing higher poverty than the country’s urban population. In 2008, unemployment rate stood at 22.7 percent. Also, at the time of appraisal in 2011, the Gini coefficient was estimated at 0.66 indicating inequality in the distribution of wealth. More than three quarters of the rural population was involved in agricultural activities. However, the agricultural sector experienced several challenges such as under-investments and poor farming practices, unsustainable land management leading to declining soil fertility and severe soil erosion, as well as extreme weather conditions such as frost, hailstorms, extreme heat, and frequent droughts. Low performance of the agricultural sector resulted in poor nutrition outcomes. For example, in 2007 the average national stunting rate was at 41.7 percent. To address these issues the government developed the National Vision 2020 which aimed to: i) combat wide-spread stock theft; ii) diversify into cash crops; iii) revise land tenure system; and iv) advance irrigation Page 4 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432) methods and effective extension service. Also, the government developed the National Strategic Development Plan (NSDP) (2011/12-2016/17) which aimed to: i) improve access to finance; ii) promote the production of high value crops and livestock products; iii) improve quality livestock breeding, seed production capacity and access to farm machinery; iv) promote investment in agro-industry and development of agri-business to increase value addition and market integration. The original and revised objectives of the project supported these government strategies. Also, the objectives were in line with the Bank’s most recent Country Partnership Framework (CPF) (FY16-20), particularly two priorities under focus area 2: i) improving the business environment and diversifying economy, and ii) improving agricultural productivity of smallholders and micro, small, and medium enterprises (MSME) in agriculture. While the original and revised PDOs remained relevant throughout project implementation a shortcoming was the lack of clarity in the PDO formulation around what outcomes would be achieved through increasing marketed output among project beneficiaries in Lesotho‘s smallholder agriculture sector. As a result, the relevance of objective rating is Substantial. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To increase marketed output among project beneficiaries in Lesotho‘s smallholder agriculture sector: Rationale This validation conducts a split rating since the PDO and targets where modified when the project received AF. Theory of Change: The project’s theory of change envisioned that project activities such as promoting agribusiness initiatives, developing market linkages, preparing and implementing Annual Investment Plans (AIPs), and providing technology packages for smallholders were to result in outputs such as Competitive Grant Program (CGP) proposals being financed, CGP sub-projects being trained, AIPs proposals being financed, and training being provided for CGP/AIP. These outputs were to result in outcomes including CGP sub-projects being commercially viable, farmers and traders using public market information, farmers having improved access to markets and farmers adopting improved technologies and practices. Finally, these intermediate outcomes were to result in the objective of increased marketed output. According to the ICR (p. 6) the project made the following assumptions (which were within the project’s control): a. Smallholder farmers are interested in working together through producer and market groups; b. Qualified service providers and contractors are available in the market; Page 5 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432) c. Marketing grant implementation modality, which provided a great degree of independence to beneficiaries and service providers in implementation of the grants and required sound skills and knowledge in financial, procurement, contract and business management, selection and supervision of equipment and works, is adequate; d. Piggery and poultry sub-projects, which required stronger cash flows and depended on availability of affordable feed (the latter dependent on weather/climate shocks), would not require different co- financing conditions. Key assumptions that were beyond the project’s control: a. Severe climatic shocks do not occur during the project life; b. Complementary International Fund for Agricultural Development (IFAD) financing assistance support would be delivered in a timely manner for scaling up project impacts. There was also the underlying assumption that training in new technologies would automatically result in the uptake of new technologies. Outputs:  878 matching grant sub-projects, of which 776 sub-projects remained operational at closing, were financed. This output did not have a target.  Out of all sub-projects, 435 sub-projects were supported in terms of production of horticulture and seedlings. Support included greenhouse (0.25 hectares per sub-project), shade nets (0.40 hectares per sub-project), development of or improved irrigation systems (mainly boreholes), adoption of drip or sprinkler irrigation system, extension of advisory services, and participation in international training programs. At project closing, 408 out of these 435 sub-projects were operational. This output did not have a target.  193 of CGP projects completed were commercially viable, surpassing the original target of 24 CGP projects.  79,006 client days of training were provided under the CGP component, surpassing the original target of 11,200 days. 47,628 client days of training were provided to female clients, achieving the original target of 47,628 training days. According to the Bank team (March 29, 2021) the training focused on: i) productivity-enhancing and natural resource management (NRM) technologies (i.e.; seeds/breeds, irrigation technologies, pest/disease management, production and post-production infrastructure and value addition, water, land and farm management practices); and ii) market-oriented, or “Farming as Business” approach (commercial training to better consider demand, costs and benefits, marketing strategies among others in decision making).  The number of targeted beneficiaries who have adopted improved production technologies/farming practices in targeted areas increased from 234 beneficiaries in 2011 to 3,271 beneficiaries in 2020, not achieving the original target of 7,500 beneficiaries.  49 percent of the targeted beneficiaries were female, almost achieving the target of 50 percent.  The percentage of farmers and traders in project districts who used public market information increased from 10 percent in 2011 to 36 percent in 2020, not achieving the original target of 50 percent.  2,318 new business contracts between farmer groups and the private sector were established as a result of the project’s market facilitation activities surpassing the original target of 100 business contracts. Page 6 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432)  42 percent of targeted beneficiaries recorded better access to markets, not achieving the target of 50 percent.  The percentage of beneficiaries satisfied with improved performance of agricultural service providers increased from 15 percent in 2011 to 93 percent in 2020, surpassing the original target of 50 percent.  35,360 client days of training under the Agriculture Investment Plan were provided, not achieving the original target of 160,000 days, and surpassing the revised target of 30,000 days. 15,970 client days of training were provided to women, not achieving the original target of 80,000 days and achieving the revised target of 15,000 days. Outcomes:  The percentage of household commercialization (measured through value of produce sold as percentage of total value of produce generated) in project areas increased from 15 percent in 2011 to 76 percent in 2020, surpassing the target of 40 percent. According to the ICR (p. 15) the project farmers produced 95 percent higher value of marketed output than the control groups resulting from higher production of cash crops and higher prices obtained per unit of marketed output and higher marketed shares of output.  Yields of major cash crops (potatoes, carrots, cabbage, and tomatoes) in the targeted area increased from zero in 2017 to 21.30 percent in 2020, surpassing the target of a 15 percent change. It is unclear what this baseline value is measuring or referring to. The ICR provided baselines for each crop but not for all major crops together. According to the ICR (p. 12) the project substantially exceeded the targets for two crops (tomatoes and cabbages). Two crops (potatoes and carrots) could not be assessed since those crops did not grow under protected farming (in greenhouses). Beans and green vegetables also achieved 8 to 13 percent higher yields. Furthermore, a 2019 Bank study “Linking Smallholders to Markets” project beneficiaries found a 40 percent higher productivity for green bell peppers for project participants compared to farmers growing peppers in open field and using traditional practices.  Smallholder livestock production experienced a net increase from zero in 2017 to 7.52 percent in 2020, not achieving the target of a 10 percent change. According to the ICR (p. 14) piggery and poultry sub-projects were the reason for not achieving the target since they were particularly vulnerable to frequent droughts resulting in animal feed shortages and price increases, which negatively impacted their productivity. The project was able to build capacity among farmers and supported the adoption of improved production technologies/farming practices. There is limited evidence that the project successfully increased marketed output among project beneficiaries as demonstrated by farmers achieving an increase in yields of two of four major crops (since the increases in yields were only measured for two major crops). The project was not able to increase the percentage of farmers and traders in project districts who used public market information and fell short of providing beneficiaries with better access to markets. Also, there were several issues in M&E implementation (see section 9b for more details) resulting in inadequate tracking towards achievement of the PDO. Based on the above, the achievement of the original objective is rated Modest. Rating Page 7 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432) Modest OBJECTIVE 1 REVISION 1 Revised Objective Increase in marketed output among project beneficiaries in Lesotho’s smallholder agriculture sector: Revised Rationale Theory of Change: The project’s theory of change envisioned that project activities such as providing emergency and recovery support to affected and strengthening response capacity was to result in outputs such as emergency support and recovery packages and training being provided as well as emergency staff being trained and disease outbreak control tools being provided. These outputs were to result in the outcome of beneficiaries benefitting from emergency and recovery support, if and when needed. The revised PDO made the same assumptions as the original PDO. Outputs:  Between 2016 and 2018, 2,075 food insecure and vulnerable households were provided with emergency special protection and integrated sustainable production support including kits of improved seeds, fertilizer, grazing vetch, shade-nets and plastic to protect vegetables against extreme cold and heat, surpassing the target of 2,000 households. This output did not have a target.  8,882 vulnerable households received support in restoring livelihoods through rehabilitation of irrigation systems, water storages, provision of livelihood kits, training on Climate Smart Agriculture (CSA), Sustainable Land Management (SLM), and nutrition sensitive technologies and practices. Of all the beneficiaries of the (CERC), 10,917 households received nutrition sensitive production support. This output did not have a target.  193 of CGP projects completed were commercially viable, achieving the revised target of 193 CGP projects.  79,006 client days of training were provided under the CGP component, surpassing the revised target of 60,000 days.  47,628 client days of training were provided to female clients, achieving the revised target of 30,000 training days  The number of targeted beneficiaries who have adopted improved production technologies/farming practices in targeted areas increased from 234 beneficiaries in 2011 to 3,271 beneficiaries in 2020, not achieving the revised target of 3,000 beneficiaries.  The percentage of farmers and traders in project districts who used public market information increased from 10 percent in 2011 to 36 percent in 2020, not achieving the revised target of 80 percent.  2,318 new business contracts between farmer groups and the private sector were established as a result of the project’s market facilitation activities surpassing the revised target of 2,000 business contracts.  The percentage of beneficiaries satisfied with improved performance of agricultural service providers increased from 15 percent in 2011 to 93 percent in 2020, surpassing the revised target of 70 percent. Page 8 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432)  35,360 client days of training under the Agriculture Investment Plan were provided surpassing the revised target of 30,000 days. 15,970 client days of training were provided to women achieving the revised target of 15,000 days. Outcomes:  23,140 beneficiaries benefited from the Contingent Emergency Response Component, which was activated due to the El Nino drought in 2015/16, surpassing the target of 5,000 beneficiaries.  An assessment by the Food and Agriculture Organization (FAO), the implementing partner of the CERC, found that beneficiaries experienced doubled crop production, diversification towards nutritious crops and 63 percent increase in maize productivity for 83 percent of beneficiaries and improved food security for 91 percent of households. Although the evidence to back these claims up is not presented in the ICR; it is plausible that the achievements at output level might have lead to increased crop production, diversification and improved food security. There is also evidence that the recipient’s capacity to respond promptly and effectively to an eligible crisis or emergency was improved as demonstrated by the large number of beneficiaries which benefited from the project’s intervention during the 2015/16 El Nino drought. Even though, the project experienced issues in regard to tracking progress towards the achievement of the PDO, with the additional evidence provided through FAO and through the El Niño response, the achievement of this objective is rated Substantial with some shortcomings. Revised Rating Substantial OBJECTIVE 2 Objective NA Rationale NA Rating Not Rated/Not Applicable OBJECTIVE 2 REVISION 1 Revised Objective To improve the recipient’s capacity to respond promptly and effectively to an eligible crisis or emergency: Revised Rationale Page 9 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432) Theory of Change: The project's theory of change envisioned that outputs such as capacity building in areas such as CSA, SLM, etc., the adoption of promoted technologies, improvement of control of livestock diseases outbreak, and improvement of emergency decision making were to result in the outcome of the recipient being able to respond promptly and effectively to an eligible crisis or emergency. Outputs:  296 extension officers of MAFS and MFRSC were trained in areas such as CSA, SLM, post-harvest management, food preservation techniques, rangeland management, fodder production, conservation agriculture and diagnosis and treatment of zoonotic diseases.  To further facilitate replication and adoption of promoted technologies, the FAO developed communication and training materials which were adapted to local conditions for extension officers. 25,000 handouts on conservation agriculture and 90,000 handouts on home gardening and nutrition were distributed to the farming communities and primary schools.  Livestock extension services were provided with veterinary kits and sterilizers to improve control of livestock disease outbreak. This allowed for the vaccination or treatment of livestock assets of 15,214 households.  Emergency decision making was improved through a) assessment, documentation and dissemination of impacts of CERC interventions and lessons learned; b) development of Sustainable Livelihoods Framework and Lesotho Livelihoods profiling exercise to support improved understanding of community dynamics and informed identification of targeted transfers and services for vulnerable households; c) thematic studies including on seed security and vulnerability; d) development of web- based Crop Information Portal under the MASF to strengthen its capacity to generate statistics for five stable crops and for timely agricultural reporting and information dissemination. Outcome: 23,140 beneficiaries benefited from the Contingent Emergency Response Component which provided emergency response financing, which was activated due to the El Nino drought in 2015/16, surpassing the target of 5,000 beneficiaries. Revised Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Achievement of the original objective “to increase in marketed output among project beneficiaries in Lesotho’s smallholder agriculture sector was Modest while the achievement of the two revised objectives were Substantial. Based on that, the project’s overall efficacy rating is Substantial with some shortcomings. Page 10 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432) Overall Efficacy Rating Substantial 5. Efficiency Economic Efficiency: The PAD did not include a traditional economic analysis and stated (p. 19) that due to the demand-driven nature of the project design, project-supported investments could not be identified with certainty prior to implementation. As a result, it was not possible to undertake a detailed ex-ante cost-benefit analysis based on a portfolio of projected sub-project investments. Instead, an economic and financial analysis was conducted using a series of representative farm models to assess the likely attractiveness of the overall project investment and to evaluate the incentives faced by beneficiary households. The key parameters used in the economic and financial analysis included participation rate, mix of cropping and livestock production activities, production increases, and productivity gains. According to the PAD (p. 19) the economic analysis estimated that if the project were to reach 5,000 to 7,000 beneficiary households within the project implementation period, and if each of these beneficiary households were to realize a minimum projected set of incremental benefits, the aggregate economic benefits generated by the project was to produce an overall internal rate of return (IRR) in excess of 12 percent. The financial analysis showed that changes in cropping patterns and productivity gains expected to result from project-supported activities were likely to generate attractive financial returns to participating households ranging from 13.3 percent to 49.0 percent. The PAD stated that the economic and financial analyses used conservative values for many key parameters indicating that the actual returns might be higher. The ICR (p. 17) modified the PAD’s analysis by including measuring returns through CGP and AIP sub-project models, using actual benefits and costs, and revising benefit and cost assumptions. The ICR used a ten-year time period for sub-projects and applied a discount rate of five percent and a financial discount rate of eight percent. For intensive vegetable production the FRR was 36.7 percent and the ERR was 44.2 percent. For intensive pig production the FRR was 16.1 percent and the ERR was 21.7 percent. For broiler and egg production the FRR was 4.4 percent and the ERR was minus 4.6 percent. For wool and mohair production the FRR was 80.8 percent and the ERR was 97 percent and for diary production the FRR was 5.5 percent and the ERR was 10.1 percent. The overall project ERR was 18.4 percent, indicating that the project was a worthwhile investment. Also, the ICR (p. 18) stated that the project’s ERR is competitive with other agriculture commercialization interventions in the South Asia and East Asia Pacific region. Operational Efficiency: The project experienced implementation delays, especially during the initial phase of implementation, due to capacity issues, demand driven nature of matching grant interventions, and lack of interest in AIP interventions and community contributions to community resource management activities. Due to an increased share of beneficiary contributions (from AIP with 20 percent beneficiary contribution to CGP with 40 percent beneficiary contribution), exchange rate savings in the amount of approximately US$0.70 Page 11 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432) million, and reimbursement of ineligible expenditures (US$270,271) the project did not disburse US$3.25 million of the IDA financing and US$2.66 million of IFAD financing. Taking everything together, the project’s overall efficiency rating is Substantial. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the original and revised objective was Substantial given the alignment with the Bank’s most recent Country Partnership Framework (CPF) (FY16-20). The efficacy of achievements of the original objective was Modest. The efficacy of achievement of the revised first objective was Substantial with some shortcomings, and efficacy of achievement of the second revised objective was Substantial. Efficiency was Substantial. Hence, the outcome of the original objective is Moderately Unsatisfactory and the outcome of the revised objective is Moderately Satisfactory. The outcome ratings for the original and revised projects are thus: Original project Revised project Relevance Substantial Efficacy Modest Substantial with shortcomings Efficiency Substantial Outcome rating Moderately Unsatisfactory Moderately Satisfactory According to IEG/OPCS harmonized guidelines, when a project’s objectives are revised, the final outcome rating is an average of outcomes before and after the revision of objectives weighted by Bank disbursements under each set of objectives. In this project 14.76% of disbursements occurred under the original objective and 85.24% under the revised objective. Page 12 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432) • Based on the original objective with an outcome of Moderately Unsatisfactory (score of 3) and a disbursement weight of 0.147 the weighted outcome score is 0.44. • Based on the revised objective with an outcome of Moderately Satisfactory (score of 4) and a disbursement weight of 0.85 the weighted outcome score is 3.4. • The combined weighted average outcome score is 3.84 which corresponds to a Moderately Satisfactory outcome rating. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The project’s risks to development outcome can be classified into the following categories: Government commitment/follow on projects: According to the ICR (p. 27) the government continues to be committed to the project’s PDO and continues to work with IFAD to receive support for sup-projects, which only became operational at project closing. Also, the government and the Bank continue to collaborate on a new project the “Lesotho Smallholder Agriculture Development Project 2”, a five-year project that became effective in August 2019 and aims to scale-up the adoption of good agricultural practices. Support from donor organizations is critical to ensure continuous financing in this area and sustaining project outcomes. Capacity building: The project was able to build institutional capacity among the implementing agencies, technical agencies, farmers and agribusinesses. According to the ICR (p. 27) the project enabled farmers to move away from producing for their own household towards commercialization and market-oriented production. The capacity built will support the sustainability of project outcomes. Climatic events: The project was able to provide farmers with better coping mechanisms and build capacity in government agencies to support farmers. However, Lesotho continues to be subject to extreme climatic events such as the droughts in 2015/16 and 2018/19, which might negatively impact the outcomes of this project. Covid-19 pandemic: According to the ICR (p. 27) the Lesotho Vulnerability Assessment Committee projected that the covid-19 pandemic is likely to increase the number of food insecure households by 179,857 by September 2020. Also, travel restrictions on food supply chains, decrease in consumer spending and purchasing power will negatively impact Lesotho’s agriculture and food systems for the beneficiaries. 8. Assessment of Bank Performance a. Quality-at-Entry Page 13 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432) According to the PAD (p. 13) the project’s design was built on lessons learned from previous projects in Lesotho such as the Bank’s Agricultural Policy and Capacity Building Project (APCBP) project. The main lessons from that project were the need to avoid an overly complex project design and the desirability of having simple and clear implementation mechanisms. The Bank identified relevant risks and rated the risk of weak implementation capacity as High. The following risks were identified as Substantial: i) size of local market and competition from South Africa; ii) weak capacities and unclear responsibilities of implementation partners, as well as tight governmental budget situation with unclear development for the future MAFS budget; iii) joint IFAD/Bank operation, though expected to provide synergies, might also lead to increased bureaucracy for the implementing partners; and iv) complex project design. The project tried to mitigate these risks by providing training and building capacity in participating government support services as well as embedding the PMU within MAFS to strengthen the skills of senior government officers. The project tried to mitigate the risk of competition from South Africa by several measures such as the project initially focusing on commodities that have a comparative advantage in the local market, supporting and training of farmers and farmer groups to gear production to market demands and establishing a competitive grants program to help create market opportunities for local production. Furthermore, design elements were to be aligned to serve both IFAD and Bank requirements as much as possible. However, while the mitigation measures for most risks were adequate, they were not sufficient in terms of capacity and according to the ICR (p. 22) the project experienced a slow start to implementation due to low technical, fiduciary and M&E capacity at the district level. Also, the PMU did not have the necessary capacity to address issues related to social and environmental safeguards and financial management resulting in safeguard compliance being rated Moderately Unsatisfactory several times throughout implementation and the project encountering cases of misappropriation of grant funds and cancellation of project proceeds and project beneficiaries and SPs (see section 10b for more details). The Bank team did not identify the risk of extreme weather events such as droughts, which Lesotho experienced between 2015-16 and 2018-19. According to the ICR (p. 23) water scarcity affected crop and livestock production and reduced productivity and animal feed shortages. As a result, the price of animal feed increased, not allowing for some sub-projects to be operationalized. Also, the Bank team did not identify the risk of insufficient demand for grants resulting in implementation delays. Finally, despite a large proportion of female farmers, the project did not take gender aspects into account. The project’s Results Framework had several shortcomings such as the majority of indicators lacking a baseline, targets being overly ambitious, and lack of guidance on impact assessment approaches to measure project outcomes etc. (see section 9a for more details). Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The Bank conducted 12 supervision missions throughout project implementation. The ICR (p. 26) stated that the Bank and IFAD supported the implementation of the project adequately and had the adequate mix of expertise. The project team provided the government with advice and identified actions to address implementation issues. For example, the recommendation by the project team to Page 14 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432) reallocate resources from underperforming AIPs to the CGPs resulted in an increase of sustainable sub- projects. According to the ICR (p. 26) implementation progress was reported candidly and the project team revised grants resources that required commitments of beneficiaries to adequately operate, maintain and manage investments. The Bank team made adaptations to the project’s Results Framework during the Mid-Term Review (MTR). However, these changes to the Results Framework were not sufficient to allow for an adequate monitoring of implementation progress. The ICR (p. 25) stated that the Bank was proactive in addressing safeguard compliance issues and proposed an action plan for the implementation of corrective actions (see section 10a for more details). However, the ICR (p. 26) also stated that safeguard compliance issues and financial management issues under the CGP were only identified during the second half of project implementation, which might indicate that Bank supervision was not sufficiently close. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The original and revised objective of the project were not sufficiently specific. The original and revised objectives did not clearly state what outcomes were to be achieved through increasing marketed output among project beneficiaries in Lesotho‘s smallholder agriculture sector. Therefore, the theory of change and how key activities and outputs were to result in the intended outcomes was not clear. According to the ICR (p. 12) the project’s Results Framework had several shortcomings: i) while all indicators included a target, the majority of indicators lacked a baseline; ii) targets were overly ambitious; iii) the baselines yields were established in 2018 and were based on ongoing general trends and changes since project appraisal in 2012; iv) representative of one year only while changing climate and weather events in Lesotho would require the use of at least three-year averages; v) baseline yields were drawn from national statistics which also included commercial farmers with access to irrigation water, technology and knowledge and did not focus on project beneficiaries; and vi) lack of guidance on impact assessment approaches to measure project outcomes. Furthermore, using an indicator on productivity with fixed proxy crops was not adequate for measuring the project’s achievement in diversification towards market driven crops. Page 15 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432) According to the PAD (p. 17) the PMU M&E Officer were to have primary responsibility for monitoring progress and outcomes based on indicators defined in the project Results Framework. b. M&E Implementation According to the ICR (p. 22) the project faced several issues regarding M&E implementation due to weak M&E capacity at the district level. As a result, M&E of sub-projects was limited. Furthermore, lack of regular monitoring resulted in a limited ability to assess the PDO achievement during implementation. The ICR (p. 24) stated that it was recommended for the M&E officer to spend more time in the field to collect data and receive feedback from sub-project clients. However, data was mainly qualitative, and the project had to rely on the Mid-Term- Review (MTR) and end line survey to assess three out of four PDOs. The ICR (p. 24) stated that after the MTR data collection, analysis, dissemination and M&E reporting improved. According to the Bank team (March 29, 2021) following the MTR recommendations, the project started capturing spillover effects of the project on indirect beneficiaries (e.g. adoption of technologies following demonstration days organized by CGPs). The project also strengthened its M&E capacity through more staff and capacity building/training for M&E specialists. It conducted CGP and AIP assessment studies, strengthened monitoring processes, planned/implemented the project completion survey. Improved monitoring capacity helped understand the reasons for low interests amongst beneficiaries to work in groups indicating lack of beneficiary inclusion. Furthermore, the project included an MTR survey and an impact assessment survey, which was conducted by an external party in a timely way. According to the ICR (p. 24) the impact assessments’ design was adequate and used the difference in difference technique to appropriately measure project. Also, the project prepared qualitative and quantitative impact assessment reports and supported the preparation of thematic studies and surveys. According to the Bank team (March 29, 2021) at the national/central level, the M&E capacity is planned to be maintained under the follow-on project. However, the M&E capacity developed at the field level (district offices) may be affected by a current higher-than-optimal staff turn-over. c. M&E Utilization According to the Bank team (March 29, 2021) despite the shortcomings mentioned above, the project’s M&E data was used to inform decision making. For example, M&E data helped to identify several issues including: i) identify the reluctance of farmers in working in groups, their interest in and potential for scaling up technology promotion and documented the significant impact of technology adoption alone on productivity enhancement; ii) identify quality and capacity issues of contractors hired by groups to implement irrigation sub-projects so that corrective measures could be taken; and iii) determine size (amount) and conditions for matching grants for both crop and livestock subprojects by providing an important lesson: livestock subprojects needed higher working capital and carried higher risk of losses during droughts. Furthermore, the M&E system also provided key data to help adjust design, as well as resolve operational and fiduciary issues, including the need for i) registration of irrigation systems, ii) restoration of damaged infrastructure, iii) provision of additional targeted training, and iv) relocation of funds. As a Page 16 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432) result of the M&E data the government, Bank and IFAD teams were able to make timely and well- informed decisions and provide adequate and timely implementation support. While M&E is rated Modest, the additional surveys and the impact assessment that were done provided sufficient evidence for the Substantial efficacy rating of the revised objectives. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as category B and triggered the Bank’s safeguard policies OP/BP 4.01 (Environmental Assessment), OP/BP 4.09 (Pest Management), OP/BP 4.11 (Physical Cultural Resources), OP/BP 4.12 (Involuntary Resettlement), and OP/BP 4.12 (International Waterways). According to the ICR (p. 25) the project prepared an Environmental and Social Management Framework (ESMF), Pest Management Plan (PMP), and Resettlement Policy Framework (RPF). Also, the Borrower conducted riparian notifications due to small-scale irrigation activities in the Orange-Senqu River basin through the Orange-Senqu River Commission to South Africa, Namibia, and Botswana. By the time the project was submitted to the Board, no responses of any of the notified riparians were received. The ICR (p. 25) stated that the project’s safeguard rating was Moderately Satisfactory until May 2018. As a result of the project’s poor compliance to environmental and social management procedures the rating was then downgraded to Moderately Unsatisfactory. Every sub-project financed by the project had to go through an environmental and social screening process. However, some sub-projects did not undergo this screening process because the farmers were not aware, and some screening processes had to be conducted retrospectively. Also, safeguard implementation was inadequate during the early phase of project implementation since prospective project sites were not screened and challenges and mitigation measures were therefore not identified. Furthermore, some AIP activities negatively impacted land and water access. Finally, the project did not document the verification process and ensure full informed consent of vulnerable groups. While the government was willing to address these issues, the PMU lacked a dedicated social and environmental specialist to follow through and ensure the project’s safeguard compliance. According to the ICR (p. 25) the Bank team addressed these issues by developing an action plan which included the recruitment of a dedicated safeguard specialist. The rating of safeguard compliance improved through: i) proper screening of sub-projects; ii) provision of training for grant recipients on Environmental and Social Safeguards; and iii) preparation and implementation of sub-project specific Environmental and Social Management Plans. As a result of these actions, the project’s safeguard rating was upgraded to Satisfactory in November 2019. However, the ICR (p. 25) stated that the project did not have a grievance mechanism to register grievances and beneficiary feedback. b. Fiduciary Compliance Page 17 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432) Financial Management: According to the ICR (p. 25) the project’s financial management performance was rated Satisfactory until January 2019. Also, the Bank team stated (March 29, 2021) that the project complied with the Bank’s financial covenants. However, due to a qualified audit report for the year ending on March 31, 2018, the project’s financial management performance was downgraded to Moderately Satisfactory. Even though the audit report for the following year was unqualified the rating remained at Moderately Satisfactory due to issues related to ineligible expenditures. The PMU addressed this issue by hiring a consultant to assess the reason for these expenditures and provide recommendations to the government resulting in the government refunding LSL2,911,350. However, the ICR stated that the project continued to have an outstanding advance which had gone beyond the application deadline and still needed to be addressed when the project closed. Procurement: According to the ICR (p. 26) the project complied with the Bank’s procurement and consultant guidelines. The project team and the procurement specialist filed all documentation related to each procurement procedure adequately using the Bank’s Systematic Tracking of Exchanges in Procurement (STEP). The project did not encounter any serious procurement issues and the project’s procurement rating was Satisfactory throughout project implementation. c. Unintended impacts (Positive or Negative) NA d. Other The project did not pay sufficient attention to gender. The indicators included in the Results Framework such as number of businesses/number of technologies adopted could have been disaggregated in terms of gender especially since approximately 70 percent of famers are female. The Results Framework only included the required Bank indicator “percentage of female beneficiaries”. However, the ICR did not state how the data was obtained and what the project did to address typical issues related to the majority of farmers being female. For example, it is not clear how the project ensured that the income earned through the agricultural activities by female farmers would not be controlled by their husbands. The project missed an important opportunity to address such gender issues. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Page 18 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432) Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The ICR (p.28-29) included the following lessons learned, which were adapted by IEG:  When a project aims to promote commercial agriculture, a mix of activities including investments and institutional support is critical. This project was able to change the mindset of smallholder agricultural producers but the ICR did not provide any evidence to back up those claims..  Assessing critical conditions during project preparation is important for ensuring the achievement of intended outcomes. In this project, the importance of farmers having incentives to work together in groups was not sufficiently identified during project preparation. As a result, AIP sub-projects were unsustainable and performed poorly.  The provision of financial capital needs to be combined with building resilience of agribusinesses to climate shocks. While this project did not address harsh weather and climatic events in its project design, it was sufficiently flexible to introduce climate-smart agriculture practices and activities during project implementation resulting in long-term climate resilience by project beneficiaries. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provided an adequate overview of project preparation and implementation. Also, the ICR included an appropriate Economic analysis. The ICR is sufficiently candid, internally consistent and outcome driven. Also, the ICR included useful lessons learned that can provide a learning experience for future projects in this area. However, the ICR did not sufficiently focus on gender. The quality of the ICR rating is Substantial. a. Quality of ICR Rating Substantial Page 19 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS - Smallholder Agriculture Dev Project (P119432) Page 20 of 20