Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK: Sindh Agricultural Growth Project (P128307) Report Number: ICRR0022977 1. Project Data Project ID Project Name P128307 PK: Sindh Agricultural Growth Project Country Practice Area(Lead) Pakistan Agriculture and Food L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-54940 30-Jun-2019 56,640,897.62 Bank Approval Date Closing Date (Actual) 07-Jul-2014 31-May-2021 IBRD/IDA (USD) Grants (USD) Original Commitment 76,400,000.00 0.00 Revised Commitment 76,400,000.00 0.00 Actual 57,435,931.06 0.00 Prepared by Reviewed by ICR Review Coordinator Group Samjhana Thapa Christopher David Christopher David Nelson IEGSD (Unit 4) Nelson 2. Project Objectives and Components DEVOBJ_TBL a. Objectives As per the Financial Agreement (FA, 2014) and the Project Appraisal Development (PAD, 2014), the Project Development Objective (PDO) of the Sindh Agricultural Growth Project was “to improve the productivity and market access of small and medium producers in important commodity value chains”. To assess the extent to which the objective of this project was achieved in this Review, the PDO will be divided into two parts, and will be referred to in Section 4 as Objectives 1 and 2 as follows: Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK: Sindh Agricultural Growth Project (P128307) Objective 1: To improve the productivity and market access of small and medium producers in key commodity value chains (Chilies, Onions, Dates, Rice). Objective 2: To improve the productivity and market access of small and medium producers in dairy value chains. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components The project has three components (PAD, pages 4-8). Component 1: Capacity Building and Institutional Development (Appraisal cost: US$18.6 million; Actual cost: US$20.6 million). This component aimed to build the capacity of producers, support the institutional development of implementing agencies, and assist in the strategic planning for the agriculture sector in the Sindh Province of Pakistan. The component included two sub-components: i. Capacity Building of producers: Based on a training needs assessment, the project would provide trainings and capacity building of farmers on good agricultural practices, harvest and post-harvest management, storage, use of equipment, marketing etc. Stakeholder forums would be organized for each of the value chains that would facilitate dialogue between the departments and farmers, and better assess their challenges and needs; and ii. Modernization of Extension Services and Agricultural Research: Support under this sub-component included: (a) technical assistance to the departments in planning and implementing the investments (Component 2); (b) financing the rehabilitation and renovation of facilities such as agricultural research centers, artificial insemination training center, veterinary care centers and semen production units; (c) modernization of extension services through ICT-based technologies (e.g. use of mobile phones, 24/7 call center, interactive websites); and (d) competitive research grants supporting research on crops and livestock. iii. Strategic Planning for the Agriculture Sector: The project would finance the development of the Provincial strategy for Sindh’s agricultural sector and prepare feasibility studies on additional crops and livestock value chains for future investments. Component 2: Investment for Agricultural Growth (Appraisal cost: US$60.1 million; Actual cost: US$34.8 million). This component consisted of four sub-components: i. Horticulture Value Chains: The project would finance three horticulture crops - dates, onions, and chilies and focus on improvements related to adoption of good agricultural practices for improved crop husbandry, and pre- and post-harvest processes that would enable market access. A key feature of this activity was the cost-sharing element of support whereby farmers would be contributing between 30-50 percent of the cost of equipment (ICR, paragraph 14). ii. Rice Post-harvest Loss Management: To address the challenges of post-harvest losses (30-40%) and improve productivity, the project would finance threshers for farmers and paddy dryers for small Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK: Sindh Agricultural Growth Project (P128307) mill operators on a 50 percent cost sharing basis. Smaller equipment such as soil and moisture testing kits, conductivity meters would also be financed with 70 percent contribution by farmers. Knowledge sharing and learning from other rice producing countries would be promoted. iii. Dairy Value Chain: The project would increase milk productivity by financing activities on improved animal health and husbandry practices, animal nutrition, milk quality and storage. Market access would be improved by forming and supporting milk producers’ groups (MPGs) for aggregating milk production for increased milk supply. iv. Demand Driven Innovation Fund: To promote any innovative ideas on technologies (not included in the project) which may be suitable for a group of producers or entrepreneurs, a fund would be established to co-finance such activities. The new technology would then enable them to provide services to a larger population. Most of the components/sub-components were implemented by project-end, except the last sub-component related to the establishment of an Innovation Fund. Due to the delays faced by the project during the early phase, capacity of the Project Management Unit was limited which led to delayed recruitment of key positions. According to the ICR (Paragraph 81), M&E Consultants were appointed only in April 2017 while the project became effective in October 2014. As a result, the Innovation Fund was not set up and the research sub-projects financed by the project was only completed just before project closing, which did not allow its results to be utilized during project implementation. Component 3: Project Management and Monitoring and Evaluation (Appraisal cost: US$10 million; Actual cost: US$7.4 million). This component would cover costs for: (i) Project Coordination Unit (PCU) at the provincial level, two Project Management Units (PMU) for the agriculture and livestock directorates, and district Project Implementation Units (PIUs); (ii) Third-party monitoring; (iii) Safeguard requirements; and (iv) Impact evaluations. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost: The appraisal estimate for the project cost was US$88.70 million and the actual cost at project closing was US$62.80 million. Actual project cost was 70.8 percent of the appraisal estimate (in US$ terms) (ICR, Annex 3). This reduced cost was due to several factors: (i) Depreciation of the Pakistani Rupee by 23 percent from PKR 100 at appraisal to an average of PKR 123 during implementation; and (ii) 28 percent inflation from 2015-2020 which was higher than the 22.3 percent depreciation of the PKR. Financing: The project was financed by an IDA credit (IAD-54940) of US$76.4 million out of which US$57.4 million was disbursed. Borrower contribution: Borrower contribution at appraisal was estimated to be US$12.3 million. At project end, the total amount contributed by the beneficiaries was US$6.8 million. Dates: The project was approved on July 07, 2014, and became effective on October 30, 2014. The project was to close on June 30, 2019, but with restructurings, the project was closed on May 31, 2021, two years after the original closing date. Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK: Sindh Agricultural Growth Project (P128307) Restructurings: The project was restructured three times (August 2016, June 2019, May 2020). The details are as follows: First Restructuring, August 2016: Based on the need of the project, a new procurement method “Framework Agreement” was requested by the Government of Sindh which was approved by the World Bank that entailed the need to amend the Financing Agreement. As implementation of the project began, it was realized that the exact quantities and timing of agriculture goods and implements to be financed by the project under Component Two on a cost-sharing basis with farmers was challenging. This required a procurement method which “allowed a long-term agreement with suppliers, contractors and providers of non-consulting services which sets out terms and conditions under which specific procurements can be made throughout the term of the agreement” (Restructuring Paper 2016). Second Restructuring, June 2019: During the first 2.5 years of implementation, the project made very slow progress due to high turnovers of Project Directors and PMU staff for the agriculture component. The project turned around and made good progress after the Mid-Term Review. Further, with the approval of a new policy for agriculture for the province, there was renewed commitment by the Government to expedite project activities. There was also a high demand for activities by the farmers, which led to a request by the Government for a one-year extension until June 30, 2020. Third Restructuring, May 2020: During the period of first extension, the project made further progress on activities. Disbursement rates in the past two and half years had improved from 29% in November 2017 to 70% in December 2019 (Restructuring Paper 2020). With the extension of project closing date by 11 months to May 31, 2021, project activities supporting crop and milk value chains of small and medium farmers were continued, which was critical in addressing the challenges faced by the rural people during the COVID-19 pandemic. 3. Relevance of Objectives Rationale Country and Sector Context. The agriculture sector’s contribution to GDP in Pakistan has declined considerably over time, but its role in poverty reduction and growth continues to be critical for the country. The sector employs about 40 percent of the labor force, generates nearly one-quarter of GDP with 60 percent of exports in value (WB 2019). The country’s top two producing provinces are Punjab and Sindh, which account for 81 percent of agriculture GDP (PAD, paragraph 4). Despite the Sindh province’s progress and contribution towards agriculture, 30-35 percent of the population live below the poverty line and most of them live in rural areas. While the major crops – rice, wheat, sugar cane and cotton - are produced by large landowners within a highly regulated market, the poor are mainly engaged in fruits and vegetables with limited attention from the Government. This has resulted into low productivity of the horticulture crops due to poor variety and quality of seed, traditional production practices with an absence of low-cost technologies, prevalence of diseases, presence of aflatoxin etc. (ICR, paragraph 4). In addition, the dairy value chain that comprises small and medium farmers, has faced challenges from poor animal husbandry practices, poor genetics, inadequate storage and processing facilities, lack of milk aggregation – factors that have contributed to low productivity and market access. Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK: Sindh Agricultural Growth Project (P128307) Government Strategy. According to the PAD (Paragraph 3), in 2011, Pakistan’s Constitution was amended which introduced devolution of agriculture services to the provinces. With limited capacity, the provinces had the mandate on policy development, its implementation and on delivery of agricultural services. This reform was an important basis for the Bank’s engagement and financing of a project at a provincial level. At closing, the project was well aligned with the key objectives of the Government of Sindh’s Agriculture Policy (2018-2030). The Policy plans to increase agricultural growth to 4-5% per annum, by focusing on both crops and livestock value chains through technological improvements in production, processing, marketing particularly of products such as fresh and processed fruits, vegetables and livestock whose demand has been on the rise in the country (Sindh Agriculture Policy 2018). Bank Strategy. During project appraisal and closing, relevant Bank strategies for the project were: (i) FY2010-2013 Pakistan Country Partnership Strategy; and (ii) FY 2015-2019 Pakistan Country Strategy (Extended to FY2020). Under the first strategy, agriculture would be addressed within the Pillar III of “Improving Infrastructure to Support Growth”, by supporting investments in irrigation and activities promoting increased productivity and value addition. The focus on agriculture continued to remain under the second strategy approved in 2014 and would contribute towards the Results Area 2 on “Private Sector Development” through the promotion of agricultural value chains. This focus was further emphasized in the Performance and Learning Review of the Country Partnership Strategy (FY15-20) (PLR 2020) which states the following: the Bank’s agriculture program will focus increasingly on creating markets for high value crops. In summary, the project’s objectives were well aligned with both the Government and World Bank strategies and addressed some of the challenges related to low productivity and limited market access faced by small and medium farmers in the Sindh region. Thus, the Relevance of this project’s objectives to Government and World Bank strategies in Pakistan at appraisal and at the project's closing were rated by this review as High. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To improve the productivity and market access of small and medium producers in key commodity value chains (Chilies, Onions, Dates). Rationale Theory of Change (ToC). A ToC was not required during appraisal; therefore, a retrospective ToC was prepared for the ICR (ICR, page 8). To achieve the objective, training and capacity needs assessment of Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK: Sindh Agricultural Growth Project (P128307) farmers would be done by organizing stakeholder forums for each of the value chains. Technical Assistance to the departments, the rehabilitation of agricultural research centers and the provision of research grants along with the setting up of an ICT-based extension system would lead to improved knowledge among farmers. Farmers would adopt improved agronomic practices for increased productivity and use low-cost technologies and equipment for reduced pre and post-harvest losses for improved market access. The ToC was found to provide a clear connection between project interventions and the expected outcomes. The ICR had included several assumptions and this review found the following key assumptions are the most relevant for achievement of this PDO: (i) Farmers use training on improved agricultural practices and adopt technologies; (ii) Technical investments improve productivity and/or reduce losses; (iii) Markets can absorb volumes and offer better prices for improved quality. Outputs As per the ICR (Annex 1.B), the project achieved the following key outputs:  Based on training assessment, crop-specific curricula and modules were developed and 80 lead trainers were trained  9,465 training days were organized reaching 111,071 farmers (male-112,360; female-13,420)  The first ICT Center networked with four local level sub-centers were established that enabled ICT focused extension system reaching 397,430 growers covering all commodities (rice, onion, chili and date)  21 ICT-enabled initiatives were delivered on the platform  Technical Advisories on improved agronomic practices and technologies were provided to 29,400 farmers  Farmer Call Center/24x7 Helpline was established reaching 187,937 growers  Village level farmer gatherings, workshops, training sessions reaching 396,530 farmers by streaming YouTube agricultural videos through Mobile Agricultural Cinemas  Range of equipment and materials tailored to each of the value chains were provided to 22,400 farmers (e.g., Tarpaulin sheets that facilitated drying of dates to reduce fungus and dirt contamination; electric chargeable spray for productivity improvement and pre-harvest spraying; rice threshers that reduced grain breakage etc.)  Five research stations and labs: Post-harvest Technology, Plant Pathology and Food Technology Laboratories at Horticulture Research Institute, Mirpurkhas; Mycology & Tissue Culture Laboratory, Plant Disease Institute and Soil and Water Testing Laboratory at Agriculture Research Institute, Tand Jam; Onion Research Station, Husri; and Chili Research Station, Umerkot were rehabilitated and renovated which offered various services (e.g., samples to detect diseases of fruits and vegetables, water and soil testing), to the farmers. Outcome To increase the productivity and market access in selected value chains, the following PDO indicators were measured:  Increase in yields for selected commodities - Onions, Dates, Chilies (production per hectare for crops) (Target: 20 percent; Actual: 20.5 percent) Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK: Sindh Agricultural Growth Project (P128307)  Percent increase in aggregate sales of selected commodities - Onions, Dates, Chilies (Target: 20 percent; Actual: 20.6 percent) The outcomes were measured based on an Impact Assessment done at the end of the project and regular monitoring reports prepared by the PMUs. The achievements were compared with the baseline survey carried out in mid-2017. The baseline survey was delayed but as no project activities were implemented on the ground until then, the survey was able to capture the situation prior to project interventions. The initial design of the project’s impact assessment was based on information to be collected from a control and treatment group. However, during implementation, it was found that project benefits had been spread to control group beneficiaries. Veterinary services, Artificial Insemination (AI), ICT-enabled extension was offered to all farmers in the province. Thus, as per the ICR (Page 12, footnote 4), “the project-end survey focused in collecting information on project’s achievements using a “before” and “after” project approach”. Based on the project-end survey, which was supplemented by using secondary data mainly for measuring yields (in the absence of a control group), the ICR reported the following results: i. Productivity: All selected commodities (onions, dates, chilies) experienced increased productivity in comparison to the baseline. The incremental increase in productivity was between 71-137% (Rice: 71%; Chili: 93%; Date: 96%; Onion: 137%) ii. Quality: Quality of commodities improved with the share of low-grade produce reduced (Chilies: from 20% to 13%; Onion: from 12% to 7%; Dates from 20% to 15%) and high-grade improved (exceeded the 20% project target for all commodities) iii. Technology Adoption: 70,000 HH’s (63% in comparison to a target of 50%) improved their crop productivity through adoption of technologies (e.g., improved agronomic practices, use of equipment and materials, ICT-enabled extension) iv. Pre- and post-harvest: 20% reduction in combined pre- and post-harvest loss for improved market access was observed v. Market access: Volume of commodities sold increased by 20.4% from 1.11 million tons to 1.34 million tons. At closing, the project surpassed most its targets for PDO and intermediate indicators. The evidence about achievements on outputs and outcome indicators reported by the ICR demonstrate that the project was successful in achieving Objective 1 to increase productivity and market access of key commodities (onions, dates, chilies). However, the project’s indicator on market access “increase in volumes sold” was found to be limited in scope. According to the ICR (Paragraph 33), the calculation used in measuring volumes of commodities sold was based on adjustments for household consumption data collected through the project- end survey. This review finds limitations with this methodology as it does not capture additional costs incurred by farmers to sell their commodities. Nonetheless, based on the successful achievements of most of the project’s outputs and outcome indicators, the efficacy with which Objective 1 was achieved is rated as Substantial. Rating Substantial Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK: Sindh Agricultural Growth Project (P128307) OBJECTIVE 2 Objective To improve the productivity and market access of small and medium producers in dairy value chains. Rationale Theory of Change (ToC): To achieve the objective of increasing milk productivity, the project would provide training on animal feed and nutrition, animal health management practices, breeding and genetic improvements etc. which would lead to adoption by farmers of good animal husbandry practices and technologies. Veterinary care centers would be strengthened which would offer animal health services (e.g., vaccination, disease prevention and treatment, Artificial Insemination). Alongside, market access would be improved by training on milk hygiene, quality, handling practices and through formation of milk producers’ groups (MPGs) for aggregating milk production for increased milk supply. The relevant assumptions outlined by the ToC that underpinned the achievement of the PDO were: (i) Farmers use the training for adoption of animal health and husbandry practices; (ii) Strengthened veterinary care centers and AI centers delivers services effectively; (iii) milk producers access health and AI services; and (iv) Milk producers see the value and form groups. Outputs  1,035 training days were organized reaching 6,800 milk producers (male- 3,965; female-2,835)  5,753 milk producers were organized into 153 Milk Producers’ group (MPGs)  Farmer workshops on Production, Collection, Processing and Marketing were organized for 420 milk producers  Training on production and processing for additional 104 milk producers  100 Veterinary Care Centers, 13 Livestock Production Offices and 6 Central Veterinary Diagnostic Laboratories sub-centers were rehabilitated and strengthened which were accessed by 6,800 milk producers and around 20,000 animals  354 milk technicians were trained to operate the collection and chilling centers  5,176 milk producers trained on milk hygiene and handling  Two Semen Production Centers were rehabilitated which distributed 37,000 semen doses (included 2,600 to Milk Producing group members)  401 AI technicians trained and 361 MPG members in AI techniques Outcome The PDO indicators measured for this objective were the following:  Increase in yields for milk (Target: 20 percent; Actual: 45 percent)  Percent increase in aggregate sales of milk (Target: 20 percent; Actual: 104 percent) Based on the baseline survey, pre-project yield for cows were 4.14 liters/day/animal and for buffaloes were 5.25 liters/day/animal which increased as follows at project-end: 5.78 liters/day (Cows); 7.12 liters/day/animal (Buffaloes). As a result, there was a 45 percent increase in yields for milk against the project target of 20 percent. The project-end survey found that nearly 50 percent of the members of the MPGs adopted good Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK: Sindh Agricultural Growth Project (P128307) animal husbandry practices, nearly 100 percent received veterinary care services, and 45 percent benefited from the Artificial Insemination (AI) of their animals for breed improvement (ICR, paragraph 39). On market access, the project measured and the ICR reported milk sale volumes resulting from an increase in production (ICR, paragraph 40). At project-end, 67 percent increase (weighted average) in sale of milk for both cows and buffaloes were observed. Following the adjustment made to household consumption, the milk that was sold increased by 104 percent (from 50,216 to 102,557 liters per day). The project measured the share of middlemen in marketing of milk which according to the project-end survey was reduced from 74 percent of total pre-project sale volume to about 30 percent at project-end. Simultaneously, there was an increase in milk sold directly by producers to MPGs; an increase to 63% from a baseline of 12% which was expected to lead to higher prices for producers (170% increase from baseline). One of the factors of higher prices received by the producers were due to the reduced role of middlemen in the milk value chain. The project met its PDO indicators, and the evidence provided in the ICR was found to be credible. Thus, the efficacy with which Objective 2 was achieved is rated as Substantial. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale In summary, the project met its targets for PDO indicators except for an under-achievement on the gender target related to beneficiaries reached (13.4% against the target of 20%). Total beneficiary coverage met was nearly 100 percent. Most of the intermediate indicator targets were also met. The incremental increase in productivity for crops were between 71-137% [Rice: 71% (Target: 95%); Chili: 93% (Target: 99%); Date: 96% (Target: 96%); Onion: 137% (Target: 106%)]. Milk productivity increased by 225 percent (Target: 20%). On market access, increase in sales volume for crops was 100% of the target, and for milk, it was five times of the project target. The evidence provided are found by this review to be sufficient. Therefore, the efficacy with which the PDO was achieved is rated as Substantial. Overall Efficacy Rating Substantial 5. Efficiency Ex Ante An economic and financial analysis was conducted during appraisal calculating the benefits of selected horticulture commodities (dates, onions, chilies, rice) and dairy. Crop models (PAD, Appendix 1) were prepared Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK: Sindh Agricultural Growth Project (P128307) “with” and “without” project interventions which estimated the benefits based on: (i) increase in yields; (ii) increase in gross revenue; (iii) Input and labor costs; and (iv) net income. With an expected rate of adoption of technologies by farmers at 50 percent for all crops, annual net economic benefit of project beneficiaries was estimated to increase by 27.5 percent. Over a period of 20 years and at a discount rate of 12 percent, the Economic Rate of Return (ERR) was estimated at 22 percent with a Net Present Value (NPV) of Rs. 4,600 million. For dairy, benefits were estimated based on an increase in yields from project activities mainly related to Artificial Insemination (AI), and Marketing through formation of Milk Producers Groups (MPGs). At a 12% discount rate over a 20-year period, the ERR for dairy was estimated to be 25.4 percent and a NPV of Rs. 686 million. Ex Post An ex post economic and financial analysis was conducted with actual project benefits and costs and followed the ex-ante methodology. Unlike the ex-ante EFA, the ex-post EFA did not separately conduct cost-benefits analysis for horticulture and dairy value chains. It conducted a single analysis aggregating the benefits of both the value chains against the project costs. The analysis used the data from the impact assessment study prepared at the end of the project. In a separate communication with IEG, the ICR team stated that the “EFA computed incremental (difference between with and without project scenario) gross benefits (without allocation of project costs) for each commodity based on the survey data, area cultivated, and cost of production metrics”. This resulted into a NPV for gross benefits for various crops as: (a) Chili – PKR 5,434.4 million; (b) Onions – PKR – 2,212.8 million; (c) Dates – PKR 1,463.4 million; (d) Rice – PKR 6,137.3 million; and (e) Dairy – PKR 812.8 million. The NPV was then calculated by deducting the total project costs from the commodity-aggregated incremental gross benefits. The NPV of the project was Rs. 11,200 million and the ERR was 36.4 percent. The financial results of the project yielded positive net income from both the horticulture and dairy value chains. Sensitivity analysis confirmed the robustness of the results, and the financial viability of the project. Administrative and Institutional Efficiency: The project was implemented over a 7- year period and included three restructurings. During the early stages of the project for about 2.5 years, there were delays in setting up the project management and implementation structures affecting timely recruitment of staff for managing project activities. In addition, the project and its activities were affected by two locust invasions in the country (2019 and 2020), and the COVID-19 pandemic’s restrictive measures that impacted farmers (ICR, paragraph 89). This led to a 23-month extension of the project’s original closing date which helped the project implement activities towards achieving the PDO. The extension did not experience any cost overruns. However, this review finds the project’s administrative cost of 12 percent to be high. In conclusion, the initial delays did impact timely implementation of project activities and the project’s administrative costs was high. However, the high ERR (36.4 percent in comparison to 25.4 percent estimated during appraisal) the project achieved at the end of the project, justify a rating of Substantial for Efficiency. Efficiency Rating Substantial Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK: Sindh Agricultural Growth Project (P128307) a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal  25.40  Not Applicable 0 ICR Estimate  36.40  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project’s objectives were well aligned with both the Government and World Bank strategies. The design of the project was simple, addressed key challenges faced by the value chains covered and therefore the Relevance of Objectives for this project was rated high. The project also met its PDO indicators and most of its intermediate indicators, and the evidence based on the impact assessments was credible which led to an Efficacy rating of Substantial. The project achieved a high ERR (36.4%) though this review notes that the administrative cost of 12% that the project incurred was high. In summary, this review concludes that the project’s overall outcome is Satisfactory. a. Outcome Rating Satisfactory 7. Risk to Development Outcome The ICR (Page 28-29) identified the following risks that could potentially impact the project's development outcome:  Farmers do not replace the equipment and materials provided by the project. During project implementation, adoption and use of the equipment and materials were high. Some of the materials (e.g., tarps, crates) were not very expensive and the farmers recognized its benefits. Therefore, the ICR considered this risk to be low.  Milk Producing Groups (MPGs) cease to operate after the project closes. The ICR sees this risk to be minimal. According to the ICR (Paragraph 122), the MPGs were making profits and had operational savings. About a third were involved in direct marketing and had opened sales outlet selling milk products as well as animal inputs. About 40 MPGs were registered as cooperatives.  Animal Breeding services do not utilize the strengthened semen production stations. This risk has been identified by the ICR to be low as well. Directorate of Animal Breeding has taken ownership of the semen production stations and its capacity has been strengthened by procuring 16 exotic bulls, among other activities. Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK: Sindh Agricultural Growth Project (P128307)  ICT-enabled agriculture extension services may not be maintained. The ICT-enabled system focused on many activities, such as advisory services, capacity building of farmers, extension, monitoring of field work etc. The ICR considered this risk to be high as the newly established extension system and its workers may not fully adopt the system and the services. Further, the equipment (e.g., laptops) provided by the project may either not get replaced or the purchase may be postponed which would impact the benefits generated from this system during project implementation. 8. Assessment of Bank Performance a. Quality-at-Entry The project was well aligned with the country and Sindh Province’s policy for agriculture which focuses in promoting agricultural growth based on technological improvements in horticulture and livestock value chains to increase productivity and market access. At the time of project preparation, the Bank had not been engaged in the agriculture sector of the country for some time and this was the first project financed by the Bank that focused on agricultural value chains. However, the design took lessons from World Bank financed projects in the horticulture sector in other countries in the South Asia region. The demand-driven approach of the project based on consultations and dialogues among stakeholders through structured stakeholder forums assessed the challenges faced by the farmers in the value chains, which helped design appropriate interventions. The project targeted the relevant producers – small and medium farmers - who were mainly engaged in the horticulture commodities selected by the project. The project design was found to be simple, and it had a clear PDO. However, the Results Framework (RF) was not ambitious enough. The project had implemented many activities through ICT-enabled agriculture extension services, but none of the activities had relevant indicators (ICR. para 77). Some of the indicators (See Section on M&E) were not clearly defined. Finally, according to the ICR (Paragraph 78), the two following risks identified during project preparation did not have adequate mitigation measures: (i) Limited Project Management Capacity; and (ii) Outdated information about producers’ demand on equipment and materials. The project faced considerable delays (2.5 years) during the initial phase of the project due to limited capacity of the Project Management Unit (PMU) resulting from timely recruitment of staff. In addition, the information about producers’ demand on equipment and materials were collected in 2011 but by the time (2017) the project was ready to implement the activity, the list was outdated and did not fully address the needs of the communities. This review concluded that Quality at entry was rated as Satisfactory with minor shortcomings. Quality-at-Entry Rating Satisfactory b. Quality of supervision Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK: Sindh Agricultural Growth Project (P128307) The project was closely supervised by the World Bank with a total of 15 missions. According to the ICR (Paragraph 114), mission documents adequately reported challenges faced by the project, particularly during the early phase when the Project Coordination Unit (PCU) and Project Management Unit (PMU) were not timely staffed that led to delayed implementation of project activities. As the project implementation progressed, Bank team were proactive in working with the Government in facilitating project implementation. Upon request of the Government, one key issue that the team addressed under the project’s first restructuring was the introduction of a new procurement method “Framework Agreement” which enabled the project to procure goods in suitable terms and simplified the procurement process for the project. The Framework Agreement was introduced in Pakistan for the first time under this project. Further, the Bank’s decision to approve the Government’s request for project-end date extensions were sound and justified, which helped the project complete its activities that contributed towards the development outcome. The project’s M&E system was useful in monitoring project activities. Despite the project’s delay in conducting a baseline survey, the information collected, and the impact assessment studies were found to be fully utilized. While project ratings over the period were largely realistic, the ICR (Paragraph 116) noted that the Bank team had “prematurely upgraded the project to Moderately Satisfactory for 1.5 years until the time of the Mid-Term Review”. At the MTR, taking into consideration the delays the project experienced during the early phase, the Bank team recommended for an 18-month extension of the project, which led to a renewed commitment from senior Government officials of the Ministry of Agriculture. To closely monitor project implementation, the ICR (Paragraph 82) pointed out that the Minister chaired monthly meetings on project progress, and the Secretary chaired bi-weekly meetings on project implementation. Further, in a separate communication with IEG, the ICR team provided the following additional information on factors that contributed to the accelerated progress on project implementation since the MTR stage: (i) Appointment of a seasoned Government official (Former Director General of Agriculture Extension) as the Project Director, who upon the request of the Bank, continued managing the project until closing; (ii) Improved information based on the completion of the baseline report and third party assessment which reported on project progress as well as on challenges that helped with decision making; (iii) Institutional arrangements for implementation was established by then with processes established for procurement and consulting contracts awarded; and (iv) Two extensions totaling 23 months led to flexibility which allowed the project to make up for the time lost due to the delays experienced during the initial phase of the project. The project’s turn-around after the Mid-term was due to both the Government’s renewed commitment as well as the proactivity and the collaborative effort from the Bank. Thus, the project’s Quality of Supervision is rated as Satisfactory. Based on the above assessment of Bank’s Quality at Entry and Quality of Supervision as Satisfactory, the Bank’s overall performance is rated Satisfactory. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK: Sindh Agricultural Growth Project (P128307) Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design A Theory of Change (ToC) was not included in the PAD as it was not required at the time of the project appraisal. Nevertheless, the ICR (Page 8) provided a retrospective ToC which presented the causal relation between the planned project interventions, its outputs, outcomes, and higher-level objectives. The project had a sound Results Framework which included three PDO level indicators, ten Component- wise intermediate-level indicators. As indicated earlier in the review, the RF could have been more ambitious and included additional indicators that captured some key project interventions such as the ICT- enabled extension services. In addition, as per the ICR (Annex 1) some of the indicators included (e.g., percentage of farmers achieving 20% reduction of on-farm pre- and post-harvest losses and wastage for selected commodities; percent of high grade produce of selected commodities by targeted beneficiaries) were difficult to compute or interpret. Nonetheless, the RF was relevant through implementation which did not require revisions and had reliable estimates on targets and ambition levels. Project’s M&E design was based on plans to conduct periodic impact assessments (2017, 2018 and 2019) which included a baseline survey and an additional final impact assessment planned at the closing of the project. The M&E system was to be managed by staff hired by the PMUs who produced quarterly progress reports on project interventions, and fiduciary reports, along with several commodity-specific studies. b. M&E Implementation Delays in establishing project management system during the early phase of the project led to interruptions in hiring M&E Specialists. The M&E Consulting Firm responsible for the baseline survey was recruited in 2017 and the baseline report was prepared in December 2017, by when half of the project duration had passed. As per the ICR (Paragraph 94), the delay with the baseline report did not impact the project. Until then, limited project activities had been implemented which allowed the baseline survey to capture the pre-project situation. However, the project did face some challenges with its monitoring during the COVID-19 Pandemic, and the locust infestation that impacted the region in 2020. During this period, the project was unable to update the implementation status and in the absence of a field mission, it was difficult for the Bank to correctly assess implementation progress. During the project period, all the impact assessments planned were completed. The data generated were found to be of good quality and the results of the final impact assessment was used for measuring the project’s impact at closing. The periodic impact assessments were also useful to determine during project implementation the limitation of the methodology i.e., control group respondents had become irrelevant due to the spillover of the benefits to large number of producers outside the project supported groups. Timely knowledge about this methodological gap allowed the project to plan a more suitable final impact assessment, which designed and implemented a survey using a “before” and “after” project approach closely linking it to the project ToC (ICR, para 96). Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK: Sindh Agricultural Growth Project (P128307) c. M&E Utilization As planned during design, the project’s RF was relevant without the need for to make any revisions made during implementation. Periodic progress reports were prepared by the PMUs and impact assessments were used for monitoring progress and assessing project’s outcomes. The project had planned for competitive research grants as well as special studies relevant for each of the value chains. This was only partially achieved with one feasibility study on chili completed. If some of these studies were timely prepared, there could have been opportunities during implementation for its utilization in the project. Nonetheless, several research grants produced useful knowledge which was expected to be beneficial for the subsequent agriculture project in Pakistan. In summary, M&E is rated as Substantial. The project’s Results Framework was sound, with appropriate indicators and targets set that were mostly achievable. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as Category B (partial assessment) under the World Bank safeguard policies. The safeguard policies on Environmental Assessment (OP/BP/GP 4.01), Natural Habitats (OP/BP/GP 4.04), and Pest Management (OP 4.09) were triggered, none were triggered on Social Safeguards. Environmental Safeguards. The PAD (Paragraph 184) stated that project investments were not expected to have significant adverse environmental and social impacts. Nonetheless, project activities particularly related to farm level agricultural practices would need to be carefully planned so that any reversible impacts can be avoided or mitigated. During project preparation, an Environmental and Social Management Framework (ESMF) and Integrated Pest Management Plan (IPMP) were prepared. The ICR (Paragraph 101) pointed out that during implementation, the project complied with safeguard requirements. However, there were recurring delays with staffing and in submission of reports. The project had hired dedicated Environment and Social Specialists for each of the Agriculture and Livestock PMUs. Each project intervention was screened based on a checklist and where relevant, Environmental and Social Management Plans (ESMPs) were prepared. In addition, field and monitoring visits used an environmental health and safety checklists that recorded compliances and non-compliances. Construction related temporary impacts (e.g., dust), drainage issues, stockpiling of surplus construction materials were observed, which were communicated with the contractors to ensure compliance. On Pest Management, based on the Integrated Pest Management Plan, trainings on awareness about pesticides and their safe handling, storage, nutrients management etc. were conducted to more than 100,000 farmers though a formal assessment was not undertaken. But field visits confirmed that safety measures such as PPEs (mask) were used for spraying and proper disposal was done of empty containers/bags of fertilizer/pesticides. Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK: Sindh Agricultural Growth Project (P128307) The Agriculture PMU had also established a Grievance Redress Mechanism which offered an interactive Voice Response System (IVR) that was available 24/7 to seek feedback or concerns from farmers about project activities, procurement etc. b. Fiduciary Compliance Financial Management and Procurement. The ICR noted that the project faced challenges related to the capacity on financial management. Hence, during project implementation, the project’s financial management was mostly rated as Moderately Satisfactory. Staffing was not adequate and though reporting requirements were met, they were delayed. The external audit conducted by the Department of Auditor General of Pakistan was performed by auditors with limited qualifications, and there were delays in addressing the recommendations of the audits. In a separate communication with IEG, the ICR team said that the Bank FM team put several measures during project implementation to help build the capacity of the PMU and Government counterparts, which were as follows: “(i) Conducted regular internal audits by an internal auditing firm which raised various issues pertaining to FM capacity for management to take appropriate action (The PMU did not take any actions to improve.); (ii) During implementation support and FM supervision missions, the PMU senior management was alerted to FM-related weaknesses; (iii) Bank arranged formal and informal training for project FMS staff; (iv) Deputation of FMS of the Livestock PMU to the Agriculture PMU on additional assignment to take care of the Agriculture PMU FM deficiencies; (v) Regularly highlighted the need for the Agriculture PMU, through the ISM Aide Memoires, to hire FMS. (The PMU continued to use their clerk for payment processing.); and (vi) To improve quality of FM operations, PMUs were advised to computerize their data. (This activity was started but could not be completed)” Procurement had similar issues with inadequate staffing. Due to lack of staff, procurement consultants had to be hired to support with procurement functions of the project. One positive development that took place during project implementation was when the procurement method was shifted towards the “Framework Agreement” which was more relevant to the requirements of the project on delivery of equipment and materials. Additionally, as per the ICR (Paragraph 11), in 2018 an innovation was introduced by a joint project team, PMU, and the World Bank’s Innovation Lab which developed a prototype of a blockchain platform. The PMU then hired Microsoft Pakistan to upgrade the prototype into a real time platform which became particularly useful during the COVID-19 pandemic for contact-less order and delivery of goods for the project. c. Unintended impacts (Positive or Negative) The dairy value chain achieved additional impacts which were not envisaged during project appraisal. Five Milk Producers’ Group (MPGs) established milk outlets selling loose milk and other milk-based products (e.g., yoghurt, bottled milk) with 50 percent financing from the project for equipment, the remaining 50 percent from their own savings. By project-end, 330 additional outlets were established. In addition to marketing outlets, several MPGs started selling inputs such as animal feed, fodder seed which has helped bring quality inputs closer to the farmers. Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK: Sindh Agricultural Growth Project (P128307) d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons Demand-driven projects should have flexible arrangements in delivery of activities. The project’s administrative document that described project activities was prepared in 2011. It listed 40 items related to equipment and materials demanded by the farmers based on consultations and studies. The project got approved in 2014, and with the delays, by the time the project was ready to implement the activity, only 10 items were relevant for the farmers. The project then faced challenges in revising the list due to the complex administrative procedures of the Government. In such circumstances, design and implementation of project activities should be flexible, particularly in demand-driven projects where interventions are based on farmers’ needs which can change due to the local circumstances. Formation of producer groups can be an important vehicle for market access. In the dairy value chain, formation of producer groups that were given the necessary training and technical support helped aggregate milk supply at the local level. With increase in quality milk supply, new opportunities came up to sell milk directly to large private buyers instead of relying on milk collectors. In the absence of middlemen, farmers received better prices and became more knowledgeable about market-related issues. Completion of baseline surveys should be a dated covenant in project’s legal agreements. The baseline survey for the project was only completed in late 2017, whereas the project was approved in 2014. Though this project’s activities were delayed and as a result, the baseline survey was still able to meet its objective of documenting pre-project baseline situation, it would have been better to include this important activity as part of a legal agreement of a project, thus giving it the needed attention to be completed on time. 13. Assessment Recommended? No Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK: Sindh Agricultural Growth Project (P128307) 14. Comments on Quality of ICR The ICR was a well-written document that provided good quality evidence on achievements based on the project’s M&E and the periodic impact assessments carried out by the project. The document adequately presented the end-project results by comparing them with baseline data and targets. The ToC made a clear causal link between project activities, outputs and PDO outcomes, and the Results Framework was adequately supplemented with details on outputs which were listed in Annex 1.B. The analysis would have benefited if definition of some of the indicators (e.g., Quality Measurements (A, B, C) and Pre- and Post-harvest loss etc.) were described which were not clear in the document but were provided by the ICR team to IEG in a separate communication. The report was candid and highlighted shortcomings about the project’s design and implementation in the relevant sections. Additional details (which was provided separately to IEG) on how the Bank helped support the weak financial management capacity of the PMU, would have been a useful example to include in assessing the Bank’s performance in supervision. Finally, the ICR was concise, and the lessons drawn were based on evidence and had broader applicability in design of demand-driven projects. Overall, the Quality of the ICR is rated Substantial since shortcomings were minor. a. Quality of ICR Rating Substantial Page 18 of 18