Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) Report Number: ICRR0021765 1. Project Data Project ID Project Name P120836 IN: Maharashtra Agric. Competitiveness Country Practice Area(Lead) India Agriculture and Food L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-48090 31-Dec-2016 83,694,431.60 Bank Approval Date Closing Date (Actual) 28-Sep-2010 31-Oct-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 100,000,000.00 0.00 Revised Commitment 88,903,239.04 0.00 Actual 83,694,431.60 0.00 Prepared by Reviewed by ICR Review Coordinator Group Hassan Wally J. W. van Holst Pellekaan Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives Page 1 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) The Project Development Objective (PDO) stated in the Project Appraisal Document (PAD, para 21) and the PDO in the Financing Agreement (p. 4) were identical and aimed to: "increase the productivity, profitability, and market access of the farming community in Maharashtra." For the purpose of this review, since the PDO includes three objectives they will be assessed separately in Section 4 as Objectives 1, 2 and 3 as follows : Objective 1: Increase Increase the productivity of the farming community in Maharashtra, Objective 2: Increase Increase the profitability of the farming community in Maharashtra. Objective 3: Increase Increase the market access of the farming community in Maharashtra. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components The project was supported by the following three components: 1. Intensification and Diversification of Market-led Production (appraisal cost: US$32.40 million, actual cost: US$26.32 million). This component took an integrated approach toward enabling farmers in the project command area to take advantage of rapidly changing consumer demand and marketing channels. This would include diversification of their farm production, introduction of enhanced productivity and post-harvest technologies and empowering farmers to better understand, adapt and actively be able to seek out and access new and alternative markets. This would be achieved by: (a) building on the existing ATMA agricultural extension program and making it more profit- and market-oriented; (b) establishing an Agri-Business Promotion Facility (ABPF) to promote producer-buyer supply linkages; (c) generating and disseminating accurate and timely market information, and (d) providing livestock support services for strengthening an important source of income for farmers. This component included four sub components: (1) Market- led agricultural technology transfer, (2) Agribusiness promotion facility, (3) Market information services, and (4) Livestock support services. Page 2 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) 2. Improving Farmer Access to Markets (appraisal cost: US$106.50 million, actual cost: US$77.09 million). This component aimed to improve farmer access to markets by establishing farmer associations and building community-level marketing capacities and infrastructure; providing access to warehouse receipt systems linked to commodity exchanges, establishing electronic trading platforms, improving market infrastructure and services of rural retail and traditional wholesale markets, and making market management more efficient and responsive to farmers’ needs. The activities to be financed would include: (a) mobilizing and building the capacity of community institutions, such as producer associations, and supporting them though grants to acquire capital assets; (b) modernizing and upgrading warehouses; (c) improving rural market infrastructure; (d) introducing of e-marketing platforms; and (e) upgrading wholesale markets. This component included two sub-components: (i) promoting alternative markets; and (ii) upgrading existing market infrastructure. 3. Project Management, Learning and Adjusting (appraisal cost: US$14.10 million, actual cost: US$13.80 million). This component would undertake project coordination and management as well as monitoring and evaluation services. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. The total cost of the project was estimated at appraisal to be US$153 million equivalent. The actual cost reported by the ICR Annex 3 was US$117.21 million which was about 77% of the appraisal estimate. The difference was mainly due to project savings of about US$16.00 million that resulted from the depreciation of the Indian Rupee combined with lower than expected borrower contribution (see below for more details). The ICR (p. 10, para 17) explained that "US$16.84 million out of the total exchange rate savings of US$27.7 million was unutilized largely due to the difficulty in obtaining one-time extension for a longer period, which would allow adequate planning and implementation, and delays in the Government’s decision making on utilization of savings." Financing. The project was financed through a Specific Investment Loan (SIL) in the amount of US$100 million equivalent on International Development Agency (IDA) credit terms. The actual disbursement amount reported by the ICR (p. 2) was US$83.69 million. This was US$16.31 million below the appraisal estimate. According to the ICR (p. 55, para 26): "the project had savings of 17% in IDA lending due to the depreciation of the Rupee." Borrower and Beneficiary Contributions. The Government of Maharashtra was expected to provide US$11.00 million of counterpart funds and US$42 million was expected to be mobilized by project beneficiaries, totaling US$53 million. The Page 3 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) actual amounts disbursed according to the ICR (p. 2) were US$9.30 million and US$24.22 million for the Government and beneficiaries, respectively. This totaled to US$33.52 million or 63% of the appraisal estimate. Dates. The project was approved on September 28, 2010 and became effective three months later on December 20, 2010. The Mid -Term Review was carried out on April 22, 2013, six months later than the date set at appraisal on October 15, 2012. The project closed on October 31, 2018, which was 22 months later than the expected closing date on December 31, 2016. According to the ICR (para 17) the extension of the closing date was "to complete key innovative activities and utilize exchange rate gains for scaling up successful interventions." The project was restructured three times, all of which were Level 2 restructurings as follows: 1. On June 26, 2014, when the amount disbursed was US$27.73 million, in order to modify the Results Framework. 2. On September 27, 2016, when the amount disbursed was US$51.14 million, in order to extend the original closing by 12 months from December 31, 2016 to January 2, 2018. 3. On July 26, 2017, when the amount disbursed was US$63.63 million, in order to extend the closing date by 10 months from January 2, 2018 to October 31, 2018. 3. Relevance of Objectives Rationale Maharashtra is the second largest state in India in terms of both area (30.7 million hectares) and population. About 58% of the state’s population lives in rural areas, most of whom are dependent on agriculture in some way for their livelihoods. The PAD (para 1) noted that the state had the largest concentration of poor in the country (32 million). Smallholder farming faced many challenges in Maharashtra including: severely limited choice in accessing markets and limited information on product quality demands. Smallholder farmers are also disadvantaged by significant (post-production) diseconomies of scale resulting in high costs of marketing and low profitability. This project would attempt to provide critical services and key infrastructure required by smallholder farmers to compete in the fast-moving agriculture markets of the future and was therefore relevant to the challenges faced by these farmers. Page 4 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) At project appraisal, the project objectives were in line with Government of Maharashtra's (GoM's) priorities for the agriculture sector as articulated in Maharashtra’s Agricultural Marketing Vision Statement. The GoM envisaged to implement a three-pronged approach to promote the development of alternative marketing options to farmers; support the top tier of regulated wholesale markets in the state to reform, invest and provide better services subsequent to which such markets were expected to compete on the basis of services rendered and value-addition; and gradually undertake further incremental regulatory reforms. Objectives were in line with the Government of India 11th Five Year Plan (2007 – 2012) which aimed to achieve an aggregate agriculture growth rate of 4% per annum. The plan featured some nascent risk management tools including: warehouse receipt financing and the use of commodity exchanges. Objectives were also in line with the first pillar of the Bank’s Country Assistance Strategy (CAS) for India (November 2008) which called for achieving rapid, inclusive growth by expanding agriculture productivity. In particular, the CAS emphasized the need to foster farmer-centered public extension systems as well as provide efficient and competitive markets. At project completion, objectives continued to be in line with the GoM priorities for the agriculture sector. Objectives were also in line with the Bank’s Country Partnership Framework for India (CPF, FY2018–22). The CPF (para 31) stated three relevant focus areas for the Bank's engagement: (i) promoting resource-efficient growth, (ii) enhancing competitiveness, and (iii) enabling job creation. These focus areas were aligned with the Government of India's priorities to end poverty and boost shared prosperity. The statement of objectives was clear and focused and relevant to Government and Bank development strategy for India. However, it lacked a strategy for achieving the government’s higher level objectives, namely improving the competitiveness of agriculture and achieving an agriculture growth rate of 4% per annum which would have provided an important context for this project. Based on the above-mentioned information Relevance of Objectives is rated Substantial. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL Page 5 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) OBJECTIVE 1 Objective Objective 1. Increase the productivity of the farming community in Maharashtra. Rationale Theory of Change. The theory of change is the relationship between activities, outputs from those activities and the final outcome (increased productivity of the farming community). The project aimed to intensify and diversify market-led production. This was to be achieved through strengthening extension services towards market-led production and improving the market intelligence services to smallholders. These activities were expected to increase the number of farmers adopting improved farming practices and technologies. This was expected to contribute to increased productivity of the farming community in Maharashtra. As a long-term outcome, increased productivity was expected to contribute to reduction in rural poverty and improving shared prosperity. According to the ICR (para 6) change would be facilitated by a number of critical elements including: (a) the project was not subject to climate change risks, (b) appraisal implementation arrangements were suitable for all APMCs, (c) release of funds by the state was on time, (d) the implementation time line was adequate for the project implementing reform-linked interventions, and (e) piecemeal approach to extension would support effective planning and implementation. The stated assumptions were general in nature and some were even beyond the control of the project. More specific assumptions could have been: interest and willingness of farmers to adopt new technology; and willingness of farmers to organize for cooperative action, both in production and output marketing, for example. Outputs The following information is from Annex 1 in the ICR unless referenced otherwise: • 9% increase in farmers adopting balanced use of fertilizers against a target of 12% and a baseline of none. • 36% increase in farmers adopting critical irrigation of crops against a target of 12% and a baseline of none. • 36.88% increase in area under critical irrigation against a target of 13% and a baseline of none. • 174 technologies were demonstrated in the project areas exceeding the target of 136. Page 6 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) Outcome The reported crop increases were a result of project interventions including promoting and demonstrating 174 technologies, and improving extension through service providers. As a result of these interventions, shares of farmers adopting critical irrigation increased to 36% against the target of 12%, and areas under critical irrigation reached 37% substantially exceeding the target of 13%. Also, 92% of farmers adopted one or more of the project-promoted technologies. Notably, the adoption of balanced use of fertilizers reached 77% among project beneficiaries. Outcomes from the adoption of improved technologies was assessed through increments in productivity (yield) of select focus crops including: maize, chickpeas, pigeon pea, soybean, onion, and tomato. The data reported in the ICR (Table 3) showed that chickpeas, maize, onions and tomatoes met or exceeded their end of project yield targets. While soybean and pigeon pea fell short of meeting their end of project target. That said, a Difference in Difference (DID) analysis (based on endline surveys by external M&E consultants) also showed that soybean (29%), chickpeas (25%), pigeon pea (22%), and onion (193%) all exceeded the 15% end of project average target for yield increase. However, the DID showed that maize yield was increased by only 9% which was below the 15% target. DID analysis for tomato was not possible due to absence of baseline value for the control group. The ICR (para 21) explained that the general yield trend for soybean, chickpea, and pigeon pea was toward a decrease in the state. This was attributed to two reasons: the deterioration of existing seed varieties, and severe droughts that affected crops in the spring season. Data reported in the same paragraph also showed that the project intervention succeeded to increase or halt drastic reductions in yields for soybean, chickpea, and pigeon pea for the project farmers when compared to the control groups. On the whole, the DID analysis of the endline survey results showed that the project achieved an average increase in crop productivity of 32% compared with a target of 15%. The project also contributed to increments in the productivity of goats through the adoption of improved animal husbandry practices, disease control, better feeding practices, improved bucks, deworming, and housing management practices to decrease mortality and increase kidding rates and live weight gain. As a result of project activities, the common interest group model (CIG) achieved a 1% increase in kidding rates and 20% reduction in mortality directly contributing to productivity increases. In the higher value goat model (HVG) model, the kidding rate increased by 25% and mortality rate decreased by 53%. Based on the above-mentioned information efficacy of the achievement of this objective is rated substantial. Page 7 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) Rating Substantial OBJECTIVE 2 Objective Increase the profitability of the farming community in Maharashtra. Rationale Theory of Change. The theory of change is the relationship between activities, outputs from those activities and the final outcome (increased profitability of the farming community). The project aimed to promote risk management through strengthening warehouses and promoting e-spot trading. The project also aimed to improve market access through improving market services, improving the efficiency of APMCs to be more responsive to farmers' needs, and strengthening farmers producer organizations. These activities were expected to result in an increase in the volume/value of agricultural produce traded and stored by farmers, increase in the volume/value of agricultural produce arriving to local markets, and increase in the volume/value of agricultural produce traded through farmer organizations. These activities were in turn expected to increase the profitability of the farming communities. However, the Results Framework lacked clear causal links between the intended activities and the expected outcome (increase profitability). Increased profitability was in the long term expected to contribute to reducing rural poverty and improving shared prosperity. Outputs The following information is from Annex 1 in the ICR unless referenced otherwise: • 73% of farmers adopted sorting and grading practices compared to a target of 13% and a baseline of none. • 1,424 farmers and agro-entrepreneurs received loans compared to a target of 1,940. • 30,000 metric tons for grains compared to a target of 50,000, and 13,278 metric tons of fruits and vegetables compared to a target of 200,000 metric tons, were marketed through Farmer Common Service Center (overlap with PDO: 3). • 43,370 metric ton per year of agricultural produce stored under warehouse receipts by farmers compared to a target of 100,603 metric ton (overlap with PDO: 3). • 11,500 demonstrations, training, and exposure visits were organized on improved marketing strategies, post-harvest value addition, and quality improvement. • 104 market forecasts for key commodities with accuracy rate of projected prices above 86% were prepared and disseminated. Page 8 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) • 178 buyer-seller meets (BSM) and 48,027 public-private partnership (PPP) demonstrations were organized with participation of 25 leading inputs suppliers and agricultural produce procurement companies. Outcome The ICR (para 24) reported that the Results Framework lacked a PDO indicator to measure profitability. The ICR also stated that the "project systematically monitored profitability." However, there was no evidence in the ICR on how profitability was monitored despite the absence of relevant outcome or intermediate outcome indicators. According to the ICR (para 24) and based on multiple thematic surveys, the project-supported intensification and diversification interventions increased farm profits by 45% on average for all crops. Also, aggregated-marketing through Farmer Producer Organizations reduced input costs by 7% and increased price realization by 22% and increased profitability of marketed shares of produce by 45%. Sorting and grading practices promoted by the projected and adopted by producers increased farm profits by 15% on average for all crops, vegetables and fruits. Profitability was also increased through establishing direct market links that resulted in increased net price realization by 38% and reduced post-harvest losses and wastes by 76% (ICR, para 18). The project achieved these results through establishing and strengthening Farmer Producer Organizations with about 15,600 members to operate and manage 406 Farmer Common Service Centers (FCSCs) for aggregated input purchase and produce marketing. The price forecasts also guided production choices for 82% of farmers, marketing choices for 59%, and timing of sales for 49%. While external thematic studies and project impact surveys pointed to potential success of the project in decreasing input costs and hence improving farmer profitability through availing market information to leading to improved market access, the ICR lacked any direct evidence of an increase in the profitability of the farming community in Maharashtra. Therefore, efficacy of the achievement of this objective is rated Modest. Page 9 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) Rating Modest OBJECTIVE 3 Objective (c) increase the market access of the farming community in Maharashtra. Rationale Theory of Change. The theory of change is the relationship between activities, outputs from those activities and the final outcome (increased access to markets).The project aimed to improve market access and strengthen supply chains. The project would support and modernize local markets known as rural haats, improve the efficiency and responsiveness of APMCs to farmers' needs, and strengthen farmers' organizations. These activities were expected to increase the value and volume of sales through rural haats, APMCs and farmers organizations. These activities combined were expected to increase access to markets. Improved market access combined with improved profitability were expected to contribute to reducing rural poverty and improving shared prosperity. Outputs The following information is from Annex 1 in the ICR unless referenced otherwise: • As the ICR (para 30) notes: "the project formed and strengthened FPOs with over 160,000 members to operate and manage 406 Farmer Producer Organizations (FCSCs) for aggregated input purchases and produce marketing. At closing, 327 FPOs availed direct marketing license and 203 FPOs obtained agriculture input marketing license." • 43,804 metric ton of agricultural produce worth Rs 99.05 crore per year were marketed through FCSCs against a target of 250,000 metric ton (grains and fruits and vegetables) and Rs 270 crore, respectively. • 23.90 metric ton per Rural Haat (RH) per week of agricultural produce worth Rs 11 lakh arrived in RH against a target of 23.56 metric ton and a baseline of 18.47 metric ton and Rs 8 lakh per RH per week and a baseline of Rs 6 lakh per RH per week. • 20,735 metric ton per APMC per month of agricultural produce worth Rs 52.67 crore were marketed through APMC against a target of 14,476 metric ton and Rs 28.7 crore per APMC per month. Page 10 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) • 43,370 metric ton per year of agricultural produce stored under warehouse receipts by farmers against a target of 100,603 metric ton per year. Storage targets were unmet because the Government reserved warehouse spaces for the state-procured grains and pulses to control prices in the aftermath of the 2016 drought (ICR, para 31). • The project modernized 472 RHs against a target of 350 (ICR, para 32). • The project successfully introduced Standardized Accounting System (SAS) to all 306 APMCs in the state. At closing, SAS was adopted by 296 APMCs, of which 150 APMCs had their new accounting books audited at closing (ICR, p. para 34). This improved accuracy of financial reports, and eliminated rooms for imposing ineligible market fees by agents. • 138 APMCs had market information displays against a target of 219. • 181 farmers used electronic spot market in the Maharashtra State Warehousing Corporation against a target of 1,120. The ICR (para 31) reported that "this was discontinued due to lack of liquidity in the operation." • 1,324 metric ton per year of agricultural produce was traded by farmers using e-spot market in in the Maharashtra State Warehousing Corporation against a target of 2,240. • The project upgraded 112 warehouses and facilitated commercial banks’ participation in the NWR program (ICR, para 31). • The project modernized market infrastructure for 81 APMCs against the target of 100 APMCs, and added 81,245 MT warehouse space, 76,638 m2 of covered auction areas (ICR, para 36). The appraisal target was not met due to severe droughts that reduced many APMCs’ ability to meet the project’s eligibility criteria in terms of arrival volumes and financial contributions, some APMCs withdrew from the project due to unsuitability of the implementation arrangements, and the implementation was delayed due to restriction of water use for construction purposes during drought years (ICR, p. 16, para 36). • The project modernized 20 livestock markets and 4 small ruminant markets (ICR, para 37). Outcome The project sought to increase market access through: (a) the promotion of alternative marketing options, and (b) through improving the market infrastructure and services of regulated wholesale markets. (a) The promotion of alternative marketing options. Market reforms were carried out as part of the project appraisal as well as reform-linked activities implemented under the project (ICR, para 28). Pre-project reforms included: removal of distance restrictions for private markets and granting Farmer Producer Organizations (FPOs) to establish and operate their own markets without making extra payments to the regulated markets. The reform-linked activities implemented under the project included: direct marketing arrangements through FPOs, contract farming arrangements between Common Interest Groups (CIGs)/FPOs and private sector, Negotiable Warehouse Receipt (NWR) financing, e-trading pilots, and improving market infrastructure and services of RHs. Page 11 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) According to the ICR (para 29) these reforms benefited private markets, direct marketing arrangements, and contract farming. By project completion, there were 51 operational private markets with more than 600 collection centers spread across the state with a turnover of US$486.8 million in FY2017–18, which represented 11% of the state’s turnover. Prior to the project, private markets were non-existent in Maharashtra. Alternative marketing channels were also found to increase the average share of farmers in retail price by 4.7% against the target of 5%. The ICR (para 29) reported that an impact study found that alternative market options led to 4.4% higher net price realization in private markets for major crops compared to traditional market outlets. The study also found that farmers were empowered to negotiate higher net price realization, accessing more efficient market transaction processes. This also forced the traditional regulated markets to improve their business processes and reduce market fees in response to increasing competition. This also led to significant reduction in marketing costs, losses, and wastes. However, the ICR did not provide evidence on the afore mentioned improvements. (b) Modernization of wholesale market and livestock market infrastructure. According to the impact survey, improved market infrastructure coupled with improved efficiency and transparency of market transaction processes were key driving factors that resulted in increased volumes and sales. According to the ICR (para 36) the improved efficiency and responsiveness in regulated wholesale markets resulted in 5% increase in the average share of farmers in wholesale price against the target of 5%. Also, respective sale targets for volume and value were met or exceeded for RHs and APMCs, but not for Farmer Producer Organizations. The impact study found that the numbers of sellers and buyers in RHs increased by 33% and 93% of respondents were satisfied with the with the project investments (ICR, para 32). The study also found that 94% of respondents, when asked about APMCs, reported substantial improvements in handling practices, timing of transactions, and availability of market information (ICR, para 36). The ICR (para 30) explained that the targets for trade volumes and values for Farmer Producer Organizations (FPOs) were not achieved due to a combination of factors: first, the unavailability of qualified service providers resulted in implementation delays, second, the state experienced severe droughts that negatively impacted crop production, and third, the appraisal targets for trade volumes and values through FPOs were unrealistic given that the irrigation coverage in the state is the lowest in the India. According to the impact survey, between 2010 and 2014 the number of animals sold in the project livestock and small livestock markets increased drastically by 1,321% against 70% increase in the control livestock and small livestock markets (ICR, para 37). However, in the following years the number of traded animals declined by 95% in the project-supported markets as a result of the Government policy that banned sales of cattle for slaughtering. The survey also found that 90% of survey respondents were satisfied with the project investments in livestock markets. The respondents noted that the project investments led to a reduction in transportation costs, improved animal disease surveillance and simplified the verification processes at road checkpoints. Also, according to the ICR (para 37) improved infrastructure reduced frequencies of thefts. However, the ICR did not provide the statistics to compare thefts before and after the project investments. Page 12 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) Based on the above-mentioned information, it is evident that the project succeeded in increasing market access despite falling short on some targets. Therefore, the efficacy of the achievement of this objective is rated substantial. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The evidence provided points to the success of the project in increasing productivity of key crops and livestock and hence the efficacy of the achievement of Objective 1 was rated substantial. This review concluded that the evidence in the ICR did not show that the project resulted in an increase in the profitability of the farming community in Maharashtra and hence the efficacy of the achievement of Objective 2 was rated modest. Finally, evidence from the project succeeded in showing that increments in market volume and value of produce sold had increased as a result of increased market access of the farming community and hence the efficacy of the achievement of Objective 3 was rated substantial. Overall, therefore the efficacy of the achievements of this project is rated substantial. Overall Efficacy Rating Substantial 5. Efficiency Economic and Financial Efficiency ex ante Page 13 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) • The economic and financial analysis (EFA) at appraisal estimated the overall project’s economic rate of return (ERR) at 19.4%. Project costs and benefits were estimated in 2010 prices for a period of 20 years and at 12% opportunity cost of capital. • Returns were projected to be derived from: intensification and diversification interventions that were expected to increase crop productivity by 15% (net), crop diversification by 10%, and cropping intensity by 10%; and from the development of alternate markets and modernization of wholesale market infrastructure that were expected to increase average shares of farmers in wholesale and retail prices. ex post • The ex post EFA in the ICR was estimated following the appraisal methodology but at 2018 prices. The overall project ERR was estimated at 18.8%, which was slightly lower than the appraisal ERR of 19.4%, but considerably higher than the 12% opportunity cost of capital. The benefits were estimated for 20 years, exclusive of project implementation period, which correspond to technical life of modernized market infrastructures, if adequate management, operation, and maintenance were provided. • The EFA included analysis of market-led production, alternative marketing options, modernization of existing markets, and a combination of market-led production and improved market access. • Despite substantial implementation delays (22 months in total), the project achieved an economic rate of return comparable to the appraisal estimate. This was possible due scaling-up of diversification and intensification activities, better results generated by Agriculture Produce Marketing Committee markets that benefited from modernized infrastructure, higher returns to goat interventions, and spending 17% lesser IDA credits owing to exchange rate gains (ICR, para 40) • The ex post EFA could have benefited from the inclusion of a sensitivity analysis to assess the impact of different factors on the efficiency of the project. Administrative and Institutional Efficiency The project closing date was extended by 22 months. Implementation delays resulted from a combination of design, administrative, and external factors. The Farmer Producer Organization (FPO) formation and strengthening was delayed due to unavailability of qualified Service Providers and droughts. Modernization of Agriculture Produce Marketing Committee (APMC) markets was delayed due to a combination of droughts and unsuitability of the appraisal and/or implementation arrangements. Livestock activities were delayed due to limited implementation capacity, unavailability of breeding bucks in the market, and delayed decision making by the Animal Husbandry Department. By project completion, US$16.84 million out of the total exchange rate savings of US$27.7 million were unutilized. According to the ICR (para 17) this was mainly due to the difficulty in obtaining Page 14 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) a one-time extension for a longer period, which would allow adequate planning and implementation, and delays in the Government’s decision making on utilization of savings Overall, efficiency is rated substantial despite implementation delays and a slightly lower ex-post ERR. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re- estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 100.00 Appraisal  19.40  Not Applicable 100.00 ICR Estimate  18.60  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of Objectives is rated Substantial. The overall Efficacy of the project is also rated Substantial. The basis for this rating is that evidence provided in the ICR points to first, the evident success of the project in increasing the productivity of key crops and livestock production and thereby increasing the productivity of the farming community in Maharashtra – substantial efficacy. Second, external thematic studies and project impact surveys pointed to potential success of the project in decreasing input costs and hence improving farmer profitability through availing market information to farmers leading to improved market access, but without any direct evidence of an increase in the profitability of the farming community in Maharashtra – modest efficacy. Third, the project succeeded in increasing market access as reflected by the increments in market volume and value of crop and livestock sales achieved by farmers and thereby increased the market access of the farming communities in Maharashtra – substantial efficacy. The efficiency with which the project was implemented is rated Substantial despite implementation delays. Page 15 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) Based on a Substantial rating for all three evaluation parameters (Relevance of Objectives, Efficacy and Efficiency), this review concluded that there were minor shortcomings in the project's achievements. The project's Overall Outcome is therefore rated Satisfactory. a. Outcome Rating Satisfactory 7. Risk to Development Outcome The ICR provided a brief discussion about the risk to development outcome referring only to the performance of FPOs as a potential concern. The ICR (para 75) reported that the FPOs formed at phase three were weak at project completion. Post- completion, these FPOs would continue to need support due to their limited capacity. According to the ICR (para 75) this represented a moderate risk to the development outcome. The following points are emphasized by IEG as areas of concern to the development outcome: • Weather/Climate related events. The project experienced severe droughts three times over the project implementation period. These resulted in delayed implementation of multiple activities, including modernization of market infrastructure. Droughts also could result in water shortages and negatively impact productivity and crop yields in project areas. • Availability of quality service providers. The unavailability of qualified service providers contributed to implementation delays. It remains unclear whether there are enough qualified service providers to support farmers' groups in future. 8. Assessment of Bank Performance a.Quality-at-Entry • This was a stand-alone project designed for a progressive state like Maharashtra with strong commitment for reforming agricultural extension services toward market-led production focus and improved agricultural marketing system (ICR, para 73). • The project design was in line with development priorities of the Government of Maharashtra and the World Bank. It benefited from the Bank’s analytical study that examined institutional, technical, and policy constraints for efficient agriculture marketing system at national and state levels (ICR, para 73). Page 16 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) • The Bank worked with the state government at the appraisal stage to create an enabling policy and regulatory environment for smooth implementation of the project (ICR, p. 25, para 73). • The design was based on a phased implementation approach to build on the lessons learned from earlier experiences of other Bank-funded projects in India, such as National Agricultural Technology Project (NATP), Uttar Pradesh Diversified Agricultural Support Project (UP DASP), Assam Agricultural Competitiveness Project (AACP). Notable lessons reflected in the design included: consolidating the Agricultural Technology Management Agency (ATMA) institutions and further strengthening these advisory services by making them more market-oriented, improving the infrastructure of both traditional and alternative markets to increase farm sale, and adopting a holistic approach to increasing farmer incomes by supporting agriculture productivity and improving market access, information and investments (PAD, p. 14). • Design suffered from an unrealistic implementation time-line given that the project was designed to implement reform-linked interventions. For example, qualified service providers to support the FPO interventions were not readily available in the market which delayed implementation and affected quality. Also, additional implementation time was required because the project was designed to follow the phased approach for testing and learning on innovative interventions. • Eleven potential risks to the project implementation and development objectives were identified at appraisal (PAD, pages 19 and 20). Ten of these had a moderate rating and one was rated low. The PAD included appropriate mitigation measures except for financial management, where the project faced constraints due to delayed release of funds in the initial three years of implementation. Also, potential external factors such as a climate change events, absence of quality service providers were overlooked as potential risks at the appraisal stage. • M&E design suffered from weaknesses with regards to the Results Framework, which could have benefited from better choice of indicators to allow accurate and effective measuring of all results and outcomes (see Section 9 for more details). Overall, Quality at Entry is rated Moderately Satisfactory due to design shortcomings. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision According to the ICR (para 74) the "Bank’s supervision was timely." The implementation benefited from frequent visits by the task team, some of whom were country-based, which allowed close and frequent implementation support. The supervision also ensured that the technical team included adequate expertise that supported the project management. Close Page 17 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) supervision and monitoring of activities by the supervision team allowed timely identification of implementation constraints and adequate measures were recommended to the Government. The ICR (para74) also noted that supervision effectively monitored the compliance with the Bank’s fiduciary and safeguards policies and necessary actions were taken on time to address any concerns. The task team leadership was said to have been effective in facilitating knowledge sharing through careful selection of suitable projects and countries with relevant experiences and best practices. While, the task team were proactive in dealing with the PDO indicators early on which ensured more accurate reporting, no relevant outcome indicators were assigned to assess profitability. Based on the above-mentioned information, Quality of Supervision is rated Satisfactory. Overall, Bank Performance is rated Satisfactory despite some design weaknesses and not fully addressing M&E weaknesses. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The PAD did not include a theory of change, which was not required at the time of appraisal. Nonetheless, the ICR included a theory of change that reflected well the relation between the project activities, outputs, outcomes and long-term outcomes. According to the PAD (paras 40 and 41) the PCU would have the operational responsibility for planning and coordinating M&E activities for the entire project with the support of the implementing departments. A dedicated M&E consultant based within the PCU would monitor and evaluate project processes, approaches, institutions and quantify short term outputs and outcomes. In addition, an M&E agency would be recruited to support the PCU and carry out a baseline survey (to validate the base line values adopted to date for the core project indicators as well as for supplemental indicators of interest to GoM) and also evaluate the project’s outcomes and impacts at mid-term and prior to project completion. The project had three objectives. The first objective (productivity) was to be assessed through changes in crop yields, the second objective (profitability) had no outcome indicator(s) to directly assess it, and the third objective (market access) would be assessed through two outcome indicators: the share of wholesale and retail price received by the farmer, and compliance with business standards by office of Director of Marketing. While two of the three outcome indicators were directly linked to the stated objectives, they were designed as an "aggregation of multiple sub-indicators and proxies making monitoring complex" (ICR, para 65). The third outcome indicator (compliance with business standards by the office of Director of Page 18 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) Marketing) did not have a clear link to the third objective and was later deleted. A notable shortcoming in the design of the Results Framework (RF) was the absence of any indicators to assess profitability (second objective). With only four intermediate outcome indicators, the RF lacked enough indicators to fully capture the impact of the project-supported activities. Also, this review finds that targets set at 5% for the increment in the share of wholesale and retail price received by the farmer was low, while those for trade volume and value for Farmer Producer Organizations were unrealistic given that the state has the lowest irrigation coverage in India. Overall, M&E design was complex and deficient. b. M&E Implementation The RF was revised to facilitate a more accurate and effective assessment of outcomes. However, the profitability outcome continued to lack any assigned indicators. According to the ICR (para 66) all surveys (baseline, MTR, and endline surveys) were conducted on time, except for the baseline for phase 1 activities. Survey analysis accuracy was underlined through using systematic monitoring and applying Difference in Difference (DID) technique combined with adequate selection of control groups to validate attribution. According to the ICR (para 66) "M&E implementation was satisfactory overall." While this review largely agrees with the assessment of the ICR, the failure to assign a relevant indicator(s) to track profitability, despite the revision of the RF during the 2014 restructuring, was a significant shortcoming. c. M&E Utilization The physical and financial reports prepared were timely, and supported informed decision making. The project also conducted various thematic studies to assess the changes brought by the project and also to support the Government’s agriculture marketing reform agenda. According to the ICR (para 67) the project's progress, results, and outcomes were all adequately monitored, and the "M&E data guided restructuring, utilization of exchange rate savings, and selection of successful interventions for scaling-up." Overall, M&E Quality is rated Substantial. There were design shortcomings and a failure to assign relevant indicators to track profitability despite the revision of the RF during the 2104 restructuring. But the incorporation of impact evaluation techniques through the DiD made assessment rigorous and revealing, thus improving the ability to measure the project's achievements. M&E Quality Rating Substantial Page 19 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) 10. Other Issues a. Safeguards The project was classified as a category B. The following safeguard policies were triggered at appraisal: Environmental Assessment (OP/BP/GP 4.01), Pest Management (OP 4.09), Involuntary Resettlement (OP/BP 4.12), and Indigenous Peoples (OP/BP 4.10). An Environment and Social Management Framework was prepared to ensure that appropriate environmental and social mitigation measures and processes were incorporated into the project design, planning and implementation stages. Also, an Indigenous Peoples’ Development Plan and a Resettlement Policy Framework were prepared. The project would focus on upgrading agricultural marketing infrastructure and improving crop productivity and diversification. While these activities were expected to generate solid waste, potential adverse environmental impacts on natural resource use and management should to be limited. The PAD (para 76) stated that the project was "not financing any direct procurement of pesticides and other agro-chemicals, except for some demonstrations of better farm practices." However, farmers were expected to increase land-based productivity through application of fertilizers, pesticides and other agro- chemicals for increasing crop yields and while diversifying cropping pattern and intensity. Environmental safeguards. According to the ICR (para 69) "environmental safeguards performance was satisfactory overall." The Environmental and Social Management Framework (ESMF) included mitigation actions that were adequately implemented. Implementation of environmental safeguards benefitted from a qualified environmental expert who monitored and reported on environmental safeguards compliance. The project supported sustainable agricultural practices, such as Integrated Pest and Nutrient Management (IPNM) and management of organic wastes in markets. The project increased awareness on Climate Smart Agriculture (CSA) through the introduction of CSA practices for 12 crops. Social safeguards. According to the ICR (para 70) "the social safeguard measures in the project were satisfactory overall." Indigenous Peoples’ Development Plans were implemented in three districts to promote integrated organic farming with the support of the Agricultural Technology Management Agency (ATMA). The project-supported weekly markets were mostly built on village council lands, and in some cases lands were donated by private individuals or taken on long-term lease, for which due processes were followed. b. Fiduciary Compliance Page 20 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) Financial Management (FM). The project included 80 accounting units to be able to cover 33 districts that included a wide variety of interventions and stakeholders. According to the ICR (para 71) "FM arrangements were responsive to implementation requirements" and "risk was mitigated through adequate measures." However, the lack of flexibility and understanding resulted in restrictions in the availability and timely release of funds. Submission of interim unaudited financial reports was regular and timely. Audits of the funds and expenditures of the project were carried out by the Accountant General of Maharashtra. The project management team's response on audit findings was "reasonable, appropriate, and timely" (ICR, para 71). Procurement. Procurement benefited from the recruitment of qualified procurement specialists and the provision of timely guidance and extensive training and capacity-building activities for the project's procurement staff. The procurement process was completed within the bid/proposal validity period, and awarded contracts were uploaded on the project website. According to the ICR (para 72) "the response and compliance of post procurement review findings were satisfactory, reasonable, and timely." c. Unintended impacts (Positive or Negative) Recognition. In 2016, the National Institution for Transforming India ranked Maharashtra first in implementation of most of the agricultural marketing reforms and for offering the best environment for doing agribusiness (ICR, para 48). Mainstreaming environmentally sustainable agricultural practices. The project supported 6,000 demonstrations of Integrated Pest and Nutrient Management (IPNM). This led to adoption, inclusion, and over 80,000 demonstrations of the practice through its integration into the Technology Kit of ATMA. Also, waste management at APMCs was transformed from a collection-disposal model to recycle-use model by producing vermicompost. This activity resulted in hygienic conditions in APMCs, reduced burden on municipal landfills, promoted use of organic inputs by farmers and created a revenue stream (ICR, para 53). d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Page 21 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) Bank Performance Satisfactory Satisfactory The inclusion of quasi-experimental methods added rigor to a flawed M&E Quality of M&E Modest Substantial design and enabled effective illustration of the project's achievements. Quality of ICR --- Substantial 12. Lessons The ICR included four lessons. The following two are emphasized with some adaptation of language: • Market-led crop diversification that is partnered with the private sector ensures sustainability beyond the project. At closing, a majority of the contract arrangements that were initiated with the project facilitation remained operational. According to the private sector entities, successful partnership between the private sector and Farmer Producer Organizations is more likely to be sustained beyond the project when partnerships are based on mutual trust, financial incentives, and when maturity of the participating organizations are established. • To ensure effective implementation, a tailored and flexible implementation time frame needs to be considered for projects that are designed for implementation of reform-linked interventions. As the project experience showed, such projects need additional time for testing and learning processes and may face difficulties with availability of qualified Service Providers. Along these lines, the project experience also showed that the piecemeal approach to extension does not support effective planning and implementation. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provided good coverage of the project activities and candidly reported on shortcomings. Discussion of outcomes was logical and relied on the achievements of the project. The ICR relied on proxy data and external information to report on profitability due to the lack of relevant profitability indicators in the M&E design. The discussion of the risk to development outcome was brief and could have benefited from further coverage of other risks that could potentially impact the outcomes, such as weather related events and availability of service providers. Also, the project data in Annex 1 were at times confusing reflecting Page 22 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Maharashtra Agric. Competitiveness (P120836) an overall complex M&E design. For example, the data reported on market access and sales volumes could have benefited from better presentation. Overall, this Review rates the Quality of the ICR as Substantial. a. Quality of ICR Rating Substantial Page 23 of 23