Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) Report Number: ICRR0022774 1. Project Data Project ID Project Name P132418 Efficient & Sustainable City Bus Service Country Practice Area(Lead) India Transport L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-18577 31-Dec-2018 9,149,054.88 Bank Approval Date Closing Date (Actual) 09-Dec-2014 30-Dec-2020 IBRD/IDA (USD) Grants (USD) Original Commitment 9,200,000.00 9,200,000.00 Revised Commitment 9,149,054.88 9,149,054.88 Actual 9,149,054.88 9,149,054.88 Prepared by Reviewed by ICR Review Coordinator Group William B. Denning Peter Nigel Freeman Victoria Alexeeva IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) and the Global Environment Objective (GEO) were identical: i.e., to improve the efficiency and attractiveness of city bus transport and reduce greenhouse gas emissions in the demonstration cities (Project Appraisal Document [PAD], page 9, and Grant Agreement, Schedule 1). Page 1 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) This review assesses the PDO in three parts: (1) improvements to the efficiency of city bus transport; (2) improvements to the attractiveness of city bus transport; and (3) the reduction of greenhouse gas emissions in the demonstration cities. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 02-Nov-2018 c. Will a split evaluation be undertaken? No d. Components There were three main components to the Efficient and Sustainable City Bus Service (ESCBS) project. There was also provision for a Project Management Unit (PMU). Component 1: National Capacity Building for Urban Bus Sector: Estimated cost: US$3.27 million (PAD par 40, but shown as US$3.5 million in the ICR Annex 3). Actual cost:US$2.37 million (ICR Annex 3). This component was to review policy, regulatory and fiscal constraints; at national, state and city levels, and to promote efficient and high-quality city bus services. It was to develop policy notes for discussion and debate among key stakeholders on areas of reform. The component included capacity building initiatives involving the development of knowledge materials, training activities, knowledge sharing and cross learning events, and dissemination of best practices in cutting edge areas aimed at the development of the overall urban bus sector in the country (PAD par 35). Component 2A: City Demonstration Projects: Physical Improvements Estimated cost: US$105.17 million (PAD para 40). Actual cost: US$77.90 million (ICR Annex 3). This component supports physical improvements targeted at modernizing the city bus services in demonstration cities including: (i) modern buses, (ii) modern depot equipment for improved maintenance and extended life of buses, and (iii) modern Intelligent Transport Systems (ITS) and Management Information Systems (MIS) - to make the services more user friendly and for improved planning and management of operations to enable optimal use of resources. Subcomponent (i) bus purchases was fully funded by counterpart financing (PAD par 36, 40). Component 2B: City Demonstration Projects: Technical Assistance and Capacity Building Estimated cost: US$2.88 million (PAD par 40). Actual cost: US$1.71 million (ICR Annex 3). This capacity building and technical assistance component supported the modernization efforts in selected demonstration cities. This included: (i) institutional strengthening, capacity building, and training; (ii) business planning including route planning and rationalization; (iii) marketing and branding; (iv) technical support with private sector participation including mainstreaming of the informal sector; (v) technical support with information technology management/Management Information Systems (ITS/MIS); (vi) vehicle and driver performance Page 2 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) management with a view to improving fuel efficiency; and (vii) incremental operational expenses (PAD par 37). Four cities (Bhopal, Chandigarh, Jaipur, and Mira Bhayander) were selected by the Ministry of Housing and Urban Affairs (MoHUA) to participate. Project Management Unit Estimated cost: US$1.7 million (PAD par 40), shown as US$1.8 million in the ICR, (ICR Annex 3). Actual cost:US$1.11 million (ICR Annex 3). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. The total actual project cost at project closure was US$84.89 million, which was US$28.11 below (75 percent) of the appraisal estimate of US$113.0 million. Note, the total project cost appraisal estimate in the PAD and the ICR were not exactly identical and the PAD figure was used. The significantly lower actual cost was primarily due to underspending in Component 2A(i), Modern Buses. This activity was estimated to cost US$54.07 million at appraisal and cost US$22.28 million at completion, 41 percent of the estimated cost. Component 2B, Technical Assistance and Capacity Building disbursed 59 percent of estimated costs; it was expected to cost US$2.88 million at appraisal but cost US$1.71 million at completion. Component 1, National Capacity Building and Technical Assistance disbursed 68 percent of its estimated costs; this was estimated to cost US$3.50 million at appraisal but the actual cost was US$2.37 million at completion. In contrast, Component 2A, Depot Modernization, disbursed 166 percent of its estimated costs; it was estimated to cost US$7.21 million at appraisal but cost US$12.00 million at completion (PAD par 40, ICR Annex 3). Financing. At appraisal the amount of Global Environmental Fund (GEF) financing for this project was estimated as US$9.20 million. At completion, the amount of GEF financing actually disbursed was US$9.15 million. Borrower Contribution. At appraisal the Borrower was expected to provide US$103.82 million, or 92 percent of the project total (PAD par 40). At completion the Borrower had disbursed US$75.65 million, or 89 percent of the project total (including US$1.8 million from the local private sector, ICR Annex 3). Most of the co-financing was from the Jawaharlal Nehru National Urban Renewal Mission (JnNURM), while required proportional contributions were provided by state and local governments. There was additional co-financing by the Energy Sector Management Assistance Program (ESMAP) under Component 1 and 2A, and by other city authorities under Component 2B (PAD Annex 6, Role of Co-financing, par 35). Dates. The project was approved on December 9, 2014 but only became effective on August 29, 2016. There was a midterm review on August 28, 2017. The original closing date of December 31, 2018 was extended three times. In December 2018 it was extended for 12 months to December 2019. An initial delay of almost 20 months in effectiveness was due to incomplete mobilization of the PMU at the national level and this had a cascading effect on project implementation (ICR par 67). In December 2019 it was extended for a further nine months to September 2019. This was to allow sufficient time for the implementation of key activities, including the construction of depots, installation of depot equipment, and the setting up of ITS/ MIS systems (ICR par 65). In September 2020 it was extended by an additional three months due to the Page 3 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) COVID-19 pandemic, which delayed on-ground implementation of ITS/MIS and the impact evaluation (ICR par 20). The project closed on December 30, 2020. Restructurings. The project was restructured five times but no changes were made to the PDOs. 1. The November 2, 2018 restructuring modified the methodology to assess the increase in ridership by more correctly measuring the change in ridership relative to the "do nothing" scenario. Two new indicators were introduced, the first to measure the improvement in overall user satisfaction and the second to record the number of city operators exposed to best practices in improved fuel efficiency. The latter replaced the previous indicator measuring the number of cities with training in improved fuel efficiency since some cities had more than one operator. 2. The December 28, 2018 restructuring finetuned the wording in some indicators to clarify the meaning. It also extended the loan closing date and the implementation schedule by 12 months. 3. The September 30, 2019 restructuring reflected changes between disbursement categories and in the implementation schedule. 4. The December 24, 2019 restructuring reflected an extension of the loan closing date and implementation schedule by 9 months. 5. The September 28, 2020 restructuring was for a further change in the loan closing date by three months and a reallocation between disbursement categories. Changes were initially made to accommodate the delay in grant effectiveness, and then to proactively accommodate the varying responses in the demonstration cities. The results framework was updated by refining the indicator methodology and end of project targets, along with changes in disbursement categories and the grant closing date. Split Rating. The PDOs were not changed. Although two indicator targets were revised downwards because of pending project delays all the others were unchanged. Since there was only a small difference between the results in the one case and in the other the original targets were exceeded, the ICR measured performance against the original targets in both cases and deemed that a split rating was unnecessary. IEG concurs with this approach. 3. Relevance of Objectives Rationale Country Context: India has been adding 10 million urban dwellers each year to its cities. This rapid growth has triggered an unprecedented urban and spatial transformation. Urban transport in Indian cities is provided by a range of modes: public, private, and personal and they have experienced low public transit modal shares, severe congestion, deteriorating road safety, and increasing air pollution and greenhouse gas emissions (GHG). The National Bus Funding Scheme 2009 of the Government of India (GOI), was the Baseline Project. Phase I included the procurement of 15,260 modern buses across the country; however, implementation of supporting infrastructure in Phase II, such as depots, workshops, and ITS/ MIS integration, was delayed. Land for bus depots was difficult to locate in cities due to multiple factors, and many of the existing depots were operated with minimal or no equipment, resulting in poorly maintained fleets, higher fuel consumption and fast deterioration of the condition of buses. Page 4 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) At the time of project preparation, clear evidence had emerged of the rapidly declining public transport modal share. With reduced efficiency, availability, and accessibility of bus services, cities witnessed a drastic increase in private vehicle usage. In cities with a formal bus system, the bus routes operated had not kept pace with the changes in travel demand. Further, awareness of the correlation between efficient urban transit and climate change mitigation, as well as an awareness of the country’s climate commitment, was still at a nascent level within city governance. The Efficient and Sustainable City Bus Services Project (ESCBS) was designed to apply to Tier 2 cities (i.e. with a population between 1 and 4 million). Bhopal, Chandigarh, Jaipur, and Mira Bhayander were selected for demonstration projects. Alignment with strategy: At appraisal, the project was in line with the World Bank Group’s (WBG’s) Country Partnership Strategy (CPS) for FY13-17 to support poverty reduction and shared prosperity. The Bank was expected to assist India in dealing with sustaining high growth, making growth inclusive, increasing the effectiveness of service delivery; and ensuring the development was sustainable and competitive. At closure the project was also well aligned with the India Country Partnership Framework (CPF) for FY 18-22. It supported the first and second pillars of “resource efficient growth” and “enhancing competitiveness and enabling job creation." The PDO was specifically aligned to Objective 1.3 of the CPF in respect of promoting policies and programs for the increased adoption of cleaner technologies that help in controlling air pollution in cities. Sector and Institutional Context: The PDO focused on improving the efficiency and attractiveness of city bus transport and reducing greenhouse gas emissions in four mid-size demonstration cities. By working on service level improvements, the project was expected to contribute to meeting India’s global commitment to GHG emission reduction. India’s Intended Nationally Determined Contribution advised in 2015, aimed to make sustained efforts in GHG emission reduction and mitigation in the transport sector, especially passenger transport. The national government aimed to promote efficient and green public bus transport and announced a national scheme at a cost of US$2.3 billion. The Bank supported this priority through a Reimbursable Advisory Service study on Improving Bus Public Transport Services with a focus on urban areas. The ESMAP training results and the technical assistance studies, undertaken as part of the ESCBS project was to be disseminated to other similar Indian cities as well as the pilot cities. The PDO remained well aligned with the government and Bank strategies, and was formulated at an outcome level to demonstrate the reduction in GHG emissions, while improving city bus transport service delivery. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To improve the efficiency of city bus transport. Page 5 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) Rationale The Theory of Change of this objective is that a GEF grant would be catalytic for the public bus transport system in that demonstration projects would show how improvements to physical infrastructure such as bus depots together with the purchase of suitable equipment, information and management systems, and coupled with capacity building initiatives including the addressing of policy, regulatory and fiscal constraints, can improve the efficiency of city bus transport in India. Outputs Component 2A and Component 1 contribute to PDO1 through: (i) building and upgrading depots, (ii) installing modern depot equipment, (iii) implementing ITS/MIS in the demonstration cities, and (iv) training bus operators on driving skills, training operators, crew, and maintenance staff on better bus maintenance practices.  Existing depots were assessed as planned and as a result seven depots were upgraded, while three cities upgraded overnight parking facilities. Some new cost saving features were introduced including solar panels and surveillance cameras. Separate toilet facilities for female staff were also introduced. Three new model depots were constructed and made operational, customized to actual needs in each case. Fully achieved.  Modern depot equipment items (131 in total) were installed and are functional in all four cities. In Bhopal, improvements to braking efficiency and overall driving were reported. Fully achieved.  Increased use of ITS and MIS data and tools in operations planning and management was implemented in three of the four cities. The fourth city, Jaipur, failed to implement the planned ITS/MIS system after issues in clarifying the scope and due to frequent changes in staff. It is now undertaking this work on its own using GOI funding. Mostly achieved.  Training/capacity building took place at all levels for 3,051 individuals (city, state and national). This covered workshops, ESMAP, expert panels, general capacity building, a focus group, a conference and a study visit. After September 2020 some events were virtual due to Covid. Mostly achieved but the design lacked an indicator to give feedback on the effectiveness of the learning. Outcomes Rate of breakdowns PDO Results Indicator 2A, Rate of breakdowns to not exceed 5 per 10,000km of bus operation, was fully achieved. Rate of breakdowns is an important metric because interruptions in service have a direct operating cost impact since service reliability has a strong impact on attracting riders. Breakdowns/10,000 k Baseline Original Target Revised Target At Completion m Bhopal 4.2 3.4 Chandigarh 6.6 0.7 5.0 Unchanged. Jaipur 7.4 2.9 Mira Bhayander 7.5 0.3 Page 6 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) PDO Results Indicator 2B is improvement in bus fuel efficiency in demonstration cities by at least 12% relative to business as usual. This indicator succeeded in achieving its revised target in the pre-Covid period (Jan.-Feb. 2020). It is now recovering from this operational disruption and by Jan.- Mar. 2021 had reached 14%, close to the original target of 15%. % fuel Baseline (l/km) Original Target Revised Target At Completion improvement Bhopal 3.2 Chandigarh 3.4 15% 12% 14% Jaipur 2.8 Mira Bhayander 2.8 Rating Substantial OBJECTIVE 2 Objective To improve the attractiveness of city bus transport. Rationale Theory of Change The Theory of Change of this objective is that a GEF grant would be catalytic for the public bus transport system in that demonstration projects would show how beyond the purchase of new buses, improvements to physical infrastructure such as bus depots together with the purchase of suitable equipment, information and management systems, and coupled with capacity building initiatives, can improve the attractiveness of city bus transport in India by improving the enabling environment. Outputs PDO2 is primarily achieved through the activities under Component 2B; the efficiencies from Component 2A and the training in Component 1 provide complementary support. These activities are: (i) Project preparation costs, (ii) TA for ITS/MIS implementation, services and business planning, marketing, fuel efficiency, (iii) Capacity building and incremental operating costs.  More efficient and user responsive bus service plans were to be prepared and adopted in at least two cities. This has been achieved in one city and is in process in two other cities. Partially achieved. Page 7 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418)  Having institutional systems in place for monitoring and evaluation of a bus fuel efficiency program in at least two cities. This was partially in place in two cities at completion and another city is in the process of implementing this through a different team. Not achieved.  All of the project cities have been introducing or implementing ITS/MIS practices for internal administration, customer-facing activities, and operational support with varying degrees of advancement. Taken together, they provide a set of examples that can be useful elsewhere. The ICR provides city level details in Annex 7. The Intermediate Indicators under Component 2B have been modestly achieved. The project was generally successful in improving the attractiveness of bus transport in the demonstration cities. Total ridership increased well, although women’s proportion of the total ridership generally did not increase as much as targeted. User satisfaction was increased both overall and for women. PDO Results Indicator 3A, Direct Project Beneficiaries and Female Beneficiaries. Baseline Original Target Revised Target At Completion 16% increase over Direct beneficiaries 10% increase over "business as usual" 180 million 235 million (Number of riders) “business as usual”* (Prior to Covid 20% was achieved) Female Beneficiaries (% of ridership) by city Bhopal 39% 39% Chandigarh 28% 32% Increase by 3% Unchanged Jaipur 39% 37% Mira Bhayander 43% 34% * According to the Project Paper on restructuring (November, 2018), the methodology to assess ridership was changed to measure the increase in ridership as a result of the project. It was modified to measure the increase in ridership relative to business as usual (i.e. without the project). The overall ridership exceeded the target, even allowing for the impact of Covid but the female ridership was not achieved and fell in three of the four demonstration cities. This likely reflected a change in working modalities during the pandemic. Results Indicator 3B, Improvement in Women’s User Satisfaction and Improvement in Overall Satisfaction with bus services. Satisfaction score Baseline Original Target Revised Target At Completion (out of 5) Women’s Satisfaction Bhopal 3.56 Increase in 4.08 Chandigarh 3.51 satisfaction in at Same as original 4.60 Jaipur 3.07 least two cities 2.64 Page 8 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) Mira Bhayander 2.42 3.25 Overall Satisfaction Bhopal 3.36 4.11 Increase in Chandigarh 3.59 4.55 satisfaction in at Same as original Jaipur 3.44 2.71 least two cities Mira Bhayander 2.62 3.30 Both overall and women's user satisfaction substantially improved in three of the four cities. The end term survey highlighted that in Jaipur both men and women were highly dissatisfied by the quality of services. Jaipur had failed to implement the planned ITS/MIS system and had suffered from frequent changes in management. The activities that supported this outcome were more service kilometers operated with improved efficiency, reliability and improved information availability. The impacts of demand responsive service planning were evident in Bhopal, where after restructuring of the routes the average daily ridership went up from 269 per bus in second half of 2016 to 494 per bus in 2017 (ICR par 43). Improvement in the user satisfaction index, both overall and for women users, was visible in Bhopal, Chandigarh and Mira Bhayander. Seat reservations for women, cleaner buses with reduced crowding, safety features on-board (such as CCTV/ GPS tracking/ panic buttons) and the introduction of women conductors/ drivers were appreciated by female riders. Bhopal set up two more helplines for grievance reporting and grievance redress; promoted a QR code-based anonymous driver rating and feedback system; and operated free bus services for women on “women’s day”. Chandigarh installed passenger information systems at all operational bus stops, modified routes, followed the social distancing norms stringently, ensured regular bus cleaning and maintenance. Jaipur’s user perception may have suffered due to reduced bus services during the COVID-19 pandemic (ICR para 46). Rating Substantial OBJECTIVE 3 Objective To reduce greenhouse gas emissions in the demonstration cities. Rationale The Theory of Change of this objective is that a GEF grant would be catalytic for the public bus transport system in that demonstration projects would show how improvements to physical infrastructure such as bus depots together with the purchase of suitable equipment, information and management systems, and coupled with capacity building initiatives including the addressing of policy, regulatory and fiscal constraints, can reduce greenhouse gas emissions of city bus transport in India. The projected GHG emission gains were estimated (over 2020-30) from cities are a combination of five activities: (i) procuring buses (higher ridership), (ii) depot creation and equipment (increased fleet life and utilization due to better maintenance and fewer breakdowns), (iii) better per vehicle fuel efficiency Page 9 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) (combination of bus maintenance and driver training), (iv) better service operations (empirical data from the ITS/ MIS processed to inform the service plans/routes resulting in increased bus kilometers and ridership), and (v) increased modal shift to city buses as compared to business as usual, due to improved service plan, more reliable service, better passenger information, and cleaner buses. Outputs Components 2A and 2B contribute directly to PDOs 1 and 2 and indirectly to PDO3. Component 1 (National Capacity Building for Urban Bus Sector) builds both strategic and local operational capacity to achieve PDO1. The review of policy, regulatory and fiscal constraints at national, state and city levels, and capacity building (training) has been carried out through studies and training undertaken by MoHUA and ESMAP.  Policy notes and recommendations were developed for sustainable city bus transportation and deliberations at the national and state levels. Fifteen city/state level workshops and two national level workshops were held; exceeding the target of at least five national and state level workshops. Substantially achieved.  Some 33 City bus operators were to be trained in fuel efficient operations through various agencies and through the technical assistance program. At completion 19 city bus operators had been trained under ESMAP. A further 20 agencies received training under the "Leaders in Urban Transport Planning and Management Program (LUTP)" and "the Centre of Excellence in Urban Transport (CEPT)." Achieved.  Five of the six studies and training initiatives are complete, covering technical subjects: bus operator (drivers) skills development, procurement specifications, bus depot design, depot level training, and services and operational planning. The outstanding study on, “Options and Recommendations for Regulatory, Institutional and Fiscal Constraints in Providing Efficient and Sustainable City Bus Services” is said to be 85% complete and the MoHUA has agreed to finish the study using its own funds. As the ICR indicates, this “Study has the potential to bring about critical regulatory and legislative reforms for sustaining the city bus sector in India” (ICR para 27, Table3). Partially achieved Overall, several outputs under Component 1 have been achieved. However, the critical national-level, more strategic study output has not been completed, and although substantial training has been undertaken, it is unclear how effective it has been. Outcomes PDO Results Indicator 1, Reduction in GHG emissions in project cities over a period of 10 years as a result of the adoption of modernization initiatives from the business as usual scenario, was theoretically achieved. Reduction in GHG Emissions (metric tons of CO2 equivalent) Original Baseline: 1,094,430 (2014-2023); Original Target: Reduction of 230,000 Revised Baseline: 1,069,063 (2014-2023); Revised Target: Reduction of 170,000 At Completion (equivalent from 2020-30) 238,600 (Scenario 1, without new buses); 454,600 (Scenario 2, with new buses) Page 10 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) The Scenario 2 savings are estimated from a theoretical model based on adding more buses (700 ICE buses and 300 e-buses) that all the four cities have procured or were planning to procure in next six months (ICR Annex 1, A.1, page 26). Project activities would result in a higher fleet utilization and optimization, thus higher service km would be operated in the project scenario as compared to business as usual. Since this was an estimate, however, and no empirical measurements of GHG were made, and given that the impacts of Covid need to be taken into account, the results must be treated with caution. Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale This is not an easy call because some of the evidence was not sufficiently rigorous and Covid impacted the results towards the end of implementation. The new depots and infrastructure improvements were all in place and new equipment was provided in all four cities. There was evidence of better fuel efficiency and less breakdowns. Three of the four cities introduced ITS/MIS systems. User satisfaction including female use satisfaction improved in three of the four cities, but there appears to have been a management problem in Jaipur. Female ridership was also below expectations, likely due to Covid and women being more dependent on public transport. The reduction in greenhouse gas emissions was estimated based on a theoretical model rather than empirical evidence and although there were a significant number of training activities, their impact has not been assessed while a strategic study with potentially national impact is unfinished. Giving equal weight to each of the objectives it appears that the efficiency and attractiveness of city bus transport has substantially improved, while the objective of reducing greenhouse gas emissions and disseminating this knowledge has at least moderately improved but there needs to be more evidence to substantiate this fully. On balance the overall efficacy is substantial, with moderate shortcomings. Overall Efficacy Rating Substantial 5. Efficiency At appraisal the economic analysis was based on incremental costs. It was claimed that it was not feasible to carry out traditional cost benefit analysis and argued that incremental cost analysis was normally used for GEF funded projects (PAD par 54). The analysis was broken down into direct and indirect effects. The direct effects comprised three activities: (i) bus depot modernization, (ii) the ESMAP fuel efficiency improvement program, and (iii) application of ITS and MIS as well as service planning and route optimization. The total was shown as “Lifetime direct GHG emissions“ and “Lifetime direct post-project GHG emissions” without defining “Lifetime” or “post-project”. By inference lifetime meant over the ten year period 2014 to 2023 (PAD Annex 6, par 7) but post- project remained less clear. The indirect effects analysis looked at the larger impact if the sustainable practices Page 11 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) piloted in this project and was to be replicated across the country (top-down) as well as in another ten cities (bottom-up). Annex 6 of the PAD laid out the logic as to how the emissions savings would be obtained. It provided the percentage improvements expected in vehicle and operating characteristics as well as the total tons of emissions savings. It did not indicate the base data by city (e.g., vehicle km operated), on which the percent improvements would be applied. At completion the ICR provided the total tons of emissions savings under two scenarios. First, savings between 2020 and 2030 with no additional bus procurement. Second, savings over the same period but including the impact of the four cities having procured an additional 1,000 buses over the following six months. It spelt out the logic of how emissions saving would be obtained and the percentage improvements expected in vehicle and operating characteristics. It also provided the total tons of emissions savings. The GEF grant of US$9.15 million resulted in reducing direct CO2 emissions by 454,600 metric tons. The GEF contribution per ton of CO2 was only US$20, compared to the US$40 estimated at project appraisal. The unit cost of CO2 reduction against the project cost of US$85 million at US$182 per ton was also significantly lower than the estimated unit cost of US$493 per ton. This is a low estimate, because the ICR only calculated direct emission benefits from the four pilot cities and did not calculate emission reductions due to the contribution of the project to accrued fuel savings from the maintenance protocols, driving, or the scale up of depot modernization on inter-city operations. Administrative and Operational Efficiency: Initial delays of 20 months affected the overall implementation timeline, while the impact of the COVID-19 pandemic on project implementation was experienced from March 2020 for a period of about six months, when national and local lockdowns and precautions were put in place. While implementation benefits from equipment and ITS systems are still being realized, most key activities were satisfactorily completed, although one major study was not completed. Knowledge sharing activities had to be carried out virtually. Overall, project implementation proved to be far more complex and time intensive than initially envisaged. In view of these factors, the two-year extension of the grant closing date was not surprising. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. Page 12 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) 6. Outcome Based on substantial relevance and substantial efficacy, with moderate shortcomings, for the PDOs, with modest efficiency, the outcome rating is Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Lack of sustained ownership and commitment: Although guidance documents and manuals have been circulated to state agencies and shared on the MOHUA website, champions are needed at the national and city levels to keep the momentum going, and the likelihood of this may be worsened by frequent changes in key officials and the lack of a permanent operations team in each of the cities. Continued and regular refresher on-job training similar to the kind provided under the project would help to strengthen commitment. Long term sustainability of the sector also requires a more enabling regulatory environment to be in place to ensure priority to city public transport (ICR page 24). Inadequate maintenance: Trained staff are necessary to operate the more sophisticated depot equipment, and proper contracting arrangements (including handing over equipment to selected private operators) are essential to ensure proper equipment usage and adequate maintenance. There is a risk that adequate funding may not be provided by the cities for the proper maintenance and upkeep of the equipment, as well as to address the expanding need for new fleets. Sustainability of ITS/MIS systems: Similarly, while ITS/ MIS are in use in two cities, with limited funding there remains a risk to their continued sustainability. Impact of Covid-19: Should the Covid-19 pandemic continue, ridership and hence revenue would suffer and stall progress. 8. Assessment of Bank Performance a. Quality-at-Entry Project preparation benefited from the background analysis undertaken in 12 cities and from a better understanding of the concerns of male and female bus users as well as non-users. Lessons from this analysis, along with previous pilot testing, were considered in the project design. Several rounds of consultation with key stakeholders were undertaken and the project was consequently designed to enhance the impact of government policies and programs. Moreover, with a view to simplifying the project design, keeping in mind the learning from the Sustainable Urban Transport Program, new depots were constructed on publicly owned sites to avoid land acquisition issues. Page 13 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) Both classroom and on-the-job learning were built into the project design to strengthen institutional capacity. The project design included the provision of technical support (in the form of the PMU) to Government agencies for data collection, organizing capacity building sessions, procurement, and review support. The design envisaged ‘learning by doing’ for blue collar staff (bus maintenance, fuel-efficient ways of driving, and best practice in depot equipment procurement). In addition, there was to be practitioner-oriented training sessions, technical assistance, and visits to demonstration projects. The project duration did not include contingencies for unanticipated operational issues (ICR page 18). The analysis of implementation capacity carried out during project appraisal indicated that the four-year project duration would be adequate, despite the design involving technical assistance at the national level and demonstration projects in four cities. There was very limited experience of working with Bank procedures and the impacts of this were seriously underestimated. Market interest for the supply of depot equipment could have been explored further; this would have saved time and effort during project implementation (ICR page 23). Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The impact of the lack of coordination, commitment, and familiarity with Bank project requirements was underestimated. A delay of almost twenty months in the grant being declared effective had a cascading effect on project implementation (ICR page 19). Because of lack of progress the dropping of two cities was considered but this was not agreed to (Project Paper, First Restructuring, par 4). At the city level, depot construction was delayed, which resulted in turn in overall delays to the setting up of equipment as well as ITS/MIS implementation. This was further exacerbated by the limited understanding of working with the World Bank’s fiduciary and safeguard requirements, as well as because of the frequent changes of officials. Delays in filling posts in the PIAs as well as frequent staff turnover impacted city level performance and was variable among the pilot cities. By the Mid-term Review (MTR) of September 2017 the project was two years behind schedule, depots and equipment were not ready, some cities still scrapped buses before their full life expectancy, and almost every city had been through two to three rounds of depot and/or ITS procurement. Project restructuring was undertaken in 2018 to address these issues. Some solutions were also found during regular Bank implementation support missions, with the PMU supporting the cities to plan and deliver better, and by escalating issues to senior officials when necessary. The grant closing date was extended by twenty-four months to allow sufficient time for the completion of key activities, including the construction of depots, installation of depot equipment, and the setting up of the ITS/ MIS systems. Following the MTR, project delivery picked up speed for the last three years of implementation. Some factors were, however, outside the control of Government and/or the implementing entities. The lack of mature domestic players for depot maintenance equipment resulted in no or weak bid responses, erroneous bids, shortages of products and delivery mix-ups. Consultants, vendors, cities, and the PMU all lacked familiarity with equipment specifications, depot planning and design, and land suitability analysis. These problems were compounded by the lack of standardization of equipment and ITS/MIS systems, delays in delivery and commissioning, and in imparting training to staff on equipment usage. The procurement of modern buses under the Baseline Project (to be procured using GoI funding) did not fully materialize. Although 788 of the Phase I buses had already been procured at the time of project appraisal, Page 14 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) only 360 of the sanctioned 886 Phase II buses were actually procured. This was primarily due to the very short timelines provided by GoI for the completion of procurement. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design At appraisal, the project included partially appropriate PDO indicators and intermediate indicators to assess the achievement of the project objectives and outputs. Some indicators had to be revised during the restructurings to remove ambiguity in measurement. For example, the methodology had to be modified to measure the increase in ridership by showing the change in ridership relative to the do nothing scenario. The baseline estimation of the overall user indicator remained unclear until April 2019, two years and eight months after effectiveness. Data quality was not discussed as an area that needed attention during preparation but during implementation this became an area of concern. The inclusion of intermediate indicators on “depot level staff training on fuel efficiency,” “buses maintained using modern equipment per depot” and “rate of equipment usage per depot/city” were useful to demonstrate the correlation to incident reduction and fuel efficiency. The project established a three-tier monitoring structure (monitoring committees), and the Quarterly Progress Report (QPR) formats were designed in consultation with the city officials and PMU/PIAs. Budgets for M&E were included for the entire duration of the project (ICR page 20). b. M&E Implementation Baselines were established (in one case late), and progress towards the PDO and key intermediate indicators were regularly tracked (ICR page 20). The initial issues in data quality, however, (especially for break-down rates and fuel efficiency), continued until project closure due to the limited capacities at the city level. Regular follow-ups and reviews by the PMU and the Bank team were necessary to ensure data consistency. The cities were then able to make the necessary adjustments and submit the QPRs timeously. Two new indicators were introduced, the first to measure the improvement in overall user satisfaction and the second to record the number of city operators exposed to best practices in improved fuel efficiency. The latter replaced the previous indicator measuring the number of cities with training in Page 15 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) improved fuel efficiency since some cities had more than one operator. There was no significant change to the project scope. The impact of the COVID-19 pandemic delayed data collection and surveys in 2020. The user surveys proposed as part of the M&E were updated by considering the COVID-19 pandemic and its impacts. Post-March 2020, the user satisfaction surveys were redesigned to capture the impacts of the COVID-19 pandemic on urban transport in the project cities. Additional questions were included to understand the changes in travel patterns, reduced economic activity, and/or user confidence in public transit. c. M&E Utilization The M&E process was used to review progress, develop mitigation measures, and agree with the cities on improvement strategies, as well as for informed decision making for overall project management. Each city, based on the maturity of its operations, was able to utilize the M&E to the extent this was feasible. For instance, by December 2019, based on the M&E data, Chandigarh realized the benefits accruing from the usage of depot equipment and obtained additional support from the Project, as well as Government funds for the purchase of equipment for other depots (ICR page 20). These findings were further shared with other cities as part of the Bank’s effort to create a demonstration effect. The implementation of ITS substantially improved the PIUs’ access to data, and therefore the opportunity to use it for more effective operations. M&E Quality Rating Modest 10. Other Issues a. Safeguards Environmental Safeguards: At appraisal, OP/BP 4.01 (Environmental Assessment), OP/BP 4.04 (Natural Habitat) and OP/BP 4.11 (Physical Cultural Resources) were triggered. The project was assigned Category ‘B’ because potential impacts were assessed to be manageable and mostly concentrated within the depot sites. These impacts included drainage, water pollution due to discharge from vehicle repairs/washing, health and safety during construction, and traffic safety related issues during the operation of the depots. In line with the Environmental and Social Management Framework (ESMF) prepared for the project, environmental assessment was undertaken along with the preparation of engineering designs for the depots. The resultant Environmental Management Plans (EMPs) were integrated with the design elements to minimize environmental footprints from depot operations. Adequate budgetary provisions were included where separate measures were to be implemented. The three new depots (two in Jaipur and one at Mira Bhayander) obtained the required licenses from the respective State Pollution Control Boards for the purchase of raw materials like sand and stone from existing approved sources (ICR page 21). Measures such as internal storm water drains, and oil, and grease traps were integrated into the depot designs to reduce the environmental footprint of the new facilities. In addition, measures with long term positive environmental benefits such as solar photovoltaic panels, rainwater harvesting systems and sanitation systems, were included in the designs. The operation of the environmental treatment Page 16 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) plants has enabled the reuse of treated wastewater for gardening or bus washing, and to recharge ground water; the use of solar PV panels has reduced dependency on fossil-based energy. These measures are contributing to environmental benefits by minimizing the project’s carbon footprint. However, the effectiveness of these facilities will depend on the implementation of the yet to be developed standard operating procedures, work for which is still underway. Implementation of the EMPs was affected by the frequent transfer of key officials. The key challenges during EMP implementation were securing access to work sites and the safety of workers during construction. In the case of the Raipur Kalan Depot in Chandigarh, the finalization of the EIA report and its disclosure took longer than expected due to delays in the administrative and technical approval of the design (ICR page 22). Compliance with the Bank’s environmental safeguards, as reported, was overall Satisfactory. Social Safeguards: OP/BP 4.12 (Involuntary Resettlement) was triggered, as the probability of displacement of squatters or encroachers on or around the boundary of depots was identified as an issue. Construction at the Ghodbunder Depot involved the relocation of 163 slum dwellers to another temporary site, and an addendum to the ESMP was prepared. The Mira Bhayander Municipal Corporation (MBMC) secured intermediate transit camps and these slum dwellers were temporarily relocated to the Transit Camp of the Basic Services for Urban Poor Scheme. Based on the guidance provided by the Bank, MBMC took steps to improve the facilities available in the transit camp site, as well as the working conditions for workers in the construction site. The final relocation to affordable housing under the Pradhan Mantri Awas Yojna scheme was scheduled for October 2021. City agencies, with guidance from the World Bank team, incorporated several measures for rehabilitation, keeping in mind gender aspects. At project closure, most gender related recommendations had been implemented. Grievance redress mechanisms had also been set up by the cities for the resolution of any complaints received. The Bank is to follow-up with the project cities to ensure that all remaining social safeguards actions have been completed in a satisfactory manner. There were delays in the preparation of the social safeguard instruments, particularly in Jaipur and Mira Bhayander, due to frequent changes in staff and the lack of capacity in the implementing agencies. The Bank provided guidance for the preparation of these instruments as well as in capacity building in implementation of the ESMF and the ESMPs. The project closed with Moderately Satisfactory rating for Involuntary Resettlement. b. Fiduciary Compliance Financial Management: The financial management capacity varied significantly in the PIAs. Bhopal City Links Ltd (BCLL) and Jaipur City Transport Services Ltd (JCTSL), the dedicated companies established for the operation and management of city bus services under the Jawaharlal Nehru National Urban Renewal Mission, had a low institutional capacity for financial management. There were repeated observations by the external auditors concerning poor documentation of fixed assets, weak internal controls, etc. Submission of audit reports Page 17 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) and Interim Unaudited Financial Reports of the PIAs by the PMU was typically on time, with audit observations on monetary values; however, these were not significant and were mostly reconciled. The financial management processes for ensuring the smooth flow of funds and proper expenditure controls at the state PIAs were either not well established or were discontinued during project implementation. The project required the creation of dedicated project bank accounts in PIAs, however discussions during implementation supervision missions with BCLL and JCTSL revealed that these bank accounts were not being used for project expenditures. The external auditors also had observations on similar lines in the project audit reports across financial years. The financial management arrangements for the project at the central level piggy backed on the institutional set-up of the Sustainable Urban Transport Project (SUTP), which closed on March 31, 2018. The two projects shared a common PMU in MoHUA, which was intended to leverage the learning on fiduciary systems and processes required under Bank projects. However, this was not the case in the state PIAs. It appears that from a financial management perspective, that the Efficient and Sustainable City Bus Services Project (ESCBS) did not receive appropriate attention from the PMU until the SUTP closed. The ESCBS project closed with a Moderately Satisfactory rating for Financial Management. Procurement: The Bank’s Procurement Guidelines (2011) applied to activities funded by the GEF grant and not to the procurement of buses and depot construction, which were funded entirely by counterpart funds. Procurement was carried out for US$16.7 million (1.7 times the GEF Grant) through 70 procurement processes. It is confirmed that there was no non-compliance in the procurement process with respect to Bank Procurement Guidelines during project implementation. However, a lack of trained staff in the city units, lack of familiarity with e-procurement or approved systems, and the frequent transfer of senior officials affected project procurement. Procurement was extremely slow, especially in Mira Bhayander and Jaipur. For many post-review procurement activities, the required data had not been updated in the system. Procurement was rated Moderately Satisfactory in the Implementation Supervision Reports and the project closed with a Moderately Satisfactory rating for Procurement. c. Unintended impacts (Positive or Negative) None. d. Other None. 11. Ratings Page 18 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The ICR developed the following useful lessons: Engaging with the state governments, along with the national and city governments, should be considered in future projects concerned with bus service efficiency and sustainability. A strong need was expressed for the creation of a State Technical Unit to bring the necessary attention and focus to the project and to support cities in planning, design, financing, and management of bus services. The Bank could also consider using the Program for Results approach when dealing with multiple cities under same operation to implement similar interventions as a part of existing or new Government programs. Centralized procurement, standardized templates and formats can all help fast-track procurement and ease the processes for medium-sized cities with weak procurement capacity. Under ESCBS, the cities with limited ability and capacity each went through a steep learning curve for the procurement of equipment and ITS/MIS. Having a centralized or standardized procurement process, with recommendations on system design specifications and a list of empaneled vendors, should be considered for future similar projects and programs. This would also eliminate the individual negotiations that the cities may have to navigate (on spares, training, etc.) in future, and will help standardize and bring about economies of scale. Lack of continuity of officials and teams can affect project outcomes and consistent capacity building efforts are critical for successful implementation. The project lost valuable time due to the frequent transfer of officials and the consequent efforts needed to restart the dialogue with the Bank. Even the key positions in the operations teams (e.g., traffic manager) were either absent/ vacant or were affected by high attrition at both city and state levels. The lack of adequate qualified staff with continuity is in large part due to the policy and regulatory environment in Indian cities not yet assigning adequate priority to urban bus transport. The Bank has already made good progress in furthering the policy dialogue and needs to continue for the long-term sustainability of the sector. A two-pronged approach to developing capabilities and the enabling environment can help to leverage private sector support. ESMAP showed greater emission reduction by involving private operators in the training. Efforts on capacity building for the private sector (such as consultants, vendors, and service providers) in bus operations, maintenance, ITS/MIS, designing/planning are critical and help in scaling up impact. At the same time, the creation of an enabling environment by putting in place appropriate risk sharing and dispute resolution mechanisms, timely payments, Page 19 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) and a collaborative approach with private operators can help support much needed market development to achieve the necessary scale up in availability of bus services in the country. The project should explicitly include the dissemination of the results in the four demonstration cities. While dissemination was envisaged under the training and TA activities, it would be a good idea for similar demonstration projects to include the dissemination of results as part of project activities. 13. Assessment Recommended? Yes ASSESSMENT_TABLE Please Explain The number of buses that became available was less than expected and more evidence is needed regarding actual GHG emissions and the impact of the dissemination of best practice. The low capacity of city bus management also indicates it may take longer than expected for the full results to materialize. Implementation was affected by the Covid pandemic and thus the true impact of the project is not yet fully realized. 14. Comments on Quality of ICR The ICR is mostly clear with some useful evidence, photographs and illustrations. It is also candid about the reasons for the delays experienced in the project. The Theory of Change, though retrofitted and conjoining three specific objectives as one, clearly shows the logical links between the project activities, their outputs and the intended outcomes. The lessons are based on the difficulties experienced, especially with low institutional capacity, and should be very helpful for future similar projects. Some small quibbles are that the base data were not provided and so it was difficult to assess the appropriateness of the final GHG savings figures. There could also have been more information in the ICR taken from the project restructuring papers clarifying the reasons for the changes in wording for some of the indicators. Some abbreviations, especially in the financial management and procurement sections, were also not defined in the abbreviations and acronyms table. a. Quality of ICR Rating Substantial Page 20 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Efficient & Sustainable City Bus Service (P132418) Page 21 of 21