Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760) Report Number : ICRR0024306 1. Project Data Project ID Program Name P163760 RMPBEE Program Country Practice Area(Lead) Serbia Transport L/C/TF Number(s) Closing Date (Original) Total Program Cost (USD) 91,770,020.18 Bank Approval Date Closing Date (Actual) 02-Nov-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 0.00 Revised Commitment 0.00 0.00 Actual 0.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ranga Rajan Vibecke Dixon Avjeet Singh IEGSD (Unit 4) Krishnamani 2. Program Context and Development Objectives DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) as stated in the Loan Agreement (Schedule 1, page 5) and in the Project Appraisal Document (PAD, page 10) is: " To improve the management and sustainability of select public infrastructure by strengthening government capacity and systems, upgrading assets and increasing expenditure efficiency ". Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760) As the program encompassed activities in the transport and energy sectors, the PDO is assessed in this Implementation Completion Report Review (ICRR) as follows: PDO 1: To improve the management and sustainability of select public infrastructure (road sector) by strengthening government capacity and systems, upgrading assets and increasing expenditure efficiency. PDO 2: To improve the management and sustainability of select public infrastructure (energy sector) by strengthening government capacity and systems, upgrading assets and increasing expenditure efficiency. b. Were the program objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 23-Dec-2023 c. Will a split evaluation be undertaken? Yes d. Components The program consisted of the following activities (LA, page 5): One: The transport sector. This component was to support the Government: (a) in maintaining 8,000 kilometers (Km) of national roads in the Government's road maintenance program (1,000 km using traditional maintenance contracts, 2,000 km using enhanced maintenance contracts and 5,000 km using performance-based maintenance contracts (PBMC) for the 3rd year), managed by the Public Enterprise "Roads of Serbia" (PERS) over the 2017 - 2020 period: and (b) supporting PERS in implementing PBMC for maintaining national roads through: (i) a service level agreement (SLA) between the Ministry of Construction, Transport and Infrastructure (MTCI) and PERS (Under this long-term agreement between the Government and PERS, PERS was to be given details about the level of fiscal support the government would provide PERS for maintaining roads with incentives for better performance); (ii) supporting PERS in developing a three year performance-based maintenance plan for 2017-2021; and (c) supporting PERS in developing an asset management plan integrating management, financial, economic and engineering practices. Two: The energy sector. This component aimed to support renovation of 234 buildings, identified as municipal priorities and approved by the government under the program for reconstruction and improvement of state-owned public facilities in the Education, Health Care and Social Protection sectors. e. Comments on Program Cost, Financing, Borrower Contribution, and Dates Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760) Program cost. The estimated program cost at appraisal was US$763.95 million. The actual cost was US$832.90 million. Financing. The program was financed by an International Bank for Reconstruction and Development (IBRD) loan of US$118.60 million. The amount disbursed was US$91.77 million. Non- World Bank (WB) contribution. The government was expected to contribute US$129.40 million. Their actual contribution was US$175.40 million. Local sources of Serbia (through toll revenues of PERS) were expected to contribute US$515.95 million. Their contribution was US$565.73 million. Dates. The program was approved and became effective on November 2, 2017 and scheduled to close on December 3, 2021. The Mid-Term Review (MTR) was held on November 14, 2022. The program closed two years behind schedule on December 31, 2023. Changes. The project was restructured twice during implementation. The following change was made through the first project restructuring on December 22, 2023.  The closing date was extended by two years to December 31, 2023, to account for the delays in the initial years and bottlenecks in the initial years of implementation, due to the COVID-19 pandemic. The PDO, the project scope and the targets of the indicators were however unchanged. The main change made through the second project restructuring on December 26, 2023.  Targets of five of the ten disbursement-linked indicators (DLIs) had not been achieved, due to the delays in the initial years. Therefore, the project scope was reduced, with targets for several indicators in the energy sector reduced to reflect the reduced project scope. Split rating. The project scope was not changed in the transport sector. However, the targets in the energy sector were significantly reduced with the second restructuring when US$71.79 of the loan was disbursed. This review uses a split rating of objective for determining the overall outcome rating. 3. Relevance TBL a. Relevance of Objectives New Rationale Country context. Serbia's public sector performance was ranked 132nd by the 2016-2017 Global Competitiveness Report, worse than its overall ranking of 94 in the report. Serbia ranked low in the government's procurement of advanced technology. When this project was under preparation, Serbia had made substantial progress towards its entry to the European Union (EU). The prospects for EU accession provided an impetus to the government for public sector reforms. Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760) Road sector context. Limited use of modern technology had contributed to deterioration of road quality. The challenges faced by the sector were: (i) a maintenance backlog; (ii) a need for modernizing management for preserving assets; (iii) a need for strengthening institutional arrangements; and (iv) a need for increasing the resilience of the network. Energy sector context. Serbia is an energy and carbon intensive country. The building sector (residential, public and commercial) dominated energy consumption, representing 45% of final energy use. The carbon intensity of Serbia was over 2.5 times that of the European Union (EU) - 28 average. (EU - 28 is an abbreviation of EU, which consists of a group of 28 countries that operate as an economic block). Much of these inefficiencies were due to the chronic lack of investment, aging building stock with insufficient thermal insulation elements and inefficient technologies. Government strategy at appraisal. The PDO was well-aligned with the government's priorities at appraisal. To address the maintenance backlog, the Government embarked on an International Finance Institutions (IFs) supported National Road Rehabilitation and Safety Program. In the energy sector, the Government adopted the Law on Efficient Use of Energy in 2013 to provide the legal foundation for energy efficiency measures. The relevance of the PDO was strengthened by the government adopting new laws and sector strategies during implementation: These included: (i) The Laws on Renewable Energy Sources and Energy Efficiency and the Rational Use of Energy in April 2021; (ii) The Long-Term building renovation strategy in February 2022; and (iii) The integrated National Energy and Climate Plan in 2024 that set specific energy efficiency targets for public buildings by 2030. World Bank (WB) strategy at appraisal. The PDO was well-aligned with the WB's Country Partnership Framework (CPF) for 2016 - 2020. The CPF identified infrastructure as one of the six areas for engagement under Focus Area One: Economic Governance and the role of the state. The CPF articulated the need for reforms aimed at improving efficiency in public spending, better maintenance of infrastructure and prioritization of public investments. Current WB strategy. The PDO continues to be relevant to the challenges identified by the 2020 Systematic Country Diagnostic Update (SCD) and is consistent with the objectives of the CPF for 2022 - 2026. The SCD update noted that government effectiveness, efficiency and accountability are critical and that continued efforts are needed to improve service delivery and make the transition to greener growth. The CPF for 2022 - 2025 focused on areas where institutional strengthening needed to be consolidated based on the experience of the previous CPF. This included greener and more resilient growth (Objective 1,2: Greener investments and transition to a low-carbon and resilient economy) and the need to address the emerging priorities stemming from the impact of the global pandemic (building back better). Previous Bank experience. The Bank has been heavily engaged in infrastructure development in Serbia through investment support to national road construction and support to energy sector reforms. On the transport side, the Bank supported technical work to mainstream Performance Based Maintenance Contracting (PBMC) in the road sector under the Transport Sector Rehabilitation Project (2004 - 2012). The Bank supported the Government in introducing DLIs in the Road Rehabilitation and Safety Project (2013 - 2020). Alongside these projects, the Bank financed the Corridor X Highway Project (2010 - 2019) to complement the government's efforts to improve the quality of roads. Previous Bank engagements in the transport sector indicated that although Serbia had a well-defined roads maintenance program, the portfolio was underfunded, contracts were not tendered competitively and there was no programmatic results-oriented approach to maintenance. Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760) In the energy sector, the Bank financed the Serbia Energy Efficiency Project (2004 - 2013), which included renovation of 82 public buildings and supported energy efficiency. This project however, did not achieve significant scale (renovating less than a percent of the public building stock) and not sustainable, as no further government spending was made available to support new investments. This Program for Results (PforR) operation aimed at continuing WB's involvement in the infrastructure sector by improving the management of the transport and energy sectors, while seeking concurrent enhancements in the public institutional capacity and long-term sustainability of the sectors. The priority areas selected were consistent with the standards that Serbia had to meet for joining the EU in the energy and transport sectors. Relevance of the PforR lending instrument. The Bank deemed the PforR as appropriate for the following reasons: (i) the operation supported ongoing government infrastructure programs, as opposed to implementing a few large investments: (ii) the Bank experience was expected to add value to improve the efficiency, effectiveness and sustainability of the program; (iii) The program results were measurable and aligned with Serbia's energy savings and greenhouse gas emission reduction targets and commercialization of road maintenance practices: (iv) Given the needs for public buildings and limited scope offered by a traditional Investment Policy Financing (IPF) operation, a PforR was expected to have the best opportunity for scale, as this was a national program; and (v) Serbia had previous experience with the lending instrument. That said, the operation which was a multi-sectoral program, encompassing energy efficiency and road maintenance objectives was overly ambitious. The unrelated scope and lack of synergy between the sectors was challenging, given the weak implementation capacities. This was particularly so since the Bank financing for this operation did not specifically earmark funds for technical assistance (TA) activities for providing implementation support. Relevance of the PDO. The PDO continues to be relevant for the Government strategy and the current WB strategy for Serbia. However, the PDO with activities in the road and energy sectors were however ambitious in the country context. Therefore, the relevance of the PDO is rated as substantial. The PforR comprised ten disbursement-linked indicators (DLIs) (six for transport and four for energy efficiency) (discussed below). Rating Relevance TBL Rating Substantial b. Relevance of DLIs New DLI RELEVANCE TBL DLI 1 DLI The Public Enterprise Roads of Serbia (PERS) completes maintenance of 1,000 km of the national road network (using the traditional approach). Rationale Under this approach, PERS had to ensure that no potholes remain open, there are no edge breaks between the pavement and shoulders and that the horizontal markings and vertical signs are installed to comply with Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760) critical road safety requirements. This indicator was pertinent to the PDO. However, this indicator was modest in terms of the level of ambition, as the traditional road maintenance are routine practices for PERS. Rating Substantial DLI 2 DLI Service Level Agreement (SLA) between PERS and the Ministry of Communication, Transport and Infrastructure (MCTI) is signed. Rationale SLA between MCTI and PERS, articulating the terms and support the MCTI would provide to PERS in return for meeting road maintenance service levels for a specified time (typically three to five years), was appropriate as it signaled the MCTI's intention to ensure sustainability of the road maintenance program. This indicator was verifiable. Rating High DLI 3 DLI PERS adopts its three year Performance based Maintenance Plan. Rationale Adoption of the three year performance based road maintenance plan, following the signing of the SLA between the MCTI and PERS was in line with the PERS Long and Medium-Term business strategies for 2017-2021. Such a plan was likely to enhance the institutional capacity for maintaining the national road network in the medium term. Rating Substantial DLI 4 DLI PERS completes maintenance of 2,000 km of the national road network using enhanced road maintenance contract. Rationale Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760) This DLI was linked to maintenance of 2,000 km of the road network using enhanced road maintenance contract in the second year of program implementation. This indicator was not clearly defined and lacked precision as to what was meant by 'enhanced road maintenance contract'. Rating Modest DLI 5 DLI PERS adopts the asset management plan. Rationale This DLI was linked to the adoption of an asset management plan integrating management, financial, economic and engineering practices. Such a plan would likely help is designing similar projects in future and was consistent with the standards Serbia had to meet for EU entry. This DLI was directly linked to the PDO of improving the institutional capacity for managing the national road network. Rating Substantial DLI 6 DLI At least 5,000 km of the national road work managed by PERS is maintained using Performance-based management contract (PBMC). Rationale This DLI was linked to maintaining 5,000 km of the road network using PBMC, following the signing of the SLA between the MPTI and PERS. The Bank financing for the operation covered the third year and the financing for the fourth and fifth year would be covered under the terms of SLA agreement between the MPTI and PERS. This indicator was appropriate as it signaled the government's intention to move towards results- oriented environment for road maintenance. The indicator although measurable, proved to be ambitious, given the weak implementation capacity. Rating Substantial DLI 7 DLI The Management Office of the Public Investment Management Office (PIMO) adopts and publishes a Program Operation Manual (POM) for the energy sector. Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760) Rationale The POM was expected to aid in setting up energy efficiency standards and procedures that were aligned with the legal framework and best practices. This could be useful for designing similar projects in the energy sector. Rating Substantial DLI 8 DLI The Government issues and publishes in the Official Gazette a decision on the PIMO's adoption of a medium- term national plan for renovation of public buildings. Rationale The medium-term plan for renovating public buildings was likely to help in institutionalizing a strategic roadmap for scaling up energy efficiency investments. This indicator was appropriate for improving the institutional capacity for managing the energy sector. Rating Substantial DLI 9 DLI The PIMO designs and operationalizes a consolidated Program Monitoring and Evaluation (M&E) system. Rationale While the medium-term plan for the energy sector would help in scaling up energy efficiency investments, a consolidated Program M&E system would likely help in complementing this effort by building the institutional capacity for implementing and managing energy projects and likely to help in the accountability of energy sector investments. Rating Substantial DLI 10 DLI Renovation of 234 public buildings that (a) had acceptance reports signed and issued; (ii) had acceptance reports that showed class C energy performance or are two classes higher than the class of the respective pre- renovated building; and (iii) had paid all the renovation invoices. Rationale Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760) This DLI was linked to renovating 234 public buildings identified by the municipalities as priorities and approved by the Government. This DLI was linked to concrete results and likely to help in enhancing the energy efficiency of public buildings and in decreasing greenhouse gas (GHG) emissions (GHC). The results could be verified. Rating High OVERALL RELEVANCE DLI TBL OVERALL RELEVANCE RATING Rationale The program included specific DLIs to foster institutional development and sustain the reforms envisaged under the government program. On the transport side DLIs 2, 3, 4, and 5 focused on the institutional framework needed to operate road maintenance on a performance-based approach to road maintenance. DLIs 7,8 and 9 aimed at developing the required planning and tools for the energy sector. For the most part, DLIs monitored critical milestones for achieving the PDO. However, there were some shortcomings. One DLI was not defined properly and some DLIs were not set at the right level of ambition. And more important, some DLIs were interlocked, affecting progress in achieving the Program objectives. By design, some DLIs were contingent upon others, that created a cascading effect that impacted the achievement of results. For instance, implementing PBMCs were contingent upon the signing of the SLA between MCTI and PERS and delays in signing the SLA affected the implementation and associated disbursements. Overall relevance is rated substantial, based on the rating for relevance of the objective and relevance of the DLIs. Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To improve the management and sustainability of select public infrastructure (road sector) by strengthening government capacity and systems, upgrading assets and increasing expenditure efficiency. Rationale Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760) Theory of change. The outputs such as maintaining 8,000 km of national roads (1,000 km using traditional maintenance, 2,000 km using enhanced maintenance and 5,000 km using PBMC) were likely to aid in strengthening the government capacity for maintaining national roads. The outputs of activities such as adopting a SLA between the MCTI and PERS, adopting a three year performance-based maintenance plan and a road asset management plan were likely to contribute to improving the government capacity for managing the sustainability of the road network. The causal links between activities, outputs and intended outcomes were clear and the outcomes were monitorable. The outcomes were predicated on these assumptions: (i) PERS toll revenues remains stable; and (ii) PERS has the capacity to implement PMBC. These assumptions proved to be unrealistic as the toll revenues of PERS did not remain stable and PERS lacked the capacity to implement PMBC. Outputs.  15,274 km of the road survey was completed, exceeding the target of 14,894 km.  The Board of PERS adopted a three year business plan in December 25, 2023 as originally targeted.  An asset management plan was adopted by PERS in December 2023 to utilize PERS physical assets in a cost-effective way by integrating evidence-based data in its management, financial, economic and engineering practices, as originally targeted.  3,000 km of the national roads were maintained, far short of the target of 8,000 km. This included 1,000 km under the traditional maintenance approach and 2,000 under the enhanced maintenance approach. As the PBMC was not implemented, the program was not able to achieve the original target of 5,000 km of roads under the PBMC.  The road safety inspection was not carried out and road safety measures were not implemented as target, as the PBMC was not implemented.  A rule requiring environmental compliance reports as a condition for payments to contractors were not completed. Outcomes. The outputs were expected to contribute to two desired outcomes: (i) enhanced motorist satisfaction through better pavement condition; and (ii) The SLA between the MCTI and PERS is implemented.  Enhanced motorist satisfaction through better pavement condition was not measured and ultimately not achieved, as the Program failed to rehabilitate 5,000 km of the road network using PBMC.  Implementation of the SLA was not achieved, although the SLA was signed on December 20, 2023. At the time of Program closure, several requirements outlined in the SLA, including the allocation of a dedicated budget, had not been met. Efficacy of this PDO is rated as Modest, as the Program fell short of achieving the PDO indicator targets. Rating Modest Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760) OBJECTIVE 1 REVISION 1 Revised Objective To improve the management and sustainability of select public infrastructure (road sector) by strengthening government capacity and systems, upgrading assets and increasing expenditure efficiency. Revised Rationale Same as above since there was no revision to the original targets. Revised Rating Modest OBJECTIVE 2 Objective PDO 2: To improve the management and sustainability of select public infrastructure (energy sector) by strengthening government capacity and systems, upgrading assets and increasing expenditure efficiency. Rationale Theory of change. The outputs such as renovating public buildings to achieve class C energy performance level were likely to aid in strengthening the government capacity for meeting the energy efficiency standards in the sector. The outputs of activities such as adopting a program operations manual, developing and adopting a medium-term national plan for renovating public buildings, implementing a comprehensive monitoring and evaluation (M&E) system, were likely to contribute to the intended objective of improving the government capacity for managing the sustainability of the energy sector. The causal links between activities, outputs and intended outcomes were clear and the outcomes were monitorable. The outcomes were predicated on these assumptions: (i) There is strong political will to support medium-term building initiatives. and (ii) Building renovation costs stay predictable during implementation. These assumptions proved to be unrealistic in the face of wavering political will and cost overruns for renovating public buildings which resulted in significant reduction in the number of buildings that were renovated. Outputs.  293 bidding documents (including technical designs) were completed and approved, exceeding the original target of 234.  Government adopted a Long-Term Building Renovation Strategy as originally targeted.  The PIMO designed and implemented a consolidated Program M&E system as originally targeted. Outcomes. The outputs described above were expected to have the following outcomes: (i) Projected lifetime energy savings due to renovation of old buildings; (ii) The number of renovated buildings that met Class C (or 2 classes higher) energy performance certificates with final acceptance report; (iii) reduce carbon dioxide emissions from energy saved in renovated building; and (iv) Number of direct project beneficiaries. Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760)  1,145,119 MWh of lifetime energy savings were realized when the operation finished, exceeding the original and revised targets of 1,100,000 MWh, respectively. (The original target for this indicator was revised downwards, with the project restructuring). However, the original target was exceeded when the project closed. The ICR (para 35) acknowledges that although lifetime energy savings exceeded the original target despite the reduction in the number of renovated buildings, suggests that the original target was underestimated.  182 buildings were renovated that met Class C standards, short of the original target of 234.  There was reduction of 18,773 tons of Carbon dioxide emissions from energy saved in renovated buildings, slightly short of the original target of 19,000.  3,674,987 people directly benefitted from Program activities, exceeding the original target of 1,381,000. 52.5% of the beneficiaries were females, exceeding the target of 51%.  85.8% of the stakeholders expressed satisfaction when the project closed, short of the original target of 98%. Efficacy of this PDO is rated as modest before project restructuring, as three of the original targets were not realized and the original target for one indicator may have been underestimated. Rating Modest OBJECTIVE 2 REVISION 1 Revised Objective The PDO was not revised. However, the targets in the energy sector were reduced with the restructuring of the project. Revised Rationale Outputs. as described above. Outcomes.  1,145,119 MWh of lifetime energy savings were realized when the operation, exceeding the revised target of 858,000 MWh.  182 buildings were renovated that met Class C standards as per the revised target.  There was reduction of 18,773 tons of Carbon dioxide emissions from energy saved in renovated buildings, exceeding the revised target of 14,820.  3,674,987 people directly benefitted from Program activities, exceeding the revised target of 1,077,181. 52.5% of the beneficiaries were females, exceeding the target of 51%.  85.8% of the stakeholders expressed satisfaction when the project closed, short of the original target of 98%. Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760) Efficacy of the PDO is rated as substantial after project restructuring, as the revised targets were realized for the most part. Revised Rating Substantial OVERALL EFFICACY Rationale The achievement of the PDO in the road sector and energy was modest before restructuring. Thus, the overall efficacy is rated as Modest. Rating Primary Reason Modest Low achievement OVERALL EFFICACY REVISION 1 Revised Rationale The achievement of the PDO in the road sector was modes after restructuring. The achievements of the PDO in the energy sector was substantial after restructuring, as the revised targets were realized for the most part. Overall, the efficacy is rated as Modest given the significant shortcomings in achievement of the first aspect of the PDO. Revised Rating Primary Reason Modest Low achievement 5. Outcome Relevance of the PDO to the Government and Bank strategy is rated as Substantial. Efficacy of PDO 1 was rated as modest, given the low achievement, both before and after restructuring. Efficacy of PDO 1 was modest before restructuring and substantial after restructuring, as the revised targets were realized. Taking the share of disbursement before and after restructuring into account, the overall outcome is Moderately Unsatisfactory. Table: Overall Outcome Rating Using Split Rating Methodology Original PDO and Revised PDO and targets targets Relevance of PDO Substantial PDO 1. Road Maintenance Modest Modest Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760) PDO 2. Energy Efficiency Modest Substantial Overall Efficacy (PDO) Modest Modest Moderately Outcome ratings Moderately Unsatisfactory Unsatisfactory Numerical value of the outcome ratings 3 3 Share of disbursement 0.74 0.26 Weighted value of the outcome rating 2.22 0.78 Final Outcome rating 3 = Moderately Unsatisfactory Outcome Rating Moderately Unsatisfactory 6. Risk to Development Outcome Political commitment. There is risk to the sustainability of development outcomes, particularly in the road sector. Although the SLA between the MCTI and PERS was signed before the operation closed, there is dedicated source of government funding yet for road maintenance activities. Given that strong political commitments in terms of allocation of resources in required for PBMC, frequent Government changes and high staff turnover could undermine the technical gains achieved through Program implementation. Financial risk. There is risk that political commitment to sensitive reforms, notably on the green transition front may waver is the face of budget constraints. 7. Assessment of Bank Performance a. Quality-at-Entry The Bank prepared this operation based on the experiences from previous Bank-financed transport projects in Serbia and from PforR operations financed by the Bank in Nepal and Uruguay. The analytical underpinnings of the operation were sound. The Bank conducted an economic analysis for the operation based on a sample of the road network and public buildings. The analysis concluded that the economic benefits far outweighed the costs. The implementation arrangements were appropriate. For the transport sector, PERS, the entity responsible for managing the road network, was in charge day-to-day road sector work. For the energy sector, the Public Investment Management Office (PIMO) was responsible for program coordination among the involved government entities. Both PERS and PIMO had previous experience in implementing similar programs and had worked with the WB. The arrangements made at appraisal for safeguards and fiduciary compliance were appropriate (discussed in section 10). There were significant shortcomings at the Quality-at-Entry: One, the multi-sector operation was overly ambitious. The unrelated scope and lack of synergy between sectors was challenging in the country context, with weak implementation capacities. Moreover, the project design did not include TA activities to improve the implementation capacities. Two, although the operation identified risks associated with Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760) technical design and implementation capacities, the mitigation measures were inadequate. This was particularly so as the Bank financing for this operation did not earmark funds for TA activities to support implementation. Three, the design included advance payments to facilitate disbursements. However, the advance payments were not earmarked to PERS. This affected the PERS cash flow and their ability to deliver the agreed results for some DLIs. Four, there were cost overruns, with the costs of renovating buildings higher during the first phase of Program implementation due to the rise in construction cost. The cost escalations placed a burden on PIMO's budgetary allocation (as the Bank contribution to the Program was fixed). The cost overruns undermined the effectiveness of the original plan. and five, there were M&E design shortcomings (discussed in section 8). Due to the shortcomings mentioned above, Bank performance at Quality-at-Entry is rated as Moderately Unsatisfactory. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision Eleven Implementation Status and Results (ISR) reports were conducted over the project lifetime, implying twice a year supervision missions. The ICR (para 77) noted that the team's familiarity with the client contributed to a stable working relations. The Bank also made the judicious decision to restructure the project twice, in the face of realities on the ground. The support provided by the team aided in fiduciary compliance (discussed in section 10). There were significant supervision shortcomings. The ICR (para 76) noted that although several missions were conducted during Program implementation, the number of Aide Memoires (AM) and Management Letters (MLs) documenting these missions was relatively low. The Energy and Transport had co-Task Team Leaders (TTLs) to ensure sector-specific technical support and supervision. However, there were frequent staff turnovers and each of the co TTLs was replaced three times from the preparation phase to the final year. Moreover, none of the co TTLs was placed in Serbia, which hindered the Bank's ability to proactively push for reforms. There were delays in holding the Mid-Term Review (MTR), with the MTR conducted just a year before project closure. The delays in holding the MTR on time undermined the Bank supervision team to take appropriate corrective actions, particularly in the road sector. Bank performance during supervision was rated as Moderately Unsatisfactory. Overall Bank performance is rated as moderately unsatisfactory due to the significant shortcomings both at Quality-at-Entry and during supervision. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760) Moderately Unsatisfactory 8. M&E Design, Implementation, & Utilization a. M&E Design There were conceptualization issues in the results framework, which compromised the final assessment of the results achieved by the Program. Some indicators were not clearly articulated. For instance, while both DLI 2 and PDO indicator focused on the SLA, the DLI was linked to "signing of the SLA" while the PDO indicator measured "implemented SLA". As a result indicator 2 was not achieved, although the related DLI was completed (the SLA was signed by project closure). The indicator pertaining to 'enhanced road maintenance" was not defined. The results indicator included some indicators that were not easy to measure such as the PDO pertaining to enhanced motorist satisfaction. b. M&E Implementation The deficiencies in M&E design were not rectified during implementation. The ICR (para 58) noted that the M&E implementation was not consistent across sectors. While annual progress reports were submitted by PIMO consistently, the lack of reporting from PERS hindered overall Program monitoring. M&E implementation was constrained by data limitations. On the transport side, it was not possible to measure some of the results indicators such as motorist satisfaction. On the energy side, the discontinuation of the social survey after 2020, made it difficult to assess progress in the energy sector. c. M&E Utilization While the progress reports were a useful information source to support M&E of implementation progress, they were not systematically used to inform Program management and decision-making. The ICR (para 61) acknowledges that while most implementation challenges were clearly noted, issues were not always promptly addressed. This resulted in cumulative delays that required higher level attention in the final year of implementation. Overall M&E is rated as Modest, due to shortcomings in design, implementation and utilization. M&E Quality Rating Modest 9. Other Issues a. Safeguards Environmental and Social Risks. The Environment and Social Systems Assessment (ESSA) that was conducted at appraisal, concluded that the Program would have long-term positive and social impacts, while the negative impacts would be short-term and linked to the rehabilitation works. On the social side, the negative impacts were expected to be site-specific (such as occupational, health and safety risks). On the Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760) environmental side, the risks related to the negative impacts of construction works, including noise/dust emissions and waste management related risks. The performance of the environmental and social systems of the Program was rated as Moderately Satisfactory during implementation (ICR, para 63). Most Project-Affected Persons (PAP) actions were implemented. However, the Program failed to conduct the annual social survey on the energy efficiency component, which was discontinued following the outbreak of the COVID-19 pandemic. On the transport side, the action related to introducing environmental compliance as an obligatory precondition for making payments to contractors was not achieved (ICR, para 65). b. Fiduciary Compliance Financial management (FM). The Bank conducted an assessment of the FM systems of the Program. The assessment concluded that the systems were adequate for the use of Program funds (PAD, para 109). FM was satisfactory during implementation (ICR, para 70). The ICR also noted that budgeting was credible, and levels of expenditure were sufficient and exceeded the amounts disbursed against the DLS. Audit reports were received in a timely fashion and these reports did not raise any issues. One weakness during implementation was that the implementing agencies did not deliver in-year financial reports to the Bank in a regular manner (a legal covenant), necessitating the Bank team to invest additional efforts for obtaining the reports. Procurement. The Bank conducted a procurement assessment of the implementing agencies at appraisal. The assessment concluded that Serbia's procurement systems for the Program were adequate for implementing the program (PAD, para 118). Procurement performance was assessed as moderately satisfactory (ICR, para 71). A new public Public Procurement Law enacted in 2020 put in place an electronic system for bid submissions. Restrictions on the use of public funds imposed by the Ministry of Finance in the same year caused delays in procurement processes, particularly affecting civil works. c. Unintended impacts (Positive or Negative) There were no unintended impacts. d. Other Not applicable. Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760) 10. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome Unsatisfactory Unsatisfactory Moderately Moderately Bank Performance Unsatisfactory Unsatisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 11. Lessons The ICR draws the following two main lessons from the experience of implementing this PforR operation, with some adaptation of language. 1. Allocation of Bank funds for Technical Assistance (TA) may raise the potential for success of PforR operations. This program relied heavily on the government's continued ownership and political commitment to the reform agenda. However, circumstances evolved rapidly and opportunities that were available at appraisal became limited during implementation. Lack of dedicated funds for TA hindered the capacity of the Program to adapt to these changes. The lesson is that TA activities may be especially helpful for PforR operation, especially in operations involving ambitious multi-sector operations. 2. Linking Program disbursements to the dedicated account of the implementing institutions can help in PforR operations. Although the design of this operation had facilities for advance payments to move the reforms forward, disbursements were not earmarked to the implementing agencies, which resulted in liquidity constraints and disincentives for the timely delivery of activities. This ultimately affected the achievement of agreed targets. The lesson is that since PforR operations rely heavily on financial incentives to drive results, ensuring that disbursements remain closely tied to the agencies that are in charge of Program implementation, PforR can better align incentives and results. 3, A careful consideration of the links between activities may help in reducing implementation challenges on the ground. This project encompassed energy efficiency and road maintenance activities. The largely unrelated scope of the activities posed significant challenges during preparation, as the activities seemed more like separate operations rather than complementary elements of a cohesive program. The lack of synergy made it difficult to develop a focused PDO, without being overly broad and to create a unified results framework that effectively addressed the objectives of both sectors. 12. Assessment Recommended? Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RMPBEE Program (P163760) No 13. Comments on Quality of ICR The ICR is clear and well-written. The theory of change articulated in the text clearly shows the causal links between the activities, outputs and intended outcomes. The theory of change explicitly states the assumptions underlying the operation. The ICR provides adequate evidence to assess the Program's outcomes. The ICR draws reasonably good lessons from the experience of implementing this operation. Overall, the quality of the ICR is rated as Substantial. a. Quality of ICR Rating Substantial Page 19 of 19