Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) Report Number: ICRR0024218 1. Project Data Project ID Project Name P120219 Cultural Heritage 2 Country Practice Area(Lead) Russian Federation Social Sustainability and Inclusion L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-79990 31-Jul-2017 95,282,694.73 Bank Approval Date Closing Date (Actual) 07-Dec-2010 30-Jun-2022 IBRD/IDA (USD) Grants (USD) Original Commitment 100,000,000.00 0.00 Revised Commitment 95,282,694.73 0.00 Actual 95,282,694.73 0.00 Prepared by Reviewed by ICR Review Coordinator Group Alejandra Eguiluz Chikako Miwa Avjeet Singh IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The PDO stated in the Loan Agreement (7999-RU, schedule I) is the following: "(i) to support the conservation, safekeeping and promotion of cultural heritage assets in Leningrad, Pskov, Novgorod and Tver oblasts of the Borrower; and (ii) to strengthen regional capacity for cultural heritage management so it supports the socio-economic development of the Participating Oblasts". This formulation is almost identical to the one in the PAD (57609-RU, page 3): "(i) to support the conservation, safekeeping, and promotion of cultural heritage assets in Leningrad, Pskov, Novgorod and Tver oblasts of the Russian Federation and (ii) to strengthen regional capacity for cultural heritage management so it supports the socio-economic development Page 1 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) of the participating oblasts". The sole difference between the two formulations is that in the PAD the mention to "the Borrower" has been replaced by “the Russian Federation”. The PDO will be parsed, for the purposes of this review, as follows:  To support the conservation and promotion of cultural heritage assets in Leningrad, Pskov, Novgorod and Tver oblasts of the Russian Federation  To support the safekeeping of cultural heritage assets in Leningrad, Pskov, Novgorod and Tver oblasts of the Russian Federation  To strengthen regional capacity for cultural heritage management so it supports the socio-economic development of the participating oblasts Differently from what the ICR proposes as PDO parsing, the two objectives of supporting the conservation and the promotion are left together under the same PDO1, given that the two goals have shared activities and outputs. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? No d. Components Component 1. Integrated Heritage Site Development (cost at appraisal: US$ 86.43 million; actual cost: US$ 81.54 million). This component was implemented through a demand-driven mechanism that supported, on a competitive basis, sub-project proposals made by participating regions (oblasts) and their Cultural Institutions (CI). It consisted of two sub-components: Subcomponent 1(a). Rehabilitation and improvement of cultural heritage sites and institutions. This subcomponent comprised the support to the subprojects for the rehabilitation, improvement, safekeeping and conservation of cultural heritage sites. The link to broader development activities had to be shown in a site development and management plan, in collaboration with all relevant stakeholders. There were two windows for funding: one for large investment programs (on infrastructures), presented by the regions and one for smaller local subprojects presented by state cultural institutions, with a more rapid implementation. Sub-component 1b: Capacity building for integrated site development. This sub-component built the capacity of oblasts, municipalities and state cultural institutions to improve comprehensive cultural development planning. It comprised activities linking cultural heritage with economic and social development; training and capacity building specific to subprojects’ investments and additional support for effective implementation; and opportunities for exchange of experiences between oblasts, municipalities and state cultural institutions. Page 2 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) Component 2: Protection of Museum Assets (cost at appraisal: US$ 7.64 million; actual cost: US$ 7 million). This component was designed to support the Ministry of Culture’s (MoC) efforts to protect and promote the moveable heritage assets. It comprised two inter-related sub-components: Sub-Component 2a: Improvement of collection management, conservation and storage, was designed to improve the management of museum collections of the Leningrad Oblast from a conservation and storage perspective. The activities included: (i) an in-depth institutional and physical needs assessment for the development of a conservation and storage strategy, (ii) targeted technical assistance to improve the management of collections and (iii) the construction of a Multifunctional Museum Facility, that would be used as a conservation, research and training center and will serve as a regional hub for providing professional art services. Sub-Component 2b: Development and testing of Museum security practice, with the following activities: (i) the elaboration of a set of practical guidelines for security by category of venues, and (ii) the installation of security equipment in a selected number of venues for testing. The institutions were classified and the specifications on security systems and training needs were integrated into the guidelines. Component 3: Project Management, Monitoring and Evaluation (at appraisal: US$ 5.93 million; actual: US$ 6.75 million). The St. Petersburg Fund for Investment projects (FISP) had been the lead agency for project preparation and was designated as the implementation agency. It worked with all oblasts to ensure effective and continuous communication with all stakeholders and close support for sub-project preparation and implementation. M&E activities were included under this component and were considered key given the demonstrative nature of the project. They included simple monitoring for each subproject. Changes to components The scope of Component 3 was expanded twice at the request of the Borrower, to include the preparation of two future related projects that were later on not financed by the Bank due to the changes in the geopolitical situation. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. At appraisal, the total project cost was estimated at US$ 100 million. A revision took place in 2022 and the cost was revised to US$ 95.28 million. The remaining amount (US$4.72 million) was canceled. The actual project cost was US$ 95.28 million, which represents 95.3% of the original amount and 100% of the revised amount. Financing. The Project was financed with an IBRD loan of US$ 100 million (7999-RU). Borrower contribution. At appraisal, the planned borrower contribution was US$ 150.50 million, according to the PAD’s financing plan. The actual borrower contribution was US$ 243.58 million (according to the ICR, page ii) Dates Page 3 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) The project was approved in December 2010, became effective in June 2011, and had the MTR in April 2014. The original closing date was July 31, 2017 and the actual closing date was June 30, 2022. In total, the project duration was extended for 59 months. In March 2022, in light of the deteriorating operational environment resulting from the sanctions imposed on Russian financial institutions by a significant number of countries, and substantial restrictions on travel and access to materials and technology, the Bank announced the suspension of disbursements. The project underwent the following 5 restructurings:  First restructuring, in July 2013, reflecting the increase in the Borrower’s contribution by US$ 3 million, to finance the preparation of a new cultural heritage investment project (Small Historic Settlements Project), activity that was included under component 3.  Second restructuring, in March 2015, to reflect an increase in the Borrower’s contribution of US$40 million, with disbursement and project implementation schedules adjusted accordingly. The Borrower’s increase was to finance: (i) the construction of the Multifunctional Museum Facility (MMF), considered by the MoC as one the key activities under the project, devoting US$ 30 million to it; (ii) provide an additional US$4.2 million for preparation of bidding documents for the contracts under the anticipated St. Petersburg Economic Development Project (SPEDP) Additional Financing (AF) that was foreseen at the time to be financed by the WB; and (iii) secure the needed funds for proper PIU functioning during the extended project implementation period. The project was extended to June 30, 2019, to secure the time required for the MMF construction, the indicators end date were updated to reflect the extended loan closing date.  Third restructuring, in February 2018, because of the increase in the Borrower’s financing by US$65 million (reaching US$258.5 million), with budget reallocation and adjustments of disbursement and project implementation schedules. The increase was to secure the completion of a total of 14 Low- Window (LW) subprojects and finance the 5th round of the Small window subprojects. The project was extended to June 30, 2022. The target values of the PDO indicators increased to capture the project results during the extended period, as follows: (i) the target value of the PDO indicator on number of subprojects implemented increased from 48 to 82; (ii) the target value of the share of stakeholders’ co-financing increased from 15 percent to 20 percent; (iii) the target value of the number of visitors to the subprojects selected during the 1st year of the Project increased from 1,935,079 visitors (calculated as 20 percent increase from the baseline of 1,612,566 visitors) to 3 million visitors (corresponds to 86 percent increase over the baseline); the target value of the level of stakeholders’ satisfaction increased from 65 percent to 75 percent. The target values of intermediate results indicators on LW and SW subprojects increased accordingly. The end dates of all indicators were updated to reflect the extended closing date.  Fourth restructuring, in January 2019, reflected the decrease in the Borrower’s contribution, by US$44.4 million, to stay at a total of US$214.1 million, in connection with the MoC decision to exclude financing of the restoration works for three of the previously approved LW subprojects, with a view to optimize the use of the federal budget funds. This brought the total number of LW financed subprojects from 14 to 11. It adjusted the target value of the PDO Indicator on number of subprojects implemented, from 82 to 79. The target values of the related intermediate results indicators were adjusted accordingly.  Fifth restructuring, in April 2020, to reflect the Borrower’s contribution increase, by US$22.3 million (reaching US$236.4 million), to secure sufficient funds for successful completion of all ongoing LW subprojects, to enable efficiencies in use of the Project funds already invested in restoration Page 4 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) contracts and generate benefits from expanding the range of services available to visitors of Cultural Heritage (CH) sites. A split evaluation is not necessary, as the successive changes in the Borrower’s contribution and project extension translated into adjustments to the targets (mainly increases) without substantially affecting the achievement of the PDO or the PDO-level indicators. 3. Relevance of Objectives Rationale Context at appraisal Following the 2008 global financial crisis, the Russian Government had adopted a strategic approach to further improve the investment climate and the quality of public administration in support of economic modernization, with specific efforts to diversify growth and share the benefits of development more widely across Russia’s regions and population. The approach considered that Russian cultural heritage (CH) could play an important role in this diversification, and stressed the importance of building on Russian regions’ cultural potential to advocate for actively preserving and conserving Russia’s CH, integrating CH into tourism development strategies, developing capacity to manage cultural sites, supporting integrated local development, promoting Russian CH abroad, increasing the number of users of cultural services in Russia, and ensuring the dynamic development of public/private partnerships to support the development of cultural activities. Due to substantial development challenges remaining in many Russian regions, the Government requested the World Bank (WB) to increase its support to the regions. The Government’s focus on CH was not new given its previous engagements with the WB. The Project aimed to support the Ministry of Culture (MoC) of the Russian Federation to implement the relevant CH objectives of the Concept for Long-Term Social and Economic Development of the Russian Federation 2020, which referred to the need to enhance the regional cultural potential as one of the regional means for socio-economic development. The Project aimed specifically at developing CH as one of the potential drivers for economic diversification of the four regions between St. Petersburg and Moscow (Leningrad, Novgorod, Pskov, and Tver Oblasts), by enhancing regional institutional capacity within the CH sector. Enhancing the use and accessibility of the CH assets would contribute to fostering the diversification of the local economy through promoting CH-based tourism and creating new job opportunities in the cultural institutions and beyond. Consistency with Country Strategy At appraisal the project was aligned with the objectives of the Concept for the Long-Term Social and Economic Development of the Russian Federation (RF) to 2020, where culture is referred to as a strategic resource for the economy, expressing support for developing Russian regions’ cultural potential (PAD, page 2). The Project objectives are directly aligned with existing federal and national programs, including the “Conservation of the Cultural and Historic Heritage” and the Program “Development of Museum Activities” that are to be carried out from 2022-2030 (ICR, page 7). It is also aligned with the national program Page 5 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) “Culture”, approved in 2018. The PDO remains relevant for the participating regions and aligned with their regional programs, such as the “State Program on preservation, promotion and protection of cultural heritage in Tver oblast for 2023-2028”, “State Program on culture, cultural heritage protection and tourism development in Pskov oblast for 2023 and planned period of 2024 and 2025”, “State Program on Culture and Archives Development in Novgorod oblast for 2019-2024”, “State Program on development of culture in Leningrad oblast for 2018-2024”. The overall increase in the Borrower’s contribution (US$150.5 million to US$243.58 million) proved the strong support from the Government and the project’s alignment with national and regional priorities. Consistency with Bank strategy The PDO is aligned to the World Bank’s Country Partnership Strategy (CPS) for Russia 2012-2016 (65115- RU, latest available), more specifically to two strategic focus areas: (i) increasing growth and diversification through better management of public finances and a stronger financial sector, and (ii) improving governance and transparency through more accountability and better service standards in public administration, procurement, and financial management. The Bank’s subsequent 2016 Systematic Country Diagnostic (110765-RU) further demonstrated the relevance of the PDO as it highlighted Russia’s need to diversify its sources of growth, inter alia, through improving public investment management. It specifically emphasized that important service providers, such as in tourism, have great potential to benefit from productivity-enhancing policies, especially the removal of barriers to allow entry of innovative and productive firms. Previous Bank experience The PAD (page 2) describes how the Bank had previously financed the St. Petersburg Economic Development Project (SPEDP) and its predecessor, the St. Petersburg City Center Rehabilitation Project. The team prepared the project with the St. Petersburg Foundation for Investment Projects (FISP), which had been the PIU in several Bank-financed projects for over a decade. The Bank had done extensive economic and sector work on the sub-national fiscal issues and development priorities, local governance and civic engagement in rural areas (Local Government and Civic Engagement project), which helped identify upfront all key stakeholders and target beneficiaries in the Participating oblasts and engage them early on to build a transparent framework for stakeholders’ participation in the subproject development and implementation. PDO level. The PDO ((i) to support the conservation, safekeeping and promotion of cultural heritage assets in Leningrad, Pskov, Novgorod and Tver oblasts of the Borrower; and (ii) to strengthen regional capacity for cultural heritage management so it supports the socio-economic development of the Participating Oblasts) was outcome oriented. The outcome of improved socio-economic development was ambitious and could have benefitted from more specificity in its definition (more jobs, increased quality/access to cultural services, etc). Overall the relevance of the objectives is deemed Substantial Rating Relevance TBL Rating Substantial Page 6 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To support the conservation and promotion of cultural heritage assets in Leningrad, Pskov, Novgorod and Tver oblasts. Rationale Theory of Change (ToC) A ToC was absent in the PAD, but the ICR presents the following (page 2): The selection and financing of subprojects to rehabilitate and improve CH sites (activity), the investments in museum collection management, conservation and storage (activity), the development and testing of security guidelines and practices (activity), and the trainings/study visits and seminars for integrated heritage site development (activity), all lead to having targeted cultural sites rehabilitated (output), the museum collections conservation and security improved (output) and improved HR and organizational capacity of public authorities and cultural institutions for investment management, preservation and promotion of CH (output). These outputs lead in turn to the outcomes of: (i) enhanced conservation of CH assets; (ii) enhanced safekeeping of the CH assets; (iii) enhanced promotion of CH assets; and (iv) strengthened regional capacity for CH management. These outcomes in turn lead to two higher outcomes: (i) improved CH public sector management and performance and (ii) diversified economy for sustainable development and growth. The underlying assumptions critical to the TOC (ICR, page 2) are the following: (i) the increased value and accessibility of CH sites can attract higher volumes of visitors from within and outside the regions; (ii) visitors are willing to pay for CH-based products; and (iii) regions and their CIs ensure sufficient funding to sustain the maintenance costs of improved CH sites. These assumptions are key and condition the possibility of the casual link established in the ToC. The chain is logical and a causal link can be established between the activities, outputs and outcomes. The outcome of "support the socio-economic development of the 4 participating regions" is not clearly deducted causally from the ToC. Several steps/concepts seem to be missing to establish that causal link: the improved management of cultural institutions would translate into more revenue for the institutions and more job creation. At the public/institutional level, the increased revenue should be able to revert to the regions in the form of more investment, or improved public services, that could be captured by specific socio-economic indicators showing the improved socio-economic development. These gaps in the ToC are linked to the assumptions established in the ToC of the ICR, thus showing the awareness of them. The ToC that applies more specifically to PDO1 is the following: The selection and financing of subprojects to rehabilitate and improve CH sites (activity), the investments in museum collection management, conservation and storage (activity), lead to having targeted cultural sites rehabilitated (output), the museum collections conservation and security improved (output), t, preservation and promotion of CH (output). These outputs led to the outcomes of: (i) enhanced conservation of CH assets; (ii) enhanced promotion of CH assets. Page 7 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) The outcomes of conversation and promotion are analyzed together as the same PDO1 because the activities and outputs leading to the two outcomes are shared Outputs  11 large window subprojects approved by the Inter-ministerial Committee (IMC), of a revised target of 11 and an original target of 8  76 small window subprojects approved by IMC, of a revised target of 68 and an original target of 40  60 implementation agreements and letter of intent signed for subprojects, of an original target of 48  28 regional museums of the region of Leningrad have improved their collections, thanks to the regional Multifunctional Museum Facility. This has given Leningrad Oblast’s cultural authorities and Leningrad Oblast Museum Agency a critical capability of improving the quality and efficiency of collection management (primarily in the field of preservation, storage and display of museum collections). Outcomes  82 cultural heritage preservation subprojects implemented, of a revised target of 79 (and an original target of 48). Achieved  2,944,251 visitors in selected supported cultural heritage sites and institutions, of a revised target of 3,000,000 (and an original target of 1,612,566). Almost achieved  40% share of stakeholder's co-financing to support the cultural heritage project investments, of a revised target of 20% and an original target of 15%. Achieved  88.1% of stakeholders are satisfied, of a revised target of 75% and an original target of 65%. Achieved Regarding the outcome indicator “number of visitors”, although the target has been almost achieved on this indicator, we don’t know if the same number of visitors would have been achieved without the project’s intervention. It would have been useful to include a counterfactual to better assess the project's contribution to the results achieved. The ICR reports that the 71 completed SW subprojects financed activities to: (a) install and utilize new visitor infrastructure, such as accessibility, welcome and orientation facilities for improved visitor comfort (16 subprojects); (b) modernize and expand museum expositions by deploying, e.g., multimedia and information and communications technologies (ICT) solutions for an increased appeal to younger audiences (29 subprojects); and (c) retrofit stage and performance equipment to broaden and enhance their cultural event offerings (25 subprojects). It is also reported that the SW subprojects leveraged co-financing to prepare new exhibits and visitor programs and other CH promotion activities. The ICR reports that the 11 LW subprojects comprised interventions on 66 structures/monuments/park estates in the 4 regions, adopting specific site management plans (SMPs) prepared in collaboration with all relevant stakeholders (i.e, regional and municipal authorities, and CIs), financial and staffing plans - thus indicating that the project helped enhance conservation of the CH. Overall, the efficacy of Objective 1 is rated as Substantial. Rating Page 8 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) Substantial OBJECTIVE 2 Objective To support the safekeeping of cultural heritage assets in Leningrad, Pskov, Novgorod and Tver oblasts. Rationale ToC The theory of change that applies to the PDO2 is the following: The selection and financing of subprojects to rehabilitate and improve CH sites (activity), the investments in museum collection management, conservation and storage (activity), and the development and testing of security guidelines and practices (activity), all lead to having targeted cultural sites rehabilitated (output), the museum collections conservation and security improved (output). These outputs lead to the outcome of enhanced safe-keeping of the CH assets. Outputs  16 pilots for security implemented  Recommendations to MOC based on testing experience of new security practices and lessons learned have been provided The 16 museums installed and pilot-tested various versions of a new comprehensive museum security system, ensuring better physical safety of their immovable and movable cultural assets against possible art crime, climate impacts, and destruction by natural disasters (fires and floods). The systems introduced mechanisms for intruder detection, automatic fire alarm and fire suppression, access control and management, video surveillance and lighting, among others. This allowed to organize new and more engaging exhibitions and enhance their cooperation and exchange of collections with other museums. Extensive operational testing of the integrated security systems in the 16 pilot museums allowed to validate the sectoral Museum Security Guidelines. They were approved by the MOC in 2014 for nationwide use by about 2500 federally owned museums of various categories for their auditing and insurance purposes. Outcome  82 cultural heritage preservation subprojects implemented, of a revised target of 79 (and an original target of 48). Achieved The ICR reports that of the 82 subprojects, the 66 built structures included appropriate access control and physical safety regimes compatible with new museum research and increased visitation activities. This allowed the CIs to organize new and more engaging exhibitions by opening to the public some of the most valuable pieces of their collections that previously could not be exhibited due to high security risks and often absence of appropriate climate control conditions. Moreover, the improved safekeeping and exhibition conditions enabled the regional CIs to host temporary/guest exhibitions from the leading Russian museums, thereby, improving the regional residents’ direct access to the nation’s most valuable CH items that was previously not available to them. The outcome and output information together give sufficient evidence of Page 9 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) efficacy for PDO2. It is clear how the subprojects enhanced safekeeping of CH. The rating is deemed Substantial. Rating Substantial OBJECTIVE 3 Objective To strengthen regional capacity for cultural heritage management so it supports the socio-economic development of the Participating Oblasts Rationale ToC The ToC for PDO3 is the following: The trainings/study visits and seminars for integrated heritage site development (activity), lead to trained personnel from the CIs and public authorities on investment management, preservation and promotion of CH (output), that in turn lead to a strengthened regional capacity for CH management (outcome) and the socioeconomic development of the participating regions (higher outcome). The causal link is clear in this ToC for the outcome of strengthened regional capacity for CH management, but it is not so evident how that translates into the socio-economic development of the regions. Outputs  1,369 participants to training and capacity building events and beneficiaries of technical assistance, of a target of 1000  709 cultural institutions/agencies of the participating oblasts for which staff has been trained and to which technical assistance has been provided, of an original target of 64. The topics of the training were the following: developing and planning cultural investment projects, private investment mobilization, cultural partnerships and network development, service quality management, as well as museum collections management, including improved storage, accounting and use of museum items for scientific, educational, art, and economic purposes. The ICR reports on the technical assistance (TA) provided to 11 beneficiary CIs to support in the preparation of the Site Management Plans (SMPs) for the LW subprojects. Each SMP included following seven sections: (i) a plan for the preservation of the CH object; (ii) a plan for improving the historical, cultural and economic value of the CH object; (iii) a plan for organizing visits to the CH site; (iv) security plan; (v) management plan for ongoing activities; (vi) funding plan; and (vii) staff training plan. The ICR also notes that 47 participating regional CIs have enhanced their capacity to conceptualize, develop and manage CH projects (ICR, page 17). Outcomes Page 10 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219)  82 cultural heritage preservation subprojects implemented, of a revised target of 79 (and an original target of 48). Achieved  40% of stakeholder's co-financing to support the cultural heritage project investments under subcomponent 1a, of a revised target of 20% and an original target of 15%. Achieved.  2,944,251 visitors in selected supported cultural heritage sites and institutions, almost achieving the revised target of 3,000,000 and exceeding the original target of 1,612,566.  88.1% of stakeholders are satisfied, of a revised target of 75% and an original target of 65%. Achieved. The team has provided the following additional information regarding the linkage of project’s activities to the socio-economic development of the regions (Email of August 7): (i) a detailed explanation on the criteria for selection of subprojects, showing how the proposals need to clearly establish the linkage with regional development activities and how regional funds need to be engaged to implement the proposed (adjacent infrastructure such as roads, parking, lighting etc), and (ii) the confirmation that the Bank team checked on the actual implementation of these adjacent infrastructure developments during the supervision missions; (iii) several examples have been provided as to how the project’s sites have been included in main large-scale regional tourism projects. The ICR reports on the results of a survey conducted by the Leontief Center in 2022, regarding the beneficiaries perception on the quality of the trainings received and the usefulness of the TA (91.2% for the TA and 92.3% for the trainings) and regarding additional qualitative impact, such as the personal and moral level- staff stimulation, the attraction of younger generation professionals, the opening up of institutional potential, the facilitation of a new management vision, and the promotion of career development opportunities. Additionally, increased cooperation among CIs and with other key stakeholders, and increased capacity among Cis to conceptualize and develop projects is reported. The ICR (page 14) mentions additional socioeconomic effects that resulted from the project interventions: the creation of 1800 temporary jobs for restoration works (short-term socio-economic effect, as these jobs presumably ended at the end of the subprojects) and a socio-economic outcome of more than 500 new permanent jobs for CIs (data provided by the CIs and by contractors and verified by the PIU). The following additional outcomes reported have their source in the Leontieff report: small and medium businesses created additional hospitality jobs in 600 new destination programs and at almost 290 new hotel and camping facilities; increased access to CH for previously underserved populations and vulnerable groups (thanks to modern IT and audiovisual equipment); enterprise development around the new sites (new hotels, restaurants etc and increased revenue and employment creation for the previously existing ones). Even though the set of outcome indicators included in the RF for PDO3 fall short in measuring PDO3, as only some of them (and very indirectly) relate to improved management and socio-economic outcomes, the ICR narrative, and the additional clarifications provided by the team (Email of August 7) provide adequate evidence to assess the PDO3 albeit without targets. Overall, the objective is rated as Substantial with moderate shortcomings. Rating Substantial Page 11 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale All three objectives were rated as Substantial, resulting in an overall efficacy rating of Substantial. Overall Efficacy Rating Substantial 5. Efficiency The PAD did not present a quantitative economic or financial analysis, although several short- and long-term benefits were expected: employment through contracts for conservation, rehabilitation and site development, economic and financial returns from tourism to local small and medium enterprises, and additional revenues to museums and other CIs. At project completion, an ex-post economic Cost-Benefit Analysis (CBA) was conducted for the overall Project activities, based on a time horizon of 30 years and a discount rate of 6 percent. Costs included all project disbursements from the WB and Borrower’s funds; investments made to improve infrastructure in the vicinity of the CH sites implemented through regional and municipal co-financing; and additional costs related to CH rehabilitation, such as maintenance, salaries of new staff, and expenses associated with new events organized in the rehabilitated sites. The benefits covered the increased social enjoyment from visiting the restored museums (use value), from having the option of visiting the museums in the future (option value), from the satisfaction from preserving them for future generations (bequest value), and from the pure knowledge that the CH sites have been rehabilitated (existence value). They were estimated for three groups of beneficiaries – visitors, residents of the four regions, and residents of the rest of the country – based on the results of a country- wide survey conducted for the purpose of this report by the Leontief Center. The results show a net present value (NPV) of US$2.1 billion, and an internal rate of return (IRR) of 32 percent. These results are in the broad range obtained for similar CH restoration efforts (IRR of 25% for restoring cultural monuments in St. Petersburg, P069063, and 47% for rehabilitating the historic core of the city of Spilt, Croatia). The results are conservative, as they do not capture several benefits, such as cost savings due to better preservation and storage derived from the use of the MMF, and the avoided losses of irreplaceable cultural artifacts due to fire or theft, that could have occurred in the absence of the security practices introduced. A financial cost-benefit analysis has been conducted, from the perspective of the beneficiary museums, using a time horizon of 30 years, and a discount rate of 6 percent. The costs include all additional expenses borne by the beneficiary institutions (costs of staff salaries, utilities, etc.). The benefits include all additional revenues net of taxes expected to be received by the beneficiary CIs, such as incremental entrance fees, gift shop sales, and revenues from new exhibitions. The results show that the net additional income of all museums would be positive (albeit very small, with NPV between 0 and 1.6), only under the assumption of a 10% increase in cost and a 10% increase in benefits, yearly (except for Korostyn Travelling Palace and the White Tower, that have negative results). If the increase in costs Page 12 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) remains at 10% and the increase in benefits is 5% or lower the results start being negative for many of the LW and for the average of the SW projects. This means that most institutions would need financial support in the future (ICR, Annex 4). Other operational and administrative factors affecting efficiency: The project duration was extended almost 5 years in total. While some of the delays were triggered due to the leadership changes and institutional capacities in the regional governments and the COVID-19 pandemic, a major part was also due to the increased scope of works with a significant increase in the number of SW subprojects and expanded number of LW subprojects. The MOF ensured adequate Project co-financing from the federal budget and was able to significantly increase this budget over time (from US$150.5 million at appraisal to US$243.58 million at closing), covering additional expenditures from predictable cost overruns in CH restoration subprojects, completion of the MMF construction, as well as additional project activities and higher targets for the number of LW and SW subprojects. The ICR notes that FISP’s strong fiduciary capacity and strong experience in managing WB-financed projects was essential in ensuring smooth day-to-day project implementation and reliable delivery of procurement, budgeting and financial management function. Even though the positive IRR of the economic ex-post analysis cannot be compared to what was expected at appraisal (no economic analysis prepared ex-ante), the ICR did compare with similar projects in the region. The ICR also notes that it is difficult to identify the full scope of works for cultural heritage projects at the design phase since these can often be uncovered only in the process of implementation when the physical state of the CH assets is known. Even though the project was extended by five years, a large part was driven by increased scope of work, in addition to the other factors such as external shock. The overall efficiency rating is deemed Substantial with some shortcomings. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 100.00 ICR Estimate  32.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Substantial Relevance, Substantial Efficacy and Substantial Efficiency result on an outcome of Satisfactory. Page 13 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) a. Outcome Rating Satisfactory 7. Risk to Development Outcome There is a financial risk associated with the weakening of the Government’s fiscal position (which started materializing during the last year of project implementation), and that may impact the regions' ability to support financially the sector of culture and tourism. There is an economic risk derived from the uncertainty of how the trends will evolve in the future regarding the increases in the costs of the maintenance of the sites and the domestic tourism trends (to ensure the benefits), that bring uncertainty to the economic sustainability of the project’s interventions. The international context (geopolitical situation in the region since February 2022) has had an important effect in international tourism, making the domestic tourism the only potential source of income. There is the risk of some stakeholders losing a sense of ownership in the project, particularly the private sector. Private investments are subject to standards of transparency and governance that need to be maintained in the future. 8. Assessment of Bank Performance a. Quality-at-Entry The policy, institutional and stakeholder engagement aspects were duly taken into account at appraisal, and environmental, social, and fiduciary arrangements built on the experience of the predecessor project (SPEDP) and complied with the Bank’s applicable requirements. The design of Sub-Component 1(a), drew on the lessons of the SPEDP with its competitive, two-phased selection of subprojects that ensured an early pipeline of high-quality subprojects fully owned by their stakeholders. Other key lessons appropriately integrated in the Project design were the simplification of implementation arrangements (such as the selection of LW subprojects from federally owned sites) and the use of contingency costing procedures for restoration works as part of processing the contract addenda. The previous strong bank experience (extensive economic and sector work carried out in Russia) helped identify upfront all key stakeholders and target beneficiaries in the Participating oblasts and engage them early on to build a transparent framework for stakeholders’ participation in the subproject development and implementation. Risk Identification was adequate (inspired by previous similar projects) and mitigation measures adopted. For instance, there were mechanisms to monitor and manage unforeseen costs increases for restoration and rehabilitation contracts during implementation. There were minor shortcomings. An ex-ante economic analysis did not take place at appraisal. Although this is understandable for what concerns the demand-driven subprojects (unknown at appraisal), some estimation for the overall project efficiency could have been made by taking the previous similar projects Page 14 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) as an example. The socioeconomic development aspect of the PDO could have been better defined with specific indicators and targets. Quality-at-Entry Rating Satisfactory b. Quality of supervision The Project benefitted from committed Bank teams with sufficient budget and technical capacity that preempted, identified, and resolved implementation challenges and risks, and was responsive to the emerging issues and client needs. Despite the changes in the TTLs, the transitions were smooth and the permanence of many core team members based in the country office maintained close collaboration with the Government and ensured institutional memory. The MTR was conducted in April 2014. The Bank provided very close support across all project management dimensions and with a range of stakeholders with different mandates and levels of authority (federal, regional, municipal) in order to ensure smooth implementation to secure cohesive and sustainable results. The team visited dozens of project sites in Leningrad, Novgorod, Pskov and Tver Oblasts, provided timely responses, actions and clearances, closely tracked the agreed follow-up steps, and reported implementation issues adequately, highlighting the key issues requiring management attention. To overcome the challenges and restrictions imposed by the COVID-19 pandemic during 2020 and the Bank arranged for virtual missions and remote supervision tools to ensure the continued Project monitoring and control of occupational health and safety aspects, and continued to verify technical quality of the restoration works through frequent virtual check-ups and exchange of photo and video materials. All that allowed for fluid communication and timely problem-solving during the COVID-19 pandemic. The shortcomings relate to the fact that none of the successive restructurings were taken as opportunities to better define and include indicators for the outcome of socio-economic development or revise the PDO. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design At appraisal, it was planned that the MoC would be responsible for the overall monitoring and evaluation of the project. The FISP was to provide assistance to the MoC to undertake M&E. A Monitoring & Evaluation Framework was to be prepared and integrated in the Operational Manual before the project effectiveness date by the FISP with the support of specialized consultants. During preparation of sub-projects, Page 15 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) participatory approaches, such as opinion surveys and focus group discussions, were to be undertaken and a baseline to be collected, for each site. Each sub-project was required to collect baseline data (before the project begins) and after project completion. The M&E system was meant to be used as a management and learning tool by the MoC, project staff, participating oblasts, municipalities and communities (PAD, page 10). The cost of M&E reporting was fully budgeted in the project management costs (Component 3). There are shortcomings at the level of the PDO description - the outcome of socio-economic regional development could have been better defined. b. M&E Implementation The PIU consistently monitored and evaluated the project and provided the Bank with semi-annual progress reports, including progress made under components and particular project-supported activities and values of the indicators in the Results Framework, level of disbursement, procurement, etc. In addition, PIU prepared reports on the project reports to the Ministry of Culture and Ministry of Finance, and Inter-Ministerial Committee chaired by the Ministry of Culture. All details on aspects monitored by the PIU are provided in the Project Operational Manual that was reviewed by the Bank and found satisfactory. All progress reports were provided by the PIU on time. Baseline values for two outcome indicators (number of visitors and level of stakeholder satisfaction) were identified during the first year of the Project, and the actual and final values were measured with use of the same methodologies. The Bank worked with the Borrower to better align the reporting with the Project outcomes. The civil works routine supervision and monitoring was provided through engagement of independent engineering supervision consultancies, project cities’ technical teams, a designated project coordinator on the ground, and day-to-day PIU supervision and monitoring with regular progress reports submitted to the Bank. Each Bank mission included visits to the project sites where the Bank representatives met with the regional and municipal authorities to discuss and ensure that they execute their obligations in line with the provisions of the LW Subproject implementation agreements, specifically in what concerns the linkage of the interventions with the regional economic development (the bank checked that the improvements of adjacent areas have been performed in line with provisions of the LW Subproject implementation agreements., as part of the regional commitments) During the last year of the project implementation, the PIU attracted Leontieff center to provide more in- depth evaluation of the project results, going beyond the Results Framework, which contributed to the preparation of the Borrower’s own completion report. A shortcoming is that during all the 5 restructurings the PDO description and RF indicators could have been modified to better capture and measure some of the project’s outcomes. c. M&E Utilization As part of the M&E use, the feedback provided by the CIs that were managing the SW sub-projects helped to inform subsequent calls for proposals and design of the Project’s capacity building activities. The report developed by the specialized consultancy firm (Leontieff center) to summarize and analyze Page 16 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) project results, allowed to put together an impressive wealth of quantitative M&E data from individual subprojects that was then used by the MoC to inform learning for other CH sites across Russia. This served the original demonstration purpose of the project. M&E Quality Rating Substantial 10. Other Issues a. Safeguards Environmental and social safeguards compliance was rated Satisfactory at project closing. The Project triggered the Bank’s safeguard policies on Environmental Assessment (OP/BP 4.01), Cultural Property (OP/BP 4.11), and Involuntary Resettlement (OP/BP 4.12). Environmental risks were overall minor and mainly associated with site-specific renovation and construction activities. They were mitigated through the Project’s Environmental Management Framework (EMF) and site-specific Environment Management Plans (EMP). Site-specific environmental risks were elevated to moderate in those few instances where the scope of planned landscape works was more significant, with relevant EMP provisions (cases of Mon Repos Park and MMF construction). Cultural property risks were addressed through project investments in the conservation and safekeeping of cultural assets, as well as through the application of the Chance Finds Procedure (CFP), in handling archaeological finds, modeled on the successful experience of the previous similar project in Russia. Social risks were minor and related to possible temporary displacement of heritage site occupants (small businesses and residents) prior to works. A Project Resettlement Policy Framework (RPF) was prepared for the Project. During implementation these risks did not occur as all subproject sites were owned and occupied by the respective CIs. There was only one case related to the boundary line of a plot for a CH site, where the adjacent plot was a small cemetery. Although none of the graves were disrupted or relocated, an abbreviated Resettlement Action Plan (RAP) was prepared in line with the Project’s RPF. The Project Operation Manual included the Environmental Management Framework, the Resettlement Policy Framework and the Change Find Procedure. The screening and scoping of subprojects for environmental and social safeguards was conducted at the subproject preparation stage by beneficiary CIs in collaboration with the local authorities with FISP guidance (with adequate permanent expertise). The Bank’s environmental and social specialists regularly reviewed compliance with safeguard requirements during site visits and conducted the Project safeguards review during the MTR, with satisfactory results. The relevant safeguards for all subprojects were properly documented and on file with FISP. As part of the social actions around the Project, FISP in collaboration with the WB Social Specialist, prepared a questionnaire and organized a workshop for CIs to provoke analysis and explore challenges and opportunities for enhanced inclusion of persons with disabilities. At Project closure there were no outstanding Environmental and/or Social safeguards compliance issues and the Project had also fully met national environmental, occupational health and safety, and labor requirements. b. Fiduciary Compliance Page 17 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) Procurement The Project was completed in line with the World Bank’s procurement guidelines and procedures set within the POM and the procurement plan. The FISP PIU was well staffed and retained additional procurement specialists in response to the increased procurement load under the SW subprojects. Procurement performance was rated Satisfactory at all times of the Project implementation. There were no cases of mis- procurement. In one case (MMF construction), when the originally selected main contractor had stopped the works, the Bank found it possible to issue a no-objection to the PIU proposal to terminate the stopped contract and, on an exceptional basis, issue a direct contract with primary subcontractor for the remaining works using the same unit prices and bills of quantities as in the original contract. This allowed a successful completion of the MMF construction in line with the required technical specifications and with due technical quality. Post reviews identified no violations of policy. Complaints were resolved by PIU in due time. Challenges included delays with securing subproject permits and approvals, changes in the scope and timetable of works, and contract prices being considerably higher than the planned budget. These challenges were mitigated by the proactive revision of procurement plans, application of contingency costing procedures, and close monitoring of contract implementation by the Bank and PIU. The Bank procurement procedures ensured selection of highly qualified designers and contractors that were critical to the success of complex restoration works. The Borrower generally expressed continued preference for the Bank procurement approaches over the national procurement law and acknowledged that the Bank procurement procedures contributed to the selection of highly qualified designers and contractors. Financial management The Project was closed with a Satisfactory Financial Management (FM) rating, with no pending FM actions. Throughout project implementation, FISP PIU carried out the FM and disbursement functions in a satisfactory manner. The PIU was well staffed with experienced FM and accounting employees. Quarterly interim unaudited financial reports were submitted on time and were satisfactory to the Bank, as well as annual audit reports. The auditors issued clean (unmodified) audit opinions and did not note any significant internal control issues. c. Unintended impacts (Positive or Negative) N/A d. Other Mobilizing private sector financing Private sector stakeholders co-financing for SW subprojects amounted to US$7.7 million, or 17 percent of the total SW subproject costs of US$45.2 million. Furthermore, the resulting increase in new tourist destination programs between 2011 and 2021 has catalyzed private sector investments in almost 290 new accommodation facilities (hotels, motels, camps, etc.), with an added capacity of 27,000 beds, to service Page 18 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) the increased visitor traffic to the improved heritage sites (an additional 1.4 million overnight visits per annum. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The following lessons are considered the most relevant, extracted from the ICR and rephrased: Small-scale investments can bring big value added might be a suitable mechanism when there is limited financial capacity. They can reach a wide range of communities, some underserved, and promote learning and social cohesion. CH projects can reach out to vulnerable youth, elderly populations, people with disabilities and marginalized communities, sharing values and history through the experience of women, Indigenous peoples, ethnic minorities etc. Introducing specific selection criteria could foster these investments. When the engagement of stakeholders is ensured (CI, public authorities at regional and municipal level, local communities, CH professionals, civil society and private sector), subproject design is more adapted and overall project sustainability is fostered. In this project, when restoring popular landmarks, special attention needs to be given to public communications and structured engagement with the surrounding community. Strong inter-sectoral coordination is also essential for expediting numerous title, zoning, engineers, and environmental clearances required for complex works such as those in CH restoration. 13. Assessment Recommended? No 14. Comments on Quality of ICR Page 19 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Cultural Heritage 2 (P120219) The ICR is well structured, parses the PDO logically and describes the components adequately. It provides a comprehensive overview of the project and good coverage of the implementation experience. The report is consistent and presents a logical linking and integration of its various parts. It is focused on results and provides substantial information beyond the indicators to provide a complete picture on the project's results. The ICR’s lessons are useful. There were some gaps in the outcomes evidence, but the team provided written responses to IEG’s questions to supplement the information provided in the ICR. Overall the ICR quality is rated as Substantial. a. Quality of ICR Rating Substantial Page 20 of 20