Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review REG DEV 1 (P126033) Report Number : ICRR0021299 1. Project Data Project ID Project Name P126033 REG DEV 1 Country Practice Area(Lead) Georgia Social, Urban, Rural and Resilience Global Practice L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-81480 31-Dec-2016 75,000,000.00 Bank Approval Date Closing Date (Actual) 20-Mar-2012 31-Dec-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 60,000,000.00 0.00 Revised Commitment 60,000,000.00 0.00 Actual 57,266,742.77 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ranga Rajan J. W. van Holst Christopher David Nelson IEGSD (Unit 4) Krishnamani Pellekaan 2. Project Objectives and Components a. Objectives The Project Development Objectives (PDOs) as stated in the Loan Agreement (Schedule 1, page 5) and in the Project Appraisal Document (PAD, page 5) were: "To improve infrastructure services and institutional capacity to support the development of tourism- based economy and cultural heritage circuits in the Kakheti region" Page 1 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review REG DEV 1 (P126033) b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components There were two components (PAD, pages 6-7). One: Infrastructure Investment. Appraisal estimate US$71.70 million. Actual cost US$66.30 million. This component provided financing on a grant basis to participating local-self-governments to carry out investment sub-projects. There were two sub-components. (a) Urban regeneration through an integrated approach for renewal of Telavi and Kvareli towns and the heritage village of Dartlo. Activities included: (i) Rehabilitation of municipal infrastructure and utilities in the central historical areas, conservation and upgrading of public spaces, cultural buildings and building facades with vernacular architecture: (ii) tourism circuit development through rehabilitation of the eleven most attractive cultural heritage sites located along the main tourism route in Kaketi, landscaping and public parking, construction of information kiosks, cafes, public toilets and access roads. and, (b) Activities in this sub-component provided financial resources to local-self-governments for carrying out public infrastructure sub-projects (such as in roads/sidewalks, water/sanitation and public facilities) to attract private sector investments in tourism and agro-processing activities. Two: Institutional Development. Appraisal estimate US$3.90 million. Actual cost US$5.90 million. This component aimed at strengthening the institutional capacity of the Georgia National Tourism Administration (GNTA), the Agency for Cultural Heritage Preservation of Georgia (ACHP), the Municipal Development Fund and other local and regional entities in activities such as: (i) destination management and promotion: (ii) setting up geo-tourism routes and tourism portal: (iii) skilled workforce development and capacity building: (iv) construction and supervision of cultural heritage areas: and (v) support for Monitoring and Evaluation (M&E). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. Appraisal estimate US$75.60 million (including baseline cost and costs of physical and price contingencies). Actual cost US$71.60 million. Project financing. The project was financed by an IBRD loan. Appraisal estimate US$60.00 million. Amount disbursed US$57.26 million. Borrower contribution. Appraisal estimate US$15.00 million. Actual contribution at closure US$14.91 million. Dates. The project was approved on March 20, 2012, became effective on May 26, 2014 and was originally scheduled for completion on December 31, 2016. Restructuring. There were two Level 2 restructurings. The following changes were made through the first restructuring on June 6, 2016. The closing date was extended by a year for completing ongoing activities that had been subject to delays for a combination of factors including, extreme weather Page 2 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review REG DEV 1 (P126033) conditions that limited access to project sites in the mountainous sub-regions, poor quality of designs at preparation which required modifications and overall slowdown in implementation following the change in government after the national elections in October 2012. The following changes were made through the second restructuring on November 28, 2017 which preceded the closing date by only one month and the ICR acknowledged "should have been made earlier” (paragraph 12): (i) Funds were reallocated between disbursement categories to increase the allocation for consultancy services and the Results Framework (RF) was modified to address the mismatch between the outcome targets and the nature of project interventions. The indicator pertaining to the increased number of hours of piped water services in the project area was dropped as the project activities envisioned investment in distribution networks and sewerage network pipes and not water system rehabilitation activities: (ii) The indicator pertaining to the "number of people in urban areas with access to all-season roads within a 500 meter range under the project" was replaced by "20 kilometer of roads (in urban areas) constructed and rehabilitated" in line with the project's goal related to enabling tourism: (iii) The target value for the number of Destination Management Offices (DMOs) in Kakheti was revised downward to one from three, as an assessment conducted in May 2017 concluded that three DMOs were not necessary in Kakheti. The project ultimately closed on December 31, 2017 which was one year after the original closing date . 3. Relevance of Objectives Rationale The PDOs were relevant to the government strategy of regional development in the country. Rural poverty and regional disparities were important issues for Georgia at appraisal. The Kaketi region, located next to the capital city of Tbilisi, was a lagging region, with Gross National Product (GNP) about 60% of the national average. The region had potential for tourism, with many historical and architectural monuments, three natural protected areas and it was the ancient center of Georgian winemaking. The Government's State Strategy on Rural Development issued in June 2010 for the 2010-2017 period highlighted the need for creating a favorable environment for regional socio-economic development and improving living standards through well-balanced socio-economic development, increased competitiveness and greater socio- economic equality among regions. The strategy identified regional development as a critical priority and identified the need for infrastructure investments for eliminating regional disparities. The Kaketi Regional Development Strategy for the 2014-2021 period identified tourism as one of the two drivers of growth. The PDOs were well-aligned with the Bank strategy. One of the two core priorities of the Country Partnership Strategy (CPS) for the 2014-2017 period was "enabling private sector led growth through improved competitiveness", The 2018-2021 Country Partnership Framework (CPF) underscored the importance of tourism as a key contributor to growth in Georgia and highlighted the need for developing the tourism sector in lagging regions with potential for tourism. Page 3 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review REG DEV 1 (P126033) Rating High 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To improve infrastructure services to support the development of tourism-based economy and cultural heritage circuits in the Kakheti region. Rationale The theory of change for this objective was that urban regeneration involving the physical improvement of tourist attractions and public infrastructure will lead to more reliable water supplies, lower transport costs, and facilitate better access and enhanced experience by tourists to tourism and cultural heritage sites and consequently contribute to a growth in tourist numbers as well as the length of their stay which in turn would contribute to economic growth in the Kakheti region. Output (ICR, pages 62-68). These activities were completed in the three hub centers of Telavi, Kvareli and Dartlo in the Kakheti region. • Rehabilitation works were undertaken on 907 piped household water connections at project closure. This exceeded the target of 500. • 26.3 kilometers (Km) of roads were rehabilitated. This exceeded the target of 20 km. • 11 cultural heritage sites were redeveloped along the tourist circuit, as targeted. • A total of 217 buildings were restored in Telavi, Kvareli and Dartlo as compared to the target of 150. • 11 Private development proposals were approved at closure. This exceeded the target of 10. • 32,319 residents in Telavi, Kvareli, Dartlo and Gujrani (including 16,867 females) directly benefitted from the infrastructure investments at project closure. This exceeded the target of 30,800 (including 16,140 female beneficiaries). Outcomes (ICR, pages 32-33). • The index of the weighted average vehicle operating costs (VOCs) on the improved urban roads in the town of Kvareli, which accounted for 25% of the total road rehabilitation, declined from 100% at the baseline to 71%, at project closure. This exceeded the revised target of 75% and, according to the ICR, was due to the improved roads. • Private sector investment (direct and indirect) in tourism increased to US$61.00 million at project closure. This exceeded the target of US$50.00 million. • According to the ICR (page 13), a beneficiary survey of the households directly benefiting from the project was conducted in 2016, on 211 households in the three hub locations of Telavi, Dartlo and Kvareli. Page 4 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review REG DEV 1 (P126033) The survey indicated that 75% of the respondents were satisfied with the project and 89% of the respondents agreed that the project had a positive impact on tourism. A survey conducted at project closure of 16 investors (who accounted for about 30% of the total investment in the region) indicated that 88% of them were aware of the urban generation works and 50% said that improvements in infrastructure and services strongly influenced their decision to invest and 38% thought that it influenced their decisions moderately. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective To improve institutional capacity to support the development of tourism-based economy and cultural heritage circuits in the Kakheti region. Rationale The theory of change is based on improving the public sector capacity building for tourism promotion and management as well as the training of employees in the private tourism industry that would lead to the strengthening of the Destination Management Office and hence enhance the strategic management of tourism assets and tourism promotion and services with the aim of supporting the private tourism industry Outputs (ICR, pages 68-71). • As agreed at the restructuring of the Municipal Development Fund MDF) only one Kakheti Destination Management Office (DMO) was established rather than three mandated in the project’s original design. This decision to have one DMO instead of three was not finalized until May 2017, five years after project approval (ICR, Page 10, Table One). The DMO model envisages promoting tourism in the region based on a collaboration between the national, regional, local governments and the private sector. Although the DMO was established just one month before project closure in November 2017 with three full time employees, DMO's operational modalities were still being finalized when the project closed (ICR, para 57). • The Visitor Management Plan and the design for the Tehavi museum supported by the project was used to prepare the first exhibition scheduled to open in May 2018. The Task Team Leader clarified that the exhibition has opened. • 440 employees of hotels and restaurants in the area were trained at project closure. • Promotional materials (including promotional maps and leaflets) for the tourism industry were developed. A Framework for the regional tourism promotion strategy was finalized. Page 5 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review REG DEV 1 (P126033) • The website for providing information on tourism in the region was completed. The ICR notes that this website was not being regularly maintained, due to lack of readership and high cost. • Capacity building support was provided to the Municipal Development Fund (MDF) in the areas of Monitoring and Evaluation and compliance with safeguards. • A Strategic Environmental, Cultural Heritage and Social Assessment (SECHSA) to complement the Regional Development in Kakheti was conducted to examine the positive or negative effects that the planned investments would have in the region. Outcomes. • The number of hotel beds in circuit route areas increased from 1,610 at the baseline to 3,588 at project closure. This exceeded the target of 1,932. There were, however, no data on occupancy rates. • Points of sale for tourism services (such as tickets, souvenir shops, restaurants, hotels, guesthouse and family houses) in renovated cultural heritage sites increased from 248 at the baseline to 337 at project closure. This exceeded the target of 323. • A survey of foreign visitors conducted at project closure through interviews with 522 people indicated that more than 90% of respondents were satisfied or highly satisfied with the quality of service in restaurants, hotels and guest houses in Kakheti. There was, however, no baseline (ICR, page 14, paragraph 19). • According to the “GeoStat survey” of hotels and hotel-type enterprises 2000-2017 which tracks region wide indicators of growth in the tourism economy, the number of foreign visitors to the Kakheti region grew by 721%, jobs in the sector grew by 300% and the average salaries of employees almost tripled. The ICR (page 15) notes that other municipalities that benefitted from the project also experienced growth in visitor numbers (visitor numbers grew 41 times in Gurjaani, six times in Lagodekhi and three times in Akhmeta). According to the data on municipal tax collection for 2000-2017 provided by the Revenue Service for Georgia, tax revenue in Telavi Municipality grew six times and in Kvareli municipality 46 times between 2000 -2017. It is difficult to attribute these listed outcomes to project activities, given that growth in tourism in Kakheti was part of the overall national trend and supported by factors external to this project such as a relatively stable political environment, national policies relaxing visa requirements and a growing number of direct airline connections to former Soviet Union countries as well as to countries in Western and Eastern Europe. Given that the operational modalities of the DMO established under the auspices of this project were not yet completed at project closure, this review concludes that the project activities made a Modest contribution to realizing this objective. Rating Modest PHREVDELTBL PHOVRLEFFRATTBL Page 6 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review REG DEV 1 (P126033) Rationale Given the number of households directly benefiting from the project and the survey of investors, this review concludes that the project activities substantially contributed to improving the infrastructure services for supporting tourism. On the other hand the operational modalities of the DMO were not yet complete at project closure and in light of the difficulties associated with the attribution of outcomes to the project's activities, this review concludes that at project completion the project investments made only a Modest contribution to improving the institutional capacity to support the development of tourism-based economy. Nevertheless, the ICR (page 21) notes that usually visitor numbers tend to grow gradually in response to tourism development initiatives, as reputation of the destination spreads and the full impact of the project on promoting tourism can only be measured in five to ten years. This review rates efficacy overall as Substantial, based not only on what was achieved but also on what can be expected to be achieved in the coming years. Overall Efficacy Rating Substantial 5. Efficiency Economic Analysis. The PAD (page 12) stated that a cost-benefit analysis was conducted for the project as a whole, rather than for each component at appraisal and at closure. The project benefits were assumed to come from an increase in domestic and international tourism arrivals, an increase in overnight stays by tourists, increase in spending by tourists, growth in private investment due to the public infrastructure investments, an increase in property value and appreciation of the rental value and creation of temporary jobs. The project's Net Present Value (NPV) at 12% discount rate was US$19.79 million at appraisal and the ex ante Economic Internal Rate of Return (EIRR) was 26%. The methodology followed for the ex post analysis was not comparable with the ex ante analysis as the ex post analysis considered the primary economic benefits from incremental spending by tourists, temporary job creation and economic benefits from secondary sales. The NPV at project closure was US$11.68 million and the estimated ex post EIRR was 21.39% (ICR, Table 4) . Although the methodology followed at closure was not comparable to the ex ante analysis, the lower than predicted daily spending by tourists and lower private sector investment than estimated at appraisal could have contributed to the relatively low ex post EIRR. Cost effectiveness. Unit costs of selected assets (representing 25% of total project costs) were compared to costs of other similar municipal, urban and regional infrastructure projects in Georgia. The analysis showed that average cost of road construction was 30% higher relative to an average of comparator projects and costs of constructing municipal buildings were 40% higher and costs of water supply pipes were 12% higher. The ICR (paragraph 23) notes that the higher unit costs could have been due to factors such as, undertaking road construction activities in remote mountainous locations and providing individual water supply solutions in several buildings in remote locations. Administrative and Operational Efficiencies. The poor quality at design contributed to the suspension of activities at a number of sites. This necessitated modification of design during implementation. The quality of works was also hampered in the initial years due to the delays in hiring the international supervision consultant and the weak capacity of the Municipal Development Fund (MDF). Inadequate capacity in Page 7 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review REG DEV 1 (P126033) government institutions caused delays in compliance with safeguards implementation which remained poor for an extended time, creating substantial risks for the livelihood of affected communities (discussed in Section 10 of this review). Although the DMO was established, the operational modalities of the DMO were not complete when the project closed and some of the tourism assets (such as the Telavi museum, Kuveri school and a number of visitor centers) were not operational at project closure. In summary, although the ex post EIRR estimated in the ICR is a positive result, it is based on the inevitable assumptions embodied in such an analysis in an industry full of uncertainties and in a region often affected by conflict in recent years. On the other hand the assessments of cost effectiveness as well as shortfalls in project administration and operations, though also not without measurement challenges, do indicate significant inefficiencies in the project's implementation. On balance this review therefore assess the overall efficiency of this project as modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Point value (%) *Coverage/Scope (%) Available? 100.00 Appraisal  26.00 Not Applicable 100.00 ICR Estimate  21.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the PDOs to the Government and Bank strategies is rated as High. Efficacy of the first objective - to improve the infrastructure services for supporting the development of a tourism-based economy - is rated Substantial. The efficacy of the second objective - to improve the institutional capacity to support the development of a tourism-based economy - is rated Modest, in terms of what was achieved. Overall efficacy, however, based on what was achieved together with what can be expected to be achieved in the next five to ten years, efficacy is rated substantial. Efficiency is rated Modest, based on the inefficiency of the project's cost effectiveness and given its administrative and operational inefficiencies, which contributed to delays during implementation and eventually non-completion of important activities at project closure. The project's performance had moderate shortcomings and hence its outcome has been rated Moderately Satisfactory. Page 8 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review REG DEV 1 (P126033) a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Institutional Risk. The Destination Management Office (DMO) of Kakheti was established under the auspices of this project. The DMO's operational model which envisions cooperation and partnership between the national, regional and the private sector was however still being finalized at the project's closure, and its sustainability will depend on the contribution of public funds and from the cooperation and partnership between the national, regional and local governments and the private sector (ICR, paragraph 57). Given that the DMO of Kakheti is still new and its operational details are still to be finalized, there is a risk to the sustainability of the DMO model. Technical risk. Assets built as a part of the project in peripheral locations to support development of tourism circuits were still not fully operational. Specifically, the project funded construction of 11 structures intended for tourist services. Only two of these structures were fully operational at project closure and arrangements for operation of these structures were still under consideration (ICR, paragraph 48). The Task Team Leader clarified that this remains the case to date. 8. Assessment of Bank Performance a. Quality-at-Entry The design of this project, which included activities combining basic infrastructure, addressing government capacity limitations and targeted support for the key tourism sector in the region to promote economic growth within a defined territory, was the first in which the Bank attempted to operationalize the conceptual framework development by the 2009 World Development Report on territorial development. Regarding the country context, the project was based on the experience gained from the prior Bank-financed projects: (Georgia Secondary and Local Roads Project and the East-West Highway Project). The project was prepared based on a two-year long period of analytical work and extended dialogue with the government and consultations with stakeholders. The risks associated with the project were rated as either Low or Moderate and the overall implementation risk was rated as Moderate at appraisal (PAD, page 11). The arrangements made at appraisal for fiduciary compliance were appropriate (discussed in Section 10). There were several shortcomings in design. As indicated in Section 5, the poor quality of design contributed to the low technical standards of civil works. This contributed to delays and necessitated modification of design during implementation. The design underestimated the risk associated with the following factors highlighted in the ICR): One, it is unclear if there was a realistic assessment of the demand for tourist services. As indicated in section 7 of this review, the project funded 11 structures for tourist services. Only two of the structures were functional at project closure and it is not clear if there would be enough demand for tourist services for sustaining all Page 9 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review REG DEV 1 (P126033) the 11 structures (paragraph 48). Two, the project’s design ambition overestimated the institutional capacity both at the national and at the local level. For instance, the process for selection of additional investments implied that subproject proposals would be put forward by the local governments and confirmed by the Supervisory Board (consisting of the Prime Minister, four ministers and other senior officials). However, this approach was not followed in practice. This resulted in several cases where investments were added to the Project without strong local government ownership and they contributed to challenges associated with operationalization of the subprojects (paragraph 48). Three, despite the preparation period of two years, the design overestimated the technical capacity in the government agencies to administer compliance with environmental and social safeguards (ICR, paragraph 31). The Municipal Development Fund (MDF) did not have the capacity or adequate institutional systems for environmental and social safeguards management. For an extended period during implementation, environmental supervision and reporting remained poor, creating substantial risks for the livelihood of affected communities (such as damage to public health due to absence of waste disposal sites). The issue was eventually addressed in the final years of implementation with guidance from the Bank supervision team (discussed in more detail below). Four, the design underestimated the risk associated with inter- ministerial coordination (ICR, paragraph 48). All of these factors contributed to delays during implementation and thereby to the non-completion of some activities at project closure. There were shortcomings in M&E design (discussed in section 10a). Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision According to the ICR eleven Implementation Status and Results Reports (ISRs) were prepared over a five- year period and supervision missions were conducted on average twice a year. The ICR (paragraph 49) notes that the supervision teams included Bank specialists and consultants with appropriate skills (including cultural heritage experts and specialists). In the first three years of the project, the Task Team Leader was based in Georgia and this helped in providing close engagement. The team provided assistance for addressing the poor quality of project design issue and this aided in resolving the poor design quality issues. According to the Borrower’s ICR (ICR, page 59) the Bank team with the task team leader, co task team leader, experts and consultants supported the MDF in project initiation, planning and project implementation. The intensified engagement of the Bank safeguards specialists with the Municipal Development Fund helped in resolving the environmental and social safeguards issues that rose during implementation and to correct the M&E shortcomings (discussed in Section 10 of this review). The extension of the project closing date during restructuring helped in completing some (though not all) activities by project closure. Given the delays in the initial years, the restructuring that was done among other things, for improving the M&E framework just a month before closure, could have been done earlier either during the Mid-Term Review or during the first project restructuring (discussed in Section 2e of this review). The team did not proactively address the challenges relating to operation and management arrangements of assets such as the visitor center facilities. The Operational Manual implied that once the sub projects were completed, they would be operated by the municipalities. However, most municipalities lacked the Page 10 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review REG DEV 1 (P126033) capacity to oversee the assets. Eventually the Ministry of Economy at the request of the municipalities took over the responsibility for managing the assets. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design AS part of the original M&E design there were five key output indicators (two pertaining to monitoring the infrastructure related activities, two pertaining to the tourism related activities and one relating to the institutional capacity. Of these, the indicators - increased volume of private sector investment and increased number of hotel beds in circuit route areas - were appropriate for monitoring performance with respect to the tourism- related activities. The indicator pertaining to reduction in vehicle operating costs due to improved urban roads was also appropriate for monitoring performance with respect to infrastructure investments. On the other hand, the indicator pertaining to the increased points of sales for tourism services in renovated cultural heritage sites was weak and inappropriate for monitoring institutional capacity building activities. The indicator measuring the increase in the number of hours per day (from 8 hours a day to 24 hours a day) of piped water services in project areas was inappropriate for monitoring infrastructure investments, because the project activities only envisioned investment in distribution networks and did not include activities pertaining to rehabilitation of water supply services that was required for the achievement of the target value of this indicator (ICR, Table 1, page 10). There were, however, baseline figures available for all the indicators proposed at appraisal. b. M&E Implementation As indicated in Section 2e, the results framework was modified, and the monitoring indicators were revised following the second restructuring of the project. According to the ICR baseline data were available for all the indicators proposed at appraisal. An international M&E firm was hired to deliver the monitoring reports and four surveys were conducted for evaluating the project results. This included surveys of beneficiaries, businesses benefiting from co-investors, international tourists and private investors in the region (paragraph 40). Nevertheless the ICR noted in paragraph 41 that delays in the collection of data negatively affected conclusions on attribution. For example, the first baseline survey of beneficiaries was conducted in 2013 by a local consultant in 2013 but by that time a number of investments had already been completed (most significantly facade improvements of the houses that the surveyed beneficiaries owned or lived in), which made it hard to interpret the changes reported between the survey in 2013 and a subsequent survey to assess the impact of project interventions. The ICR went on to state, however, that an international M&E Page 11 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review REG DEV 1 (P126033) consultant was mobilized and consequently "additional surveys of investors and beneficiaries informed the assessment of the project results presented" in the ICR. c. M&E Utilization According to the ICR the reporting on indicators was used for monitoring problematic issues during implementation and for monitoring project performance at closure (paragraph 42). M&E Quality Rating Modest 10. Other Issues a. Safeguards Four safeguard policies were triggered at appraisal: Environmental Assessment (OP/BP 4.01): Natural Habitats (OP/BP 4.04): Physical Cultural Resources (OP/BP 4.11): and, Involuntary Resettlement (OP/BP 4.12). Compliance with these safeguards is discussed in paragraph 45 of the ICR. The conclusions on compliance are summarized below. Environmental Safeguards and Safeguards pertaining to Natural Habitats and Physical Cultural Resources: The ICR notes that an Environmental Management Framework (EMF) was prepared at appraisal and each subproject was screened for the potential, environmental, social and cultural heritage impacts, and site-specific documents were prepared. Environmental Management Plans (EMPs) and were prepared and publicly disclosed. There were concerns associated with absence of waste disposal arrangements during implementation. The ICR states that all significant issues were addressed and at closure there was no damage to either public health or the natural environment because of the project. There was also compliance with safeguards pertaining to natural habitats as the project did not affect natural habitats either inside or outside the formally designated project areas. Although there were delays during implementation, the safeguard issues pertaining to physical cultural issues were eventually resolved and there was compliance with safeguards pertaining to physical cultural resources. Involuntary Resettlement. Resettlement Policy Framework (RPF) and an Abbreviated Resettlement Action Plan (ARAP) were prepared by the Municipal Development Fund. The ICR notes that resettlement impacts occurred in three sub projects (in Kvareli and Telavi). A Kvareli ARAP covered temporary income losses of 33 people. A Telavi ARAP covered temporary relocation/business interruption impacts for 91 households and 84 business. Page 12 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review REG DEV 1 (P126033) b. Fiduciary Compliance The Municipal Development Fund (MDF) was responsible for fiduciary management. The MDF had executed Bank-financed projects before and was familiar with the Bank's financial and procurement procedures. . Financial Management. A financial management assessment conducted at appraisal, concluded that the financial management arrangements were satisfactory. The ICR does not provide information on quality of financial management performance during implementation. The Borrower's ICR (page 57) notes, however, that the MDF had qualified financial accounting staff and compliance with financial management issues was verified regularly by independent auditors. The Task Team Leader clarified that the audits were unqualified. Procurement. A procurement management assessment conducted at appraisal, concluded that the procurement risk was Moderate (PAD, page 15). The MDF prepared an initial Procurement Plan at appraisal and this plan was deemed to be satisfactory by the Bank. The ICR (page 24) notes that there were procurement delays during implementation, due primarily to the lack of capacity within the implementing agency. These were however resolved and there was compliance with Bank procurement guidelines. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome --- Satisfactory Satisfactory There were some significant Moderately Bank Performance Satisfactory shortcomings at Quality-at- Satisfactory Entry. Quality of M&E Modest Modest --- Quality of ICR Substantial --- 12. Lessons The ICR (pages 29-31) draws nine lessons from the experience of implementing this project. Of these, this review found that three lessons were most important with broad application to similar projects that might be financed by the Bank. These lessons are summarized below with some editing. Page 13 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review REG DEV 1 (P126033) (1) A deep understanding of country context is required for projects introducing an innovative design. Although this project was prepared based on extensive research and close collaboration with government and other stakeholders, implementation was significantly delayed in the initial years due to the weak implementation capacity of the Municipal Development Fund. Further, the potential issues associated with operating and managing the assets were not identified by the preparation team although eventually these were resolved by the Municipal Development Fund in the final years of the implementation period. The ICR points out that lesson is that shortcomings of project design that create implementation problems "can be largely related to the rushed finalization of project preparation, thus further underlying importance of patience in preparing innovative operations". (2). Given that forecasts of tourist demand are extremely difficult, the design of tourist development projects needs to provide flexible implementation plans to accommodate uncertain demand. In this project, some investments associated with tourism visitor centers were deemed to be not necessary, because of the relatively weak demand for tourism services and commercial space in specific locations. The point is that such investments should be delayed despite their inclusion in the project design. Indeed such investments were, appropriately, not pursued in this project. (3) The project demonstrated the need for developing appropriate methodology on M&E frameworks for tourism development projects. Some of the indicators used in this project to measure the extent to which the project activities contributed to improving the institutional capacity to support the development of a tourist- based economy could not be used because the attribution of the development of a tourist-based economic to improvements in institutional capacity in the tourist industry could not be established. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is reasonably clear and candidly discusses the problems faced during implementation due to inadequate preparation. It is also candid in discussing issues relating to some investments that may not have been necessary, given the existing demand for tourism services in the project intervened areas. It's ratings are consistent with the guidelines and it draws good general lessons and specific lessons for Georgia from the experience of implementing this project. There were, however, some shortcomings. Annex 3 contained no information on the project’s financing which this annex requires. The main text of the ICR provides no information on financial management standards during implementation. Nevertheless, the Borrower's ICR (ICR, page 57) provided some details on financial management during implementation. The text of the ICR is unduly long and would have benefitted from better editing. For instance, the footnote 20 for "Calculations based on GeoStat survey of hotels and hotel enterprises 2000-2017, it is assumed that every stay in a hotel represents a separate visit" is repeated five times in the same page. Page 14 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review REG DEV 1 (P126033) a. Quality of ICR Rating Substantial Page 15 of 15