39293 THE WORLD BANK INDEPENDENT EVALUATION GROUP An Independent Evaluation of the World Bank's Support of Regional Programs Case Study of the Organization of Eastern Caribbean States Telecommunications Reform Project Deepa Chakrapani Director-General: Vinod Thomas Director: Ajay Chhibber Manager: Victoria Elliott Task Manager: Catherine Gwin 2006 This paper is available upon request from IEG. The World Bank Washington, D.C. ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION The Independent Evaluation Group (IEG) is an independent unit within the World Bank; it reports directly to the Bank's Board of Executive Directors. IEG assesses what works, and what does not; how a borrower plans to run and maintain a project; and the lasting contribution of the Bank to a country's overall development. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank's work, and to provide accountability in the achievement of its objectives. It also improves Bank work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. IEG Working Papers are an informal series to disseminate the findings of work in progress to encourage the exchange of ideas about development effectiveness through evaluation. The findings, interpretations, and conclusions expressed here are those of the author(s) and do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent. The World Bank cannot guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply on the part of the World Bank any judgment of the legal status of any territory or the endorsement or acceptance of such boundaries. Contact: Independent Evaluation Group Knowledge Programs and Evaluation Capacity Development (IEGKE) e-mail: eline@worldbank.org Telephone: 202-458-4497 Facsimile: 202-522-3125 http:/www.worldbank.org/IEG Contents Acronyms..........................................................................................................................iii Preface............................................................................................................................... iv Evaluation Objectives and Methodology................................................................iv Evaluation Criteria.................................................................................................iv Executive Summary......................................................................................................... vi 1. Introduction..................................................................................................................1 Challenges Facing the Sector..................................................................................1 Regional Program Summary Description................................................................1 2. Relevance: Rationale, Alignment, and Design ..........................................................3 Subsidiarity Principle ..............................................................................................3 Alignment with Country, Regional, and Bank Goals and Strategies.......................4 Regional Consensus.................................................................................................5 Design of the Regional Program .............................................................................6 Clarity and Monitorability of Objectives.................................................................7 3. Efficacy: Outcomes, Impacts, and Sustainability .....................................................9 Achievement of Objectives.......................................................................................9 Capacity Building ..................................................................................................12 Realized Distribution of Costs and Benefits ..........................................................13 Risks to Outcomes and Impacts .............................................................................13 4. Efficiency: Governance, Management, and Financing ..........................................15 Efficient Use of Resources .....................................................................................15 Governance, Management and Legitimacy ...........................................................16 Financing...............................................................................................................18 Donor Performance ...............................................................................................19 5. Monitoring and Evaluation.......................................................................................20 6. World Bank Performance.........................................................................................21 Comparative Advantage.........................................................................................21 The Bank's Coordinating Role with Other Donors ...............................................21 i Quality of Support and Oversight..........................................................................21 Structures and Incentives.......................................................................................22 Linkages to Other Bank Country Operations ........................................................22 7. Country Participation................................................................................................24 8. Conclusions.................................................................................................................25 Summary of Findings.............................................................................................25 Implications for Effective Support .........................................................................25 Annex A: Governance and Management Arrangements.............................................27 Annex B: Goals, Objectives, Outcomes, Outputs, and Activities................................30 Annex C: References........................................................................................................33 Annex D: Persons Consulted...........................................................................................34 Tables Table 3.1: Successful Achievment of Outcomes...............................................................10 Table 4.1: Project Costs by Component ............................................................................15 Table 4.2: Project Costs by Source of Finance..................................................................15 ii Acronyms C&W Cable & Wireless CAS Country Assistance Strategy CTU Caribbean Telecommunications Union ECTEL Eastern Caribbean Telecommunications Authority IBRD International Bank for Reconstruction and Development ICR Implementation Completion Report ICT Informatics and Communications Technology IDA International Development Association IEG Independent Evaluation group IMF International Monetary Fund ITU International Telecommunications Union NTRC National Telecommunications Regulatory Commissions OECS Organization of Eastern Caribbean States OIC OECS Telecommunications Implementation Committee PAD Project Appraisal Documents PCP Price Cap Plan PHRD Japan Human Resources and Development Fund PMU Project Management Unit PPIAF Public Private Infrastructure Advisory Facility RSMMS Radio Spectrum Monitoring and Measurement System TPCG Telecommunications Policy Coordination Group USAID U.S. Agency for International Development WTO World Trade Organization iii Preface EVALUATION OBJECTIVES AND METHODOLOGY This review of the Organization of Eastern Caribbean States (OECS) Telecommunications Reform project is one of 19 desk reviews undertaken as part of an independent evaluation by the Independent Evaluation Group (IEG) of the relevance, effectiveness, and efficiency of World Bank support for regional programs over past 10 years (1995-2004). Twelve of the reviews, including this assessment of the OECS Telecommunication Reform Project, are desk reviews; the other seven reviews are in- depth field studies. All reviews draw on core program documentation, progress reports, self- and/or independent program evaluations, related Bank country assistance strategies (CAS) and sector strategies, and interviews with key Bank staff. This documentation does not assess the transparency and efficiency with which the Eastern Caribbean Telecommunications Authority, the regional regulator established under the project, operates. And such an assessment is beyond the scope of this review. This desk review has involved reviewing: (a) the 1998 and 2001 CAS; (b) relevant project documents, including the project appraisal document (PAD, and the implementation completion report (ICR) and ICR review by IEG; and (c) other relevant internal reviews of the country program and project. In addition, it draws on an impact assessment, carried out by USAID and a consulting corporation (CARANA), which was commissioned by the regional telecommunications authority. See Annex C and D for a list of references and people interviewed respectively. EVALUATION CRITERIA The 19 reviews use the Operations Evaluation Department (OED) evaluation criteria of relevance, efficacy, and efficiency. In addition, they assess the Bank's performance and examine the performance of the regional program's participating countries. The key evaluative questions addressed under these criteria--designed to deal with the special characteristics of multicountry programs--are as follows. Relevance · Subsidiarity: To what extent is the program being organized and carried out at the lowest level effective, and how does it complement, substitute for, or compete with Bank country or global programs? · Alignment: To what extent does the program arise out of a regional consensus, formal or informal, concerning the main regional challenges in the sector and the need for collective action? To what extent is it consistent with the strategies and priorities of the region/subregion, countries, and the Bank? · Design of the regional program: To what extent is program design technically sound, and to what extent does it take into account the different levels of development and interests of iv participating countries, foster the confidence and trust among participants necessary for program implementation, and have clear and monitorable objectives? Efficacy · Achievement of objectives: To what extent has the program achieved, or is it likely to achieve, its stated objectives, including its intended distribution of benefits and costs among participating countries? · Capacity building: To what extent has the program contributed to building capacities at the regional and/or participating country levels? · Risk to outcomes and impacts: To what extent are the outcomes and impacts of the program likely to be resilient to risk over time? To what extent have the risks to project outcomes been identified and have measures to integrate them been undertaken? · Monitoring and Evaluation: Has the program incorporated adequate monitoring and evaluation processes and addressed available findings? Efficiency · Efficient use of resources: To what extent has the program realized, or is it expected to realize, benefits by using reasonable levels of time and money? · Governance, management, and legitimacy: To what extent have the governance and management arrangements clearly defined key roles and responsibilities; fostered effective exercise of voice by program participants and coordination among donors; contributed to or impeded the implementation of the program and achievement of its objectives; and entailed adequate monitoring of program performance and evaluation of results? · Financing: To what extent have financing arrangements affected positively or negatively the strategic direction, outcomes, and sustainability of the program? World Bank's Performance · Comparative advantage and coordination: To what extent has the Bank exercised its comparative advantage in relation to other parties in the project and worked to harmonize its support with other donors? · Quality of support and oversight: To what extent has the Bank provided adequate strategic and technical support to the program, established relevant linkages between the program and other Bank country operations, exercised sufficient oversight of its engagement, and developed an appropriate disengagement strategy for the program? · Structures and Incentives: To what extent have Bank policies, processes, and procedures contributed to, or impeded, the success of the program? Participating Countries' Performance · Commitments and/or capacities of participating countries: How have the commitments and/or capacities of participating countries contributed to or impeded the success of the program? Have one or more countries exercised a primary leadership role? · Program coordination within countries: To what extent have there been adequate linkages between the regional program's country-level activities and related national activities? v Executive Summary Background 1. In 1998, five member countries of the Organization of Eastern Caribbean States (OECS)--Dominica, Grenada, St. Kitts and Nevis, St. Lucia, and St. Vincent and the Grenadines--had expensive and erratic telephone systems. Their national legislation was outdated, their regulatory capacity was weak, and a private firm--Cable & Wireless-- had a monopoly on service. As a result, the cost of doing business was high, and this impeded private sector activities and economic growth. Launched in 1998, the OECS Telecommunication Reform Project brought the countries together to enhance efforts for telecommunication reform at the regional and national levels. Program Summary Description 2. Objectives: The project, which was the first World Bank operation to support reforms in telecommunications in the five OECS countries--comprised nine International Development Association (IDA) credits and International Bank for Reconstruction and Development (IBRD) loans, totaling $10.85 million.1 Conceived at a time of sweeping technological changes (involving use of Internet, cellular, and Voice-Over Internet services), the project's two main objectives were to develop pro-competition reforms in the sector and to increase supply of workers with informatics-related skills, such as computer and network specialists, software developers, and data processors. More than 80 percent of the project's funds supported reform and modernization. The project financed technical assistance and equipment to help: · Implement changes in regulations and design new telecommunications legislation; · Establish a coherent licensing regime, and a regional regulatory authority with supporting national commissions; · Encourage member participation in Caribbean-wide radio spectrum management; · Support and provide start-up financing for the regional regulatory authority, spectrum management, and other needed equipment; · Undertake a public information campaign on policy changes; and · Provide regional and national training on licensing, interconnection policy, number allocation, and dispute resolution. 3. In addition, midway through the project, the skill-building and training component was restructured to focus on Informatics Policy due to increasing private sector capacity in providing information. The revised component aimed to develop: (a) a regional Informatics and Communications Technology (ICT) strategy, and (b) an investment promotion strategy to increase investments in ICT, including private-public partnerships. 1Four OECS members did not participate in the project, namely Montserrat, Anguilla, Antigua and Barbuda and the British Virgin islands. Antigua, the sole World Bank member of the group was in arrears to the IMF. vi Rationale for a Regional Program 4. The regional project was relevant because countries had shared interests in reforming telecommunications and in enhancing national efforts by working together. Normally, regulation of telecommunication is addressed at the national level, with international cooperation, under the aegis of the International Telecommunications Union (ITU). In this case, countries believed that a regional approach was warranted for two reasons. First, they believed that by acting together, they would be in a better position to renegotiate their own contracts with Cable & Wireless, which ran until 2024 in St. Kitts and Nevis. Second, there were economies of scale to be gained by establishing a common regulatory authority rather than a separate authority in each small country; and there was a shortage of skilled labor in all countries for performing the needed regulatory functions. Moreover, the countries had previous experience working regionally and had pre-existing commitments towards economic integration. They also had previously established a successful regional regulatory body for civil aviation. Quality of Design & Implementation 5. Good design and implementation: The project was well-designed and well- implemented. It had clear goals and a defined scope. The Bank had proposed a broad program that aimed to strengthen private sector development, but quickly acquiesced to the five governments' demand for a discrete telecommunications operation that could be processed quickly. Thus the project was able to provide timely support to enable countries to meet impending deadlines on their contract renegotiations and deal with the key issue of strengthening their regulatory processes. 6. The technical design of the project drew on the Bank's significant analytical work on exports, private sector development, public utilities modernization, and capacity building for informatics. The project conducted an up-front risk assessment that detailed strategies for dealing with alternate scenarios including the risk of breakdown of regional cooperation and the possibility of one country reforming faster than another. The project also adopted a nuanced strategy that went beyond an all-or-nothing approach to reform. This strategy included arrangements for better regulating services that could be dominated by one or a few player(s) even after reform. 7. Institutional arrangements: The institutional arrangements for the project established clear roles, responsibilities, and reporting relationships between the governance and management entities. Ministers of communications and the OECS Secretariat served on the project's governing body, the Telecommunications Policy Coordination Group, which was responsible for setting policy, monitoring performance, and appointing the project's independent financial auditors. The project also established a working group, the Telecommunications Implementation Committee, comprising senior technical officials from the countries to deal with the day-to-day operational issues. The OECS Secretariat served as interim project manager, until the new regional regulatory authority--the Eastern Caribbean Telecommunications Authority (ECTEL)--was established and took over project management. vii Program Performance 8. The project achieved its objective of developing pro-competition reforms in telecommunications. It successfully: · Facilitated agreement between the heads of state for telecommunications reform, and helped create ECTEL as the world's first regional telecommunications authority. The OECS Secretariat, as the primary institution responsible for regional cooperation, facilitated attention to telecommunications reform during semi-annual heads of state meetings. · Enabled countries to agree on modalities for sector regulation, including on: (a) the scope of ECTEL's activities; (b) the development of national telecom legislation that was harmonized on broad policy and technical content, but adapted to suit local contexts; and (c) the creation of national telecommunications regulatory commissions in each country that would act nationally, but be guided by ECTEL's technical advice. · Provided and financed internationally renowned consultants and experts to help each country negotiate its contract with Cable & Wireless, dismantle the long standing monopoly, and allow for entry of new players. 9. An impact assessment showed increased access to telecom services, reduced prices, and increased employment opportunities. Prices of all telecom services fell by at least 50 percent, subscription growth rates tripled between 1996-2000 and 2000-2004, and 800 jobs were created in the region by new service providers and in Internet and customer equipment services.2 Would the sweeping technologies alone have led these countries to reform their telecommunications sector? While the project benefited from technological changes, ECTEL and Bank staff believes that this alone could not have introduced competition or advanced technologies as quickly, without the project's promotion of regional collaboration. Effectiveness of World Bank Performance 10. The analytical work established the Bank's credibility in the countries and the project benefited from having a high quality task team that balanced sector knowledge globally and in the OECS with regional experience necessary to guide implementation. The Bank maintained close dialogue with ECTEL and the OECS Secretariat to ensure alignment of project and national goals. Though the project was managed by the Global ICT Department, it benefited from having a single country director for the OECS who is responsible for all individual and multicountry programs in the subregion. 2E. Caribbean Information & Communication Technology Impact Assessment Report. CARANA/USAID. viii 1. Introduction CHALLENGES FACING THE SECTOR 1.1 Prior to the inception of the OECS Telecommunications Reform project, the telecom sector in the OECS was weak and inefficient. The 2001 OECS country assistance strategy (CAS) noted that unregulated monopolies and monopolistic pricing in utilities (power, water, and telecom) made the cost of doing business relatively high and hindered private sector activities, thereby impeding growth. Countries in the subregion shared an expensive and erratic telephone system. Their telecom legislation was inadequate and there was no regulatory capacity. A private firm, Cable & Wireless (C&W), had a monopoly in the sector and was subject to minimal government regulation. 1.2 The project was conceived at a time when there were sweeping technological changes occurring in telecommunications, such as the increasing use of internet, cellular, and Voice-Over-Internet Protocols services. By creating a regional regulatory authority, modernizing telecom legislation, and encouraging greater competition, the governments hoped to reduce telecommunications costs and spur private sector development. Moreover, the governments "hoped that by acting together, they could better address the issue of renegotiation of monopoly agreements with C&W and establish a more robust legal and regulatory framework."3 REGIONAL PROGRAM SUMMARY DESCRIPTION 1.3 Objectives: This regional project, approved in 1998 for $10.85 million, was the first Bank project in the telecommunications sector in the OECS subregion. Its main objectives were to develop: (a) pro-competition reforms in the telecommunications sector, and (b) an increased supply of informatics-related skills including software developers, computer and network specialists, data processors, and other technical experts. 1.4 More than 80 percent of the funds supported telecommunications sector reform and modernization. This component aimed to establish a regional framework to revise telecommunications regulation in participating states, modernize radio spectrum allocation, licensing and regulation, and establish a regional telecommunication regulatory authority. More specifically, the sector reforms involved the following elements: · Implementing changes in regulations, designing new telecommunications legislation, and establishing a coherent licensing regime and a regional regulatory authority with supporting national commissions; · Encouraging OECS participation in Caribbean-wide radio spectrum management; 3Implementation Completion Report. Page 2. 1 · Undertaking a public information campaign on the policy changes; · Supporting and providing start-up financing for the regulatory authority, spectrum management and other needed equipment; and · Training at the regional and national level on licensing, interconnection policy, number allocation, and dispute resolution. 1.5 Midway through the project, the skill-building and training component was restructured to deal with informatics policy. The revised component aimed to develop: (a) a regional ICT strategy; and (b) an investment promotion strategy to increase investments in ICT, including private-public partnerships. 1.6 Project financing: The original project cost was estimated at $10.1 million and comprised five International Development Association (IDA) credits and four International Bank for Reconstruction and Development (IBRD), totaling $3.6 and $2.35 million, respectively. The five OECS countries involved contributed equal amounts totaling $0.83 million. The countries borrowed the rest of the $1.19 million (50 percent through an IDA credit and 50 percent through an IBRD loan), with the exception of St. Kitts and Nevis, which did not have access to IDA credits and borrowed the full amount through an IBRD loan. The project closed in December 2004. 2 2. Relevance: Rationale, Alignment, and Design 2.1 Overall, the project was relevant to the economic needs and goals of the participating countries, and well-aligned with country and Bank development goals. Its objectives and scope gained widespread consensus from country counterparts. And it was designed to be narrow enough in scope to enhance the likelihood of success. SUBSIDIARITY PRINCIPLE 2.2 The project had to be undertaken in a multicountry setting for telecommunication policy and regulation to be economically viable and effective for the participating island states. The project appraisal document (PAD) notes that normally, regulation of telecommunications is addressed at the national level, with international cooperation on specific issues--such as spectrum allocation--that usually involves the multilateral International Telecommunications Union (ITU). A country-by-country approach would have avoided the time and effort required to achieve regional consensus on policy directions, and to deal with the political and administrative complexities of establishing a multinational cooperative mechanism. However, in this case, the countries (and the Bank) believed that a subregional approach was warranted for the following reasons: (a) There was a clear economic justification for working on a sub-regional basis: · Common issues: The issues facing each island were similar. In each, Cable & Wireless (C&W) was established as a legal monopoly and telecom reform would require renegotiation of existing contracts. · Cost sharing: It would be too expensive to establish effective policy and regulation mechanisms for each island, given the small market size and the fact that sharing the costs of regulation was in all the countries' interests. · Human capacity: There was a shortage of skilled labor in the sector in all countries. The creation of a regional regulator made it possible for the countries to hire internationally experienced consultants to help in the negotiation of contracts with C&W and to support the design and start-up of the regional regulator. Bank staff and the ECTEL management believe that this expertise would not have been available to countries individually. (b) Need to terminate monopoly. The countries believed that by acting subregionally, member governments would be in a better position to negotiate their own exclusive agreements on telecom services with C&W. The common cause of terminating a foreign monopoly provided strong motivation for countries to work together. (c) Timing coincided with technological changes. The timing was appropriate. Sweeping technological changes, which had significant potential to reduce telecommunication costs, strengthened the countries' will for reform. ECTEL staff noted that had the C&W monopoly continued, the new technology may not have been made available as quickly to the countries. The countries decided to use the opportunity presented by St. Lucia's 3 licensing renegotiation with C&W as a lever and springboard to launch the telecommunications reform process in all island states. 2.3 Experience with regional cooperation: It was also easier for countries to consider working on a sub-regional level, given: · The pre-existing commitment on the part of the member countries to economic integration in the subregion as demonstrated by a common currency and central bank, inter-island free trade, and recent steps toward labor market reform. · The existence of a prior model for a successful regional regulatory body--the OECS Civil Aviation Authority. 2.4 In essence, while the economies of scale drove the argument toward the need for a multicountry or subregional approach, member countries were also comfortable with that approach given their experience in working as a subregion on other issues. 2.5 But while the OECS comprises nine member states, four did not participate in the project.4 Since the establishment of the Eastern Caribbean Telecommunications Authority (ECTEL), three countries including the British Virgins Islands, Anguilla, and Montserrat have expressed interest in joining it. 2.6 Another option discussed during project design was the possibility of using the Caribbean Telecommunications Union (CTU) based in Trinidad, as a focal point for regulatory activities. This option was discarded because member countries felt that the CTU was too "`remote from OECS-specific needs" and would not be "subject to effective local control."5 ALIGNMENT WITH COUNTRY, REGIONAL, AND BANK GOALS AND STRATEGIES 2.7 From the onset, the project has been led by the heads of state of the participating OECS countries. This ensured that the project was well-aligned with the goals of the countries. For instance, the prime ministers of St. Kitts and Nevis, St. Lucia, and St. Vincent and the Grenadines publicly announced modernization of the telecommunication framework (that is, introducing competition in the sector) as a top political priority. 2.8 Before the project began, both Dominica and Grenada had made commitments under the World Trade Organization (WTO) Basic Agreement on Telecommunications to liberalize certain telecommunications services, and this was a key incentive for the two countries to support telecommunications reform. Thus, telecom sector reform was an important national and political priority for all participating states. The PAD notes that the commitment to prepare the project came "first and foremost from the political leaders of the five countries." 4Anguilla and British Virgin islands, two associate members of the OECS, and Montserrat did not participate. Antigua and Barbuda the sole World Bank member state that did not participate was in arrears with the IMF at the time of project design. 5Project Appraisal Document. Page 9. 4 2.9 The project was closely aligned with the Bank's strategic goals at the corporate and regional levels. At the corporate level, telecommunications reform was considered important for aiding growth and private sector development, a central Bank objective, as well as facilitating knowledge creation and sharing. The project is consistent with the Bank's strategy in the telecommunication sector, which calls for "shifting the government's role from ownership and operator to policy making and regulation, promoting efficiency and service quality, increased private sector participation in investments and provision of services, and promoting markets in the use of radio spectrum.6" 2.10 At the subregional level, the project is consistent with the 1995 OECS CAS goal of "diversifying the economic base through improvement in competitiveness and provision of infrastructure." The project also is aligned with the 2001 OECS CAS objective of "promoting sustainable, private sector-led economic diversification by providing a supportive environment for promising, newly emerging service industries, improving the provision of public services to the population, and strengthening the broader functioning of the public sector."7 REGIONAL CONSENSUS 2.11 The project benefited from consistent and unwavering commitment from the governments of the five participating countries at all stages of the project. The heads of state not only agreed on the need to reform telecommunications, but also on the need for a regional approach to ensure that: (a) costs remained manageable, and (b) there was adequate bargaining power to renegotiate contracts with C&W. "There was unanimity among OECS governments that combined telecommunications and regulatory activity on behalf of the region is appropriate and should be set up within the next three years."8 2.12 Countries faced similar issues. As noted in paragraph 1.3, the challenges facing the telecommunications sector in each of the five participating countries were similar. There was strong subregional consensus from government officials both on the need for sector reform and the use of a regional approach. The OECS heads of state in a public statement clarified their intention to proceed on telecommunications reform on a regional basis. The OECS Secretariat appointed staff to the project management unit (PMU) before project appraisal.9 2.13 Shared commitment to reform. The strong political support for sector reform was the key to the successful accomplishment of project goals. Prior to negotiations, countries even developed a detailed project implementation plan and a complete package of Terms of Reference for consultancy services. St. Lucia and St. Kitts and Nevis were the most 6Project Appraisal Document. Page 9. 7Implementation Completion Report. Page 4. 8Project Appraisal Document. Page 11. 9Prior to negotiations, the OECS Secretariat, on behalf of the five borrowers developed a detailed project implementation plan and a complete package of Terms of Reference for consultancy services. 5 active participants. St. Lucia, whose contract with C&W ran until 2000, first took on the Japan Policy and Human Resources Development Fund (PHRD) grant for project preparation and hosted ECTEL. St Kitts and Nevis, interested in re-negotiating its contract with Cable & Wireless which ran until 2024, was the first to pass new telecommunications legislation and set up a national telecommunications regulatory commission. DESIGN OF THE REGIONAL PROGRAM 2.14 Overall, the design of the project was satisfactory and the project benefited from having specific goals and clearly defined scope. 2.15 Initially, the Bank aimed to prepare an operation that would address private sector development constraints more generally through an Enterprise Development Program that would provide matching grants to "co-finance private firms' investments in product improvement and diversification, as well as market development."10 However, this suggestion was not supported by the participating countries for two reasons: · The OECS governments preferred to use the European Union's Stabex grant funds for a larger program.11 · The governments wanted a discrete telecommunications operation, which could be processed quickly, to take advantage of the momentum toward change and provide timely support to some OECS countries facing telecommunications decisions in the near future as a result of WTO commitments and the impending deadlines of monopoly contracts. 2.16 Thus, given the limited capacity of the governments to address many complex issues concurrently, this limitation in scope increased the project's likelihood of success, although the Bank cannot be credited for this element of design. 2.17 Other important design elements that contributed to project success were: (a) the significant analytical work undertaken by the Bank, and (b) the establishment (ultimately) of an experienced and knowledgeable Bank task team (see paragraph 6.6). 2.18 Bank analytical work: The Bank's previously conducted economic and sector work on service exports, private sector development, and public sector modernization. This significant analytical work, together with on-going policy dialogue, showed that high telecommunication and data communication costs were obstacles to developing non- tourism service exports that had the potential to generate significant employment, especially outsourced information processing. An Institutional Development Fund for capacity building for the Eastern Caribbean Investment Promotion Service--intended to develop the capacity to market informatics opportunities in the OECS--provided the 10Project Appraisal Document. Page 9. 11The expanded scope project framework and operating guidelines that had been developed and funded by a PHRD grant were used by the EU and the OECS governments. 6 basis for the training component. The Bank, at the time of project design, also was working closely with the OECS governments on the WTO Basic Agreement on Telecommunications, and in developing appropriate organizational structures to ensure coordination and cooperation among the OECS countries in regulatory matters. Bank internal review found that sector knowledge and background work done prior to processing the project was of high quality. Review also found that the project design focused on the right issues and included sufficient flexibility to handle varied concerns of the five participating countries. 2.19 The project also benefited from the task team's knowledge of information technology and telecommunications sectors, both globally and in the OECS countries, and their links to the macro-economy. This was balanced with adequate regional experience necessary to guide project implementation. 2.20 Training: The restructured training component also performed well. The training component was designed during pre-appraisal, replaced midway with a component that aimed to develop informatics and communications technology (ICT) strategies, and has led to a follow-up project on ICT development and reform (see paragraph 3.8). 2.21 Risk management: The risk management strategy for the project was satisfactory. The Bank clearly identified the project as having substantial risk. And the quality at entry assessment commends the Bank for having a "well thought-out strategy for dealing with alternative scenarios" including the implication of risks of the break down of regional cooperation and of one country reforming faster than another. The project was also commended for its nuanced strategy to achieve liberalization and regional reform that went beyond an all-or-nothing approach. One criticism has been that the project did not identify the risk of regulatory capture and suggest mechanisms to guard against the possibility.12 But interviews with Bank staff and ECTEL noted that the strong political will and support for sector liberalization minimized this risk. CLARITY AND MONITORABILITY OF OBJECTIVES 2.22 Limited emphasis on monitoring and evaluation: The project had developed performance indicators at both output and outcome levels, but monitoring and evaluation (M&E) was not a specific area of emphasis during design. After an internal quality assessment, these indicators were improved to include not only institution-building objectives but also market-improvement outcomes resulting from these institutional measures (see Annex B). Monitoring and evaluation arrangements were criticized as not adequately linked to the risks identified and to alternative strategies that had been established at design. For instance, the monitoring indicators did not specify how goals should be changed if some of the risks materialize, such as the risk that some countries would work faster than others to set up national telecommunications laws and policies, and the risk that negotiation with C&W would not be fruitful in all countries. 12Regulatory capture refers to a situation where the regulator's independence is diminished through inappropriate close associations with the firms being regulated. 7 2.23 Performance measures through impact assessment: In the latter stages of implementation, the project commissioned an impact assessment that was conducted jointly by USAID and CARANA Corporation. This assessment provided quantitative estimates of the project's benefits and costs on several levels. Such an assessment was possible due to the nature of the telecommunications sector and readily available information on subscriptions, prices and other market characteristics. The ICR drew on these assessments to report on quantitative performance measures established during design. Thus, despite the limited emphasis on monitoring and evaluation, there is reliable quantitative information on project impacts and outcomes. The follow-on project however, is expected to make a more concerted effort to monitor and evaluate outcomes and impact. 8 3. Efficacy: Outcomes, Impacts, and Sustainability 3.1 Overall, the project was successful in (a) creating the world's first regional telecommunications authority, (b) dismantling Cable & Wireless' long-standing monopoly in telecommunication services in the subregion, (c) allowing entry of new players, and (d) expanding telecommunication services rapidly and reducing costs. It also achieved the equal distribution of benefits which it sought, and the risks to its outcomes and impact is low. The project benefited from the technological changes sweeping the sector at the time and it is likely that some sector reform would have occurred even if the Bank has not been involved. How much of this is strictly attributable to the project is hard to estimate; still, the project's achievements were considerable. ACHIEVEMENT OF OBJECTIVES 3.2 Objectives: The project's goals were two-fold: to develop (a) pro-competition reforms in the telecommunications sector, and (b) increase supply of informatics related skills. The first goal of sector reform accounted for 83 percent of project costs and 80 percent of Bank financing. The second goal, training, was restructured midway because a large number of private firms and educational institutes had started to provide quality training on informatics, and both the Bank and participating governments believed that this development negated the need for training. The training goal was thus replaced by an "informatics policy" goal that aimed to develop two regional strategies: one for ICT development and another to promote and increase investment in ICT, including private- public partnerships. 3.3 Achievement of Outcomes: The project achieved its objective of developing pro- competition reforms in the telecommunications sector and the associated outcomes noted in Table 3.1. Access to telecommunications services improved significantly (triple the number of subscribers); the mobile users market grew rapidly (highest cellular penetration in the Caribbean); and competition in other segments, such as Internet and fixed lines, increased. The project successfully: · Facilitated high-level policy dialogue (among heads of governments), which was necessary to obtain consensus on the need for sector reform, and the means to achieve it. The OECS Secretariat, as the primary institution responsible for regional cooperation as well as the interim project management agency, helped focus political attention at the highest levels on telecommunications issues. · Enabled countries to develop a common view of how the sector would be regulated after de-monopolization. This included agreement on: (a) the need for a regional telecommunications regulator and the scope and nature of its activities; (b) the development of national telecom legislation, which while adapted to the local context, would be harmonized on broad policy and technical content; and (c) the creation of national telecom regulatory commissions (NTRC) in each country that would act nationally, but under the technical advice and guidance of the regional regulator. 9 · Provided technical expertise--that is, internationally renowned consultants and experts--to help each country negotiate its contract with C&W. This, in the view of Bank staff, was critical factor for the success of the project. · Although negotiations with C&W were conducted on each national contract, the simultaneous negotiation of the contracts of all five participating countries added to the collective bargaining power of the states. The project facilitated the coordination and technical assistance necessary to conduct negotiations simultaneously. Interviews with Bank staff suggest that financing technical assistance for the negotiation, as well as helping participating states obtain an experienced international consultant, were critical for success. Table 3.1: Successful Achievement of Project Outcomes Outcomes Achievements Competitive prices Prices: a. Dramatic reduction in international (67 percent), national for bandwidth and (48 percent), and mobile (50 percent) tariffs between 1998 and data intensive 2003. services b. Under the Price Cap Plan (PCP) mentioned below, ECTEL states paid 22 percent less on fixed-to-fixed local calls from Jan. 1, 2005, with a further reduction of 20 percent from December, 2005. c. The PCP also provides for further reductions in local tariff rates (free minutes and discounts). d. Renegotiations of C&W's exclusive agreement through 2000 with St. Lucia and through 2024 with St Kitts and Nevis. Increased access: Cellular penetration increased from 2.5 percent in 2000 to 72 percent in 2004. Subscriber growth rate tripled between 1996-2000 and 2000-2004. Increased competition: 14 entrants were granted licenses to provide fixed and mobile services, with an additional 114 licenses issued in other areas. Methodologies for Where there is competition (for example, mobile markets), cost price-setting in oriented methodologies do not apply. For regulated markets, the regulated markets PCP sets the maximum prices for services offered by a telecommunications company with dominant or monopoly advantage. It does this by imposing constraints on the extent to which the rates for a "basket" of services can increase or decrease. In this case, the PCP provides a methodology for setting basic prices for 18 categories of services that contains 200 rate elements. 3.4 The project met its output indicators (listed in Annex G) for establishing the Eastern Caribbean Telecommunications Authority (ECTEL) and improving the sector's regulatory framework. The project prepared new telecom legislation for all countries that has since been approved by the national legislatures. As a result, all five participating countries now have a modern regulatory framework that is fully consistent with that of the other participating OECS countries. 3.5 On radio spectrum management, despite significant delays in procurement of the necessary equipment (see paragraph 4.6), there is now an operational regional Radio Spectrum Monitoring and Management System (RSMMS). The RSMMS issues licenses 10 in accordance with new International Telecommunications Union standards and monitors compliance, deals with complaints on allocation in a timely manner, and proactively works with NTRCs to prevent radio interference, and addresses cross-border frequency issues. 3.6 The restructured informatics and communications technology (ICT) component, though modest by design, was accomplished. A new regional ICT strategy was developed and adopted by countries in 2002. And an investment promotion strategy, as well as a pilot ICT project, for funding and investment was identified. These policy-level strategies have been endorsed by the OECS and are the basis for the development of national ICT action plans in the participating countries. This component has become the basis on which a follow-on Bank multicountry project, the Telecommunications and ICT Development Project, was approved in August 2005. This follow-on project aims to enhance the access, quality, and use of telecom services--that is, to promote universal access to ICT services and utilization of broad-band applications. 3.7 Beyond outcomes, into impact: Beyond reduced prices and increased access to telecom services, there was an accompanying expansion in employment opportunities as newly licensed competitor companies started operations. Direct employment by new service providers increased from 377 in 2001 to an expected 693 by the end of 2004. Over 500 new jobs also were created in the region in Internet services and customer premises equipment services, along with additional jobs in companies that gained from C&W outsourcing a number of operations13. This suggests that the overall impact of reform and development of the ICT sector on labor growth within the sector alone was positive. In addition, direct investment in the sector increased from $40 million in 2001 to more than $87 million in 2003, and was expected to increase to $94 million in 2004. 3.8 Overall, "the initial $10 million World Bank-financed project can take significant credit for an annual consumer benefit of approximately $20 million in terms of lower fixed line prices delivered to date, as well as the considerable rollout and price reductions seen in other market segments including mobile and Internet." Both the initial project and additional consulting support valued at $0.55 million were "instrumental in negotiating the recent price cap agreement which is scheduled to deliver $7.5 million in additional price reductions over the next two years alone.14" Competitive ICT service pricing is also expected to have created significant new opportunities for businesses in the economy as a whole. Further, the economic benefits do not include the significant social benefits of connectivity in building communities and standards, as well as improving service delivery and quality of governance in the region. 3.9 While the project can be credited with facilitating sector liberalization, in part, the project was able to capitalize on changes and technological developments in the telecom industry. For instance, monopoly positions would have become harder to maintain with evolving direct satellite telecommunication linkages. Income from sale of spectrum rights 13The Eastern Caribbean Information and Communication Technology (ECICT) Impact Assessment Report produced by CARANA/USAID. 14Ibid, 15. 11 is declining worldwide with new technologies that enable "spectrum hopping". Nonetheless, while the project benefited from these changes, there is a view among ECTEL and Bank staff that technology alone would not have been able to introduce competition or advanced technologies in the island states as quickly without the project's promotion of regional collaboration. 3.10 Despite the project's considerable achievements, it is beyond the scope of this review to assess the efficiency of ECTEL's performance or the transparency of its operations. CAPACITY BUILDING 3.11 The main capacity building dimension of this project is the establishment of a new regional telecommunications regulator, ECTEL, based in St. Lucia, which is the first of its kind in the world. ECTEL's goal is to ensure "transparent, objective, competitive, investor-friendly licensing and regulatory regimes" in the participating countries and to serve as a "forum for coordination" of sub-regional telecom policies and regulations. National telecommunications regulatory commissions (NTRCs) were established in each country under revised or new national telecommunications laws. The NTRCs are expected to implement regional regulations and policies at the national level, with technical assistance from ECTEL. The first NTRC was created in St. Kitts and Nevis in 2001 and a common mission statement for all NTRCs was set in 2003. 3.12 Financing for ECTEL: The project successfully created and launched ECTEL based on the treaty of May 2000. ECTEL was conceived as a financially sustainable organization that receives its financing from royalties, spectrum and license fees, and other income as the council of ministers may decide and that is provided for in the telecommunications act passed in each country and the ECTEL treaty. Its financial sustainability and day-to-day operations are designed to be independent of the political authorities in the member countries (see paragraph 4.9). Its annual expenditures and revenues are reviewed by a budget advisory committee, and a fee collection mechanism is in place to ensure timely deposits from the participating countries to ECTEL. 3.13 Technical capacity: In accordance with a 2002 agreement on ECTEL's staffing structure, all staff are on time bound service contracts and technical expertise is concentrated at the regional (not national) level. ECTEL's board comprises representatives of participating countries who are appointed for a three-year term by the ministers of telecommunications in consultation with the cabinet. The representatives are expected to have professional training in law, finance, economics, or other relevant professions, and the ICR concludes that ECTEL "satisfies most internationally accepted criteria for a competent telecommunication regulatory agency." 3.14 Cross-country information sharing: The project also built mechanisms for cross- country learning and sharing of experiences into its institutional structure (see paragraph 4.7). A working group called the OECS Telecommunications Implementation Committee (OIC) comprising senior officials from the participating countries met regularly, and this facilitated sharing of information and experience. Moreover, these officials were involved 12 in implementing the project at the national level in their countries and were empowered to make regional decisions on behalf of their member governments. REALIZED DISTRIBUTION OF COSTS AND BENEFITS 3.15 All participating governments shared in financing the project equally, and expected the benefits to apply equally. At appraisal, the project identified the risk that some countries would reform faster than others and that the negotiations with monopoly telecom providers may not be successful in all countries. These risks did not materialize. 3.16 The high level of borrower commitment from all participants and government-to- government peer pressure ensured timely completion and approval by parliaments of the necessary legislation in all cases. And even in St. Kitts, where the agreement at the time extended for four more years (until 2004), the project was successful in re-negotiating the monopoly agreements with C&W, despite some initial set backs. 3.17 One key factor in the project's success is the balance it sought between regional regulatory control and national control. The ministers of telecommunications in all participating countries are represented on the governing board of ECTEL. They approve the ECTEL sponsored legislation and at the country level. Each national government remains responsible for implementing its own national telecommunications legislation and maintains full authority over its own national telecommunications regulatory commission (NTRC), which implements national regulations (based on the premise of the regional regulations), and receives technical advice from ECTEL. Thus, participating countries did not perceive a significant loss of national control. For instance, ECTEL is responsible for the final approval of spectrum licenses. National legislation stipulates that the government issue a license only on recommendation from ECTEL. By treaty, ECTEL has the authority to set price caps for a variety of telecom services in markets where there are very few large service providers and competition remains limited. NTRCs are responsible for implementing and abiding by the price caps set by ECTEL. Interviews with ECTEL staff suggest that while some countries had pushed for a stronger national role and weaker regional role, the project achieved an appropriate compromise, with the roles of the regional regulators and national governments governed through the ECTEL treaty. RISKS TO OUTCOMES AND IMPACTS 3.18 There appear to be few risks to project outcomes and impact. Based on the actions taken to-date as well as the countries' agreement to launch a follow-up project,15 there appears to be little risk of countries reversing policy reforms. Moreover, countries that previously did not participate in ECTEL have indicated greater interest in joining. This expansion would further enhance the project's impact and the sustainability of ECTEL 15Project Appraisal Document. 2005. 13 and sector reform in the subregion16. In addition, the increased use of mobile communications in the islands is expected to positively affect funding for ECTEL. 3.19 There are two reasons for caution on the assessment of risks in the long run. One issue raised by ECTEL and Bank staff is the delay of some countries in transferring licensing fees and revenues collected on behalf of ECTEL. In the short term, this is not a significant risk, since ECTEL has established a special fund from spectrum management fees to protect itself from such delays. However, ECTEL staff noted that prolonged delays could be problematic. Second, though ECTEL has enjoyed strong political commitment to-date, there is a risk of political interference in the efficient and transparent functioning of ECTEL. 16Anguilla, British Virgin Islands, and Montserrat are expected to accede to ECTEL. 14 4. Efficiency: Governance, Management, and Financing 4.1 Overall, the project achieved all its major objectives with minor delays in costs and time. It garnered legitimacy from the most important actors--the heads of state of participating countries--to enable successful sector reform. While the financing arrangements were elaborate and caused some inefficiency, they were adequate for achieving the project's objectives. EFFICIENT USE OF RESOURCES 4.2 The project was completed successfully though not within the planned time frame and resources. The project was slightly over-budget and closed a year later than planned. 4.3 Project cost: The original project cost was estimated at $10.1 million. Of this, $5.95 million was financed through IBRD loans and IDA credits, totaling $3.6 and $2.35 million, respectively.17 As shown in Tables 4.1 and 4.2, actual project costs were marginally over-budget by $0.75 million (7.4 percent), and the actual portion of loan and credit used was under-budget by $0.42 million (5.4 percent). In addition to the $10 million, the Bank funded technical assistance worth $0.55 million to fund consultancies for the negotiations on a Price Cap Plan (PCP) where a dominant provider (such as C&W) remained in effect. Table 4.1: Project Cost by Component (in $ million) Component Planned Actual Actual (% planned) A Telecom Regulatory Framework 8.50 10.14 119 B. Informatics 0.85 0.11 13 C. Project Management 0.75 0.60 80 Total Project Cost 10.10 10.85 Additional TA Financing 0.55 Source: Implementation Completion Report. Table 4.2: Project Cost by Source of Financing (in $ million) Component Planned Actual Actual (% plan) Bank Govt. Bank Govt. Bank Govt. A Telecom Regulatory Framework 4.80 3.70 5.38 4.76 112.1 128.6 B. Informatics 0.60 0.25 0.11 0.00 18.3 C. Project Management 0.55 0.20 0.14 0.46 25.5 Total Project Cost 5.95 4.15 5.63 5.22 Additional TA Financing 0.55 * Project preparation includes studies and training financed through a Japanese PHRD Trust Fund (JPN 29611) Source: Implementation Completion Report. 17The five OECS countries contributed $0.8 million equally, and borrowed $1.2 million, with a split of 50 percent IDA credit and 50 percent IBRD loan, with the exception of St. Kitts & Nevis which did not have access to IDA credits and borrowed the full amount through an IBRD loan. 15 4.4 Long preparation period: The project concept changed significantly from inception, leading to a relatively long preparation period of 20 months and a high level of resources (189 staff weeks at a cost of $840,000).18 On balance, these staff and consultant expenditures were not thought to be excessive, given the difficulties of working across several small countries. 4.5 Delays on the RSMMS: The project experienced delays in procuring, testing, and implementing the enabling hardware and software for the RSMMS. The ICR notes that "after intensive supervision, the RSMMS was brought on line in a fully satisfactory manner about 12 months later than expected." The delay was attributed to the complexity of undertaking procurement using an intricate system in five countries, delays in issuing letters of credit, and additional delays by contractors in certifying acceptance. These slippages led to the extension of the project closing date to December 2004, instead of the planned September 2003. The ICR notes that ECTEL lost revenue as a result of these delays: "Had the spectrum management equipment been installed in a timely manner, ECTEL would have had a greater accuracy in assessing spectrum fees and could have issued invoices faster." 4.6 Complex coordination: Despite the best of efforts from the Project Management Unit (PMU) and the high level of government commitment, the coordination required for the project remained complex and stretched project management and supervision resources thin. This project involved five ministers of communications, five prime ministers, and two international organizations (the OECS Secretariat and ECTEL) at all critical junctures of implementation. Even with the best intentions, there were delays, particularly in obtaining signatures from countries on the two extensions to the project's closing date. GOVERNANCE, MANAGEMENT AND LEGITIMACY 4.7 Governance: As a Bank project, the governance of the OECS Telecommunications Reform Project during its implementation rested ultimately with the Bank's Board of Governors.19 In addition, the project established governance and management arrangements to facilitate implementation (see Annex A). The main coordinating/governing body was the newly created OECS Telecommunications Policy Coordination Group (TPCG), comprising ministers of communications of the participating countries. The TPCG represents and reports directly to the member governments and is responsible for policy setting and monitoring project performance. It also appoints the independent financial auditors for the project. In addition to the TPCG, the project also established a working group called the OECS Telecommunications Implementation Committee (OIC), comprising senior officials from the participating 18Bank budget of $201,000, Japanese Policy and Human resource Development Trust Funds of $440,000, and $80,000 from Project Preparation Facility. 19Since then, ECTEL has its own governance and management system (see 4.20 below). 16 countries who are empowered to make regional decisions on behalf of their member governments.20 4.8 Management: The responsibility for project management rested with the OECS Secretariat in the initial stages. It was expected that the regional telecommunications authority or ECTEL would be established and by midway (within 2.5 years), the management would then be transferred to it. A separate PMU was set up in the OECS Secretariat, under the Division for Economic Affairs, comprising a project manager and support staff. An Educational Reform Unit was also set up under the PMU to manage implementation of the training component. The project manager was responsible for implementing the telecom modernization component, overseeing the training component, and serving as a liaison with the OIC. 4.9 The project facilitated the development of a clear governance and management structure for ECTEL, whose life extends beyond the project. ECTEL is governed by a Council of Ministers from the participating countries and a representative of the OECS Secretariat. The council of ministers appoints and directs ECTEL's Board of Directors, approves its budget, and determines its internal structure. The Board comprises one member from each state appointed by the minister for a one-year term, and the managing director as an ex-officio member. It is responsible for advising management and making recommendations to the council on all telecom matters, and establishing rules and procedures consistent with the treaty for management and operation of ECTEL. ECTEL's day-to-day management rests with the Directorate, made up of the managing director, and professional, technical, and support staff.21 4.10 Successful institutional arrangements: The institutional arrangements for the project were clear from the start, and the roles and responsibilities as well as reporting relationships among the various entities involved in the project were clearly established. The arrangements were transparent and provided for regular reports to the policy and implementation committees, and thereby, to the governments of the participating countries. There are several elements of the project's institutional arrangements that facilitated successful implementation: · The use of a committee of senior officials, who were empowered to make decisions on behalf of their government, facilitated adequate and continued PMU coordination with the member states and ensured that the project's goals remained consistent with the goals of the individual member countries. · Signaling the strong political commitment, the OECS Secretariat appointed PMU staff before project appraisal. The staff hired for the PMU was adequately qualified. That "continuity of staff, both in the OECS Secretariat, the PMU and 20Each country would establish a national ministry for telecommunications with overall responsibility for telecommunications policy and some mechanism for licensing and regulation. 21ECTEL Website: http://www.ectel.int/ectelnew/documentdoc/liberalisation/Media.pdf. 17 ECTEL, fostered a close working relationship and a productive channel of communication between the Bank, ECTEL and the OECS Secretariat.22" · The management structure of ECTEL was performance-oriented. As mentioned in paragraph 3.13, all ECTEL staff was hired on time-bound service contracts. Even the Board of Directors, who were appointed for a three-year term by the ministers of telecommunications, were expected to have professional training in law, finance, economics, or other relevant professions. · ECTEL was conceived at the time of design as an independent and financially sustainable entity that receives its financing from royalties, spectrum and license fees, and whose regular operations are independent of the political authorities of the participating countries (see paragraph 3.12 above). 4.11 The project did not explicitly involve communication with the public in each country, and there was little direct involvement of the ultimate beneficiaries (that is, the consumers) in the project's design and implementation. But the Bank consulted with public sector, private sector, and nongovernmental organization (NGO) representatives from all five participating countries during the course of preparing a private sector development project (that was ultimately overtaken by the European Union). At these interviews, the stakeholders unanimously supported telecommunications reform and voiced their support for Bank involvement in the same. This, along with the easy and direct visibility of benefits from sector de-monopolization --in terms of reduced rates and enhanced telecom access to the consumers--gave legitimacy to the design and implementation. FINANCING 4.12 Administrative costs: The financing arrangements were tedious and this significantly increased the administrative costs of coordinating and managing the project. In all, the project involved five IBRD loans and four IDA credits to the participating countries. This meant larger Bank administrative costs in preparing and tracking individual loan and credit agreements and difficulties in coordinating the approval on all nine agreements when the project's closing date was extended for a year (as discussed in paragraph 4.6). In addition, the Bank used technical assistance from the Private-Public Infrastructure Advisory Facility (PPIAF) to finance consultancies required to develop an approach paper on the modalities for co-operation and accession to ECTEL by the various countries that were not originally part of the project but that subsequently expressed an interest in joining ECTEL. 4.13 Project preparation: The lack of a single counterpart agency that could borrow from the Bank required cumbersome burden-sharing arrangements for the handling of grant resources for project preparation. St. Lucia obtained a Japan PHRD grant for project preparation on behalf of the others. Grenada and St. Kitts and Nevis took the 22Implementation Completion Report. Page 11. 18 Project Preparation Fund credit advances for preparation of country-specific issues. The OECS Secretariat secured an additional grant from the Caribbean Development Bank to aid preparation, implemented all these grant activities, and committed staff time and resources for project preparation. DONOR PERFORMANCE 4.14 There are no other donors in this project and hence this section does not apply. See paragraph 6.11 for the Bank's role in coordinating with other donors active in the sector in the OECS countries. 19 5. Monitoring and Evaluation 5.1 Although monitoring and evaluation (M&E) was given little attention during the design of the project (see paragraph 2.10 above), it did prepare sound quantitative output and outcome indicators, and towards the latter stages of implementation, ECTEL commissioned an Eastern Caribbean informatics and communications technology (ICT) impact assessment. This assessment, financed partly by the Bank, was conducted by USAID and the CARANA Corporation to determine the project's impact on the direct costs and access to telecommunication services, as well as other economic benefits like growth and employment. The results of this assessment have been detailed in paragraphs 3.10-3.14 above. 5.2 In part, the nature of the telecommunications sector enabled Bank to report on progress against the quantitative indicators set out at design. The sector characteristics also enabled the CARANA assessment to determine the visible impacts on the economy. 20 6. World Bank Performance 6.1 Overall, the quality of Bank support and oversight was good, and the Bank successfully disengaged after the achievement of the project's objectives. The project was supported adequately by internal Bank structures and incentives. Its performance on coordinating with the other donors, such as the European Union (EU), which was active in infrastructure development, has been modest. COMPARATIVE ADVANTAGE 6.2 At the time of project design, the Bank had been actively engaged in policy dialogue with all participating countries on the WTO Basic Agreement on Telecommunications, and in developing mechanisms for coordination and cooperation among the OECS countries on regulatory issues (see paragraph 2.14). Furthermore, this dialogue was supported with grant funding for preparatory work on infrastructure (including telecom) regulation and reforms. This interaction, along with in-depth analytical work on related private sector development and public sector modernization issues, provided the Bank with considerable knowledge about telecommunications in the subregion, which the clients valued. Clients perceived the Bank to have global experience in establishing regulatory authorities, upon which they could draw. And according to Bank staff, clients stated that the Bank's presence lent international credibility to the transparency of the process. THE BANK'S COORDINATING ROLE WITH OTHER DONORS 6.3 There were no other co-financiers or partners to the project. Internal review concluded that the team could have done more to seek partnerships on the discontinued training component (with the International Finance Corporation or IFC), and to improve coordination with other donors working in the region (for example, USAID and the EU). The Bank did increasingly engage with other donors midway through the project when the informatics and communications technology (ICT) gained greater emphasis. But as more donors become actively involved in ICT-related areas, the OECS Secretariat and the Eastern Caribbean Technologies Authority (ECTEL) noted the need for better donor coordination to facilitate scaling up ICT programs in the region. Thus, the Bank's performance in this regard was modest. QUALITY OF SUPPORT AND OVERSIGHT 6.4 The quality of Bank support was good. The Bank picked up on the opportunity presented by sweeping technological changes in the sector to activate the countries' political commitment for reform. The Bank mobilized and supported significant economic and sector work (ESW) in response to client demand. Thus, in the clients' view, the Bank had: (a) developed considerable sector knowledge on the issues facing the subregions, and (b) established its credibility well before the project commenced. Moreover, when the initial design of a private sector development project did not meet 21 country demands, the Bank was willing to limit the project's scope to the telecom sector only. 6.5 The project also provided close supervision by a consistent team with adequate technical expertise. Despite the frequent changes in task managers in the early conception and design stages (see paragraph 5.6), the Bank maintained close dialogue with the OECS Secretariat, ECTEL and the Telecommunications Policy Coordination Group (TPCG) to ensure that the project's goals were directly in line with the sector goals of the participating countries. This along with an effective and consistent staff mix in a stable task team during the implementation phase was "important for keeping the Bank's client relationship productive and effective in resolving urgent issues."23 6.6 The OECS Secretariat and ECTEL have noted the "harmonious" and fruitful working relationship" the Bank maintained with all parties. Also, clients have valued the "continued efforts on procurement and disbursements of ex-post reviews in supervision missions."24 STRUCTURES AND INCENTIVES 6.7 The development of a regional project is easier in the OECS than in some regions, because the countries involved identify themselves as belonging to a subregion and the Bank's internal administrative structure for the OECS countries corresponds to the classification used by the countries. The Bank has a single country director for the OECS who is responsible not only for individual country programs but also for multicountry subregional programs. This combination of a pre-existing country grouping and a Bank structure aligned to it facilitates the development of regional projects in the OECS states. 6.8 The ownership for the program within the Bank's subregional country assistance strategy (CAS), and direct support from the country director were noted as a critical for project success in interviews with Bank staff. It enabled the Bank to speak with "one voice" to the clients and also made available to the task team the Bank's subregional resources for supporting areas such as administrative, legal, procurement, and financial management services. And though the project was administered by the Global ICT Department, which is not part of the country team, the department did have a close relationship with the country team and briefed the country director before and after supervision missions. LINKAGES TO OTHER BANK COUNTRY OPERATIONS 6.9 There was no other Bank project in telecommunications in the OECS at the time of design, although this project was clearly aligned with improvements being sought though Bank support. 23ICR Section on Bank Performance. 24See ICR Section on Borrower Comments. Page 11. 22 6.10 Five other regional Bank projects listed in the 2001 CAS did not materialize due inadequate government commitment to a regional approach.25 Countries were hesitant about some subregional interventions as shown by their preference to deal with donors, individually on country programming, and by their reluctance to finance and empower the OECS Secretariat for regional activities or common policies. The 2005 OECS CAS notes that "although the OECS Development Charter broadly charts a way forward, it has proved too general to serve as an adequate framework for either leveraging regionalism or facilitating donor co-ordination around a set of agreed priorities. ... The reluctance of OECS heads of government to give responsibility to the OECS Secretariat for coordinating donor assistance has meant that ad hoc national agendas have continued to undermine the formulation and implementation of a strong regional agenda."26 6.11 But despite the unsuccessful experience with other regional and subregional projects, this project benefited from a high degree of political commitment that out- weighed the difficulties and implementation challenges of operating subregionally. 25OECS CAS Completion Report Review. 2005. 26OECS CAS Completion Report. Page 53. 23 7. Country Participation 7.1 The participating countries were heavily involved in the conceptualization, design, and implementation of the project. In addition to statements made by the heads of state, the ministers of communications of the five countries were instrumental during preparation in defining the terms of reference for initial technical studies of the telecom sectors in the countries, indicating the high level of political commitment. 7.2 Further, even before the counterpart funding on the regional project, each country contributed $30,000 to expand the scope of the initial technical work to be undertaken with the Japan Human Resources and Development (PHRD) grant and Caribbean Development Bank grant financing. The project continued to enjoy full commitment from participating member countries and was viewed as a flagship project by the OECS Secretariat. The countries paid their counterpart funding in full and on time. 24 8. Conclusions SUMMARY OF FINDINGS 8.1 Overall, this project was relevant to the needs of the participating countries and arose as a result of direct demand from them. This project was initiated at a time of high demand for reforms, and when the technological changes across the telecommunications sector were significant enough to drive and sustain country interest. The high degree of political commitment through-out resulted in successful project implementation to create an independent and financially sustainable regional regulator. IMPLICATIONS FOR EFFECTIVE SUPPORT · Countries' commitment contributes to project success. Regional programs that arise out of demands from participant countries tend to enjoy greater political commitment, and therefore are more likely to succeed. In this case, political commitment was driven by the common goal of introducing competition into the telecommunications sector. This, along with WTO obligations and the technological changes sweeping the sector, provided a strong economic rationale for reform. Without this commitment, it would have been difficult to renegotiate Cable & Wireless' (C&W) exclusive arrangements for all telecommunication services (only through 2000 in St. Lucia, but continuing through 2024 in St. Kitts and Nevis). Getting the five OECS countries to act on this commitment by passing the ECTEL treaty in 2000, and enacting new telecommunications laws beginning with St. Kitts in June 2000, was also important for the success of the project. · Limited scope and goals sustain political commitment. Countries chose to limit the project to clearly defined goals that represented the areas of common interest, and this was important to sustain political commitment. · Multicountry projects are complex, even if supported with strong political commitment . While a high level of political commitment is necessary for moving forward on a regional project, it does not reduce the complexity of its preparation or oversight. · Rationale for a regional approach must be strong. The rationale to support a regional approach should be strong enough to overcome the perceived costs (complexities and coordinating difficulties) and to sustain country ownership. In this case, the rationales were: (a) All participant countries shared a common cause or goal (lower telecommunications costs in this case, and a common large monopoly provider (C&W) against which countries needed to fight. (b) The regional approach had a clear economic justification that included lack of adequate skills and human resources (in this case, it would have 25 been expensive for each island to have a telecom regulator given the small market size and limited human resource skills). · Regional cooperation is easier when benefits can be seen quickly. Regional cooperation is facilitated in an area such as telecommunication reforms, where the benefits to the population can be demonstrated in a fairly short period of time. · Governance and management arrangements are important to success of regional project. Governance and management arrangements are key factors that can contribute to or impede the success of a regional project. (a) In addition to specific institutional structures for project implementation, regional projects need to specify mechanisms for engaging senior levels of government on key policy issues. In this instance, the OECS Secretariat facilitated the discussion of the high level telecom policy issues critical to the success of project activities with the heads of state. (b) All participating countries must perceive that they have adequate voice and decision making authority at all stages of preparation and implementation. They also need to be assured that there is adequate transparency regarding agreed upon actions at the country and regional level. · A regional project benefits from being located within the Bank under a single country director. There was one interlocutor with clients, and it made available the country team's legal, procurement and financial management support. 26 Annex A: Governance and Management Arrangements Management Agencies for Project Implementation 1. Management arrangements PMU was first established under the OECS Secretariat, under the for project Economic Affairs Department, and subsequently moved to the implementation. regional telecom authority, ECTEL. See chart below. Regional Coordinating Body(ies) 2. Name, location, and Eastern Caribbean Telecommunications Authority (ECTEL) internet address of the Vide Boutielle, P.O Box 1886, Castries, St. Lucia regional implementing Telephone: (758) 1701/1702 Fax: (759) 458 1698 agencies/coordinating Web: www.ectel.int body for the project. E-mail: ectel@ectel.int 3. Current size 4. Current membership Five countries: See table below 5. Membership criteria Signatories of the ECTEL treaty of May 2000 6. Membership Under the treaty, the contracting member states undertake to: responsibilities (a) collaborate and co-ordinate with each other and with ECTEL; (b) take all appropriate measures for ensuring implementation of the policy and recommendations of ECTEL; (c) Meet the financial and other commitments under this treaty to ensure the efficient operations of ECTEL. 7. Functions of the (a) Coordinate with and advice contracting states on the conduct and coordinating body regulation of telecommunications and ancillary matters. (b) Prepare and maintain a harmonized regional radio spectrum plan. (c) Recommend a regional policy for conduct of telecommunications, in particular in relation to universal service, interconnection, pricing and numbering, and to monitor implementation in contracting states. (d) Recommend to contracting states the type of telecommunications networks or services that should be subject to individual licenses and class licenses, respectively, and exemptions, if any. (e) Recommend to the contracting states the type of communications networks, services and radio equipment which should be subject to frequency authorization, and exemption, if any. (f) Prepare and recommend to the contracting states forms, other instruments and regulations for the adoption by contracting states for the purpose of harmonization of telecommunications, including application forms and other forms in respect of licenses, frequency authorizations and tender documents. (g) Design and operate open tender procedures for individual licenses as requested by contracting states. (h) Review applications for individual licenses submitted by contracting states, and recommend applicants who satisfy the relevant technical and financial requirements for individual licenses. (i) Recommend to contracting states terms and conditions to be included in a license, particularly with respect to the provision of universal service; (j) Make recommendations to contracting states on matters relating to the management of frequency authorizations, including the sale, if any, by auction. 27 (k) Monitor, in collaboration with the contracting state, effectiveness of the license for the purpose of this treaty and make the appropriate recommendation to the contracting state including recommending suspension or revocation of the license. (l) Recommend to the contracting states an appropriate fee structure for licenses or other matters for or in relation to the conduct or regulation of telecommunications. (m) Recommend a regional cost-based pricing regime for implementation by each contracting state. (n) Recommend technical standards and procedures for the approval of equipment, including radio equipment for use in the operation of telecommunications, in each contracting state. (o) Coordinate activities with relevant international organizations, states or other bodies or persons for the promotion and implementation of this treaty. (p) Advise contracting states on the management of the Universal Service Fund and make recommendations on applications for disbursements from these funds. (q) Prepare annual reports for submission to the Council on the execution of its functions. (r) Perform such other functions as are assigned to it by resolution of the Council. 8. Meeting frequency 9. Name, location, and Organization of Eastern Caribbean States internet address of other Morne Fortune, P. O. Box 179 implementing Castries, St. Lucia agencies/coordinating Phone: (758) 452-2537 body for the project. Fax : (758) 453-1628 Email: oesec@oecs.org Country Council of Ministers Board of Directors Commonwealth of Dominica Honourable Reginald Austrie Mr. Clifford Parillon Grenada Honorable Gregory Bowen Mr. Michael Pierre St. Kitts and Nevis Honorable Nigel Carty Dr. Ken Ballantyne St. Lucia Honorable Felix Ministered Mr. Embert Charles St Vincent and the Grenadines Honorable Jerrol Thompson Mr. Isaac Solomon Senior Management Position/Title Email address Mr. Eluid Williams Managing Director ewilliams@ectel.int Mr. Donnie De Freitas Director of Technical Services ddefreitas@ectel.int Mr. Derrick Redman General Counsel dredman@ectel.int Mr. Roy Humes Spectrum Management Specialist rhumes@ectel.int Ms. Cheryl Hector Senior Financial Analyst chector@ectel.int Ms. Heather Blanchard Accountant hblanchard@ectel.int 28 Chart A.1: Institutional Arrangement for the Project OECS Telecommunications Policy Coordination Group (TPCG) OECS Secretariat's Director Independent of Economic Affairs Division Auditors Project Manager at PMU (OECS Secretariat, ECTEL) Accountant, Legal Officer, Educational Reform OECS Telecommunications & Admin. asst. Unit Officer Implementation Committee 29 Annex B: Goals, Objectives, Outcomes, Outputs, and Activities Goals/Mission Program Intended Outcomes Outputs Inputs/Activities Objectives Diversify the Pro- * Competitive prices for * Establish RTA or Regional 1. Regional Telecom Sector economic base competition critical bandwidth and Telecommunications Authority Policy and Strategy through reforms in the data intensive telecom Performance Indicators: 2. Regulatory authority structure improving telecommunica services 1. Submission of a plan to agreed to by countries and related competitiveness tions sector Performance Indicators: establish RTA training. in the provision None 2. Legislation establishing RTA of passed in at least 3 key countries infrastructure. * Development of cost- within 2 years of effectiveness. oriented methodologies for services. * Improve/modernize telecom Performance Indicators: regulatory framework. 1. Report on Regulatory Options % of services priced Performance Indicators: using cost-based 1. Newly-negotiated license (St. methodology Lucia) to be consistent with new ----year---- 1 2 3 4 framework. telephone 30 50 100 2. Review current sector Internet 40 60 100 legislation within a year of leased lines 35 40 50 100 effectiveness. 3. New sector legislation passed in each country within three years of effectiveness. Improve capacity to enforce 1. Spectrum management licensed use of radio frequency equipment and capability spectrum. Performance Indicators: 1. Improve performance on issuing of licenses according to ITU standards. 2. Reduce spectrum allocation complaints by 50 percent within a year of effectiveness. 30 Goals/Mission Program Intended Outcomes Outputs Inputs/Activities Objectives Increase supply Increase supply of * Increase in number of trained 1. Training needs assessment of of informatics informatics related skills. informatics specialists. informatics industry. related skills. 2. Survey of suppliers of (restructured) Performance Indicators: informatics training. 1,000 people trained within two 3. Establishment of demand- years of effectiveness. based voucher scheme with procedures aimed at efficiency and transparency. Enhance development of Develop (a) a regional informatics the ICT sector. and communications technology strategy, and (b) an investment promotion strategy to increase investments in ICT, including private-public partnerships. Components A. Telecommunications Regulatory Framework Modernization (Total 8.5 m, Bank 4.8 m): Establish framework for sector reform, including: * Implementing proposed changes in regulations, designing new telecommunications legislation, and establishing a coherent licensing regime and an appropriate regional regulatory authority; * OECS participation in Caribbean-wide radio spectrum management workshop; * Undertaking public information campaign regarding policy changes; * Supporting and start-up financing for the regulatory authority, including spectrum management and other equipment and training on licensing, interconnection policy, number allocation, and dispute resolution. B. Pilot for Informatics Training (Total 0.9 m, Bank 0.6 m): * Technical assistance to assess current supply and demand of informatics training. * A demand-based voucher system to provide a well-focused and limited subsidy for informatics training. Trainee/employer would contribute a portion of the total training costs. The voucher system would have: > clearly defined eligibility criteria to (a) identify voucher recipients; (b) select individuals; and (c) identify suppliers of informatics training; 31 > partial pre-allocation of voucher funds among participating OECS countries (together with a non-allocated amount so as to encourage competition among OECS countries for these funds). The training goal was restructured into an `Informatics Policy' objective that aimed to develop: (a) A regional informatics and communications technology (ICT) strategy, and (b) An investment promotion strategy to increase investments in ICT, including private-public partnerships. C. Project Management (Total 0.8m, Bank 0.6 m): Financial support to the OECS Secretariat for project implementation costs including: * Monitoring and audit * Training and increasing implementation capacity of the OECS Secretariat and other critical public agencies responsible for implementing and sustaining telecom sector reform. 32 Annex C: References Eastern Caribbean Telecommunications Authority (ECTEL) Treaty and information on the ECTEL web site. Internal Documentation. Public Private Infrastructure Advisory Facility (PPIAF). 2003. Gridlines on Regional Telecommunications Regulator for the Caribbean. Oct 2003. USAID/CARANA ECICT. 2004. Impact Assessment: ECTEL States. USAID/CARANA ECICT Project. January 2004. http://www.carana.com/ecict/documents/Impact_Assessment.pdf World Bank. 2005a. Implementation Completion Report on Credits and Loans in the Amount of SDR 1.8 Million and USD 3.6 Million, Respectively to the Commonwealth of Dominica. Grenada, St. Kitts and Nevis, St. Lucia, and St. Vincent and the Grenadines for an Organization of Eastern Caribbean States (OECS) Telecommunications Reform Project. Report No. 31780. June 14, 2005. _________. 2005b. Organization of Eastern Caribbean States Country Assistance Strategy (CAS). September 6, 2005. _________. 2005c. Organization of Eastern Caribbean States Country Assistance Strategy (CAS) Completion Report. March 2005. _________. 2005d. Project Appraisal Document for Dominica, Grenada, St. Kitts and Nevis, St. Lucia, and St. Vincent and the Grenadines - Telecommunications and ICT Development Project. Project Appraisal Document. Report No. 31395. April 2005. _________. 2001. Organization of Eastern Caribbean States Country Assistance Strategy (CAS) Completion Report. June 4, 2001. _________. 1998. Project Appraisal Document for Dominica, Grenada, St. Kitts & Nevis, St. Lucia, St. Vincent & the Grenadines - OECS Telecommunications Reform Project. Report No. 17772. May 1998. World Bank and ECTEL. Undated. Telecommunications Reform in the OECS ­ Impacts on Prices and Services. Report No. 36232. 33 Annex D: Persons Consulted Robert Schware ­ TTL. Mr. Eluid T. Williams, managing Director and senior staff of ECTEL, Castries, St. Lucia. 34