Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00006074 IMPLEMENTATION COMPLETION AND RESULTS REPORT IDA-H8830 ON A GRANT IN THE AMOUNT OF SDR 1.50 MILLION (US$2.15 MILLION EQUIVALENT) AND TF-A3687 ON A GRANT IN THE AMOUNT OF US$0.95 MILLION AND IDA-D3190 ON A GRANT IN THE AMOUNT OF SDR 2.5 MILLION (US$3.55 MILLION EQUIVALENT) TO THE PACIFIC AVIATION SAFETY OFFICE FOR THE PACIFIC AVIATION SAFETY OFFICE REFORM PROJECT May 31, 2023 Transport Global Practice East Asia and Pacific Region CURRENCY EQUIVALENTS (Exchange Rate Effective December 31, 2022) Currency Unit = United States Dollar US$ 1.33 = SDR 1 FISCAL YEAR July 1 – June 30 Regional Vice President: Manuela V. Ferro Country Director: Stephen N. Ndegwa Regional Director: Ranjit J. Lamech Practice Manager: Benedictus Eijbergen Task Team Leader: Dung Anh Hoang ICR Main Contributor: Keelye Hanmer ABBREVIATIONS AND ACRONYMS ADB Asian Development Bank AWP Annual Work Program CAA Civil Aviation Authority CAASP Civil Aviation Authority Strengthening Program CPF Country Partnership Framework DFAT Australian Department of Foreign Affairs and Trade E&S Environmental and Social FM Financial Management GASOS Global Aviation Safety Oversight System GEMS GASOS Electronic Management System ICAO International Civil Aviation Organization ICR Implementation Completion Report ICT Information Communication Technology IDA International Development Association IFR Interim Financial Report IRI Intermediate Results Indicator ISR Implementation Status and Results Report KPI Key Performance Indicator MDTF Multi-Donor Trust Fund M&E Monitoring and Evaluation MFAT Ministry of Foreign Affairs and Trade (New Zealand) PAD Project Appraisal Document PAIP Pacific Aviation Investment Program PASO Pacific Aviation Safety Office PDO Project Development Objective PIC Pacific Island Country PICASST Pacific Islands Civil Aviation Safety and Security Treaty PIDP Pacific Inspector Development Program PIFS Pacific Islands Forum Secretariat PNG Papua New Guinea PRAS Pacific Regional Aviation Strategy PRIF Pacific Regional Infrastructure Facility RAMM Regional Aviation Ministers Meeting ROI Register of Inspectors RSOO Regional Safety Oversight Organization SDR Special Drawing Rights SLA Service Level Agreement TAIP Tonga Aviation Investment Project TAL Tonga Airports Limited TFSU Technical and Fiduciary Service Unit USD United States Dollar USAP Universal Security Audit Program USOAP Universal Safety Oversight Audit Program VSAT Very Small Aperture Terminal WB-RFP World Bank Regional Partnership Framework WPPM World Plan Percentage Model TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5 A. CONTEXT AT APPRAISAL .........................................................................................................5 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION ..............................................................10 II. OUTCOME .................................................................................................................... 15 A. RELEVANCE OF PDOs ............................................................................................................15 C. EFFICIENCY ...........................................................................................................................22 D. JUSTIFICATION OF OVERALL OUTCOME RATING ....................................................................24 E. OTHER OUTCOMES AND IMPACTS .........................................................................................24 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 27 A. KEY FACTORS DURING PREPARATION ...................................................................................27 B. KEY FACTORS DURING IMPLEMENTATION .............................................................................28 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 29 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................29 B. FIDUCIARY, ENVIRONMENTAL AND SOCIAL (E&S) COMPLIANCE.............................................31 C. BANK PERFORMANCE ...........................................................................................................33 D. RISK TO DEVELOPMENT OUTCOME .......................................................................................34 V. LESSONS AND RECOMMENDATIONS ............................................................................. 35 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 48 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 50 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 51 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 52 ANNEX 6. SUPPORTING DOCUMENTS .................................................................................. 53 The World Bank Pacific Aviation Safety Office Reform Project (P145057) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P145057 Pacific Aviation Safety Office Reform Project Country Financing Instrument Pacific Islands Investment Project Financing Original EA Category Revised EA Category Not Required (C) Not Required (C) Organizations Borrower Implementing Agency Pacific Aviation Safety Office Pacific Aviation Safety Office Project Development Objective (PDO) Original PDO To ensure effective regional delivery of aviation safety and security oversight in Pacific Island Countries by strengthening thePacific Aviation Safety Office’s technical and coordination capacity. Revised PDO The project development objective is to strengthen the coordination capacity of the Pacific Aviation Safety Office to deliver regional aviation safety and security oversight, and technical and advisory services to the Pacific Island Countries. Page 1 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 2,150,000 2,111,013 2,128,142 IDA-H8830 950,000 633,815 633,815 TF-A3687 3,550,000 3,550,000 3,376,535 IDA-D3190 Total 6,650,000 6,294,828 6,138,492 Non-World Bank Financing 0 0 0 Borrower/Recipient 0 0 0 Total 0 0 0 Total Project Cost 6,650,000 6,294,828 6,138,492 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 30-Sep-2013 12-Dec-2013 20-Apr-2016 31-Dec-2018 31-Dec-2022 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 05-Jan-2017 1.60 Additional Financing 23-May-2018 2.23 Additional Financing 18-Dec-2018 2.68 Change in Loan Closing Date(s) 20-Dec-2019 3.62 Change in Loan Closing Date(s) Change in Legal Covenants Change in Institutional Arrangements Other Change(s) 03-Dec-2021 5.13 Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Reallocation between Disbursement Categories Change in Procurement Change in Implementation Schedule Page 2 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Moderately Satisfactory Modest RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 16-Dec-2013 Satisfactory Satisfactory 0 02 07-Jun-2014 Satisfactory Satisfactory .54 03 23-Jan-2015 Satisfactory Moderately Satisfactory .69 04 19-May-2015 Moderately Satisfactory Moderately Satisfactory .69 05 05-Dec-2015 Moderately Satisfactory Moderately Satisfactory .69 06 16-Jun-2016 Moderately Satisfactory Moderately Satisfactory 1.39 07 22-Dec-2016 Moderately Satisfactory Moderately Satisfactory 1.60 08 12-Jun-2017 Moderately Satisfactory Moderately Satisfactory 1.60 09 28-Nov-2017 Moderately Satisfactory Moderately Satisfactory 1.97 10 25-May-2018 Moderately Satisfactory Moderately Satisfactory 2.23 11 11-Dec-2018 Moderately Satisfactory Moderately Satisfactory 2.68 12 14-Jun-2019 Moderately Satisfactory Moderately Satisfactory 2.91 13 05-Feb-2020 Moderately Satisfactory Moderately Satisfactory 3.62 14 19-Sep-2020 Moderately Satisfactory Moderately Satisfactory 4.40 15 24-Dec-2020 Moderately Satisfactory Moderately Satisfactory 4.51 16 24-Jun-2021 Moderately Satisfactory Moderately Satisfactory 4.73 17 14-Aug-2021 Moderately Satisfactory Moderately Satisfactory 4.73 18 11-Feb-2022 Moderately Satisfactory Moderately Satisfactory 5.13 19 18-Aug-2022 Moderately Satisfactory Satisfactory 5.77 20 07-Dec-2022 Satisfactory Satisfactory 6.19 Page 3 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) SECTORS AND THEMES Sectors Major Sector/Sector (%) Information and Communications Technologies 5 Public Administration - Information and 1 Communications Technologies ICT Services 3 Other Information and Communications Technologies 1 Transportation 95 Public Administration - Transportation 83 Aviation 12 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Private Sector Development 56 Regional Integration 56 Public Sector Management 44 Public Administration 44 Administrative and Civil Service Reform 44 ADM STAFF Role At Approval At ICR Regional Vice President: Axel van Trotsenburg Manuela V. Ferro Country Director: Franz R. Drees-Gross Stephen N. Ndegwa Director: John A. Roome Ranjit J. Lamech Practice Manager: Michel Kerf Benedictus Eijbergen Task Team Leader(s): Christopher J. De Serio Dung Anh Hoang ICR Contributing Author: Keelye Rinchen Hanmer Page 4 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. The aviation sector provides vital national, regional, and international connectivity for Pacific Island Countries (PICs). Air connectivity serves as a lifeline for PICs, allowing for the import and export of essential goods, as well as for education and medical needs to be met. Air services enable effective regional integration and are a pre-requisite for tourism development, the main economic driver for PICs. A reliable network of air links – within and among island countries, and to major hubs such as Australia and New Zealand and beyond – is vital for the viability of PICs from humanitarian, political and economic perspectives. Therefore, an effective aviation safety and security oversight system is required to ensure that services are maintained and, where possible, expanded to facilitate the development of tourism and the economies of PICs. 2. As part of a broader effort by PICs to increase regional integration and address the issue of scarce, highly specialized human resources in small island states, the Pacific Aviation Safety Office (PASO) 1 was formally established, with ratification by the Pacific Islands Civil Aviation Safety and Security Treaty (PICASST), in 2006. 2 Formed through an Asian Development Bank (ADB) loan that was guaranteed by four member states (Kiribati, Papua New Guinea, Samoa, and Vanuatu), PASO’s role is to provide technical advisory services to State Civil Aviation Authorities (CAAs), as well as aviation safety and security oversight. It performs entry control and auditing of airport and airline operators and aims to assist with aviation capacity building, so that they may meet the international standards required under the International Civil Aviation Organization (ICAO) convention. All developing member countries of PASO have agreed to regulatory harmonization, 3 as well as to an annual levy to help finance PASO and the implementation of a PASO oversight program. 3. Compliance with aviation safety and security regulation is a non-negotiable obligation under international treaties. Assessments of the aviation sector in the South Pacific by ICAO have found that many of the countries lack the proper policies, regulations, and infrastructure to comply with ICAO’s Standards and Recommended Practices. PICs are required to maintain international and domestic air services as part of a safe and secure air transport system. Therefore, ensuring that PASO remains a viable entity is an urgent priority. Unfortunately, most PASO members are unable to fully meet all of their oversight obligations, and any interruption to PASO’s services would be detrimental to the provision of safe and secure air transport services across the region. 4. PASO provides advisory, inspection, and oversight services covering the areas of: (i) aerodrome assessment; (ii) security; (iii) aircraft airworthiness; (iv) flight operations; and (v) personnel licensing. However, the subscription and fee-for-service business model has not proven to be sustainable since: (i) countries have not purchased the necessary safety oversight services; (ii) there is a real, or perceived, lack of qualified technical specialists in PASO to perform the technical services, affecting demand; (iii) some countries are in arrears on member subscription fees; and, (iv) salaries and cost structures for PASO do not correctly reflect the demand for services. 1 www.paso.aero. 2 www.forumsec.org/wp-content/uploads/2018/02/Pacific-Islands-Civil-Aviation-Safety-and-Security-Treaty-PICASST.pdf 3 The current Pacific Islands Civil Aviation Safety and Security Treaty (PICASST) signatories are the Pacific nations of the Cook Islands, Kiribati, Nauru, Niue, Papua New Guinea, Samoa, Solomon Islands, Tonga, Tuvalu, and Vanuatu. Associate members of PASO are Australia, Fiji, and New Zealand. Government representatives from these nations make up the PASO Council. Page 5 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) 5. In December 2011, the World Bank approved the Pacific Aviation Investment Program (PAIP), an adaptable regional program that consisted of a series of projects designed to ensure that critical aviation infrastructure met operational safety requirements and to strengthen the regulatory compliance of international air transport in participating countries in the region. Major elements of PAIP focused on upgrading airport infrastructure and aeronautical equipment and strengthening policy and regulatory ability. Phase 1 of PAIP included projects in Kiribati, Tonga, and Tuvalu. Phase 2 included Samoa in April 2014. Phase 3 and Phase 4 covered Vanuatu in March 2015 and the Solomon Islands in March 2019, respectively. PAIP also found a need to strengthen compliance with international safety regulations and PASO was recognized as a potential PAIP beneficiary of regional World Bank funding during Phase 1. 6. The World Bank Regional Partnership Framework (WB-RPF) FY17–FY21 4 for nine PICs was structured around four focus areas: (i) fully exploiting available economic opportunities; (ii) enhancing access to employment opportunities; (iii) protecting incomes and livelihoods; and (iv) strengthening the enablers of growth and opportunities (macro- economic management, infrastructure, and addressing knowledge gaps). The WB-RPF highlighted that the economic geography of PICs, which is characterized by internal dispersion and remoteness, is a major constraint to ending absolute poverty and increasing shared prosperity. Furthermore, the WB-RPF highlighted the need to explore opportunities to further regional approaches for the Pacific region, given that many of the solutions to development challenges are regional in nature, e.g., connectivity through aviation, and that regional approaches would provide benefits from economies of scale, centralized implementation arrangements, and the efficient use of World Bank resources by supporting multiple countries through consolidated engagement. Under Objective 4.2: Increased access to basic services and improved connective infrastructure, the WB-RFP specifically highlighted that the World Bank would continue to work to improve operational safety and oversight of international air transport infrastructure, building on the PAIP. 7. At the time of project identification, PASO’s effectiveness and efficiency was faced with several limitations: (i) PASO was constrained by its governance arrangements. It did not have delegated authority and enforcement status associated with a state regulatory body, but rather provided states with advice and recommendations to be assessed and acted upon by the respective state regulatory body, as deemed appropriate. (ii) PASO was a not‐for‐profit regional entity operating on a subscription‐based fee‐for-services business model. Since its inception, PASO has functioned as a professional services supplier with insufficient chargeable days and, therefore, insufficient revenue to cover its full‐time consultant salaries or the remaining fixed costs. In the absence of appropriate corporate systems and internal controls, the organization continued to operate with on‐ going operational losses through to 2013. (iii) Some states expressed concern about PASO’s ability to undertake its core functions, while at the same time holding unrealistic expectations about the delivery of these services on demand. PASO must, therefore, be adequately staffed with qualified and experienced personnel and be backed up by appropriate systems and documentation. 8. When established with funding from the ADB loan, it was envisaged that PASO would be financially self- sustaining through a combination of annual subscriptions and service fees. Unfortunately, PASO has operated at an annual financial loss since its inception. A PASO Financial Review funded by the Pacific Regional Infrastructure Facility (PRIF) contains a short-term cash flow forecast that projected insolvency before the end of calendar year 2013. 5 At this point, it was determined that if PASO was to disappear, or its operations further weaken, several member states would face significant challenges in meeting national and international regulatory obligations with practical and affordable 4 IBRD/IDA/IFC/MIGA Regional Partnership Framework for Kiribati, Nauru, the Republic of Marshall Islands, Federated States of Micronesia, Palau, Independent State of Samoa, Kingdom of Tonga, Tuvalu, and Vanuatu for the period FY17–FY21 (Report No. 100997- EAP). 5 CAPA Consultants. PASO Financial Review. June 2013. Page 6 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) service alternatives in the short to medium term. Furthermore, if PASO became insolvent, Kiribati, Papua New Guinea, Samoa, and Vanuatu, as guarantors of the ADB loan that established PASO, would incur repayment obligations of US$0.4 million to ADB. 9. In July 2012, with funding from PRIF, a consulting firm was engaged under PAIP to prepare the PASO Business Plan. This assignment assessed PASO’s capacity and capability to deliver its regulatory services as intended. This included a review of the services offered by PASO; its promotion, pricing, and delivery mechanism; and how these aligned with the aviation sectors’ need, client demand, and PASO’s founding objectives. The PASO Business Plan was approved and implemented in May 2013, with the key findings and recommendations incorporated into the basis for the World Bank PASO Reform Project. In 2013, funding for PASO came from three main sources: (i) payments under the Government of Vanuatu Hosting Agreement; (ii) member subscriptions from 10 member states and three associate member states; and (iii) fee for service payments from member states for inspection work and other services. During project appraisal, a major challenge to PASO’s governance identified was the inadequate attention devoted to financial management planning and reporting. In addition, the envisaged commitment from member states to purchase PASO services or pay subscription fees was not realized. 10. Considering the urgent need for ensuring the continuation of PASO services in the region, the PASO Reform Project was prepared within a condensed timeframe with the aim to provide transitional financial support to a change management process and identify alternative funding mechanisms to ensure its future operations (Component A), provide technical assistance for the identification and implementation of a new organizational framework (Component B), ensure appropriate quality management systems to provide assurance that oversight is being managed by member states (Component C). Figure 1. PASO governance arrangements 11. The PASO Reform Project operated within the governance arrangements highlighted in Figure 1. The PASO Council, which must convene annually, is guided by the PASO Constitution, which sets out the intent and procedural norms of the Council. The Council is comprised of a member state or associate member delegate, often the director of civil aviation or a ministerial level representative. The role of the PASO Council is to provide guidance and oversight to PASO and to set the strategic direction of project activities, review overall project performance, and oversee financial expenditure against planned investment. Theory of Change 12. At project appraisal, there was no theory of change in the Project Appraisal Document (PAD). However, the Results Framework implied that PASO was designed based on the theory of change presented in Figure 2. Page 7 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) Figure 2. Theory of change at project appraisal Critical assumptions: 1. The original implementation period is sufficient to complete project activities. 2. Adequate technical design. 3. PASO capacity risks mitigated through additional regional support from the PAIP. Project Development Objectives (PDOs) 13. At project appraisal, the PDO, as stated in the Financing Agreement, was to “ensure effective regional delivery of aviation safety and security oversight in Pacific Island Countries by strengthening the Pacific Aviation Safety Office’s technical and coordination capacity.” 14. The PDO was revised to “strengthen the coordination capacity of the Pacific Aviation Safety Office to deliver regional aviation safety and security oversight, and technical and advisory services to the Pacific Island Countries.” Key Expected Outcomes and Outcome Indicators 15. At project appraisal, the PDO was expected to be measured through three indicators: (i) successful recruitment of technical coordinators with expertise in airworthiness, flight operations, security, and aerodromes; (ii) establishment of a register of inspectors (ROI); and (iii) percentage of recommended annual work program implemented by PASO member states. Page 8 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) Components 16. At project appraisal, PASO consisted of three components and the total project cost was estimated at US$2.15 million. Table 1 provides a summary of the components and financing allocations at the time. Table 1. Project components and financing allocation at project appraisal Component Description Activities Financing allocation Component A: Comprises a (i) Implementing a new organizational framework and staff US$1.65 million Transitional program of changes, including financing of severance packages for (76.5% of total Management activities to termination of existing contracts of staff financing at and Support assist PASO (ii) Conducting legal reviews for strengthening its governance appraisal) through the and management restructuring (iii) Evaluating the PASO funding approach, including finding new process revenue sources including, (iv) Conducting workshops and seminars for PASO Council financing the members costs (v) Developing a communications strategy (vi) Providing transitional support to implement the new Business Plan, including engagement of the Technical and Fiduciary Service Unit (TFSU) for provision of support to PASO as per terms of a Service Agreement (vii) Carrying out annual financial audits of the Project as agreed with the Association Component B: Supports the (i) Establishing a comprehensive ROI for the region available for US$0.269 Establishment establishment PASO-related services million of a Pool of of a pool of (ii) Establishing a Regional Program of Inspectors to build and (12.5% of total Regional inspectors to assess the capacity of inspectors from member states financing at Aviation be used by (iii) Provision of necessary training for inspectors to ensure that appraisal) Inspectors PASO to they meet skill levels necessary to provide PASO services oversee (iv) Provision of training to PASO staff to develop the human aviation safety resources to deliver the necessary services and security Component C: Helps to (i) Establishing a quality assurance system for PASO to ensure US$0.236 Quality strengthen that all operations meet an appropriate standard million (11.0% Management overall quality (ii) Implementing a PASO Management Information System with of total management an appropriate IT network, including a document management financing at of PASO system appraisal) Page 9 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) B. SIGNIFICANT CHANGES DURING IMPLEMENTATION Revised PDO Indicators 17. Revisions were made to PDO-level indicators and corresponding targets during project implementation, as highlighted in Table 2. Details related to the changes are described under Section II of this Implementation Completion Report (ICR). 18. The PDO revision did not represent a significant difference to the intended objectives of the project. As noted above, the project appraisal PDO to “ensure effective regional delivery of aviation safety and security oversight in Pacific Island Countries by strengthening the Pacific Aviation Safety Office’s technical and coordination capacity” was revised to “strengthen the coordination capacity of the Pacific Aviation Safety Office to deliver regional aviation safety and security oversight, and technical and advisory services to the Pacific Island Countries”. The objectives of delivering on aviation safety and security oversight and strengthening technical and coordination capacity remained; the change in the PDO was to recognize PASO as the primary beneficiary of the Regional International Development Association (IDA) Grant. 19. Changes to PDO indicators were made largely to be able to better demonstrate project achievements and to better reflect changes in the project path to deliver the same expected outcomes. PDO 1 was revised to capture the new organizational framework approved by the PASO Council and to capture the successful recruitment of these positions. PDO 2 remained unchanged. PDO 3 was not revised; however, the end target value was adjusted in 2020 due to COVID-19 challenges. Annex 1 provides details on the Results Framework and achievement of indicators. Table 2. Changes to PDO-level Indicators PDO indicator Indicator at time of Appraisal Changes during implementation PDO 1 • Successful recruitment of technical coordinators • Successful recruitment of general with expertise in airworthiness, flight operations and manager, operations manager, corporate security/aerodromes services manager PDO 2 • Establishment of an ROI • No change PDO 3 • Recommended annual work program implemented • No change to indicator, target revised by PASO members PDO 4 • N/A • ICAO Regional Safety Oversight Organization Service Level • PASO addresses at least 80% of the Global Aviation Safety Oversight System (GASOS) functions for a Level 2 Regional Safety Oversight Organization (RSOO) 6 Revised Components 20. A new component was added during project implementation. Component D: Supporting Regional Aviation Infrastructure was added to the project as part of a restructuring, approved in 2017. Although this component was added and the scope of the project expanded to be more ambitious in terms of delivery, the inclusion of Component D and the 6The revision was made as part of a Project Restructuring in 2021 to reflect the results that would be achieved by PASO at project closing (see Table 4). Page 10 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) provision of infrastructure was not core to the success of the project. Table 3 provides details on the additional component and rationale. Details on the impact of these changes are described in Section II below. Table 3. Changes to project components during project implementation Component Description Activities Financing allocation Component D: A new component was approved in a January (i) Very Small Aperture US$0.99 Supporting 2017 project restructuring and additional Terminal (VSAT) installations in million Regional Aviation financing was received from PRIF for the Cook Islands and Niue Infrastructure implementation of the infrastructure activities. Project Restructuring and Additional Finance 21. The PASO Reform Project was financed by Regional IDA allocations and the PRIF Multi-Donor Trust Fund (MDTF). The original IDA Grant (IDA-H8830) in the amount of SDR 1.5 million (US$2.15 million equivalent) was approved by the World Bank on September 30, 2013 and became effective on December 12, 2013. This grant was fully disbursed by the original closing date of December 31, 2021. The additional finance and restructuring are detailed in Table 4. Table 4. Additional finance and project restructuring Restructuring/date Key revision(s) Rationale Project Restructuring • Revise the PDO • The change to the PDO recognizes PASO as the – Level 1 and • Include a Component D: primary beneficiary of the regional IDA Grant. Additional Finance 1 ‘Supporting Regional Aviation • Component D was financed through the PRIF (TF0A3687) Infrastructure’ to provide VSAT to MDTF and administered by the World Bank. As the Niue and the Cook Islands Cook Islands and Niue are self-governing Approved on January • The PRIF MDTF grant funding for territories in free association with New Zealand, a 5, 2017 PASO in the amount of US$0.95 non-client member of the World Bank, World Bank million constituted the first Board approval was required as New Zealand is Effective on February additional financing. not eligible to benefit from World Bank financing. 13, 2017 • The Results Framework was updated to reflect the level of conformance by member states to their agreed PASO annual work plans to include the Cook Islands and Papua New Guinea, which were inadvertently excluded, and to include an indicator to capture the addition of Component D. Page 11 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) Project Restructuring • The second additional financing of • Funding resources were not sufficient to meet the – Level 2 and SDR 2.5 million (US$3.55 million project’s investment expectations. Additional Finance 2 equivalent) • The project period was extended to enable the (IDA-D3190) 7 • A three-year extension of the new financing modality and additional project closing date of the IDA Grant from resources were provided to support member state Approved by World December 31, 2018 to December readiness with the new model and, therefore, help Bank on May 23, 2018 31, 2021 to ensure that the PASO is increasingly financially • Revise the Implementation viable. Effective on Schedule September 3, 2018. • Adjust the Results Framework • Revise the Disbursement Projections • Change components Project Restructuring • Extend 12-month extension of the • An extension was required to ensure final delivery – Level 2 closing date to December 31, 2019, and installation of VSAT in the Cook Islands and for the PRIF MDTF Niue. Approved by World • Amend Disbursement Category 1 • Operating costs were removed as part of the Bank on December of the Financing Agreement (IDA- second additional financing and reintroduced as 18, 2018 H883/IDA-D319) to reintroduce part of this restructuring. operating costs as an eligible expenditure Project Restructuring • Extend the closing date by six • Costs savings of just under US$300,000 were – Level 2 months from December 31, 2019 achieved on VSAT activities. PASO required more to June 30, 2020 for the PRIF MDTF time to determine how to best spend these Approved by World • Change the project’s institutional savings. As the PRIF Trust Fund closed in 2020, the Bank on December and implementation PRIF grant could only be extended up to June 30, 20, 2019 arrangements/ support mechanism 2020. (no change to Implementing • Change in implementation arrangements due to Agency) Tonga Airports Limited (TAL) no longer hosting the TFSU after the Tonga Aviation Investment Project (TAIP) closed on December 31, 2019. 7 P164468 – Cost Overrun, Scale Up. Page 12 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) Project Restructuring • Extend the closing date for IDA • The extension was proposed only for IDA Grant – Level 2 Grant No. D3190 to December 31, No. D3190. No extension was required for IDA 2022 (first time) Grant No. H8830, as it had been fully disbursed, Approved by World • Revise the Results Framework while the PRIF Grant No. TF A3687 closed on June Bank on December 3, • Reallocation between 30, 2020. 2021 disbursement categories under IDA • The 12-month extension was necessary for PASO Grant No.H883 and IDA Grant No. to achieve PDO 3 and PDO 4, and to continue D319 providing services to its member states to • Changes to components and costs maintain post-COVID aviation operations in the including cancellation of about region. US$0.32 million from the PRIF • The end target for PDO 3 and intermediate Grant TF-A3687 indicators were revised to better reflect the impacts of COVID-19. • The description of PDO 4 indicator was revised to “PASO addresses at least 80% of the GASOS functions for a Level 2 RSOO”. Revised Intermediate Results Indicators (IRIs) 22. The restructuring of PASO also added three IRIs. All original IRIs remained throughout project implementation. Details on the IRIs can be found in the Results Framework in Annex 1. (i) Supporting Regional Aviation Infrastructure (VSAT). This indicator was added to account for additional financing that was secured for the VSAT in 2017. While the installation of a VSAT was not originally included in the project design, additional financing was secured from the PRIF MTFD to carry out this activity. (ii) Beneficiaries of job-focused interventions. This indicator was added in 2018, with a target of 10 in total and an intermediate target of 2 females. At the time there were no female regulatory inspectors, and the PASO Reform Project aimed to expand the ROI to include at least 2 female candidate inspectors for requisite training. (iii) Project supported organization(s) publish reports on effect of collaboration on project/program/policies. This indicator was added in 2018 as a measure of citizen engagement. PASO made a commitment to improve public access to information through the online publication of relevant organizational documents and records (such as the PASO Strategic Plan 2023–2032, the annual PASO business plan, annual reports, quarterly reports), as part of a website update. Other Changes 23. Changes in institutional arrangements (no change to implementing agency). Under PAIP, the TFSU was established as a central project management unit to support the various countries under the PAIP. The TFSU, hosted by TAL, was providing project management, technical and fiduciary support services to the PASO Reform Project, as detailed in the Service Agreement between TAL and PASO. However, when TAIP 8 closed on December 31, 2019, a new institutional arrangement for implementation support was required. The project investigated the option of replicating the arrangements with the TFSU and having it hosted by another agency within a PAIP country or by the PASO. The Ministry of Civil Aviation for the Government of the Solomon Islands was amenable to the option of another PAIP country hosting the TFSU, but this would have required state-level agreements between the Solomon Islands and other PAIP countries. If PASO was to host the TFSU, a new treaty would have been needed to replace the Service Agreement. Therefore, it was 8 TAIP (P128939) constituted Tonga’s participation in PAIP. Page 13 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) agreed that the three remaining PAIP recipients 9 that required the continued technical and fiduciary support would adopt new arrangements for the TFSU services, namely, directly recruiting individual consultants from the existing TFSU staff to provide these services, rather than having the TFSU recruited through a single entity (i.e., through TAL, as per the previous arrangement). Each recipient would be responsible for the management of their respective consultants. PASO engaged three core consultants – a program manager, a procurement specialist, and a financial manager – as part of its Project Implementation Support Team. Figure 3 shows the project implementation arrangements, including the change from the TFSU to the Project Implementation Support Team. Figure 3. Project implementation arrangements Rationale for Changes and their Implication for the Original Theory of Change 24. The PASO Reform Project was flexible enough to facilitate the transformation of PASO. The PASO Reform Project was the first World Bank engagement with PASO. In response to changing needs within PASO and the services provided to member states, new components and activities were added, the Results Framework was revised, and extensions of the project closing date were processed. This had implications for the original theory of change. Figure 4 represents the theory of change considering the additional finance and restructuring (changes from the theory of change at time of project appraisal are underlined). 9 PASO, Solomon Islands and Tuvalu. Page 14 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) Figure 4. Revised theory of change Critical assumptions: 1. The original implementation period is sufficient to complete project activities. 2. Adequate technical design. 3. PASO capacity risks mitigated through additional regional support from the PAIP. II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 25. The WB-RPF was extended to FY23 10 through a Performance and Learning Review, which confirmed that the FY17–FY21 RPF continues to be relevant and well aligned with development priorities. The Performance Learning Review highlighted that World Bank support to PASO has been instrumental in enabling this regional safety and security oversight organization to overcome its early financial problems and strengthen its capabilities. PASO has now established a pool of regional aviation inspectors and supported its country members to update legislation and improve regulatory oversight from a regional perspective. PASO’s services are more accessible and more widely used by its member states. Capacity across the region has improved through the analysis and prioritization of training needs and the provision of training in specific areas of aviation expertise. PASO serves as a platform for coordination and dialogue at the regional 10IBRD/IDA/IFC/MIGA Regional Partnership Framework for Kiribati, Nauru, the Republic of Marshall Islands, Federated States of Micronesia, Palau, Independent State of Samoa, Kingdom of Tonga, Tuvalu, and Vanuatu for the period FY17–FY23 (Report No. 120479 – EAP). Page 15 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) level, from technicians to decision-makers, and with regional stakeholders such as the ADB, Australia, New Zealand, and other donors. Quarterly meetings of the Program Steering Committee (comprising at least one high-level representative of each participating country) provide another forum for participating countries to create synergies, exchange experiences across the region, and support each other technically. 26. Papua New Guinea and the Solomon Islands are not covered by the WB-RPF for nine PICs but are informed by their country partnership frameworks (CPFs). The importance of improving connectivity and service delivery was highlighted in both the Papua New Guinea CPF FY19–FY23 11 (Focus Area 2: Ensuring More Effective and Inclusive Service Delivery) and the Solomon Islands CPF FY18–FY23, extended to FY24 through a Performance Learning Review 12 (Objective 3.1: Improve connectivity). 27. Three recently developed strategic documents reflect the vital importance of ensuring a coordinated approach to air transport connectivity and the key role that a strengthened multi-functional regional aviation organization plays in achieving the strategic goals for the region. The 2050 Strategy for the Blue Pacific Continent, endorsed at the 51st Pacific Islands Forum Leaders Meeting, notes “As Pacific Leaders, our vision is for a resilient Pacific region of peace, harmony, security, social inclusion and prosperity, that ensures all Pacific peoples can lead free, healthy and productive lives.” The Pacific Regional Aviation Strategy (PRAS) 2022–2023 outlines a framework for aviation in the Pacific and continued commitment to comply with ICAO standards, maintain a safe aviation system, strengthen regulatory oversight capacity, and increase connectivity in support of economic growth. The strategy recognizes the need for a harmonized, collaborative, and connected Pacific aviation system. The PASO Strategic Plan 2023–2032 sets the 10-year strategic focus of PASO and operationalizes the regional priorities for the aviation sector. 28. The focus on ensuring that the aviation sector remains safe and secure continues to be relevant for the Pacific region. Given the geographic dispersion in the region, the highly specialized nature of aviation services, and lessons learned from COVID-19, the operation of a collective shared resource to provide support to the sector in a harmonized, compliant, and efficient manner continues to remain a focus as part of the Pacific regional approach. As discussed above and presented in the theory of change, the components and activities were highly relevant to the development priorities of the region and supported the priorities outlined in the WB-RPF. 29. The relevance of the PDO is rated as High. B. ACHIEVEMENT OF PDOs (EFFICACY) 30. Progress towards achieving the PDO and implementation progress have been rated as Moderately Satisfactory or better throughout project implementation. The sections below summarize the status of project activities at project close and their contribution to the overall achievement of the objectives. Assessment of Achievement of Each Objective/Outcome 31. Efficacy is evaluated against the revised PDO and Results Framework. While the PDO and outcome targets were adjusted during project implementation, a split rating evaluation has not been applied by the ICR. The change to the PDO was not material, but grammatical, as the objective of the IDA Grant was to strengthen and enhance PASO’s ability to 11 IBRD/IDA/IFC/MIGA Country Partnership Framework for The Independent State of Papua New Guinea for the Period FY19-FY23 (Report No. 128471-PG). 12 IDA/IFC/MIGA Performance and Learning Review of The Country Partnership Framework for the Solomon Islands for the Period FY 18-FY23 (Report No. 174891-SB). Page 16 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) deliver regional safety and security services to PICs. Table 5 highlights the achievement of PDO indicators at project close, while the following section highlights key project achievements against the two outcomes: (i) strengthening PASO coordination capacity; and (ii) delivering regional aviation safety and security oversight, and technical and advisory services to PICs. Table 5. PDO indicator status at project close End target as of PDO indicator Status December 2022 Outcome: Strengthening PASO coordination capacity PDO 1: Successful recruitment of general manager, operations manager, Positions filled Achieved corporate services manager PDO 4: PASO addresses at least 80% of the GASOS functions for a Level 2 RSOO 80 percent Achieved Outcome: Deliver regional aviation safety and security oversight, and technical and advisory services to PICs PDO 2: Establishment of an ROI ROI in place Achieved Partially PDO 3: Recommended annual work program implemented by PASO members 60 percent Achieved 13 Outcome: Strengthening PASO coordination capacity 32. Coordination capacity relates to strengthening the organizational, procedural, and administrative governance arrangements through which PASO is better able to collaborate and cooperate with member states. Activities to strengthen coordination capacity have in turn strengthened the accountability and transparency of the organization. The PASO Reform Project supported the strengthening of PASO’s coordination capacity through: (i) a revised governance structure that provided adequate staffing arrangements for PASO to meet core oversight requirements as well as strengthen organizational management and ownership; 14 (ii) strengthening the role of PASO as an RSOO; 15 (iii) implementing a new financing mechanism 16 and enhancing quality management; 17 and (iv) providing a clear strategy and vision for PASO moving forward. 33. The project supported a revised governance structure that provided adequate staffing arrangements for PASO to meet core oversight requirements as well as strengthen organizational management and ownership. Project activities included a review of the PASO Constitution and recommended updates to its governance structure. The revised draft Constitution was completed in 2020 and adopted at the Regional Aviation Ministers Meeting (RAMM) in June 2021. While the PASO Reform Project originally envisaged recruiting three technical staff to coordinate oversight areas in airworthiness, flight operations, and security/aerodromes, the PASO Council endorsed a new organizational framework 13 Indicator was not achieved in all years. 14 Supported through PDO 1 and IRI: Revised governance structure (technical and financial committee composition), as per PASO Constitution. 15 Supported through PDO 4. 16 Supported through the IRI: Sustainable financing mechanism in place. 17 Supported through the IRI: Establishment of quality assurance system. Page 17 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) that was recommended as part of the “Strengthening PASO Governance and Management” assignment completed under the project in 2016. As such, a general manager, operations manager, and corporate services manager were put in place, rather than appointing technical experts as fulltime inspectorate staff, which was supported by the project. PASO implemented these staffing changes in line with the respective PASO Council decision in November 2016. In addition, the project supported the appointment of a transition manager to help facilitate the organizational changes and to oversee initial activities and procurements. Figure 5 highlights the organizational changes over the implementation period of the project. Figure 5. PASO organizational changes 34. The PASO Reform Project has put PASO in a position to be one of the first organization globally to achieve Level 2 RSOO accreditation and the organization is now better equipped to address GASOS functions. In 2017, ICAO presented a new conceptual framework, referred to as GASOS, that represented an evolving effort to improve aviation safety oversight given that next generation aircraft and air navigation platforms presented new safety and security challenges for the sector. Numerous civil aviation authorities (CAAs) worldwide were unable to keep up with changes in the aviation industry, due both to a lack of financial resources and highly specialized skillsets. In the unique case of limited capacity environments, such as those found in small island developing states, these shortcomings were identified as usually more frequent and disruptive. ICAO’s vision was centered on the collaborative support of member states to realize an integrated global system that could deliver on the aviation safety oversight mandates, achieved through strengthening and empowering RSOOs (such as PASO). RSOOs would constitute the pillars of a new, global safety oversight system, and would need to demonstrate their ability to deliver adequate levels of service and, thus, qualify as an ICAO-recognized safety oversight provider. 18 There are three levels of the RSOO status, 19 and while PASO has achieved a Level 1 RSOO it is working toward becoming a Level 2 RSOO. 18 There are currently 17 RSOOs, including PASO, with a range of functions – from relatively informal safety oversight advisory services to more institutionalized organizations that perform safety oversight through formal delegation. 19 Three levels of delegation/empowerment have been defined based on increasing complexity of tasks from 1–3 and functions performed. Level 1: Advisory, consultancy and coordinating tasks and functions; Level 2: Operational assistance tasks and functions; Level 3: Certifying agency tasks and functions (www.icao.int/ESAF/EASA-ICAO-RSOO-2017/Documents/Presentations/Theme%205-RSOOimprovements.pdf). Page 18 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) 35. PASO is one of the first organizations to pursue Level 2 RSOO recognition. PASO has worked through a process of naming GASOS functions that are applicable to the Pacific and established a workplan and framework to complete documentation of these functions. PASO is working on proposals to have the remaining 20 percent completed in 2023 and ready for assessment by ICAO. PASO will be allocating resources for 2023 for the review and integration of the documents for the achievement of Level 2 RSOO, beyond the closing date of the PASO Reform Project. PASO currently provides technical advice under its Level 1 status; however, under Level 2, PASO would provide greater operational assistance as well as carry out some functions on behalf of member states. This would be critical for small PICs, which are unable to conduct their work independently. Table 6. GASOS functions for PASO Progress towards GASOS function GASOS function Comments completion number Total number of GASOS functions 128 Not all functions apply to PASO/Pacific Islands. Number of GASOS functions ineligible There are 21 GASOS functions that have not been or not available to complete uploaded to the GASOS Electronic Management System 21 (GEMS) by ICAO or do not apply to the PICs, therefore, PASO is not able to address them. Number of GASOS functions available There are 107 GASOS functions that are available for 107 to complete by PASO PASO to complete. Number of GASOS functions that will This is 80 percent of the number of GASOS functions that 86 be completed by December 2022 is available for PASO to complete. Number of GASOS functions to be This is 20 percent of the GASOS functions that PASO will 21 completed in 2023 need to complete to receive Level 2 RSOO recognition. Table 7. PASO Level 2 RSOO accreditation plan Steps for level 2 RSOO accreditation Timeframe Status GASOS Pilot Assessment and Corrective Action Plan received (PASO only August 2019 Complete underwent a partial assessment) Pilot Assessment and Corrective Action Plan responses July–Dec 2021 Complete Development of GASOS documentation for 86 functions (80 percent) Jan–Dec 2022 Complete 20 Development of GASOS documentation for remainder of the 21 GASOS Not started, 2023 functions unfunded Not started, Internal PASO review with PASO inspectors 2023 unfunded ICAO assessment of PASO documents. ICAO is slowly reviewing documents as they are being uploaded to GEMS; however, ICAO has indicated that they do not have the resources required to complete it in a 2021–2023 In progress timely manner. PASO will receive another Corrective Action Plan after the assessment. Not started, PASO addresses the Corrective Action Plan after full assessment. 2023 unfunded 20 Reflected in PDO 4, as of December 31, 2022. Page 19 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) GASOS implementation – Once PASO receives recognition as a Level 2 RSOO, PASO will engage with member states to conduct trainings and Not started, 2023 – 2024 integration on the changes involved in working with PASO as a recognised unfunded Level 2 RSOO. GASOS sustainability – For PASO to keep the Level 2 RSOO recognition, it is essential to incorporate measures for sustainability. A GASOS specialist Not started, 2023–Onwards would ensure that PASO keeps the necessary documentation up-to-date. unfunded The documentation would need to be reviewed and updated each year 36. Another critical activity supporting the strengthening of PASO was the introduction of the revised financing mechanism. PASO developed the Work Plan Percentage Model (WPPM), which established a way forward to allow prepayment by member states for specific baseline services and delivery of these work plans through service level agreements signed between member states and PASO. The WPPM removes the need to have subscription fees and a fee-for-service model and replaces it with an agreed budget against the PASO Strategic Plan, under which member states’ individual contributions are calculated as a percentage using their work plans against the total costs required to fund PASO. At project appraisal, member states' confidence in PASO to deliver services was low; however, with positive momentum and the completion of project activities, confidence in PASO has grown and the capability to deliver inspectorate services to member states has increased through the WPPM. The introduction of the WPPM represents a significant step forward in the ongoing resilience and sustainability of PASO, as it has stabilized PASO’s revenue and improved its cashflow surety, and, therefore, contributed to supporting the future operation of PASO in the region. 37. PASO systems have improved through strengthening quality management and assurance systems. The project successfully supported technological improvements to improve quality management and transparency. This included the establishment of a Quality Assurance System, as well as the modernization of communication technology, with an upgraded information communication technology (ICT) system in place and in use by PASO. PASO also improved public access to information through the online publication of relevant organizational documents and records as part of a planned website update that strengthened transparency and accountability. 38. The PASO Reform Project has supported the preparation of documentation to provide PASO with a clear strategy and vision moving forward. The project supported the development of the marketing and communications strategy and plan, the PASO Strategic Plan 2023–2032, and the PRAS 2023–2032. The marketing and communications strategy and plan for PASO reflected the transformation of PASO in its regional aviation role. The assignment included a further refresh of PASO branding, website content, and public relations processes. Support was provided for ongoing communications and marketing advice and support for press releases, media materials and has been key in establishing an improved media presence and internal processes for managing PASO communications. The PASO Strategic Plan 2023– 2032 takes the goals and priorities determined at the regional level and operationalizes them with specific actions and performance measures to enable PASO to track progress and success. The PRAS 2023–2032, is designed to support Pacific States and the aviation industry to align with ICAO safety and security requirements and support regional compliance. 39. A variety of training and capacity development opportunities within the PASO Reform Project further supported enhanced coordination capacity. The Civil Aviation Authorities Strengthening Program (CAASP) provided training for CAA personnel to ensure that they can effectively contribute to their operations. Between 2019 and 2022, 10 trainings were conducted for 66 participants. Financing was allocated against identified training needs, including ICAO training and RPAS training, which provided PASO members with an opportunity to strengthen their understanding of drone regulation. As the project progressed and PASO transformed, the PASO Council incorporated governance training into the project to improve the capacity of Council members and the organization in relation to the oversight Page 20 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) responsibilities, given the role of PASO as a regional organization. Outcome: Delivering regional aviation safety and security oversight, and technical advisory services to PICs 40. This outcome is focused on service delivery to member states. The PASO Reform Project supported enhanced delivery of regional aviation safety and security oversight, and technical advisory services to PICs through: (i) supporting an increase in the scope and breadth of inspector expertise; 21 and (ii) increasing implementation of the annual work program. 22 41. The project supported an increase in the scope and breadth of inspector expertise, with an ROI established in 2014 and endorsed by the PASO Council. PASO has recruited several different inspectors to support the annual work plans for the member states. PASO has also recently expanded its services and is now offering surveillance support as well as audits and entry into service support. 23 The pool includes inspectors from several different technical areas including airworthiness, aerodromes, air navigation, flight operations, aviation security, and personnel licensing. In addition, PASO has also recently been offering aviation meteorological support. The size and composition of the ROI has responded to the needs of member states, inspection and compliance schedules, and the annual work program, with the pool currently comprising 22 inspectors. 42. The implementation of the annual work program and adherence to agreed annual work plans are core indicators of PASO’s success, as they show the effort and work that PASO has completed in each member state, including the utilization of key PASO staff. Measuring the implementation of the recommended annual work program demonstrates the percentage of PASO services utilized by member states. Adherence to agreed annual work plans is a key measure of PASO services, as it shows each member states’ engagement in the planned annual work program, i.e., the utilization of PASO staff and inspectors. PASO’s overall work program has roughly doubled over the project period, which reflects the fact that their expanding role continues to be crucial in supporting member states in safety and security oversight (Table 8). Between 2014 and 2019, many member states achieved the target for adherence to annual work plans (Table 9), with 2020 onwards affected by the impact of COVID-19 travel restrictions. Table 8. Average annual work program performance Year Target (%) Average work program (%) Total workdays 2014 80 72.1 272 2015 80 49.9 107 2016 80 61.7 316 2017 80 80.1 426 2018 80 88.8 413 2019 80 78.9 506 2020 60 24 21.9 239 2021 60 62.4 680 2022 60 52.2 530 21 Supported through PDO 2. 22 Supported through PDO 3 and IRI: Adherence to Agreed Annual Work Plans. 23 Entry into service refers to providing guidance and training on to an operator new to the aircraft or aviation. 24 Reduced to 60 percent from 2020 onwards due to COVID-19 travel restrictions. Page 21 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) Table 9. Adherence to agreed annual work plans by PASO members 25 Member End target of 80% End target of 60% Average Baseline State 2013 2014 2015 2016 2017 2018 2019 2020 26 2021 2022 % Cook Islands 0 0 0 0 33 76 100 100 0 100 100 51 Kiribati 130 130 59 59 51 68 82 75 0 0 45 57 Nauru 0 0 106 106 0 0 0 0 0 0 13 23 Niue 0 0 85 85 0 0 0 60 0 0 16 25 Papua New 0 0 1,000 0 17 100 42 100 0 0 37 130 Guinea Samoa 54 54 228 228 100 100 100 100 15.9 25.47 49 100 Solomon 24 24 0 0 0 0 0 29 14.8 23.3 59 15 Islands 27 Tonga 62 62 458 458 100 100 100 100 38.9 0 100 152 Tuvalu 0 0 85 85 100 33 54 100 0 0 3 46 Vanuatu 27 27 21 21 100 100 100 100 0 23.45 100 59 Data value Nov Dec Apr Nov Oct Jan Dec Oct May Nov date 28 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 Justification for Overall Efficacy Rating 43. The overall efficacy is rated as Substantial. C. EFFICIENCY Assessment of Efficiency and Rating 44. Economic and financial analyses was not undertaken at the time of project appraisal. PASO is a service organization and while established as a non-for-profit organization (i.e., it does not generate profits for distribution to its owners), it must, as all non-for-profits, generate a profit to be financially viable and, as per the PASO Constitution, operate cost effectively, especially as it now has equity. The technical assistance activities included in the PASO Reform Project were focused on strengthening its business model as a service provider, increasing efficiency, and keeping the organization financially viable. A key concern during project appraisal was the imminent threat of PASO becoming insolvent and being unable to repay the ADB loan used to establish the organization. Investments were, therefore, included in the project to strengthen the governance arrangements. A key achievement of PASO has been the strengthening of financial reporting and controls, which have meant that PASO is better able to manage its resources. Through the reforms and strengthening of governance arrangements, PASO has been able to make interest and loan 25 Values were taken from archived Implementation Status and Results Reports (ISRs). Values were taken as close to the end of calendar year as possible; however, in some cases these were not available. 26 COVID-19 affected the achievement of the indicator for many member states between 2020 and 2022. 27 The Solomon Islands opted not to engage PASO services between 2015–2017. 28 The achievement of these sub-indicators was to be evaluated at the end of each calendar year. Page 22 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) repayments on the ADB loan, as and when they fall due. 29 This, Figure 6. PASO funding support 2021 (top) and 2022 in turn, has reduced the risk that the member states, as (bottom) guarantors of the ADB loan, would incur the repayment obligations. The audited financial statements for year ended Member December 31, 2021 30 reported that PASO financial statements Subscriptions and have been prepared on a going concern basis, which assumes Other Revenue the continuity of normal business and the settlement of Vanuatu Hosting liabilities in the ordinary course of business, and as per the 17% Agreement 26% Council representatives, the going concern principle is 6% World Bank applicable given the ongoing financial support provided mainly by donors. The PASO Reform Project has had the immediate and 19% ongoing effect of assisting PASO operations to breakeven and MFAT generate small profits (as well as occasional small losses). 32% 45. As shown in Figure 6, PASO service delivery is reliant DFAT on external sources of funds; however, the PASO Reform Project has directly supported the transformation of PASO and instilled changes for future sustainability. With World Bank support, PASO has strengthened its cash management Member forecasting and financial management practices. The Subscriptions and strengthened system has assisted PASO in maintaining financial other revenue integrity and liquidity, including over a period of financial Vanuatu Hosting uncertainty, as debtors and members faced fiscal constraints 19% 14% Agreement due to COVID-19. The introduction of the WPPM and 5% improvements in grant fund recording and management further World Bank enhanced PASO as a trusted partner within the aviation sector. In addition, the strengthening of PASO as a Level 1 RSOO, and 31% 31% MFAT positioning of the organization to achieve Level 2 RSOO, has further solidified its role in the region. DFAT 46. While there were extensions to the project timeframe and additional funds required, these were not related to inefficient project implementation, but the further deepening of the World Bank’s support to PASO. At project appraisal, costs were estimated at US$2.15 million; however, the revised cost at project close was US$6.33 million. As noted above, two rounds of additional financing – comprising the PRIF MDTF (US$950,000) and a Regional IDA Grant (US$3.55million) – were processed to support additional activities. The project implementation period spanned nine years, an additional four years from the originally envisaged five-year implementation period. Extensions of time and additional resources reflect the increased level of engagement between PASO and the World Bank over time and allowed for delays in implementation because of COVID-19. The PDO results were achieved with a 194 percent increase in the total financing and an 80 percent increase in duration. 47. Given the small project investment, the transformation of PASO with the resources provided represents strong 29 The 2022 accounts note that the loan ends in 2033 and reported an outstanding balance of AU$816,000. With negative net assets of approximately AU$700,000, PASO current does not have sufficient resources to cover the loan, with minimal profits/breakeven to be able to pay off early. 30 The audit report for year ended December 31, 2022 is due on June 30, 2023. Page 23 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) value for money. Considering a ‘without-project’ scenario, without World Bank support through the PASO Reform Project, PASO would not have been able to provide uninterrupted aviation safety and security services for member states in the Pacific region and be the functioning Level 1 RSOO that it is today. This would have resulted in a significant gap in services, given that PASO is the sole international organization responsible for regional regulatory aviation safety oversight services for its 10 member states. The PASO Reform Project has supported a more effective and efficient PASO, making it easier for members, donors, and stakeholders to do business and engage with PASO. 48. The overall efficiency is rated as Substantial. D. JUSTIFICATION OF OVERALL OUTCOME RATING 49. The PASO Reform Project has supported the transformation of PASO from a small organization comprising four staff to a regional organization supporting member states across the Pacific region and has contributed to institutional capacity building. While there were extensions to the project timeframe and additional financing, these were not related to inefficient project implementation, but to the deepening of World Bank support to PASO. Given the small amount of project investment, this represents strong value for money. The Reform Project was successful in achieving the objectives, costs were controlled, and the project provided extended support to PASO based on the needs and opportunities that were identified during implementation. Service delivery to the Pacific region has improved, and the project has been instrumental in providing technical assistance in regional aviation safety and security oversight to PASO member states, especially during COVID-19. 50. The overall outcome is Satisfactory. E. OTHER OUTCOMES AND IMPACTS 51. The PASO Reform Project contributed to enhancing the climate resilience of the sector. Project investments supported enhanced resilience, including the installation of the VSAT, provision of training, and strengthening of safety. Figure 7. VSAT installation (i) The project successfully supported the installation of VSATs in the Cook Islands and Niue. PASO was requested to provide the VSATs, given that it is a regional organization that could provide access to this activity for Niue and Cook Islands, which were not part of PAIP. This allowed Niue and the Cook Islands to benefit from funding available to support infrastructure improvements and take advantage of previous knowledge and suppliers under PAIP. PASO successfully delivered two important aviation infrastructure equipment to Niue and the Cook Islands to support the improvement of safe aircraft operations through the design, supply, and installation of VSAT equipment at Hanan and Rarotonga airports. The equipment extended the coverage of improved communication across the region started under PAIP to allow countries to transmit important aviation data in the event of poor weather or local network outages. The final commissioning and handover for both airports was completed in June 2020 and supported the delivery of positive impacts Page 24 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) regionally. 31 The installation of the VSAT communications systems created redundancy and provided an alternative channel for member states to connect in the event of extreme weather events. (ii) The Remotely Piloted Aircraft System (RPAS) training also supported the enhanced provision of services, including during extreme weather events. Participants from PASO member states took part in RPAS professional training, to help strengthen their understanding of drone regulations. Such training on the use of RPAS can help pilots to avoid other traffic and dangerous situations, such as potential collisions with other airspace users or obstacles and severe weather conditions. (iii) Weather can significantly affect aircraft operation and improvements in safety contribute to strengthening the climate resilience of air transport services. Project investments have supported resilience by helping to ensure operational safety in low visibility situations or during extreme weather events. Gender 52. At the time of project appraisal, the PASO Reform Project was ‘gender-informed’. Through a review of active aviation inspectors in the region it was identified that there were no women involved in technical oversight roles in PASO or in any of the PASO member states. However, it was noted that by preserving the safety of aviation services in PICs through a strong PASO, that there would be benefits to both men and women by providing safe and secure aviation services. As gender development was not an intended outcome of the project, no specific project indicators were incorporated into the Results Framework. Opportunities for women to be engaged and/or trained as part of these programs were considered through the project. The 2018 Project Restructuring Paper noted “There are currently no female regulatory inspectors. The project will aim to expand the register of inspectors to include at least two female candidate inspectors for requisite training programs under PASO’s Pacific Inspector Development Program.” During implementation, five beneficiaries were identified to be trained as inspectors (three males and two females); unfortunately, trainings were interrupted by the COVID-19 pandemic. When travel resumed, there was no time to do a new assessment of CAA members to find new candidates to be trained. This is reflected in the IRI ‘Beneficiaries of job-focused interventions’, which was not achieved. Institutional Strengthening 53. In addition to the training and capacity development activities, the PASO Reform Project has resulted in key institutional and regulatory changes, including the establishment of the RAMM, revisions to the PICASST, and the PRAS 2023–2032. (i) RAMM: In 2018, the PASO Council of Directors endorsed PASO to work with the Pacific Islands Forum Secretariat (PIFS) to establish a regional meeting of aviation ministers to identify collaborative and unified approaches that could benefit the region. The RAMM was a strategic opportunity for Pacific Island Forum member states to discuss the wide-ranging aviation needs of the region to ensure a safe, sustainable, and enhanced aviation environment for the Pacific and a focus on strengthening aviation safety and security compliance by enhancing current institutional arrangements and establishing a long-term shared vision for the region. The inaugural RAMM was held, virtually, on June 30, 2021 and was the first such high-level regional aviation meeting since the initiation of the PICASST in 2004. (ii) Revisions to the PICASST: PICASST is the founding document for PASO, which established PASO as a regional entity to support aviation safety and security in the Pacific region. As the reform of PASO continued and 31 It is noted that the delivery of the infrastructure was outside of the PASO mandate and role. This created a potential conflict given that PASO offers advisory services to member states’ civil aviation regulators and, therefore, the installation of the equipment blurred the roles and responsibilities with oversight and sign off. While PASO successfully installed the equipment, the scope of PASO services needs to be carefully considered in the future to ensure that aviation compliance conflicts are not created in the delivery of activities by PASO. Page 25 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) strategies were developed, PASO found that amendments to the PICASST were required to provide PASO with the mandate to be an RSOO and provide the types of services required within the regulatory framework of each member state. Under the PASO Reform Project, a review and amendment to the PICASST was completed in February 2020. These amendments were approved and ratified by the RAMM2 and the resulting actions arising from the PICASST amendments will be implemented by member states commencing in 2022. (iii) PRAS 2023–2032: In 2021, RAMM1 agreed on a framework for aviation in the Pacific, reaffirming a commitment to comply with ICAO standards, maintain a safe, secure, resilient, and sustainable aviation system, and strengthen regulatory oversight capability and effectiveness. In 2022, RAMM2 endorsed the PRAS 2023–2032, which was prepared by the PASO Reform Project on behalf of PASO, to be implemented in conjunction with the amendments to the PICASST. Role of Other Donors and International Finance Institutions 54. The unified strategy to address regional regulatory challenges through the project provided a platform for coordinating development partners’ involvement in the sector, thereby increasing the impact of the World Bank’s investment. This includes: (i) ADB, which are providing technical assistance for strengthening aviation regulation; (ii) the Australian Department of Foreign Affairs and Trade (DFAT) and the New Zealand Ministry of Foreign Affairs and Trade (MFAT), which provided support during the COVID-19 pandemic; (iii) support provided by ICAO, including providing comments and recommendations on the design of the Business Plan and offering its support during implementation of the transition arrangements; and (iv) PRIF support, which was provided to five PAIP states (Kiribati, Samoa, Tonga, Tuvalu, and Vanuatu) for coverage of operator certification fees. Poverty Reduction and Shared Prosperity 55. The PASO Reform Project supported the World Bank Group’s twin goals of eliminating extreme poverty and boosting shared prosperity by increasing regional integration, mitigating the effects of economic isolation, and supporting human resource development and mobility. Through supporting PASO and strengthening services, the project has provided a level of confidence in air service providers and encouraged the expansion of services. Protecting and expanding connectivity provided by air transport also increases resilience to climate change by enabling rapid response to emergencies and natural disasters. Page 26 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) Other Unintended Outcomes and Impacts 56. While COVID-19 presented many challenges it also offered opportunities for Figure 8. PASO Master Class enhancing service delivery, with restrictions in mobility allowing for greater use of ICT tools and remote support. In response to COVID-19, in June 2020, PASO began delivering virtual training opportunities for civil aviation and airport personnel in member states, as part of the ‘PASO Master Class Series’. Training focused on strategic areas, such as: (i) Pacific aviation COVID safety restart training; (ii) Runway condition reporting – ICAO Global Reporting Framework; and (iii) Safety oversight and the PASO model. The training provided a continued link with member states and ongoing capacity building, despite the impacts to global travel, and highlighted opportunities for the use of technology to deliver capacity building across the Pacific region. In addition, due to travel restrictions, PASO inspectors were unable to travel to undertake their work in 2020, resulting in a signficiant reduction in previous levels of safety and security oversight. Member states use the Audit and Inspection Program for certification of aviation operators, which usually requires a physical onsite inspection. As the completion of oversight tasks to meet international standards was not possible in 2020, PASO invested in remote auditing capabilities, with nearly all work being conducted remotely in 2021. Through the increasing use of ICT tools and connectivity improvements, moving forward, the need to travel may decrease, resulting in a reduction in related costs. The delivery of virtual support has demonstrated the possibility of improving client relationships and more efficient PASO business services delivery. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 57. There were several issues that limited the ability of PASO to provide the intended services effectively. Given that PASO is a regional organization, geopolitical influences and pressures across the Pacific were bound to affect the project. In addition, when the PASO Reform Project was being prepared, PASO was at a crossroads in terms of whether services would continue. These key factors affected the preparation of the PASO Reform Project, as well as changes made during implementation. 58. The poor financial sustainability and business performance of PASO was a major challenge during project preparation, in addition to identified institutional risks, and was reflected in the project design. Pacific member state debt resulted in limited resources and unsustainable revenue to support oversight activities. PASO services were being provided to member states; however, compensation remained outstanding. As a result, financial resources within PASO were constrained, limiting the ability of PASO to deliver intended services. Not only did this affect PASO’s operations, but it contributed to strained relationships between PASO and its member states. When the PASO Reform Project was being prepared, there were concerns regarding transparency and accountability within the PASO organization. For example, financial reporting had demonstrated deficiencies related to the presentation of overall entity financial positions. 32 The fiduciary risk associated with PASO’s fiscal performance issues meant that the project size was originally limited to approximately US$2.15 million equivalent. In addition, capacity was provided through the TFSU, and the new governance arrangements and financing mechanisms, which were an integral element of the project 32 PASO management increased the frequency of financial reporting to the Council to improve transparency. Page 27 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) design, were proactive measures that sought to ensure the sustainability of the investments. 59. The preparation of the PASO Reform Project benefitted from upstream work by other donors and shortened preparation timelines. The financial review funded by PRIF in 2013 highlighted an urgent need to provide financing for PASO in the short-term to ensure that it would remain a viable entity. Upon recognition that PASO’s short-term cash flow forecast, and upcoming liabilities projected impending insolvency, member states requested World Bank support. The project was prepared based on a Situation of Urgent Need of Assistance or Capacity Constraints. The project preparation timeframe was condensed, with five months between the Concept (April 2, 2013) and World Bank Board approval (September 30, 2013). The average preparation timeframe in the Pacific averages around 12 months, therefore, the PASO Reform Project benefitted from the expedited process. The project also became effective shortly thereafter, on December 12, 2013. The fast delivery of the project was critical to ensuring that member states were able to meet their international regulatory requirements in the aviation sector. B. KEY FACTORS DURING IMPLEMENTATION Factors subject to the control of the World Bank 60. The World Bank provided additional financing to support the successful implementation of the reform agenda. With additional financing of US$3.55 million equivalent from the Regional IDA envelope, the financing support for PASO almost doubled. As part of the rationale for the additional financing, it was noted that the appraisal estimates for the recommended activity interventions defined in the 2013 Business Plan were not sufficiently robust; however, during project implementation, it became clear that funding resources were not sufficient to meet project expectations, particularly around capacity development for the inspectorate pool. Furthermore, the additional financing and associated extension of time allowed the introduction of the new financing modality and support for member states readiness to implement the new model. The second additional financing allowed for the successful completion of the reform activities. The further support provided by the World Bank at the time was also critical, as the new financing mechanism was not yet in place, hence, there was no fiscal space within PASO’s own budget to cover the costs. Factors subject to the control of the implementing agency 61. Baseline reviews of PASO’s financial sustainability in 2015–16 identified several issues relating to the existing funding methodology of PASO that impacted on their operational and financial performance, including: (i) Services were being provided to operators/document holders and billed via the member state’s finance ministry; however, there was slow or non-payment for services undertaken. (ii) The revenue generated was insufficient to cover operational costs or provide funds to foster institutional knowledge and systems within PASO and member state civil aviation departments. (iii) The risk of member state’s PASO finances being diverted to other immediate demands will always be present in the Pacific region, where funds are scarce and critical short-term demands continue to arise. (iv) Program push-back and deferment will always be present under financing methodology, due to scarcity of funds and document holders looking to save on costs. (v) Program deferment and rescheduling was leading to inefficient and low utilization use of PASO inspectorate staff, resulting in poor financial performance. 62. Increasing confidence in PASO’s capabilities and improved governance supported additional donor financing. Financing support for the corporate services manager and operations manager were considered as part of the second additional financing, including funding their annual salaries for two years – 100 percent for the corporate services manager and 50 percent for the operations manager. However, this support was provided by MFAT. In addition, as Page 28 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) noted above, support provided by MFAT and DFAT supported the achievement of PDO indicators PDO 3 and PDO 4. Over the course of project implementation, PASO has been better equipped to leverage additional donor financing in support of delivering core activities. Factors outside of the control of the implementing agency 63. The COVID-19 pandemic resulted in unforeseen challenges during implementation and PASO revenue decreased significantly due to the associated travel restrictions. Project implementation had been progressing as planned, until the COVID-19 pandemic in March 2020. However, COVID-19 travel restrictions, limited fiscal capacity of PASO member states, and PASO’s decreased revenue resulted in challenges to project implementation. Key activities were either put on hold due to travel restrictions or continued to progress slowly. Vanuatu re-opened its border on July 1, 2022. PASO’s in-person aviation inspectors resumed in the Cook Islands in May 2022 and in Vanuatu in July 2022. During 2020, with less work plan demand and non-payment of subscription fees, revenue from member states decreased by almost 40 percent. PASO maintained liquidity throughout 2020 (and achieved a small surplus) through careful cost management. PASO was also supported by MFAT, through a grant of NZ$4.2 million (US$3.05 million equivalent), as part of the ‘PASO Support Program’, valid from July 2018 to June 2021. This MFAT grant included core PASO funding support. In December 2020, DFAT supported PASO with a grant of AU$2 million (US$1.54 million equivalent), as part of the ‘COVID-19 PASO Package’, valid until January 2022. This DFAT grant supported fees for inspection services, which, pre-pandemic, member states had paid to PASO from their aviation revenue. While the DFAT and MFAT funding were not part of the PASO Reform Project, it is important to note that their activities were closely linked to the World Bank project and supported the achievement of the PDO indicators PDO 3 and PDO 4. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 64. The M&E design was generally adequate to capture project progress appropriately; however, as highlighted in the sections above, there were changes to the M&E (including Results Framework) over the implementation period. The following shortcomings in the M&E framework were observed. 65. A key concern highlighted at project appraisal was the financial sustainability of PASO, but there was a lack of clarity regarding how financial sustainability would be measured through the project. (i) A key concern highlighted at project appraisal was the financial sustainability of PASO, given it was at risk of insolvency and had had an annual financial loss since its inception. However, no indicators were included at the PDO level, only a single IRI. (ii) The M&E framework did not clearly define how sustainability would be measured. The IRI ‘Revised financing mechanism for sustainability’ had the end target of ‘Sustainable financing mechanism in place’. While the revised financing mechanism is in place (i.e., the WPPM), the inclusion of a revised financing mechanism does not directly correlate with sustainable operations. (iii) The M&E framework did not explicitly state that the project would result in the financial independence of PASO. While this is inferred from the PAD, which states “The technical assistance activities to be financed by the grant are not investments designed to generate financial returns for PASO, but rather strengthen its business model as a service provider and make it financially sustainable”, the PAD and subsequent project papers could have made it clearer how sustainability would be assessed. As noted, PASO is still reliant on Page 29 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) external donor financing, which could be taken to mean that the goal of financial sustainability was not achieved. The ICR has assessed this indicator as achieved, based on the adoption of the revised financing mechanism, which has been supported by a stabilization of PASO cashflow, clearer work planning, and the prioritized delivery of support to member states. (iv) In addition, as seen through COVID-19, there are various exogenous factors out of the control of the project that could result in changes to financial revenue, which could affect the financial sustainability of PASO. 66. While only a few changes to PDO indicators were made, some changes were made in quick succession, indicating that the original indicator was not appropriate. For example, the PDO 4 indicator ‘ICAO Regional Safety Oversight Organization Service Level’ was added to the project in 2018. The indicator was revised in 2021 to ‘PASO addresses at least 80% of the GASOS functions for a Level 2 RSOO’. In addition, while this activity has been included as a PDO level indicator, the PASO Reform Project did not directly include project activities nor was funding allocated to supporting PASO achieving Level 2 RSOO status. In addition, parallel support from other donors also contributed to the achievement of this indicator. For example, the DFAT grant was used to support addressing GASOS functions. 33 67. While the project included an indicator to identify opportunities to support women through the project, gender disaggregated data was not collected during the project. As the indicator on beneficiaries of job-focused interventions was not achieved, given the fact that the Pacific Inspector Development Program (PIDP) was on hold due to travel restrictions, the project could not report on other project activities (such as other training) that may have supported women, as gender disaggregated data was not collected. In addition, in May 2021 the achievement of this IRI was zero out of the end target of 10. However, in June 2022, the achievement was noted as five individuals. The comments noted that “There were five beneficiaries being trained to become inspectors (three males and two female). However, the trainings were interrupted by COVID-19. Now that travel has resumed, there is no time to do a new assessment of members to find new candidates to be trained. The target of 10 beneficiaries will not be met by project closure.” However, the change in the reporting related to the achievement of the five individuals was not clear. M&E Implementation 68. M&E reporting was undertaken by PASO, with the support of the TFSU, in the form of Quarterly Project Implementation Status Reports. The TFSU was expected to be available over the implementation period to support M&E. However, when TAIP closed and the new institutional arrangement for implementation support was implemented, responsibility shifted to the Project Implementation Support Team. Project reports were initially delayed, and the quality of the reports delivered required improvement, as it was noted that they lacked the level of detail required to adequately understand project implementation status and several rounds of review were required by the World Bank Task Team before the final reports were accepted. It was also noted that project reports did not always present solutions to the challenges that PASO was experiencing at the time. The institutionalization of all project management structures including the PASO Council, the PASO Steering Committee, and the PASO management supported the improvements in M&E reporting and transparency and accountability in delivering project results over the course of the project. Towards the end of the project, project reports were generally submitted on time. The recipient prepared an ICR, which was provided to the World Bank in November 2023. Effectiveness of governance arrangements 69. Updates and reports were to be provided to the PASO Council by the general manager or project manager, as part of project reporting (monthly, quarterly, and semesterly basis). The PASO Council had a key role in key project 33 DFAT financing supported addressing 34 percent of the total GASOS functions. Page 30 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) deliverables and outcomes, such as the introduction of the WPPM, as well as working with member states to improve the performance of annual work plans. However, it is noted that additional management and technical support were necessary to assist the Council in meeting its obligations, particularly in the early phases of the project, which were supported through the project. The PASO Steering Committee, established by the PASO Council, was intended to improve engagement and direction for the project. Additional management and technical support were necessary during project implementation to assist the Committee in meeting its obligations and to conduct regular meetings. A challenge during project implementation was that the PASO Steering Committee meetings did not always occur frequently, and project progress reports were not regularly presented to provide updates of progress. Effectiveness of project management arrangements 70. During project implementation, there was no change in the implementing agency (PASO); however, the project management arrangements were adjusted during implementation, as noted in the sections above. The TFSU provided technical and fiduciary support until the closure of TAIP in December 2019, following which the Project Implementation Support Team was established, comprising international consultants (a program manager, a procurement specialist, and a financial manager). Three PASO project managers were engaged to provide implementation support. 71. The costs for project implementation support were captured under Component A and included financing for TFSU services, 34 the Project Implementation Support Team, and support for the PASO Steering Committee. Over the course of implementation these ‘project management’ costs totaled about US$1.3 million, approximately 22 percent of total project costs. 35 Project managements costs increased along with the extension of the project to support implementation until closure, particularly for international consultants financed through the project. Based on a review of the activities included within the project (see Procurement below), the implementation structure does not appear to have been commensurable with the nature and scale of activities included in the project. M&E Utilization 72. Throughout the project lifetime, indicators in the Results Framework were updated frequently and reported in the World Bank Aide Memoires and Implementation Status and Results Reports (ISRs) after the implementation support missions. Over the course of project implementation, 20 ISRs were submitted. In some cases, the Results Framework in the ISRs did not include updated results, but the results from the previous reporting period. The submission of the project reports, and continuous World Bank supervision reports, including detailed action plans for PASO follow up, supported M&E efforts. Justification for Overall Rating of Quality of M&E 73. Based on the above, the overall quality of M&E is rated as Modest. B. FIDUCIARY, ENVIRONMENTAL AND SOCIAL (E&S) COMPLIANCE 74. Financial management was rated as Moderately Satisfactory. The overall financial management arrangements for the project met the minimum requirements stipulated in the World Bank Policy on Investment Project Financing. For all ISRs submitted across project implementation, 36 financial management was rated as at least Moderately 34 Cost shared implementation and support services provided by TFSU between 2012 and 2019. 35 PASO IFR Q3, 2022. This does not include costs such as incremental operating costs. 36 20 ISRs were reviewed. Page 31 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) Satisfactory for 70 percent of the time. 75. The designated account was not active at times and the audit reports were unqualified (clean), but the interim financial reports (IFRs) were generally submitted on time and accepted. In addition, in November 2014, the financial management review identified potential ineligible expenditure amounting to approximately US$220,000, which was drawn down to finance PASO operating costs. The ineligible expenditure was incurred due to a misunderstanding about the use of grant funds to cover business and operating costs. Evidence of fraud or corruption was not identified, as these were business as usual organizational expenses, and the funds were reimbursed to the project. The final audit report for year ended December 31, 2022 will be due on June 30, 2023. 76. Procurement was rated as Moderately Satisfactory. Procurement was centrally managed, with the TFSU providing these services from 2013 to 2019, and the PAIP procurement specialist providing these services from 2020 to 2022. Over the implementation period, the project undertook 52 successful procurements in compliance with the World Bank Procurement Guidelines and Regulations. For all ISRs submitted across project implementation, 37 procurement was rated as Moderately Satisfactory or Satisfactory 65 percent of the time. 77. Challenges included ineffective work planning, the length of time to prepare terms of reference, protracted evaluations due to limited resources, and the time taken to establish evaluation committees, which affected the timeliness of procurement. This was mitigated through the inclusion of an Implementation Plan by Component and Activity as well as greater coordination between the PASO Council and the World Bank on the strategic direction of the project; the absence of this plan in the early stages of implementation did affect project implementation. Over the course of implementation there were 27 procurement plans approved by the World Bank, meaning that over the nine- year implementation period, the procurement plan was adjusted close to once every quarter. While this reflects a reactive approach to programming, it also reflects the changes in project direction over time and the addition of new project activities. 78. Many of the procurement transactions throughout the lifetime of the project were low value, high transactional procurements. More than 50 percent of procurement activities were related to the hiring of individual consultants, which included contracts for ongoing specialist support to PASO. PASO noted a preference for individual consultants in terms of the level of interaction and overall effectiveness, with a concern being that firms would struggle to understand the regional context and scale of PASO, in addition to the complexity of the geopolitical space in the Pacific, which had implications for PASO’s operations. From 2020, procurement performance was reported on through the quarterly reports and six-monthly semester reports, against agreed key performance indicators (KPIs). For the value of contracts signed versus planned, changes in market rates resulted in differences between the cost estimate as approved and the actual procurement. 37 20 ISRs were reviewed. Page 32 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) Table 10. Contracts by type and component Component Contract type Contract type total A B C D Consulting services Individual 27 5 2 34 Firms 6 2 3 11 Goods 5 1 6 Works 1 1 Component total 33 5 9 5 52 Table 11. Overall achievement of KPIs 2020–2022 KPIs Target (%) Average results (%) Number of contracts signed vs planned 90 90 Value of contracts signed vs planned 90 75 Number of contracts signed within estimated cost out of all contracts signed 90 77 during the period Non-ad hoc procurement activities 85 87 Contracts signed within the planned date 80 77 Percentage of contracts (USD value/number) done via open tendering out of 80 83 contracts signed during the period Percentage of contract awards publicly advertised out of contracts signed during 100 100 the period 79. E&S policies were not triggered. As the financing of PASO supports institutional capacity development, E&S policies were not triggered, and the project was classified as ‘Category C’. The installation of the VSAT did not trigger additional E&S policies, as there were no large scale or irreversible impacts associated with the installation of the equipment. C. BANK PERFORMANCE Quality at Entry 80. The strategic relevance and approach and institutional aspects were clearly identified at entry. This was the World Bank’s first engagement with PASO as an implementing agency. Arguably, without World Bank support, PASO would not have been able to continue its services and support to the Pacific region and there was uncertainty regarding PASO’s ability to remain in existence. The engagement and financial support from the World Bank resulted in a change in environment, in which PASO transformed into a reputable organization and solidified further support from the donor community. At entry, the support from the World Bank was highly targeted and constrained, with a small initial investment that grew over time. The project was delivered under condensed procedures and provided much needed resources in a timely manner. While the strategic relevance and institutional aspects were clearly identified, aspects related to governance and politics, as well as human resources and organizational capacity, weakened quality at entry. Page 33 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) For example, the amount of time needed for decisions to be made within the complex multi-faceted regional framework was over-estimated, as was the expectation that key stakeholders had a firm grasp of World Bank policies and procedures. Quality of Supervision 81. The World Bank Task Team provided proactive support to help the project adapt to changing needs during implementation, and overall implementation support was adequate. Implementation support missions were carried out on a frequent basis, and the Task Team closely monitored progress and addressed challenges in a proactive manner. Task Team members participated in annual/semi-annual PASO Council meetings to ensure engagement with Council members on key decision points in the reform process. As a result of COVID-19, face-to-face missions were not conducted during 2020 and 2021; however, these resumed in 2022. A mid-term review of the project was completed in April 2016 as part of the five-year implementation cycle to evaluate project performance. 82. Opportunities for improvement and to alter the scope of the project because of changing needs were acted upon. The World Bank Task Team undertook two additional financings (including project restructurings) and three stand- alone project restructurings over the nine-year implementation period. Project restructurings from 2017 occurred on a close to annual basis. Although project papers outlined changes in project needs, the rationale and storyline for the number and succession of project restructurings were not always adequately detailed or candid about why changes were required. In addition, during project implementation, there were multiple changes in the composition of the World Bank Task Team, including the Task Team Leader. Transition arrangements were occasionally not made clear to the implementing agency, resulted in a loss of institutional knowledge, or a change in approach to supervision arrangements. Furthermore, often, Task Team leaders did not have a technical background in regulatory aviation, which would have been beneficial to the project to further support implementation. Justification for Overall Rating of World Bank Performance 83. The overall rating for World Bank performance is Moderately Satisfactory. D. RISK TO DEVELOPMENT OUTCOME 84. At project completion, the risk to development outcome was considered Substantial, particularly in view of the sustainability of investments. While aviation is intended to be a self-sustaining industry with states and operators collecting revenue to maintain their assets and staffing departments, this continues to be a challenge. COVID-19 highlighted the fragility of the aviation sector, particularly in the Pacific region, and the need for states to improve their own financial sustainability moving forward. While the PASO Reform Project has contributed to enhancing the financial sustainability of PASO and enhancing donor coordination and support, PASO still relies on external donors for operations and to expand support in the region. Key focus areas for PASO include an implementation plan for how services will continue to be provided to member states, how to manage a growing program and additional scope for PASO (particularly considering the aim of reaching a Level 2 RSOO), and how to continue to implement a financing mechanism that will ensure stable services with decreased reliance on donor financing to support day-to-day operations. Page 34 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) V. LESSONS AND RECOMMENDATIONS 85. Lessons and recommendations from the project cover the areas of: (i) governance arrangements and institutional engagement; (ii) regional organizations and strengthening donor coordination; and (iii) supporting the climate resilience agenda. 86. Governance arrangements and institutional engagement. Institutional engagement usually requires a longer implementation period than a typical World Bank project of 5 years. Through the PASO Reform Project, 10 member states were involved in the governance process, which affected the timeliness of decision making. Furthermore, greater flexibility in project designs also allows for course corrections to be made during implementation and to support changing needs. Projects should consider the capacity of stakeholders and develop this further in support of project implementation, and the capacity of members states in the areas of governance and aviation safety and security could be further enhanced. For example, PASO Council members play a key role in the success of PASO’s ability to provide aviation safety and security in the region. Therefore, moving forward, ensuring that Council members have capacity in the areas of governance, strategy, finance, and leadership is essential to decision making for PASO. In addition, in recognizing that skills in aviation are thin and there is likely to be a shortage of aviation professionals in the future, ICAO has launched a Next Generation of Aviation Professionals initiative. PASO can further build capacity in the aviation sector through focusing on the development of a pool of talent. This may include, for example, investigating options for supporting outreach programs and accreditation in the Pacific region. Also, the PASO Reform Project did not include a specific focus on supporting women in the aviation sector, which could be investigated as part of the next generation development. Lessons for future training and capacity building should be taken from the PASO Reform Project, particularly with regards to achievability. An example is the PIDP, which was set up to train, support, and mentor appropriately qualified and experienced CAA personnel in member states to become ICAO-recognized inspectors for the Pacific region. PASO and its member states identified candidates for this program with project funds to support training and mentoring, with the aim that the candidate would be accepted to the PASO Pool of Inspectors. The high entry requirements for the PIDP meant that members states would regularly nominate candidates, but most of the nominations did not meet the criteria of the PIDP. There were only four candidates who were able to meet the criteria and they were all considered to be ‘Developing Inspectors’. Those candidates attended some ICAO trainings; however, this was disrupted due to COVID-19 travel restrictions. Unfortunately, all four candidates no longer work for their state CAAs and hence, no longer qualify for further training. 87. Supporting regional organizations and strengthening donor coordination. The region contains many small island nations, often spread over large geographic regions, and therefore aviation is vital for domestic and international connectivity as well as being a critical economic enabler. Many Pacific states face challenges to independently deliver their own internationally compliant aviation oversight and resolve safety deficiencies for a variety of reasons including insufficient financial, technical, and/or human resources. These challenges provided the catalyst for the creation of a regional approach to address common challenges and highlighted the need for a regional organization to provide a collective approach to aviation. When working with regional organizations, a lessons learned is to consider taking a phased approached to avoid programing becoming over ambitious and to ensure that outcomes are realistic and can be achieved within the project timeframe. In addition, where there are prominent donors in a sector, projects should coordinate to maximize project investments, enable organizational strengthening, and support continuous service delivery. 88. Addressing climate change and environmental sustainability in aviation. In June 2022, PIFS launched the Blue Pacific Strategy 2050. This Strategy outlines the importance of addressing climate change and environmental issues in the Pacific region. Looking forward, PASO could provide support on how the Pacific region could participate in addressing Page 35 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) climate change and environmental issues, which was not considered as part of the PASO Reform Project. The aviation sector is expected to grow significantly in the coming decades, and the strategic focus of transport sector investments within the Pacific region will continue to be on improving connectivity. Sustainable aviation, i.e., efforts and initiatives taken by the aviation industry to reduce its carbon footprint and minimize the negative impact of air travel on the environment, will be at the forefront. Options for air travel will include the use of biofuels and flight optimization. PASO could further support environmental sustainability by supporting feasibility assessments and implementation plans for rolling out changes across the region, such as the introduction of biofuels. . Page 36 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Strengthen PASO coordination capacity Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Indicator One: Successful Text No positions filled Lead Technical Expert General Manager, General Manager, recruitment of General appointed/ Two Operations Manager, Operations Manager, Manager, Operations technical experts Corporate Services Corporate Services Manager, Corporate Services appointed Manager in place. Manager in place. Manager 01-Nov-2013 31-Dec-2018 23-May-2018 31-Jul-2022 Comments (achievements against targets): This indicator was achieved in 2017. The Project originally envisaged recruiting three technical staff to coordinate oversight areas (airworthiness, flight operations, and aerodromes/security). The PASO Council endorsed a new organizational framework that was recommended as part of the “PASO Governance and Management” assignment, which sought to have a General Manager, an Operations Manager, and a Corporate Services Manager rather than appointing technical experts as full-time inspectorate staff. This indicator was subsequently revised to reflect this as part of the second Additional Finance. In addition, the project financed the appointment of a Transition Manager, to help facilitate the organizational changes and to oversee the initial activities and procurements. Page 37 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Indicator Four: PASO Percentage 34.00 80.00 80.00 addresses at least 80% of the GASOS functions for a Level 2 31-Oct-2021 31-Dec-2022 31-Dec-2022 RSOO Comments (achievements against targets): The description of the PDO4 indicator was revised to “PASO addresses at least 80% of the GASOS functions for a Level 2 RSOO” as part of the Restructuring approved on December 3, 2021. This indicator was achieved, with 107 GASOS Functions available for PASO to complete, and 86 completed (80 percent). Objective/Outcome: Deliver regional aviation safety & security oversight, and technical and advisory services to PICs Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Indicator Two: Establishment Text No ROI Updated ROI in place Updated ROI in place. of a Register of Inspectors (ROI) 01-Nov-2013 31-Dec-2018 31-Jul-2022 Comments (achievements against targets): This indicator was achieved in June 2014, with the register of inspectors established and endorsed by the PASO Council. The registry comprises qualified technical experts from Australia, New Zealand, Fiji, and Tonga across the following specializations; aerodromes, airworthiness, air navigation services, and flight operations. There are currently 22 individuals and organizations within the pool. Page 38 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Indicator Three: Percentage 12.50 80.00 60.00 52.20 Recommended Annual Work Program Implemented by 01-Nov-2013 31-Dec-2018 03-Dec-2021 30-Jun-2022 PASO Members Comments (achievements against targets): The target was achieved three times. PASO exceeded the target of 80 percent of State work plans achieved in 2017 (80.1 percent) and 2018 (88.8 percent). The 2019 figure (76.4 percent) was affected by the strict application of PASO’s Debt Management Policy that led to the suspension of services to five Member States that had arrears with PASO of over 70 days. In 2020 (44.4 percent), this indicator was not achieved due to COVID-19 border closures affecting all PICs. The utilization of remote-based inspections allowed work to continue but at a much slower pace (Remote auditing was the primary way that PASO completed its work plans during the COVID-19 travel restrictions). The original target of 80 percent was also revised to 60 percent from 2020 onwards due to COVID-19 travel restrictions. The revised target of 60 percent was considered more reasonable as an average against all and for individual Member States. During 2021 and 2022 the work plans continued to be below the target, although the trend of work plans is moving toward this target as border restrictions have eased and tourism has picked up pace in Member States. The end target was revised as part of the Restructuring approved on December 3, 2021. A.2 Intermediate Results Indicators Component: Component A: Transitional Management and Support Page 39 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Component A: Intermediate Text PASO Constitution Updated PASO Updated PASO Results Indicator One: Constitution Constitution approved Revised governance by PASO Council structure (technical and financial committee 01-Aug-2013 31-Dec-2018 31-Jul-2022 composition), as per PASO Constitution Comments (achievements against targets): The draft Constitution was completed in February 2020, approved by the PASO Council on July 6, 2021, and adopted at RAMM in June 2021. Revised governance structure addressed by PASO. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Component A: Intermediate Text Existing funding Sustainable Financing Sustainable Financing Results Indicator Two: mechanism Mechanism in place Mechanism in place Revised financing mechanism inadequate for sustainability 01-Aug-2013 31-Dec-2018 31-Jul-2022 Comments (achievements against targets): This indicator was achieved in June 2021 with the revised financing mechanism (i.e., the WPPM) being approved by the PASO Council. The adoption of the WPPM has seen a stabilizing of PASO cashflow, clear work planning to support State compliance with ICAO, and prioritized delivery of technical support to Member States. Page 40 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) Component: Component B: Establishment of a Pool of Regional Aviation Inspectors Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Component B: Intermediate Percentage 43.80 80.00 60.00 52.20 Results Indicator One: Adherence to Agreed Annual 01-Jan-2013 31-Dec-2018 03-Dec-2021 31-Dec-2022 Work Plans Annual Work Plan Percentage 130.00 80.00 60.00 45.00 Implemented - Kiribati 01-Jan-2013 31-Dec-2018 03-Dec-2021 31-Dec-2022 Annual Work Plan Percentage 0.00 80.00 60.00 100.00 Implemented - Cook Islands 01-Jan-2013 31-Dec-2018 03-Dec-2021 31-Dec-2022 Annual Work Plan Percentage 0.00 80.00 60.00 13.00 Implemented - Nauru 01-Jan-2013 31-Dec-2018 03-Dec-2021 31-Dec-2022 Annual Work Plan Percentage 0.00 80.00 60.00 16.00 Implemented - Niue 01-Jan-2013 31-Dec-2018 03-Dec-2021 31-Dec-2022 Page 41 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) Annual Work Plan Percentage 0.00 80.00 60.00 37.00 Implemented - Papua New Guinea 01-Jan-2013 31-Dec-2018 03-Dec-2021 30-Jun-2022 Annual Work Plan Percentage 54.00 80.00 60.00 49.00 Implemented - Samoa 01-Jan-2013 31-Dec-2018 03-Dec-2021 31-Dec-2022 Annual Work Plan Percentage 24.00 80.00 60.00 59.00 Implemented - Solomon Islands 01-Jan-2013 31-Dec-2018 03-Dec-2021 31-Dec-2022 Annual Work Plan Percentage 62.00 80.00 60.00 100.00 Implemented - Tonga 01-Jan-2013 31-Dec-2018 03-Dec-2021 31-Dec-2022 Annual Work Plan Percentage 0.00 80.00 60.00 3.00 Implemented - Tuvalu 01-Jan-2013 31-Dec-2018 03-Dec-2021 31-Dec-2022 Annual Work Plan Percentage 27.00 80.00 60.00 100.00 Implemented - Vanuatu 01-Jan-2013 31-Dec-2018 03-Dec-2021 31-Dec-2022 Comments (achievements against targets): Sub-indicators reflect each Member States' engagement of PASO in the planned annual work program. The average percentage across PASO is captured in the PDO-level indicator. Solomon Islands opted not to engage PASO services between 2015-2017 but re-engaged in 2018. Page 42 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) End target values were revised as part of the Restructuring approved on December 3, 2021, to better reflect the impacts of COVID-19. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Beneficiaries of job-focused Number 0.00 10.00 0.00 interventions 01-Jun-2018 31-Dec-2019 31-Dec-2022 Beneficiaries of job-focused Number 0.00 2.00 0.00 interventions - Female Comments (achievements against targets): This indicator was added as part of the Restructuring approved on December 18, 2018. There were 5 beneficiaries being trained to become inspectors (3 males and 2 females). However, the training taking place in Singapore was interrupted by COVID-19. Once travel resumed, there was no time to do a new assessment of CAA members to find new candidates to be trained. The target of 10 beneficiaries was not met by project closure. Component: Component C: Quality Management Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Component C: Intermediate Text No Quality Assurance Quality Assurance Quality Assurance Results Indicator One: System System in Use System in use. Establishment of Quality Assurance System 01-Aug-2013 31-Dec-2018 31-Jul-2022 Page 43 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) Comments (achievements against targets): This indicator was achieved in December 2022. The Quality Assurance System was developed incorporating updates required to meet Level 2 RSOO. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Component C: Intermediate Text Inadequate ICT system Upgraded ICT System Upgraded ICT system Results Indicator Two: in use in use in use Modernization of communication technology 01-Aug-2013 31-Dec-2018 31-Jul-2022 Comments (achievements against targets): This indicator was achieved, with the upgraded ICT system installed and in use. The financial system has been updated to the use of Xero. Microsoft Office has been enhanced to include upgrades including SharePoint. These are fully integrated into PASO’s daily use. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Project-supported Yes/No No Yes Yes organization(s) publish reports on effect of 01-Jun-2018 31-Dec-2019 31-Jul-2022 collaboration on project/program/policies (Yes/No) Page 44 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) Comments (achievements against targets): This indicator was added as part of the Restructuring approved on December 18, 2018. This indicator was achieved in June 2022. The PASO website includes an area for the public to view all reports via online publication of relevant organizational documents and records (including the Strategic Plan, the Annual Business Plan, Annual Reports, and Quarterly Reports). Component: Component D: Safety Equipment and Infrastructure Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Component D: Supporting Text No VSAT VSAT operating VSAT operating Regional Aviation Infrastructure 02-Jan-2017 31-Dec-2018 31-Jul-2022 Comments (achievements against targets): This indicator was achieved in June 2021, with the VSAT installed and operating. This indicator was included as part of the Project Restructuring and first Additional Finance, approved on January 5, 2017. Page 45 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1: Strengthen PASO coordination capacity 1. Successful recruitment of general manager, operations manager, corporate Outcome Indicators services manager 2. PASO addresses at least 80% of the GASOS functions for a Level 2 RSOO Component A 1. Revised governance structure (technical and financial committee composition), as per PASO Constitution IRIs 2. Revised financing mechanism for sustainability Component C 3. Establishment of quality assurance system 4. Modernization of communication technology 1. General manager, operations manager, corporate services manager in place 2. 80% of required GASOS functions achieved by 2022 Key Outputs by Component (linked to the achievement of the 3. PASO constitution approved by PASO Council Objective/Outcome 1) 4. Financing mechanism approved and in place 5. New quality assurance system in use 6. Upgraded ICT system in use Objective/Outcome 2: Deliver regional aviation safety & security oversight, and technical and advisory services to PICs 1. Establishment of an ROI Outcome Indicators 2. Recommended annual work program implemented by PASO members Component B 1. Adherence to agreed annual work plans IRIs Component D 2. Supporting regional aviation infrastructure Page 46 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) Citizen engagement 3. Project-supported organization(s) publish reports on effect of collaboration on project/program/policies 4. Beneficiaries of job-focused interventions 1. ROI in place Key Outputs by Component (linked to the achievement of the 2. Recommended annual work program implemented by PASO members Objective/Outcome 2) 3. VSAT installed and operating Page 47 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Chrisopher Bennett Task Team Leader Christopher J. De Serio Co-Task Team Leader(s) Charles Schlumberger Lead Air Transport Specialist Jinan Shi Senior Procurement Specialist Cristiano Costa e Silva Nunes Senior Procurement Specialist David Bruce Whitehead Financial Management Specialist Robert Gilfoyle Senior Financial Management Specialist Beverley Ann McLean Social Specialist Penelope Ruth Ferguson Social Specialist Marjorie Mpundu Senior Counsel Megan Rose Schlotjes Team Member Nora Weisskopf Team Member Shruti Pandya Team Member Ross James Butler Social Specialist Daniel Toga Infrastructure Hien Minh Vu Administative Core Services Shruti Pandya Administative Core Services Kuana Tupu Administative Core Services Supervision/ICR Dung Anh Hoang Task Team Leader Yi Yang Co-Task Team Leader Cristiano Costa e Silva Nunes Procurement Specialist(s) Janet Virginia Gamarra Rupa Financial Management Specialist Vivianti Rambe Environmental Specialist Leisande Otto Team Member Satoshi Ogita Team Member Laisa Duri Boedoro Procurement Team Keelye Hanmer Team Member (ICR Author) Rachelle Therese Marburg Social Specialist Nicholas Gerard Williams Procurement Team Page 48 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY13 0 1,574.45 FY14 1.900 8,422.03 FY16 0 1,558.26 Total 1.90 11,554.74 Supervision/ICR FY14 7.335 72,949.07 FY15 14.525 106,114.58 FY16 9.588 58,928.72 FY17 13.450 76,900.85 FY18 12.978 90,482.18 FY19 10.973 62,651.37 FY20 20.728 130,107.98 FY21 18.878 119,049.14 FY22 10.999 56,011.31 FY23 7.509 43,023.72 Total 126.96 816,218.92 Page 49 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) ANNEX 3. PROJECT COST BY COMPONENT Table A3.1. Comparison of Appraisal and Closing Completion Costs Financed Project Costs Additional Additional Total Adjustment Amount at Actual at Percentage finance 1 finance 2 (US$ (US$M) from PRIF Components approval project closing of approval (US$M) M) cancellation (US$M) (US$M) (US$M) (US$M) Component A: Transitional 1.65 0.10 1.05 2.80 1.23 4.03 144 Management and Support Component B: Establishment of a Pool 0.27 0.00 1.21 1.48 -0.54 0.94 248 of Regional Aviation Inspectors Component C: Quality 0.23 0.00 0.79 1.02 -0.65 0.37 61 Management Component D: Safety Equipment and 0 0.87 0.50 1.35 -0.36 0.99 0 Infrastructure Total 2.15 0.95 3.55 6.65 -0.32 38 6.33 194 38 A cancellation of US$320,000 from the PRIF MDTF was processed. Page 50 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) ANNEX 4. EFFICIENCY ANALYSIS Economic and financial analysis were not undertaken during project appraisal. The following summary was included in the PAD: PASO is a service organization with few assets and is not intended to be a profit-making entity. The technical assistance activities to be financed by the grant are not investments designed to generate financial returns for PASO, but rather strengthen its business model as a service provider and make it financially sustainable. The reform process is intended to modify fiscal governance and management arrangements, while introducing new fee structures that will allow PASO to offer the type of services that are more matched to regulatory needs – direct backstopping of member states civil aviation departments and authorities and coordination of inspection services and training, without being constrained by the predominant fee-for-service financing model. Page 51 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS According to the guidelines, the ICR was submitted to the Recipient for comments. As of the submission date, comments have not been received. Page 52 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) ANNEX 6. SUPPORTING DOCUMENTS • Pacific Aviation Safety Office. 2022. ICR Report, PASO Reform Project, Implementation Completion and Results for the reform of the Pacific Aviation Safety Office funded by the World Bank 2013–2022. • Pacific Aviation Safety Office. 2022. Pacific Regional Aviation Strategy. https://paso.aero/wp- content/uploads/Pacific-Regional-Aviation-Strategy-Report-web.pdf • Pacific Aviation Safety Office. 2022. PASO Strategic Plan 2023–2032. https://paso.aero/wp- content/uploads/PASO-Strategic-Plan-2022.pdf • Pacific Islands Forum Secretariat. 2022. 2050 Strategy for the Blue Pacific Continent. https://www.forumsec.org/wp-content/uploads/2022/08/PIFS-2050-Strategy-Blue-Pacific-Continent-WEB- 5Aug2022.pdf • World Bank. Aide Memoires for PASO. • World Bank. Implementation Status and Results Reports for PASO. • World Bank. 2013. Project Appraisal Document – Pacific Aviation Safety Office Reform - P145057. Report No: PAD532. https://documents1.worldbank.org/curated/en/521161468063253212/pdf/PAD5320PAD0Pac0100OUO0900Bo x379827B.pdf • World Bank. 2017. Regional Partnership Framework for Kiribati, Nauru, the Republic of Marshall Islands, Federated States of Micronesia, Palau, Independent State of Samoa, Kingdom of Tonga, Tuvalu and Vanuatu for the period FY17–21. Report No: 100997. https://thedocs.worldbank.org/en/doc/118921488323842915- 0070022017/original/R20170028.pdf • World Bank. 2018. Country Partnership Framework for Solomon Islands FY18–23. Report No: 122600-SB. https://documents1.worldbank.org/curated/en/187191533591240759/pdf/122600-CORR-PUBLIC-IDA-R2018- 0249-2.pdf • World Bank. 2018. Disclosable Restructuring Paper – Pacific Aviation Safety Office Reform – P145057. Report No: RES34743. https://documents1.worldbank.org/curated/en/135981545173291844/pdf/Disclosable- Restructuring-Paper-Pacific-Aviation-Safety-Office-Reform-P145057.pdf • World Bank. 2019. Disclosable Restructuring Paper – Pacific Aviation Safety Office Reform – P145057. Report No: RES39732. https://documents1.worldbank.org/curated/en/138451576820445104/pdf/Disclosable- Restructuring-Paper-Pacific-Aviation-Safety-Office-Reform-P145057.pdf Page 53 of 54 The World Bank Pacific Aviation Safety Office Reform Project (P145057) • World Bank. 2019. Country Partnership Framework for Papua New Guinea FY19–23. https://documents1.worldbank.org/curated/en/986831558749746464/pdf/Papua-New-Guinea-Country- Partnership-Framework-for-the-Period-FY19-FY23.pdf • World Bank. 2021. Disclosable Restructuring Paper – Pacific Aviation Safety Office Reform – P145057. Report No: RES41527. https://documents1.worldbank.org/curated/en/099920112032122420/pdf/Disclosable0Re0rm0Project000P145 057.pdf • World Bank. 2020. Performance and Learning Review of the Regional Partnership Framework for Kiribati, Nauru, the Republic of Marshall Islands, Federated States of Micronesia, Palau, Independent State of Samoa, Kingdom of Tonga, Tuvalu and Vanuatu for the period FY17–21. Report No: 120479. https://www.theprif.org/sites/default/files/documents/Nine-Pacific-Island-Countries-PIC9-Kiribati-Nauru- Marshall-Islands-Federated-States-of-Micronesia-Palau-Samoa-Tonga-Tuvalu-and-Vanuatu-Performance-and- Learning-Review-for-the-Period-FY2017-2021.pdf • World Bank. 2022. Performance and Learning Review of the Country Partnership Framework for Solomon Islands FY18–23. Page 54 of 54