82183 January 10, 2005 OPERATIONS EVALUATION DEPARTMENT Review of the Financial Sector Assessment Program (FSAP) Approach Paper I. Introduction 1. The Financial Sector Assessment Program (FSAP) is a major initiative, undertaken jointly by the World Bank and the IMF, in response to the financial crises of the late 1990s. The FSAP was set up in May 1999, initially as a 12-country pilot exercise to facilitate early detection of financial sector vulnerabilities and identification of financial sector development needs, as well as support an improved and coordinated dialogue among the national authorities, the Bank, and the Fund. As of July 2004, over 80 country assessments have been completed or are in the process, and the program has involved a significant deployment of resources at the Bank. 2. The FSAP provides findings and recommendations to country authorities both orally, in a concluding session with senior national authorities, and in a number of documents. The documents include an aide mémoire, previously referred to as a FSAP report; detailed assessments of compliance with selected standards and codes (and associated ROSCs); and supporting technical notes. In addition, drawing on the FSAP findings for a country, Bank staff are to prepare a Financial Sector Assessment (FSA), summarizing major findings of the FSAP mission; the IMF staff are to prepare a Financial System Stability Assessment (FSSA), which summarizes the findings of relevance to Fund surveillance as part of the regular Article IV consultation. Main objectives of FSAP 3. Various policy documents and review papers have indicated that the ultimate objectives of the FSAP comprise: (i) crisis prevention and mitigation—through the identification and resolution of financial sector vulnerabilities and their macroeconomic stability implications—contributing to global financial stability, and (ii) fostering financial sector development and its contribution to economic growth. 4. The rationale for placing development and crisis prevention objectives under the same program architecture (and thus having a joint Bank-Fund initiative) has been elaborated over time. Some reasons were operational—to optimize the use of the limited pool of expert resources, to avoid duplication of efforts, to promote consistency of the two institutions’ analysis and advice on financial sector issues. Other reasons reflected the recognition—reinforced by the early pilot experience—that most countries face both vulnerability and development issues, and considerable synergies might be achieved by addressing them jointly (e.g., institutional development aspects that may affect financial stability). These factors argued for an integrated approach to financial sector assessment. In terms of responsibilities, policy documents indicate that because of institutional expertise and mandate the IMF would take the lead on stability issues, while the Bank 2 would lead on development issues. However, since development aspects may have stability implications, coordination between the two institutions is key.1 Design and implementation of the FSAP 5. In September 1998, the Financial Sector Liaison Committee (FSLC) was set up, comprising senior staff from the Bank and Fund, to enhance operational coordination between both institutions on financial sector issues, including policy advice and support. Subsequently, the FSLC took managerial responsibilities over the FSAP, including country selection, assigning lead responsibility between the Fund and the Bank in each country, and resolving contentious issues in specific cases. The FSAP was launched formally in May 1999, initially as a pilot of 12 country cases and later extended to 20 cases for FY2001.2 6. Fundamentally, the FSAP was initially conceived as a diagnostic and policy advice tool which would provide: (i) confidential advice to country authorities, (ii) information for the Bank and the Fund on development and stability issues, and (iii) in some cases, information to the public and private sector. 7. The main design features of the FSAP architecture comprise the following: (a) Voluntary participation. Policy papers have indicated that, over time, all countries were expected to participate, although the mechanisms to achieve that have not been made explicit. In addition, agreement to participate involves only an initial assessment, not a permanent commitment to participate in follow-ups and reassessments. The voluntary participation and limited scope of the commitment might diminish the contribution of the FSAP to global financial stability if some countries of systemic importance do not participate. In practice, resource constraints have required some scaling back of the number of FSAP assessments—to about 17-20 a year— which would imply that a comprehensive assessment for the entire Bank/Fund membership would take place over about a 10-year period.3 In principle, priority would be given to systemically important countries, and other selection criteria would include external sector weakness or financial vulnerability, likelihood of major reform programs, or where features of the exchange rate and monetary policy regime suggest potential vulnerabilities. (b) Different outputs for different purposes including: (a) comprehensive and confidential FSAP reports to advise country authorities; and (b) separate summary reports—the Fund’s FSSA and Bank’s FSA—to inform the Boards. The initiative provides for voluntary publication of the FSAs, FSSAs, and other non-confidential FSAP documents to broaden 1 In the case of industrialized countries, the exercise was carried out by the Fund, as financial development issues were not considered to be significant. 2 See Annex 2 for the list of assessments and updates. 3 More broadly, resource constraints have raised the issue of whether the design of FSAP should aim for fewer cases with a comprehensive coverage of risks and vulnerabilities, or a larger number of more tailored assessments. 3 dissemination and inform the private sector.4 As of July 14, 2004, 70 FSAPs had been completed, 41 FSSAs had been published, and 14 FSAs had been published. (c) Standardization of coverage and assessment of financial sector issues and their impact on development and macroeconomic stability. However, initial policy papers also emphasized the need for selectivity and focus based on country circumstances—including on which standards to assess. The tensions and tradeoffs between standardization and tailoring were recognized in later documents and the need to move to a more country tailored approach was highlighted. (d) Joint Bank-Fund initiative, with shared responsibilities between the IMF and World Bank, with lead responsibility depending on country circumstances. In the case of industrial countries, assessments are typically carried out exclusively by the IMF. 8. Implementation of the program would consist of: (i) the identification of financial sector risks, vulnerabilities, and financial sector development needs by a team of Fund and Bank staff and outside experts from cooperating official bodies (mostly from central banks and supervisory authorities) through a series of field visits and using a variety of tools and methodologies; (ii) the articulation of findings and prioritized recommendations; and (iii) follow-up activities to assist in and assess the implementation of recommended measures. In addition, reassessments are to be conducted to identify new sources of vulnerabilities and development needs. (a) The assessment of vulnerabilities and financial sector development issues include a review of prudential regulation and supervision, liquidity management (financial policies, instruments, and market arrangements), arrangements for crisis management (financial safety nets, frameworks for bank and corporate restructuring and bankruptcy), market structure, maturity and development of market infrastructure, maturity and institutional capacity of banking and non-banking financial institutions, development of the securities market, legal framework, depth of financial intermediation; access to credit and other financial services; financial market integrity (anti-money laundering and the combating the financing of terrorism); and links to the development of the real sectors. (b) FSAP findings and prioritized recommendations are articulated in the report to the authorities (aide memoire) in detail, and summarized in the FSA and FSSA. (c) Follow-up should be carried out to ensure that a proper program is designed to assist in addressing vulnerabilities and development issues. The Fund uses the FSAP findings as a basis for discussions with country authorities at the time of the Article IV consultation. FSAP findings are also intended to 4 Publication policies have evolved over time; the pilot exercise barred explicitly publication, but later policy documents have permitted voluntary publication of the FSSA, FSA, and some background material. 4 help shape the Bank’s work program. In addition, reassessments were initially conceived to take place every four to five years. Later staff documents put the frequency at seven to ten years because of resource constraints and their implications for the pace of the program. II. Issues for the Evaluation 9. This is one of three OED evaluations of the Bank’s financial sector work. The first was a review of Bank Lines of Credit; the second was a review of Bank assistance in the Financial Sector Reform, excluding Lines of Credit and Financial Sector Assessments. The IMF’s Independent Evaluation Office (IEO) is undertaking a parallel evaluation of the FSAP. It is expected that the IEO evaluation will concentrate more heavily on stability aspects, in line with the Fund’s main focus in the FSAP, while the OED review will focus more on development issues.5 10. Given the brief time that the FSAP has been in place, this FSAP evaluation will focus on reviewing the aspects of the FSAP which would affect the objectives, specifically, the inputs and outputs (see Figure 1 for illustrative framework).6 In addition, this evaluation will review the relevance and effectiveness of the program and the implementation issues related to the program. The general questions to be reviewed are: (a) Inputs: Have the program and the assessments been well-designed to address the stability and development issues, including identifying development issues, financial vulnerabilities, and sources of risk? Have findings and recommendations been clearly articulated and prioritized? (b) Outputs: Have the Bank and Fund worked effectively with the country authorities and other donors to ensure that recommendations have led to policy and institutional changes that significantly improved financial sector development and reduced financial vulnerabilities? (c) Implementation issues: Has the FSAP been implemented efficiently? How do the costs of the FSAP compare with other financial sector AAA? Does the FSAP displace or complement other financial sector work? Has Bank- Fund coordination been proceeding well? Is the work well integrated with other Bank activities? Are the results of the FSAP integrated into the CAS? 5 There have been several internal and external reviews of the FSAP program; the main findings are summarized in Annex 1. Given the relative newness of the program, however, those reviews focused mainly on inputs, outputs, and process, and very little on outcomes. 6 Ideally, this FSAP evaluation would measure whether the ultimate objectives of increased financial stability and the prevention of crises, and fostering financial sector development and its contribution to economic growth have been achieved. But, given the limited time period, limited sample, and the voluntary nature of the program, such an analysis is premature. 5 Figure 1. The FSAP’s Framework Inputs Diagnosis • Assessment of financial sector risks and vulnerabilities analysis of institutional, structural, and market features; stress testing; FSI; standards and codes. • Re-assessments Outputs Findings, recommendations and prioritization • FSAP reports to country authorities • FSA to Executive Board Follow-up on recommendations • Bank and other donor programs for lending/AAA • Government actions: policy, legal and institutional change Outcomes • Reduction of financial vulnerabilities • Improved financial sector depth and efficiency Impact • Crisis prevention • Financial sector development leading to economic growth Assessing Inputs 11. Effectiveness of the assessments includes whether the coverage of countries assessed reflects appropriate priorities, and whether, at the individual country level, the coverage of financial development issues, sources of risks and the identification of vulnerability is adequate. Issues include: (a) Scope of country coverage. • Has the actual coverage of FSAPs across countries followed appropriate priorities, taking into account cost effectiveness vis-à-vis potential risks and benefits? Have countries of systemic importance, especially emerging market ones, volunteered and received higher priority as envisaged in policy documents? Has voluntary participation of countries involved early dialogue with the authorities on assessment issues and coverage? 6 (b) Identification of development issues, major sources of risks and vulnerabilities at the country level. • Do the general framework and design of the study support the overall objectives of the program? • Has the coverage of issues within specific countries been balanced between comprehensiveness and tailoring to country circumstances? • Does the FSAP identify the potential synergies and tradeoffs between financial development and stability issues? • Is the analysis cast broadly enough to capture less obvious vulnerabilities and development issues (for example transfer of risks between government and financial sector, perceptions of implicit government guarantees)? • Does the analysis adequately address related development issues, such as development of the corporate sector, small and medium enterprise (SME) development, and other “real” sector development issues? • How has the assessment process dealt with the distinction between de jure processes and de facto implementation, including issues such as regulatory capture, past failures to follow pre-arranged crisis-resolution mechanisms, proper regulatory framework but pervasive governance problems, etc.? (c) Identification and usefulness of recommendations. • Have FSAPs proposed specific and actionable measures to address identified vulnerabilities and development issues, and have such issues and measures been prioritized? Have recommendations been articulated as a coherent “roadmap” for reform with regard to suitability, sequencing, and implementation capacity? Did these “roadmaps” provide a basis for the Bank and other donors to incorporate the recommendations into their plans for TA/advisory/lending programs? • Have FSAP products—aide mémoires, FSSAs, FSAs—been delivered in a timely fashion and served a clear purpose? Have FSAP products been relevant to their different audiences, i.e. country authorities, the Board, Bank operations, markets, etc.? Is the current degree of transparency appropriate? Assessing Outputs • Have recommended actions/reforms been effectively implemented by country authorities? Aspects to be reviewed could include: - Improvements in governance: (i) independence of central bank; (ii) independence of the banking sector from government influence; and (iii) independence from large borrower influence (State-owned enterprises, related companies). 7 - Improvements in regulatory and legal frameworks and implementation. - Improvements in skills training and professionalized management, both at the central bank and financial institutions. • Has the Bank, with the IMF and other donors, supported these actions and reforms through follow-up work with the authorities? • Have the FSAP diagnoses and reports been helpful in recent cases of financial crises? Assessing Implementation Issues • How well has Bank-Fund coordination been working? What has been the division of labor and coordination with the World Bank, including on lead responsibilities and on balancing potential tradeoffs between financial development and stability issues? Has consideration of institutional priorities been balanced in the conduct of FSAPs? Has coordination extended to follow-up work? • How well is the FSAP integrated with other Bank programs? Has the FSAP received adequate attention and “ownership” internally in the Bank? Are FSAP recommendations taken into account when designing a Country Assistance Strategy, as well as specific lending or AAA projects? • Is the FSAP given appropriate access to resources, particularly access to the needed specialists? Are there any constraints on the required skills mix to ensure that FSAPs are staffed properly? Are team members also available for related work, including preparatory and follow-up work on the FSAPs? • How efficient is the use of FSAP funds? How do the costs of the FSAP compare with other financial sector AAA? Does the FSAP displace or complement other financial sector work? III. Methodology of the Evaluation 12. The evaluation will use a variety of types of evidence to address the questions posed in Section II. The primary “level” at which an issue is addressed (e.g., cross-section analysis of all FSAP cases or detailed case study) will depend on the nature of the question. Most of these inputs will be developed in conjunction with IEO. 13. Cross-country analysis. Using a database of indicators, this will address issues such as how FSAP priorities were implemented in practice, and will compare countries which have participated in FSAP with those which have not, to see whether there are indications that participation in the FSAP affects financial sector development. 8 14. Desk reviews and in-depth studies. Desk reviews on all FSAPs would be conducted, including evidence on effectiveness of assessment in country, articulation of recommendations and linkage to TA, policy and institutional change, and integration with other Bank work. In addition, more thorough reviews would be completed for a limited set of countries,7 including cases under special country circumstances such as cases at the brink of major macroeconomic crisis; cases where there was a banking crisis following the FSAP; cases of FSAPs following financial sector crisis; and cases with major shifts in the market structure (privatization programs, opening of markets to foreign financial institutions), and cases that the staff and others consider "successful experiences," such as cases where the FSAP had an impact on policy changes and development strategies. Desk reviews would be augmented as need with interviews with key counterparts. 15. Reviews of other evaluations and studies. These would include other evaluations on the factors influencing financial sector stability and development. (See Annex I for a summary of some existing evaluations) 16. Interviews and Surveys. The interviews would gather insights from diverse groups of stakeholders, including Fund and Bank staff, financial market participants, and country authorities. They would be complemented by surveys to obtain evidence systematically on the views from various stakeholders on a more specific range of issues. IV. Timetable 17. Timetable. Field visits for some of the in-depth country studies would take place through the first quarter of 2005. The final report is expected to be presented to the Board in Fall 2005. 18. Collaboration with the IEO. The modalities for coordination with IEO’s evaluation are being discussed, but the aim will be to cooperate on various ‘joint’ inputs—including joint country studies and collaboration on survey exercises. While IEO and OED will share findings and drafts, each unit will prepare a separate report. 19. Peer Reviewers. Ms. Laurie Effron (OEDCR) and Ms.Gloria Grandolini (BCFBD) are the peer reviewers. 7 Approximately 19-20 in-depth studies will be carried out jointly with IEO. IEO will carry out an additional 5-6 studies of developed countries, which OED will not participate in. Of the joint in-depth studies, it is envisioned that 4-5 studies will include field visits and additional work. 9 Annex 1 Internal and External Reviews of the FSAP: Summary Findings A number of internal and external reviews have been carried out on the FSAP. The main findings are summarized below: • An internal evaluation by the Fund was carried out as part of a review that was set up to assess the relevance and appropriate balance of the wide range of activities of the then Monetary and Exchange Affairs (MAE) department,8 as well as the effectiveness and efficiency of its operations. The group produced a report in late 2002 that covered in part the FSAP, including issues of quality control, number of countries assessed yearly, prioritization of systemic countries, and integration with Article IV consultations.9 Its main recommendations with respect to the FSAP included reducing the number of FSAP targeted each year; greater priority on emerging markets and financial centers, and on perceived risks and systemic importance; more selective approach to issues and standards covered; and greater involvement of area departments in the FSAP process. • In November 2002, the Bank’s Quality Assurance Group (QAG) rated four of the individual FSAPs. Since this was a limited sample, care must be taken in interpreting findings at this stage. QAG found the FSAP work in the sample to be of good quality and strategically relevant. However, the reviews indicated that Bank staff were concerned that: (i) development issues were not as strongly emphasized as stability issues; (ii) integration with the “real” sectors, especially corporate sector, was not adequately addressed; (iii) the value-added of doing three papers (the FSAP aide memoire, the FSSA, and the FSA) was questionable; and (iv) the FSAP was more strongly driven by Fund requirements, both in terms of logistics (such as integrating timing with Article IV Surveillance) and content (spending too much time on Standards and Codes relative to development issues). • In February 2003, a comprehensive internal review of the FSAP was produced jointly by the staff of the Fund and the Bank.10 The key recommendations of the report, broadly endorsed by the Board, comprised the need to tailor the scope of FSAPs to country circumstances (i.e., selectively narrow the number of issues and standards assessed); reduce the number of FSAP assessments and reassessments initiated each year; deploy other instruments for follow-ups and ongoing monitoring (e.g., focused FSAP updates, expert participation in Article IV consultations); and streamline the production and review of documents. 8 Now the Monetary and Financial Systems Department (MFD). 9 Review of the Monetary and Exchange Affairs Department, November 2002. 10 Early progress reports and update in 2000 focused their attention mainly on the set up and initial implementation of the initiative. 10 Annex 1 • The FSAP has also been subject to external reviews, including the June 2003 report of the General Accounting Office (GAO) of the United States.11 The main findings of GAO’s assessment comprise the need to improve the timeliness in the production and publication of reports, to expand the coverage and frequency of follow-ups, and to increase participation including through making the program mandatory. • The FSAP was also included as part of OED’s review of global initiatives in June 2004.12 Since the FSAP is still relatively new compared to most of the global programs, and since this study was to be carried out shortly, a detailed analysis was deferred. However, the global programs paper did note some of the same concerns that were raised in the QAG reviews; i.e., that the FSAP might serve the needs of the Fund more than those of the Bank, and that there was more focus on macroeconomic stabilization than on assisting countries on generating investment funds, improving allocation of funds, and accessing international capital. 11 International Financial Crises—Challenges Remaining in IMF’s ability to Anticipate, Prevent, and Resolve Financial Crises. Report to the Chairman, Committee on Financial Services, and to the Vice Chairman, Joint Economic Committee, House of Representatives, June 2003. United States General Accounting Office, GAO-03-734. 12 “Addressing Challenges of Globalization: An Independent Evaluation of the World Bank’s Approach to Global Programs – Phase 2 Report”; June 21, 2004. 11 Annex 2 FSAP: Completed and Ongoing/planned (in italics) per Fiscal Year13 2000 2001 2002 2003 2004 2005 Columbia* Ghana* Gabon Kyrgyz Republic Macedonia Belarus Lebanon Guatemala Switzerland Japan** Jordan* Sudan Canada Poland Lithuania Bangladesh Kuwait Norway South Africa* Armenia Luxembourg Hong Kong New Belgium Zealand** El Savador Israel Sweden Honduras Kenya Italy Hungary Peru Philippines* Malta ECCU14 Paraguay Iran Yemen Korea** Mauritius Ecuador Rwanda Kazakhstan* Senegal Costa Rica* Singapore** Azerbaijan Serbia Ireland** Slovenia* Bulgaria* Oman Austria Albania Cameroon* Iceland Sri Lanka* Germany** Netherlands Jamaica Estonia Czech Morocco Mozambique Nicaragua Trinidad and Republic Tobago India* Uganda Nigeria Tanzania Chile* Spain United Romania* Saudi Arabia Bahrain Kingdom United Arab Slovak Algeria France Emirates Republic Latvia Barbados Bolivia Pakistan Tunisia* Brazil* Moldova Finland Ukraine México* Russia* Croatia Egypt* Georgia Zambia Total: 12 Total: 20 Total: 20 Total: 15 Total: 16 Total: 14 (Pilot (preliminary) countries) FSAP Updates Lebanon Hungary Iceland Ghana Senegal South Africa Slovenia Colombia Kazakhstan Uganda El Salvador Nigeria Peru Hungary Armenia Total: 0 Total: 2 Total: 1 Total: 1 Total: 4 Total: 4 (preliminary) Note: The FSAP for Argentina (FY 2001) and Uruguay (FY 2002) were not completed. The FSAP for Cote d’Ivoire (FY 2002) has not yet been completed due to security reasons. *Denotes country under consideration for joint desk study with IEO **Denotes country under consideration for desk study by IEO 13 Within the column of each fiscal year the order of countries is determined by the order of first missions. 14 ECCU stands for the Eastern Caribbean Currency Union.