Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CEMAC Regional Institutions Support (P099833) Report Number : ICRR0020705 1. Project Data Project ID Project Name P099833 CEMAC Regional Institutions Support Country Practice Area(Lead) Africa Finance & Markets L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-45290,IDA-H4340 30-Jun-2014 64,300,000.00 Bank Approval Date Closing Date (Actual) 04-Dec-2008 29-Feb-2016 IBRD/IDA (USD) Grants (USD) Original Commitment 50,000,000.00 0.00 Revised Commitment 50,000,000.00 0.00 Actual 39,153,298.86 0.00 Prepared by Reviewed by ICR Review Coordinator Group Nestor Ntungwanayo John R. Eriksson Christopher David Nelson IEGFP (Unit 3) 2. Project Objectives and Components a. Objectives The objective of the project is to encourage an expanded and better governed regional market and a more transparent, better regulated, and more competitive financial system, which will in turn permit increased access to finance and greater availability of resources, particularly facilitating the re-utilization of surplus oil revenues in Central Africa (Financing Agreement, p.6). The project objectives in the Project Appraisal Document (PAD) on page 9 were stated in a slightly different way, but the thrust of the project objectives was identical. This review assesses project performance against the objectives as stated in the Financing Agreement. Page 1 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CEMAC Regional Institutions Support (P099833) b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components The project has four major components as detailed below (the first three with several sub-components): (A) Strengthening the Central Bank of Central Africa Region (BEAC): (Cost at appraisal of US$14.5 million, against actual cost of US$14.6 million): This component was to fund the implementation of the Central Bank's strategic plan, including in relation to the areas of its statutory mandate, and its internal support functions. Toward increasing the effectiveness of monetary policy, activities envisaged, included the following: (i) refining monetary programming by updating the methodology and introducing it at the regional level; (ii) strengthening the statistical base for policy-making; and (iii) introducing government paper. In addition, work launched on a credit bureau and a system consolidating financial statements for key enterprises was to be completed. Toward encouraging the use of the regional payments system to expand access to financial services, activities include measures to: (i) simplify access to the system; (ii) encourage the use of the systems by microfinance institutions; (iii) discourage paper-based transactions and encourage electronic means of payment; and (iv) introduce a system to track “incidents” of payments. (B) Encouraging Regional Investment Financing (Cost at appraisal of US$15.0 million, against actual cost of US$7.71 million): This component was to support (a) the implementation of the 2008-12 strategic plan of the Central African States Development Bank (BDEAC), and (b) the establishment of a fund providing Study Sub-financings to Beneficiaries for the purposes of carrying out feasibility and pre- investment studies in the area of regional integration. Key activities were: (a) to support a fund that will help finance feasibility and pre-investment studies, (b) to support BDEAC to implement its business plan for the next five years with the following activities: (i) strengthening policies, procedures and internal controls; (ii) strengthening project evaluation and portfolio management; (iii) putting in place new policies on investments, asset-liability management; treasury operations; and risk assessment of financial intermediaries; (iv) definition of a strategy for resource mobilization; (v) establish guidelines for auditors in line with international practices; (vi) improving management tools and designing and implementing a medium-term information systems plan; and (vii) preparing and putting in place a comprehensive training program to upgrade staff skills. (C) Improving Regional Economic Policy Coordination, Financial Sector Supervision and Financial Integrity (Cost at appraisal of US$14.0 million, against actual cost of US$12.5 million): The component aimed to enhance (a) capacity building of the Commission of the Economic and Monetary Community of Central Africa (CEMAC commission); (b) capacity building of the Central Africa Banking Commission (COBAC); (c) development of the financial markets in Central Africa; and (d) capacity building of the Action Group Against Money Laundering in Central Africa (GABAC). Specific activities are delineated below: (i) To strengthen the CEMAC commission, the project will finance advisory services, materials, supplies, training and hardware to support the following activities: (i) harmonization of procedures for the elaboration Page 2 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CEMAC Regional Institutions Support (P099833) of economic statistics; (ii) implementation of an analytical tool to assess practices in transport corridors; (iii) establishment of a data base for CEMAC ensuring cooperation with BEAC and BDEAC; (iv) implementation o f a training program to upgrade CEMAC Commission staff skills. (ii) To beef up the COBAC, activities will be carried out: (i) revision of banking prudential regulations; (ii) upgrading the banking supervision system; (iii) improvements in the regulation, and designing and putting in place a system for supervision of microfinance institutions; (iv) putting in place a management information system adapted to COBAC’s requirements; (v) carrying out studies promoting banking sector integration and development; and (vi) implementing a staff training program. (iii) To strengthen the financial markets, IDA financing be limited to a study on the feasibility of the financial market, targeting harmonization of the legal and regulatory frameworks and the technical interconnections between the existing exchanges. (iv) To support GABAC, the project will support GABAC to carry out the following activities in its action plan: (i) a regional public information campaign on money laundering and financing of terrorism to make stakeholders, including senior government officials, aware of the policy issues, existing legislation and national requirements; (ii) updating as needed the legislation and regulations applicable to money laundering combating financing of terrorism; (iii) strengthening the regulatory framework by GABAC’s General Secretary by establishing appropriate evaluation methodology, carrying out staff training, and development of information systems. (D) Support for Project Implementation (Cost at appraisal of US$3.0 million, against actual cost of US$2.70 million): This component was to fund activities related to project coordination, management, and monitoring and evaluation, through the acquisition of goods, provision of consultants' services, carrying out of training, and financing of operating costs. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Cost: There are discrepancies among the costs reported in the ICR, in the Bank's SAP system, and the Client Connection. The project team did not provide satisfactory explanation related to the source of those discrepancies, despite a formal request to that effect. Financing: The same discrepancies mentioned above are found in the numbers related to the financing of the project, and were not clarified by the project team. Borrower Contribution: The appraised amount of US$12.30 from the Borrower was totally disbursed. Dates: The project was approved on December 04, 2008 and became effective on April 27, 2009. It was restructured twice in February 2014 and in February 2016 to extend the project closing dates, and to reallocate credit proceeds across expenditure categories. 3. Relevance of Objectives & Design a. Relevance of Objectives Page 3 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CEMAC Regional Institutions Support (P099833) The project design drew from the priorities of the 2003 Central Africa Regional Integration Assistance Strategy (RIAS) and the 2008 Sub-Saharan RIAS, which all identified regional financial sector integration as a key channel for assistance from the WBG. Similarly, the 2011 progress report of the 2008 RIAS confirmed that regional business and financial sector reforms can facilitate the achievement of optimal economic scale and spatial distribution across borders. Finally, most Bank country partnership strategies developed after the approval of the project (Cameroon-2010, Gabon-2012, and Republic of Congo-2012) stressed the catalytic role of a stable financial sector in deepening economic growth in the member countries and the region as a whole. A recently approved Country Partnership Framework (CPF) for Cameroon for the period of FY17-FY21 (Cameroon is the largest economy of the CEMAC region) has a pillar devoted to fostering infrastructure and private sector development, which stresses that the WBG will continue to work with the BEAC and other CEMAC institutions to improve access to finance in the region, and that it will prepare a successor project to this one, which is expected to support regional statistics, payments systems, oversight of banking institutions, financial inclusion, mobile banking, as well as strengthened governance of the BEAC (Cameroon CPF-2017). Overall, the project was and is still consistent with the CEMAC countries' search for deepened financial integration, and continues to be part of the Bank's drive to support regional integration through the financial sector. Rating High b. Relevance of Design The statement of the project objectives was complex, as it provided a cascade of objectives at twolevels. The project objectives were to encourage: (i) an expanded and better governed regional market and (ii) a more transparent, better regulated, more competitive financial system. These two objectives were to facilitate the achievement of a higher objective of an increased access to finance and greater availability of resources, particularly facilitating the re-utilization of surplus oil revenues in Central Africa. There was a possibility to simplify and clarify the statement of the project objectives, which would have started from a more precise description of the activities that were intended to be funded, and therefore identify the outputs that could be generated, and the outcomes and objectives that could be achieved after the completion of the activities detailed under the project's components. The description of some of the activities to be implemented was imprecise (see Subcomponents B and C of the Component 2), and this became a hurdle when the implementation phase arrived. While there was a logical theory of change between the identified activities and the first-level objectives, it was unclear how those activities could lead to the achievement of the higher level objective. The results framework in Annex 3 of the PAD was qualified by the ICR as unsuitable to link the activities to the expected outcomes and the stated objectives of the project. Because of the shortcomings in designing the results framework, the relevance of design is rated as modest. Rating Page 4 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CEMAC Regional Institutions Support (P099833) Modest 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective (i) To encourage an expanded and better governed regional market. Rationale Outputs (i) BEAC: The project provided technical assistance to improve BEAC's performance in its missions of defining monetary policy and managing foreign reserves. It provided IT solutions through developing databases on the balance of payments and on economic, financial, and monetary issues, and established a registry of payment irregularities in the CEMAC banking system. Internally, BEAC benefited from improved administrative management, including enhancement of internal audit systems and human resources management, implementation support of a new accounting system, and development of IT and accounting plans. (ii) BDEAC: The project provided a combination of technical assistance, capacity building, and IT hardware and software solutions, telecommunication infrastructure and vehicles. The project upgraded the aging IT infrastructure of the BDEAC, including the implementation of core banking information systems and provided connectivity to the regional payment systems, (SYSTAC and SYGMA), including the connection to the SWIFT system. The project provided support on the improvement of credit underwriting procedures including environmental safeguards plans, Anti-money laundering (AML) analysis tools, and cost-benefits study methodologies. Finally, the project supported the implementation of the human resources strategy of the BDEAC, and the staff of the BDEAC benefited from multiple capacity-building opportunities. However, key IT equipment was acquired just prior to the project closure and its efficacy has not been yet tested. (iii) CEMAC Commission: The project provided IT hardware, software, and telecommunication solutions, and developed an information system to cover administrative, financial, and accounting matters. The project provided also technical assistance to enhance CEMAC with the development of economic policy convergence reports, macroeconomic framework modeling, and the creation of an investment climate observatory. Finally, staff from the CEMAC benefited from the multiple training opportunities including results-based management and methods for GDP calculations. However, most of the investments with regard to hardware procured through this project were vandalized during the civil unrest, and the systems and tools developed are not currently being used. Outcomes (intermediate and final): BEAC: (i) In the area of monetary policy, all CEMAC member countries issue public securities since 2011 to finance their budgets, and the amounts of those securities have increased considerably over the years, limiting the use of monetary financing of budgets. (ii) Toward strengthening the management of foreign exchange reserves, the training of a forex marketplace team allowed an optimal management of the share of the foreign exchange reserves held outside the open account of the French Treasury. This share has exceeded 50 percent of the total amount of reserves, Page 5 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CEMAC Regional Institutions Support (P099833) and the rate of return it generates reached an average of 2.3 percent over the period 2008-2014, despite the erratic evolution of international financial markets. (iii) As a result of enhanced payment systems, the amount of electronic payments through the wholesale payment system (SYGMA) increased by more than 3.9 percent between 2008 and August 2015, far exceeding the target set by the project. This performance will be further enhanced by the implementation of the "Payments Incident Center - CIP", which is operational since February 2016. (iv) A Human Resources Master Plan (HRDP) for BEAC has been deployed and is being backed by a modern IT solution, and supplemented by a Multi-Year Training Plan (MTP), and a three-year program of human resources capacity building. (v) BEAC's internal auditors have been trained and are in charge of the new high-tech activities with new audit methods, providing the BEAC a world-level auditing performance. BDEAC (i) BDEAC governance improved with the creation of an Audit Committee and the appointment of Independent Directors, the adoption of new management policies and procedures, the management of associated risks, and the strengthening of internal control. (ii) BDEAC mechanisms to mobilize resources improved with the establishment of a structure for mobilizing concessional resources, adoption of new procedures for cost-benefit analysis of projects, implementation of new environmental and social standards, improvement of the legal framework, and establishment of a system of Compliance procedures and tools. (iii) BDEAC information systems were upgraded, including the integrated Banking Software, and the Management and Communication System, which have given the institution the necessary tools to reinforce the management of projects and associated risks. (iv) The BDEAC is implementing a new framework of human resources management with the aim of creating expertise that will contribute to the achievement of its strategic objectives in the short and medium term, which was mainstreamed in the new Bank's recruitment, evaluation and career management policy. CEMAC Commission (i) The project helped to develop and implement two tools, namely "The Macroeconomic Framework Model" and "The Investment Climate Observatory", as well as a reform of the multilateral surveillance system. These tools contributed to improvement in the convergence of the member countries' economies and the development of intra-CEMAC trade, and strengthened sub-regional economic integration. (ii) The convergence of CEMAC economies was reinforced by the new harmonized calculation of GDP as well as the reform of public finances through the six Directives on the harmonization of public financial management in the Member States, (iii) The project provided human capacity building, with 16 executives trained in results-based management, and hardware with the acquisition of computer and software equipment and a Very Small Aperture Terminal (VSAT) kit to improve the communication system. This outcome was affected by the insecurity that hit the country harboring CEMAC headquarters (Central African Republic). Contribution to higher level objective: See discussion at the end of Sub-objective 2 Rating Page 6 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CEMAC Regional Institutions Support (P099833) Modest PHREVDELTBL PHEFFICACYTBL Objective 2 Objective (ii) To encourage a more transparent, better regulated, and more competitive financial system, Rationale Outputs: (i) COBAC: Regarding Microfinance Institutions (MFIs), the project funded technical assistance in the areas of microfinance regulation and supervision and designed an accounting plan for the MFI institutions, as well as the creation of a system that manages the reporting requirements of MFIs (SESAME), which currently allows the filing of financial reports online. The system is successfully being used by 70 percent of the MFIs in the CEMAC region, while the remaining 30 percent are not covered, because of poor internet connectivity. Regarding the supervision of banking institutions, the key achievement was the implementation of an IT system that allows COBAC to supervise, in real time, all the banks operating in the CEMAC region. Moreover, the project supported a diagnostic of the compliance of COBAC with Basel II requirements and the development of an action plan. Finally, the staff of COBAC benefited from multiple capacity-building opportunities including accounting, consolidated supervision, stress tests, International Financial Reporting Standards (IFRS), International Accounting Standards (IAS), and internal audit. (ii) COSUMAF (Central Africa Commission for the Regulation of Securities Market): In this area, the project supported the completion of a study of the depth of the capital markets. Technical assistance was provided to support the COSUMAF during the IOSCO accession process, which led to the COSUMAF membership of the IOSCO. In addition, the project facilitated the organization of multiple dissemination events and capacity- building workshops to train stakeholders. Finally, IT equipment was acquired to modernize the IT infrastructure of the COSUMAF. (iii) GABAC: The project supported the development and dissemination of Anti-Money Laundering/Combatting the Financing of Terrorism (AML/CFT) campaigns, and provided TA to update the legal and regulatory environment. It developed (i) a manual to conduct mutual evaluations between countries, and (ii) an internal administrative, accounting, and financial manual. The project helped to implement an action plan to prepare GABAC to access the Financial Action Task Force (FATF) as a regional member. The project provided capacity-building opportunities to 24 evaluators from CEMAC members countries and supported the organization of typology reports for Chad and the Republic of Congo. Finally, IT equipment was procured to upgrade the IT capabilities of GABAC. However, GABAC had to evacuate the Central African Republic, and its procured IT equipment was vandalized and destroyed. Outcomes (intermediate and final): COBAC (i) There was improved ability of the COBAC to supervise, in real time, all the banks operating in the CEMAC region, and this allows the regulator to monitor, in real time, all CEMAC banks and to take corrective actions as needed. (ii) COBAC's workforce has more than doubled, making it possible to strengthen preventive supervision, and implement a program to strengthen human capacities and equipment. (iii) The new chart of accounts has significantly improved banks' compliance with prudential standards. As Page 7 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CEMAC Regional Institutions Support (P099833) regards the microfinance sector, a comprehensive monitoring system was set up, enabling over 700 authorized establishments to make regular reports on their activities. COSUMAF The development of internal and functional procedures for this institution, the popularization of the stock market culture, and the preparation and implementation of internal and external financial regulations allowed the institution to be admitted as a full member of the International Organization of Securities Commissions (IOSCO). The COSUMAF now has international recognition as a regulator of capital markets which means that the institution is compliant with the basic foundational requirements for securities regulators. GABAC (i) Public awareness campaigns organized in all member countries were accompanied by typology exercises to study the channels used by launderers and the conduct of mutual evaluations devices to ensure their effectiveness in anti-money laundering. New texts have also been adopted to reinforce the legal and regulatory framework for the fight against money laundering in the CEMAC zone. (ii) All the member countries have set up National Agencies of Financial Investigation (ANIFs) which are operational. A framework for the exchange of experience was set up for consultation between the ANIFs, the Permanent Secretariat of GABAC and the partners who provide support in this area. (iii) The GABAC institution was admitted as an observer member of the Financial Action Task Force (FATAF), which is an acknowledgment of the capabilities it has to effectively coordinate the fight against money laundering in the CEMAC space. Higher level objective: To permit increased access to finance and greater availability of resources, particularly facilitating the re-utilization of surplus oil revenues in Central Africa Some of the results and outcomes achieved under the two objectives contributed a great deal toward the higher level objective of an increased access to finance and greater availability of resources, and facilitating the re-utilization of surplus oil revenues in Central Africa. For instance, an optimal management of the share of the foreign exchange reserves by the BEAC, and enhanced payment systems have allowed quicker availability of resources for the CEMAC region. Moreover, the strengthening of the COBAC and the COSUMAF have helped the regional banking system to operate more efficiently, thus supporting it to avail financial resources for the CEMAC economies. However, assessment of the achieved outcome is difficult to determine given the weaknesses of the results framework throughout the project implementation period. Rating Modest PHREVDELTBL PHREVISEDTBL 5. Efficiency Economic and financial efficiency: This was a technical assistance project, and there was no standard economic and financial analysis conducted either at project appraisal or closure. Instead the PAD and the ICR identified a set of high-level potential outcomes that could be linked to the effective implementation of the Page 8 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CEMAC Regional Institutions Support (P099833) project, which included: (i) improved monetary policy, foreign reserves management, and cross-border flows, (ii) improved financial supervision and more financial stability of monetary reserves, and (iii) improved access to finance for companies and households. However, the ICR indicated (p.17-18) that some of those outcomes (foreign reserves management, cross-border payments, and regional investment flows) presented serious attribution uncertainties such that the recent trends of those outcomes cannot solely reflect the sole economic and financial efficiency of the project. Given the connectivity challenges of the CEMAC region and the human resource limitations of COBAC before the project approval, the implementation of IT solutions significantly improved its supervisory activity as the current IT system provides information to COBAC in real time about the financial situation of all banks in the region. On-site visits by COBAC supervisors currently last three weeks instead of six weeks as previously. Moreover, close to 70 percent of the MFIs are filing their financial statements online. Administrative and operational efficiency: There is a lack of clarity in the disbursement of respective windows of the project resources. The ICR is unclear as to why there were different rates of disbursement from the Borrower, the IDA credit and grants resources, and it was not indicated which resources funded which activities. Furthermore, early implementation was marred by weak project design and quick rotation of TTLs. Most importantly, the M&E design was not revised despite recommendations by key stakeholders to do so in order to facilitate implementation and better tracking of results. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of objectives is high, but the relevance of design is modest. Efficacy is modest for both objectives. First, toward an expanded and better governed regional market, key institutions (BEAC, BDEAC, and CEMAC Commission) benefited from a range of outputs, and achieved a number of outcomes as described under Section 4. Second, toward a more transparent, better regulated, and more competitive financial system, supported institutions (COBAC, COSUMAF, and GABAC) benefited from a range of outputs and achieved outcomes describe under Section 4. Although the results and outcomes generated at the level of the two sub- objectives fed into the higher objective, a systematic assessment of achievement of Sub-Objectives 1 and 2 at Page 9 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CEMAC Regional Institutions Support (P099833) the Outcome level was precluded given the weaknesses inherent in the results framework throughout the project implementation period. Therefore, their respective efficacies are rated Modest (see Sections 3b and 10 for further discussion). Efficiency is modest, and the overall outcome is rated as Unsatisfactory.” a. Outcome Rating Unsatisfactory 7. Rationale for Risk to Development Outcome Rating As most of the assistance was used to generate outputs (equipment, IT systems, technical assistance and capacity building) to the six beneficiary institutions, the key risk is that budget allocation will not be sufficient to maintain the systems, this being particularly critical for COBAC, the COSUMAF, and GABAC, which have lower operating budgets. Another risk is that of losing the knowledge acquired through all the capacity-building opportunities as a result of the staff turnover, but most of the beneficiary institutions have a comparatively better workers' compensation status. Technical knowledge provided to the PIU to support project implementation will remain in the project after project closure, as the staff are on the BEAC’s payroll. a. Risk to Development Outcome Rating Modest 8. Assessment of Bank Performance a. Quality-at-Entry While the project initiation was in line with the key recommendations of the 2006 CEMAC Financial Sector Assessment Program (FSAP), and the World Bank Group's country and regional strategies, lessons learned from the implementation of similar operations in other parts of the world were not sufficiently incorporated. Most importantly, the project was very complex and the flexibility in the design of the operation and the lax description of the potential activities ended up being a handicap during the implementation phase. Moreover, the project risks were not properly identified and mitigation measures were inappropriate, and implementation readiness was lacking after project approval. There were weaknesses in the crafting of M&E and other implementation arrangements. The ICR indicated that the initial results framework was inappropriate for tracking and reporting results, and it was not revised throughout the period of project implementation. Despite the complexity of the project, the team did not set up a strong coordinating unit to oversee the interaction of the multiple counterparts with different levels of implementation capacity. The engagement level with country counterparts was limited and key documentation of the project was provided in French only two years after the project approval. Page 10 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CEMAC Regional Institutions Support (P099833) Quality-at-Entry Rating Unsatisfactory b. Quality of supervision The rapid rotation of TTLs during the first two years affected significantly the project launch, and no ISRs were filed until two years after project approval. When the fourth TTL took over, there were close to 30 no-objection requests that were pending for more than six months. After agreement on the TTL arrangements in 2011, measures toward effective implementation included the translation of the operation-related documents into French, which permitted the reengagement with the six implementing counterparts. The composition of the supervision team was also strengthened as global experts from different disciplines were brought on board to ensure higher levels of quality, and the Bank team ensured that the World Bank Group procurement rules were well understood to avoid further delays. Coordination among key project stakeholders had been weak since the beginning as illustrated by the project suspension decided in 2010, but only discovered in 2012 by both the World Bank Group project team and the project counterparts when a disbursement request was rejected, although the fiduciary team did not find any problems with regard to the use of the funds by the beneficiary institutions. Despite shortcomings in the project design and M&E framework, the Bank decided not to restructure the project or to conduct a midterm review, given that building consensus among the six beneficiary institutions on a restructured program was judged politically difficult to strike, and the project was left without effective key performance indicators to evaluate its impact. Toward the second half of the life of the project, implementation and disbursement ratios improved dramatically, although implementation quality was affected, as was the case in COBAC whereby the IT system on banking supervision was implemented just a few weeks before the project closure, and required fine tuning to maximize the use of the IT tools and software. At the BEAC, the accounting reform was not yet completed and the payment incidents, balance of payments and financial statistics databases were not operational. Finally, the ISRs were prepared and filed only intermittently, and their ratings were in many instances too optimistic, and were most of the time dictated by the level of disbursement, and not by the progress toward outcome. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Unsatisfactory 9. Assessment of Borrower Performance a. Government Performance The senior management of the project, composed of official representatives of the CEMAC member states, was supportive and committed throughout the project life. However, the borrower (the BEAC) delayed in operationalizing the PIU as required by the Legal Agreement, and the steering committee in charge of the high-level oversight of the project only met twice during the life of the project. Finally, the project coordinators at the beneficiary institutions level were appointed with delay and contributed little to the implementation of the project. Page 11 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CEMAC Regional Institutions Support (P099833) Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance Project implementation was handled by both the Unité de Gestion des Réformes des Institutions Financières (UGRIF), which was the coordinating entity, and each beneficiary entity was accountable for implementing its own programs. Although made operational with delay, the PIU performed its tasks effectively. Project, financial and procurement management were rated between Moderately Satisfactory and Satisfactory. However, disbursement was suspended during 2010-2012, and the ICR did not provide any adequate explanation. The PIU had a very challenging mandate as it had to coordinate the implementation of a significant number of activities among six institutions, although the technical implementation was managed by each beneficiary institution. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The results framework and monitoring in Annex 3 of the PAD were difficult to implement, mainly because of the complexity of the stated objectives. The M&E arrangements were not carefully crafted as many of the selected indicators had attribution problems and data were sometimes not available to keep track of progress. UGRIF had the responsibility for coordinating M&E activities while data collection and monitoring were to be supplied by each implementing institution. The relevant departments of each implementing institution were responsible for data collection, management and reporting of their respective component and subcomponents. UGRIF was charged with consolidating information and data received from each of the agencies. b. M&E Implementation Because the M&E indicators presented attribution problems and sometimes were difficult to track, both the Bank and the beneficiary entities did not effectively monitor these indicators and focused more on the implementation of operation. Despite the recognition that the results framework was inadequate, it was not revised throughout the period of project implementation, and while a Mid-Term Review (MTR) was planned, it did not take place. A number of ISRs mention the inadequacy of the indicators; however, no actions were taken by the Bank to revise them with more relevant and measurable indicators. UGRIF did not fulfill its M&E Page 12 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CEMAC Regional Institutions Support (P099833) responsibility.Throughout the project life, the PIU did not have a full-time M&E specialist. Only a short-term consultant was hired at the beginning of the project to help with the M&E. c. M&E Utilization The ICR did not provide any information related to M&E utilization. M&E Quality Rating Negligible 11. Other Issues a. Safeguards The project was designated as a category C, and no safeguard policies were triggered. b. Fiduciary Compliance Financial management: There were only minor delays in the submission of interim and annual financial statements and financial management varied between MS and S throughout the project implementation period. However, the ICR did not provide conclusions of the final external financial audit . Moreover, there were discrepancies among Bank internal sources of information related to the cost and financing of the project. Finally, the ICR did not provide a clear story related to the lack of disbursement and the project suspension during the period 2010-2012. Procurement: Technical procurement capacity of the six different beneficiary entities was uneven, and it took time for the procurement specialists to become comfortable with the Bank procedures. Procurement related to the BDEAC component suffered from management interference, causing delays in the implementation of the activities of this component. The Bank procurement team provided hands on advice to the beneficiaries, and project procurement was always rated as Satisfactory during the whole project life. c. Unintended impacts (Positive or Negative) The ICR did not find any. d. Other The ICR did not find any. Page 13 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CEMAC Regional Institutions Support (P099833) 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Relevance of objectives is high, and that of design is modest. Efficacy is rated Moderately Outcome Unsatisfactory modest for both sub- Unsatisfactory objectives, because of M&E weaknesses, and effciency is modest. Risk to Development Modest Modest --- Outcome Quality at entry and Quality of Supervision is respectively U and MU. Per the Harmonized Moderately Bank Performance Unsatisfactory Criteria agreed by IEG and Unsatisfactory OPCS, the Overall Bank Performance rating is Unsatisfactory. Ratings of Government and Implementing Agency Performance are respectively Moderately Moderately MU and MS. Per the Borrower Performance Satisfactory Unsatisfactory Harmonized Criteria, the Overall Borrower Performance rating is Moderately Unsatisfactory. Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons IEG agrees with the lessons learned and presented in the ICR, which underscore the following main themes: (i) the importance of a good theory of change and a results framework to track and report progress, (ii) the central role of adequate implementation arrangements, (iii) the need for simplicity in the project design and the rigorous selection of implementable activities, (iv) the irreplaceable role of Bank's internal coordination to ensure that every stakeholder in the Bank's Global practices provide the expertise and experience that the Bank can bring to bear for each operation, and (v) the implementation of regional projects needs special attention, because Page 14 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CEMAC Regional Institutions Support (P099833) broader geographical coverage present specific ownership and coordination challenges. An IEG lesson arising from project implementation is that shortcomings in project design lead to major consequences for a project's performance. When a project is not well designed, its success will almost always require a restructuring for any chance for the project to generate the optimal results. When the restructuring does not take place like in this case, project performance will likely be affected. 14. Assessment Recommended? No 15. Comments on Quality of ICR The ICR is comprehensive, but needed to be concise (33 pages against 15 pages recommended by OPCS). The report focused on describing implemented activities and achieved outputs, but assessment of outcomes was hampered by a weak result framework and M&E arrangements. However, there is congruence among the ICR and the Borrower's input that results were achieved toward the objectives, which is also partially illustrated by the Annex 2 summarizing completed outputs, but their assessment is difficult to strike because of weak benchmarking. The lessons and recommendations presented by the report are generally sound and realistic. Three aspects needed deeper analysis and better presentation: (i) the story related to fiduciary management and suspension of disbursement of project resources needed to be told with more clarity and precision, (ii) an explanation should have been provided to clarify the discrepancies among the numbers related to the costs and the financing of the project, and (iii) the assessment of the relevance of project design could have been centered around the theory of change to better illustrate the achievements and challenges for the project. a. Quality of ICR Rating Substantial Page 15 of 15