ICRR 13299 Report Number : ICRR13299 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 03/23/2010 PROJ ID : P084736 Appraisal Actual Project Name : Public Admin US$M ): Project Costs (US$M): 27.00 27.40 Capacity Building Project Country : Afghanistan Loan/ US$M): Loan /Credit (US$M): 27.00 27.40 Sector Board : PS US$M): Cofinancing (US$M ): Sector (s): Central government administration (80%) Law and justice (15%) Sub-national government administration (5%) Theme (s): Public expenditure financial management and procurement (67% - P) Administrative and civil service reform (33% - S) L/C Number : CH144 Board Approval Date : 01/27/2005 Partners involved : Closing Date : 06/30/2009 06/30/2009 Evaluator : Panel Reviewer : Group Manager : Group : Marcelo J. Bueno Gita Gopal Ismail Arslan IEGCR 2. Project Objectives and Components: a. Objectives: The original Project Development Objective (PDO) was to assist the authorities to develop the capacity to manage public expenditures in an effective, transparent, and accountable manner in order to achieve satisfactory and sustainable development outcomes in Afghanistan, with particular focus on enhancing the performance of public sector procurement, financial management, and accountability systems . b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 02/27/2007 c. Components (or Key Conditions in the case of DPLs, as appropriate): Component 1: Strengthening Procurement Capacity within the government (appraisal cost US$9.0 million; actual cost US$8.45 million). This component was to provide technical advisory services, equipment, and training to the Government of Afghanistan (GOA), specifically to the Afghanistan Reconstruction and Development Services (ARDS) Procurement Unit to: (i) support line ministries with direct procurement processes, particularly in relation to donor programs; (ii) undertake a procurement capacity building program within line ministries for them to be accredited with increased delegated authority for procurement management; (iii) support the promulgation and implementation of the new Procurement Law and regulations; and (iv) help establish a Procurement Policy Unit (PPU) within the GOA. . Component 2: Strengthening Financial Management Capacity (appraisal cost US$8.0 million; actual cost US$13.62 million). Building on the assistance provided to the Ministry of Finance (MOF)/Treasury under the Bank's First and Second Emergency Public Administration Projects (EPAP I and EPAP II), this component was to provide technical advisory support, equipment, and managerial and professional development training to achieve core (MOF) and line ministries' financial management objectives of high efficiency in budget execution and fiscal reporting, including the financing of a comprehensive Afghanistan Financial Management Information System (AFMIS) architecture study for the GOA to implement over the medium -long term horizon. Component 3: Strengthening the Internal Audit Capacity (appraisal cost US$4.00 million; actual cost US$1.87 million). This component was to provide technical advisory services, training, and equipment to support the internal audit function within the GOA along generally recognized international standards of internal auditing and control in order to improve public sector accountability and governance, establish sound financial management within the public sector, and effectively execute the government budget . To achieve these objectives, support was to be provided for establishing an audit committee for the oversight of the GOA's internal auditing service, creating an organizational structure to remove line ministry influence over the work of internal audit, and facilitating the transition to stronger internal audit working practices on operational reviews and evaluation of internal control systems. Component 4: Strengthening the External Audit Capacity (appraisal cost US$4.50 million; actual cost US$2.48). This component was to provide technical advisory services, equipment, and training to support and strengthen the capacity of the Control and Audit Office (CAO) in providing comprehensive external audit services along the auditing standards of the International Organization of Supreme Audit Institutions (IOSAI), as well as to strengthen professional accounting and audit standards throughout the country . Support was for the (i) undertaking of core audit operations to acceptable international standards and providing on -the-job training for CAO staff; (ii) strengthening the audit environment and capacity of the CAO to conduct audits consistent with internationally accepted standards; (iii) managing a managerial and professional development training program for CAO staff; (iv) development of professional accounting and auditing capacity in both the public and private sectors; and (v) creation of a scrutiny and response system to audit reports emanating from involved line ministries and the Parliament's Public Accounts Committee . Component 5: Other Institutional Strengthening and Capacity Building (appraisal cost US$1.50 million; actual cost US$0.98 million). This component was to provide additional support for the administrative reforms implemented through the Independent Administrative Reform and Civil Service Commission (IARCSC) under EPAP II. Support included ministry rationalization by the GOA, government -wide pay and grading review and implementation, management of the Public Administration Reform (PAR) program and donor-funded TA, and the transformation of the MOF into a strategically -focused, service-oriented, professionally-staffed, and best-practice institutional leader for the government sector . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The project was approved on January 27, 2005 for US$27 million (grant) and closed on schedule on June 30, 2009. The difference between the appraisal and actual amounts is due to exchange rate changes . At restructuring in February 2007, the PDO was modified to read: "To enhance the performance of public sector procurement, financial management, and accountability systems which will contribute to a more effective, transparent and accountable state ." There were two reasons for restructuring . First, it was recognized that capacity building would take much more time than originally envisaged. Second, training activities were delayed for a number of reasons, including lagging recruitment of trainable staff and difficulties in finding qualified trainers in view of the security situation . As a result expenditures under components 3 and 4 were reduced and funds reallocated to component 2. 3. Relevance of Objectives & Design: Relevance of Objectives . Public sector capacity building was the goal of Afghanistan's National Development Framework (NDF) that strongly undergirded the original PDO . The NDF was designed to provide a comprehensive development strategy and a strong agenda to rebuild institutions and implement structural reforms within a stable macroeconomic and investment framework . With the goal to achieve the government's Millennium Development Goals by 2015, and to transition from buying-in core public administration capacity to coordinate donor resources and sustain key functions within the GOA, the PDO was highly relevant . It was consistent with the reform of the country's public administration sector under the Bank's transitional support strategy and IDA's support under EPAP I & II of a strategy to assist the GOA to demonstrate its ability to manage donor resources in an effective and transparent manner along rules -based governance and effective capacity building, while addressing key constraints in areas as procurement, treasury operations, audit capacity, and financial management. The revision of the PDO at restructuring provided no major strategic change in the overall relevance of the PDO and the monitoring indicators associated with the same remained basically aligned with the planned activities under the project . Relevance of Design . Project design took into account lessons learned from previous Bank -assisted projects in the public administration sector including findings from and progress made in EPAP I and EPAP II on standards of fiduciary control, buy-in capacity within the Government, public management reforms, and capacity building strategies that addressed legislative, regulatory and institutional reforms, and management control and human resource systems. Given the identified key constraints and weaknesses in government systems, appropriate mitigation measures were incorporated into the design of the project components . The timeframe to develop capacity for managing government systems, however, was underestimated and some major risks were not fully factored in. These risks included the fragile political and security environment, the weak recruitment of counterpart staff by the implementing agencies, and the lack of clarity over functional responsibilities in implementation, inhibiting progress and creating overlap and complications . 4. Achievement of Objectives (Efficacy): Given the country context and the enormous challenges faced by the GOA during project preparation and implementation, the efficacy of the project was substantial . The five project components, taken jointly, enhanced and strengthened the capacity and performance of public sector procurement, financial management, and accountability. Procurement capacity within the government was enhanced . Procurement activities in the ARDS had been legally institutionalized with the enactment of the Procurement Law, and are now consistent with international standards, while its management system has been certified by the International Organization for Standardization (ISO), the first ever in Afghanistan . Procurement training courses at different levels i .e. basic, intermediate, and advanced, have been completed for both the public and the private sectors, while a procurement MIS had been launched and is being improved further in e-reporting and monitoring by the PPU. Additionally, procurement functions in line ministries have been appropriately staffed to facilitate the effective handling of increased demand for procurement services as well as to strengthen their procurement control functions for reporting and transparency purposes. Several line ministries, initially in 4 provinces, are awaiting Priority Reform and Restructuring (PRR) status which will speed up the institutional development and procurement and financial management capacity of these ministries, further enhancing the government's procurement capacity . FM) systems and capacity have improved . The GOA's financial management (FM) While the majority of the Treasury's more sophisticated operations are still being managed by local and international consultants, new twinning arrangements have been implemented to ensure the passing of skills to the PRR staff, including the introduction of civil service recruitment and on -the-job training programs to address the insufficient numbers of PRR staff to take over the AFMIS, which had been rolled-out to the line ministries and Mustofiats (provincial officers of the Treasury Department ). Financial management transparency and comprehensiveness was also improved with the development of a classification system that tracks expenditures on organizational, economic, and functional programs, the promulgation of financial regulations, enhancements in the Chart of Accounts, and improvements in the quality of budgetary information on extra -budgetary funds and donor funding . The internal audit capacity and operations of the GOA had been strengthened with improvements in internal controls and sound financial management practices consistent with generally accepted international standards of internal audit, including a more effective and accountable way of executing fiscal policy . An Audit Committee (AC) was put in place to formulate policies and procedures, and provide regulatory oversight of internal audit activities while assuring full audit coverage of the entire financial and operational activities of the MOF. Program audits have also been implemented including audit testing to carry -out effective operational reviews and correct deficiencies of internal control systems, the provision of internal audit manuals and handbooks, and the creation of a fraud investigation unit within the MOF and an IT audit unit to facilitate the transition. Technical advice on legislative reforms provided the external audit capacity framework to enable full effective audit coverage of the GOA's financial system, allowing the CAO to shift from transaction -oriented compliance audits to performance audits, while raising the quality of CAO audits . Notwithstanding these achievements, however, several important actions still needed further attention, including the creation of a legislative Public Accounts Committee (PAC) to scrutinize and respond to audit reports from involved line ministries, and the adoption of the proposed Bank -supported Audit Law which is still under consideration . Training and capacity building of about 390 Mustofiats over the life of the project in areas of financial and human resource management, communications, budgeting, leadership, etc . had been completed, including the training of MOF middle-management staff in accounting, management, basic computer skills, and English. With added funding, the tasks of ministry rationalization and the implementation of government-wide pay and grading review were included in the Bank support program to provide the incentive to attract and hold unto qualified staff . In addition, support was provided to assist the IARCSC in managing the public administration reform program, and help the GOA in managing the design and delivery of the appropriate TA to other government ministries . Public sector governance and accountability were enhanced . Public sector governance and accountability were enhanced, overall, through the above results . The establishment of a regulatory framework consistent with international standards was a first important step in strengthening public procurement efficiency while establishing clear lines of accountability . With increased capacity, monitoring is likely to further strengthen the accountability of public expenditures . Enhancements of the internal and external audit systems and capacity of the GOA will further improve accountability, as will the strengthened CAO, the newly enhanced Chart of Accounts and financial regulations, the public expenditure tracking system, and improved fiscal and budget information . Public sector governance and accountability was also enhanced by institutionalizing the annual budget process with inputs from the political sector to ensure strategic prioritization and transparency in resource allocation, predictability and control in budget execution, accountability in banking recording and reporting, and the legal and regulatory assurances of a comprehensive annual external audit and public reporting of public finance expenditures . 5. Efficiency (not applicable to DPLs): The PACB was a capacity building and technical assistance project, and estimates of the net present value and economic/financial rate of return as a measure of efficiency would not be applicable in this case . The resources and benefits from the project nevertheless helped develop and enhanced the GOA's procurement capacity, financial management system, and accountability regimen . With the necessary regulatory frameworks strengthening the GOA's internal and external audit capacity, the MOF has been transformed as the public sector's lead agency, having attained acceptable standards in the conduct of public financial management while placing diminishing reliance on international TA for basic and routine audit matters . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal % % ICR estimate % % * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project was relevant, though with some shortcomings in the overambitious design, while achievements were substantial. With the legal and regulatory frameworks in place and under implementation for enhancing procurement, financial management, and accountability, public sector performance has clearly performed relatively satisfactorily including the processing and control of budget reporting and the management of public expenditures, consistent with standards set under EPAPs I & II . Satisfactory progress was also noted in the external and internal audit capacity and performance of IAD and CAO with the completion of the internal audits of key line ministries (e.g. MOF, Defense, Local Governments). At the same time, insufficient progress was made in some areas, including in developing an effective audit function and creating sustainable cadres of permanent staff. a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: The risks to sustaining the development outcome are significant for the following reasons: (i) with the MOF and most line ministries still lacking in capacity to effectively manage procurement issues, support of donors and guidance of international consultants will still be required for several years, notwithstanding the progress already achieved in capacity building of national staff and procurement liaison officers (PLOs); (ii) without the support of the project, there is a risk that the GOA may cease paying the current level of salaries to the PLOs, resulting in a high probability of many departing government service for better opportunities in the private sector; (iii) the lack of IT capacity and trained staff in accounting and treasury to operate the AFMIS poses a big problem for the GOA to deal with after the contractor leaves the project given the weak recruitment capacity in the local environment; (iv) the lack of independence of the CAO makes its role in the accountability and transparency framework less effective having no inputs and guidance on accountability issues from the less -than-pro-active National Assembly, thus the urgency to pass the proposed Bank -supported draft Audit Law which gives independence to the CAO; and (v) there is a risk that the Government might cease support for continuing the training programs and study tours in procurement, auditing and accounting, IT, and English . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: Despite some shortcomings in the supervision of the project, and the complexity of project design, Bank performance was satisfactory overall, in ensuring quality at entry while taking pro -active measures in project supervision and implementation . Quality at Entry . EPAP I and EPAP II, including analytical work, policy dialogue in administrative reform, and the Programmatic Support for Institutional Building adjustment operations provided the overall strategic framework for the project. Project preparation was carried out with an adequate complement of technical skill mix and expertise in addressing sector and country context issues . But the original project design was overambitious in areas such as capacity building and training . Quality of Supervision . Major efforts were made during project supervision to remedy shortcomings identified earlier as well as by a 2006 QAG panel, which noted deficiencies in communications between the Bank and the client, weak coordination of Bank and donor programs, and the need for a larger supervision budget to obtain more specialist input . While little was done towards better donor coordination, strengthened supervision included (i) aide-memoires alerting the government and implementing agency to problems with project execution along with remedial action; (ii) monitoring of fiduciary compliance; (iii) focus on performance ratings of Implementation Status Reports (ISRs) in terms of PDO achievements and implementation progress; (iv) disseminating rules and procedures to be applied concerning selection of consultants, accounting and audits; and (v) having the task team leader (TTL) on the ground in Kabul, given the weak implementation capacity of the government . at -Entry :Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Satisfactory 9. Assessment of Borrower Performance: The Borrower's performance was satisfactory . Government Performance . With strong champions in senior levels of government, the Borrower, in pacticular the MOF, was heavily involved in project preparation and worked very closely with the Bank project team in monitoring and supervising the project . Throughout the life cycle of the project, the GOA demonstrated and maintained full commitment to its objectives, while making every effort to comply with Bank covenants, despite the country context in Afghanistan and difficulties in finding and recruiting qualified staff. Execution of the project and active intervention by the government was satisfactory and successful including in developing M&E, coordinating knowledge transfer, and making the appropriate project course corrections when warranted. Implementing Agency Performance . The performance of the MOF's Project Management Committee (PMC) was satisfactory in monitoring progress of the various project components and ensuring coordination in financial management, payments and audit, procurement, financial reporting arrangements, and the M&E of implementation progress and results being carried out by three MOF departments i .e. (i) the Treasury in strengthening financial management; (ii) the ARDS in strengthening procurement; and (iii) the CAO in strengthening the external and internal audit functions of the MOF . a. Government Performance :Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Satisfactory 10. M&E Design, Implementation, & Utilization: While no rating was provided in the ICR on the project's M&E design, implementation, and utilization, this review rates the M&E as modest overall. . The M&E design was based on qualitative indicators of progress in legislative, policy, institutional and human resource development. There were no quantified indicators of progress, although specific indicators were agreed during implementation on capacity building objectives, even in the absence of baseline data . Implementation of M&E was facilitated by the creation of a Reform Implementation Management Unit, which gathered information on project implementation and prepared annual performance reports . These M&E findings were utilized to inform future decision making and, in particular, to assist in preparing a follow -up Public Financial Management Reform project. a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): The ICR reported that there were no significant deviations or waivers from the Bank's fiduciary policies and procedures during the implementation of the project . Unintended outcomes included the following . On the positive side, a very useful website was developed in ARDS, incorporating all business opportunities being handled by the ARDS as well as directly by the line ministries . Bidders are able to download complete bid documents (amendments, pre-bid and bid-opening minutes, etc.) directly from the website, including the registration of firms with ARDS. Local bidders are thus able to get more knowledge and experience in bidding and procurement. On the negative side, the Internal Audit (IA) department of the MOF, earlier intended to be the sole authority in carrying out internal audits of government agencies had been relegated by Presidential order to auditing only the financial activities of the MOF . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Satisfactory Risk to Development Significant Significant Outcome : Bank Performance : Satisfactory Satisfactory Borrower Performance : Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: The ICR contains useful lessons applicable to countries such as Afghanistan, among which the following stand out. - Capacity building is a slow process, requiring time and patience, as witnessed by bidders having difficulty submitting bids in proper format and audit staff requiring continued on the job training to be effective . - Sustainability is a major issue because, in the absence of permanent cadres of personnel, it is difficult to retain qualified staff. - Development of an effective audit function requires full cooperation and involvement of all levels of government. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: Overall, the quality of the ICR is satisfactory . It's assessment of the prevailing economic and political environment under which the project was designed, implemented, and supervised, is candid, and it presents considerable information on all important actions taken associated with the project objectives, design relevance, and the implementation and supervision of each component . The ICR could have been improved with a more in-depth discussion of the sustainability of achievements and Bank follow -up programs to help remedy the risks involved. a.Quality of ICR Rating : Satisfactory