ICRR 13288 Report Number : ICRR13288 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 02/25/2010 PROJ ID : P082610 Appraisal Actual Project Name : Emergency Public US$M ): Project Costs (US$M): 10.9 11.97 Administration Project II Country : Afghanistan Loan/ US$M): Loan /Credit (US$M): 8.4 9.05 Sector Board : PS Cofinancing (US$M ): US$M): 2.5 2.92 Sector (s): General public administration sector (100%) Theme (s): Administrative and civil service reform (34% - P) Macroeconomic management (33% - P) Other public sector governance (33% - P) L/C Number : CH055 Board Approval Date : 06/24/2003 Partners involved : Dfid (cofinancing); Closing Date : 03/31/2006 09/30/2008 EC, US Government, IMF (parallel financing) Evaluator : Panel Reviewer : Group Manager : Group : Clay Wescott Fareed M. A. Hassan Ismail Arslan IEGCR 2. Project Objectives and Components: a. Objectives: The objective is to assist the authorities to enhance the accountability and transparency in the management of public resources and implement its National Development Budget . Specifically, this includes continued emphasis on the objective of the first Emergency Public Administration Project (EPAP I) - putting in place, on an emergency basis, capacity in key public administration areas to facilitate better use of public resources, including donor funds, for urgent reconstruction and development efforts . A further emphasis of this project is to begin to lay the foundation for more broad -based civil service reform directed at increasing the efficiency with which basic services are delivered to the population . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): The project had 3 components: omponent 1: Supplemental Support for EPAP contracts (appraisal cost US$ 4.3 million; actual cost US$ 4.8 Component million ). Financing necessary to complete the originally envisioned work program for the procurement and financial management agents, resulting from Afghanistan ’s difficult security and logistical post -conflict environment. Component 2: Civil Service and Administrative Reform . Co- Co -financed with DfID, there were three sub- sub -components providing technical assistance in various areas (appraisal cost US$ 2.6 million; actual cost US$ 4.25 million ) . First, to assist the Administrative Reform and Civil Service Commission (ARCSC) develop its US$4 capacity in: (i) appointments and appeals; (ii) civil service management; and (iii) management of the public administrative reform program. There was also support to ARCSC to provide support to line ministries for costing proposals when they seek priority reform and restructuring (PRR) status. Secondly, to assist ministries that have approved restructuring plans under the PRR decree to implement those plans; and to provide gap financing for critical activities within the Public Administration Reform and Economic Management (PAREM) development budget that are not funded by other donors . Thirdly, to carry out a six-monthly review of activities under the first two sub-components, with an aim to assess progress and make recommendations to enhance the effectiveness o f the work planning and programming . Component 3: Studies and Institutional Strengthening (appraisal cost US$ 1.5 million; actual cost US$ 0.0 million ). The intention was to provide the Transitional Islamic State of Afghanistan (TISA) with flexibility to commission studies that could generate quick impact or which deal with key policy issues . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: There were three extensions, presumably due to the difficulties of working in the security and logistical postconflict environment. The ICR points out a change in the procurement component that may have contributed to delays: the Afghanistan Assistance Coordination Authority was disbanded in 2003; the procurement function was eventually transferred to the Afghanistan Reconstruction and Development Services (ARDS). The DfID trust fund was eventually larger in US$ terms than envisioned due to the strength of the British Pound, allowing for some extension of project activities beyond that contemplated at appraisal . The closing date of March 31, 2006 was extended three times, with a final closing of September 30, 2008. The reason was reported difficulties in contracting suitably qualified firms to work on civil service reform, along with the availability of unanticipated DFID trust funds due to the strength of the British Pound . 3. Relevance of Objectives & Design: The objectives of EPAP II were relevant . They are consistent with one of the key Transitional Support Strategy (TSS) focus areas, supporting the continued development of a well -functioning state that is accountable to its citizenry and able to ensure equitable, efficient and effective service delivery, which carries over to the Interim Strategy Note (FY09-11). They are also consistent with the Government ’s National Development Strategy, which calls for encouraging public institutions to perform effectively with a focus on results, through pay and grading restructuring, ensuring and expanding merit based appointments, rationalizing and restructuring government institutions to reflect core functions and responsibilities, and increasing training and capacity development . The strategy is to identify pragmatic opportunities for action in the short term which will be supportive of longer term improvements. This approach was appropriate, given the great uncertainties of the country context . Because of the emergency nature of the operation, the project was approved before formal AAA was completed . The design was appropriately based on a rapid appraisal of the political and administrative context, informed by a watching brief the Bank had been keeping during the Taliban era . The PDOs were closely aligned with EPAP I, the Programmatic support for Institution Building Series (3 operations), the Public Administration Capacity Building Project, and the Public Financial Management Reform project, and the IMF ’s PRGF, which were all ongoing during EPAP II. In addition, civil service capacity building was supported by UNDP, ADB, EC, and UNAMA, in addition to the DFID cofinancing to EPAP II . The Bank’s leadership of the Afghanistan Reconstruction Trust Fund put it in a strong position to engage in policy dialog with the Government, and to coordinate the work of other international partners. However, the ICR notes that because of the emergency nature of the assistance, and delays in the procurement of TA contracts for the civil service component, broader institutional development and capacity building had not proceeded as quickly as expected in the design (although lack of project implementation capacity was indicated as a risk ). Also unanticipated was: 1) undermining of the process of selecting civil servants by pressure by ministries on, and budgetary problems faced by, the Appointments Board; 2) HR procedures developed by ministries different from procedures and standards of the Civil Service Management Department, leading to inconsistent implementation across the Civil Service . Although three outcome indicators are provided in the PAD, there is no results framework, and incomplete reporting on results in ISRs. 4. Achievement of Objectives (Efficacy): Based on a review of the achievement of the three performance indicators stated in the PAD, the review finds that the project achieved substantial efficacy in assisting the authorities to enhance the accountability and transparency in the management of public resources and implement its National Development Budget, taking into account the severe constraints of working in the Afghan context . (a) The delivery of goods, services and works in line with the agreed procurement plan in a transparent manner (Substantial ). The Public Procurement Law was approved in 2005 and procurement regulations in compliance with the law adopted, replacing the Procurement Regulations of 1987 and Purchase Regulation of 2001. As of March 2006, the cost of EPAP II procurement services was less than one percent of the total value of contracts completed or underway using Bank or internationally accepted standards . Despite progress, challenges include : 1) Pervasive bribery as matter of course in public procurement; 2) weak competition because of unwillingness of international firms to bid for work in Afghanistan; 3) lack of capacity and risk to longer term sustainability; despite EPAP training, many staff trained were not subsequently involved in actual procurements by their ministry; it is not possible to determine the actual results of the training provided (see section 10); 4) World Bank procurement rules adopted don’t make provisions to encourage Afghan businesses to participate in contracts, thus preventing the buildup of technical capacity in the country; 5) some ministries claimed they were unaware of how to access EPAP II procurement services; and 6) although provincial administrations are constitutionally part of the central government, many suffer from lack of financial transfers from Kabul, making them dependent on funds from local commanders, and weakening capacity vis a vis Taliban shadow governments active in some provinces . (b) Accurate accounting and timely reporting on budget execution, including detailed reports submitted to ( Substantial ). donors (to the extent that their resources flow through the budget ) (Substantial Achievements included: 1) adoption of a computerized Treasury payment and accounting system, the Afghanistan Financial Management Information System (AFMIS) enabling tighter control of budget execution against budget allotments along with recording of revenues, operational in all line ministries and in 12 provinces; 2) implementation of a verified payroll covering salary payments to 24,000 staff in 17 ministries in Kabul (out of a total 130,000 civil servants being paid nationwide excluding teachers, police and armed forces ), and pilot payroll databases in two ministries; 3) establishment of a Grant Management Unit in the MOF to meet donor requirements for more detailed information than is available from AFMIS; 4) adoption of a Public Finance and Expenditure Management Law leading to availability of increasing number of budget reports, and wide adherence to and understanding of budget procedures throughout government; 5) adoption of Treasury Single Account; 6) improvements in the central bank ’s payment systems; 7) an updated chart of accounts in line with international standards; 8) reconciliation of bank and advance accounts is carried out on a quarterly and timely basis, and a consolidated statement of financial accounts prepared annually for timely approval by the Cabinet, starting in 2004; and 9) associated training. These achievements were facilitated by the 2004 Constitution which provides a general legal framework for public finance management . Although the Bank’s emergency support was initiated before formal analytical work could be completed, there has been ample work carried out since then to help guide and monitor the impact of Bank support, including two PEFA assessments, and a 2006 Public Finance Management Review, combining a Public Expenditure Review, a Country Financial Accountability Assessment, and a Country Procurement Assessment Review . Based on the 2008 PEFA assessment, the country’s ratings are better than the average for other low income countries .However, many challenges remain, including: 1) many of these improvements depend on continuing provision of international technical assistance (estimated to be 70 international advisors and 250 local advisors, compared with 1400 MOF staff in Kabul and 5000 in total); the results of training provided cannot be determined; 2) the domestic revenue captured by the formal system amounts to only 7 percent of GDP, which is low even for low income, post -conflict countries; 3) there remains a substantial gap between the planned and executed budgets, and a failure to integrate operating and development budgets called for as part of the program budgeting reform; 4) there has been little progress outside of the MOF in setting up internal audit to a professional standard, nor in follow -up of audit recommendations; 5) the country has a high risk of distressed debt, although a Highly -Indebted Poor Country process begun in July 2007 is expected to improve the situation; 6) an estimated 52 percent of the country’s GDP derives from the external (donor funded) budget, over which the government has little control since it is largely disbursed directly by donors outside the Treasury system; 7) large sections of the economy including much of the value of land transactions and drug trafficking (itself estimated at 1/3 of formal GDP) escape the formal PFM system; and 8) it is impossible to tell how many officials obtain multiple salaries from different ministries and departments. (c) The Administrative Reform and Civil Service Commission gaining capacity to carry out its core functions (modest ). Achievements detailed in the ICR’s Table 1 include: 1) Structure of Government – number of ministries reduced from 31 to 26; Cabinet Secretariat and Parliamentary Secretariat established and functioning; 2) Legal and regulatory framework – Civil Service Law approved 2005, and key regulations drafted; 3) Ministry and agency restructuring – Priority Reform and Restructuring program approves 36,700 positions; pilot restructuring of 3 provincial and one district Governor ’s offices; 4) Human resources management – new eight grade structure designed; 1429 Grade Two and above (and 7260 Grade Three and below) merit based appointments processed by ministries and agencies and monitored by Appointments Board; 279 appeals evaluated by Appeals Board; 5) Capacity Development – 193 persons appointed through expatriate and lateral entry programs; new Civil Service Training Institute establishment, and civil servants training based on training and development strategy; and 6) Reform oversight – Cabinet Sub-Committee, Public Administrative Reform Steering Committee, and Independent Administrative Reform and Civil Service Commission established . The revamped public administration can point to achievements such as the return of millions of children to school, the repatriation of 2.5 million refugees and resettlement of 600,000 internally displaced peoples, the introduction of a new currency and stable exchange rate, and the doubling of GDP per capita since 2002. Key challenges include: 1) the direct financing by donors consultants in key positions has had a limited impact on ministry capacity; as in the other components, the results of training provided can't be determined; 2) inability of government to recruit and retain qualified staff (being addressed by complex pay and grading reform underway ); 3) weak leadership of the reform effort over the last few years; 4) reforms have yet to take hold within key agencies such as the Afghan National Police, where factional networks and drug alliances vie for posts, particularly remunerative ones covering oversight of smuggling routes; and 5) reforms haven’t yet reached the sub-national level, where most government services should be delivered; an Independent Directorate of Local Governance was established in 2007, but systemic weaknesses and unclear service delivery responsibilities among government agencies and donors has impeded progress; however, a positive step forward is the National Solidarity Program, which is providing block grants for nearly 8,000 towns and villages to address community defined needs, complementing the work of other government programs supporting education and health . 5. Efficiency (not applicable to DPLs): No ERR/FRR was estimated in PAD or ICR. Given the security and other challenges, it would be hard to evaluate the efficiency of the consulting services used . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The overall outcome of the project is moderately satisfactory . The objectives and design were relevant for the country context and Bank strategy both at the time of appraisal, and evaluation, although it didn ’t anticipate delays in the Civil Service component due to procurement delays . It also didn't anticipate pressure on the Appointments Board by ministries to select candidates on other than a merit basis, and inconsistent implementation of HR procedures by ministries . The components are relevant to the objective . The approach of identifying pragmatic opportunities for action in the short term was appropriate . However, a moderate shortcoming is that it isn't possible to determine the results of the training provided under the project . In addition, there was no results framework, although three performance indicators are stated in the PAD . There was progress through the provision of emergency capacity towards meeting the overall objective, but less, measurable progress in fostering broader institutional development . There are major remaining challenges, but this is not surprising given the context . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: Outcomes face high risk for the reasons stated in the ICR, and for the following additional reasons : a) the ongoing civil war has recently worsened, and the current regime relies on foreign military forces to retain power; countries providing these forces have indicated their patience is running out, and the US recently announced a deadline of 2011 for beginning to withdraw its forces; b ) many subnational jurisdictions have dual administrations, with a shadow Taliban government reportedly more decisive and less corrupt than the formal government; and c) as foreign forces leave, the regime would likely find it necessary to roll back administrative improvements achieved to date, in order to make greater use of public resources to pay off elite supporters willing to oppose Taliban, bypassing administrative controls put in place to ensure that resources are used to provide public goods to the broader citizenry . In addition, civil service jobs would likely be used to a greater extent by elites to reward clients for loyal political support . These risks are likely to materialize, and would have a high impact on the operation’s development outcome. a. Risk to Development Outcome Rating : High 8. Assessment of Bank Performance: Quality at project entry was moderately satisfactory . Design of PDOs and components was broadly satisfactory, based on a rapid appraisal of the political and administrative context . The design took into account legacy systems, such as vote books used throughout the 1980s and 1990s. The asymmetric approach adopted was vital in getting things moving at the start; initial, modest steps provided the groundwork for more ambitious reforms later on, such as pay and grading reform . The lack of a results framework is a shortcoming. Quality of project supervision was moderately satisfactory . The Bank earned leadership among donors in public administration reform by the excellence of specialists on the ground, and quality of their work . A key role was also played by former Bank staff, who played a useful bridging rule with their links both to donors and to rival political factions. Implementation Status and Results Reports were perfunctory . Although a 2004 Quality of Supervision Assessment carried out by a QAG panel rated the overall quality of supervision as satisfactory, this Review agrees with the ICR that there were subsequent shortcomings, including the four changes in TTL, and inadequate reporting of the civil service component, and of results achieved from training. at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: Government performance is considered to be moderately satisfactory . There was substantial commitment at the design stage from the Minister of Finance . Government support continued through implementation in carrying out monitoring, knowledge transfer, revising regulations, and restructuring to align organizational capabilities with reform priorities. In some cases, promising government proposals that could have addressed some of the challenges discussed above were not accepted by international partners . For example, the Ministry of Finance proposed that a biometric national database be set up in organizing the 2004 and 2005 elections, but international partners decided that a system based on paper cards and verification by ink be used instead . Shortcomings included weak financial reporting, and late submission of annual audited financial statements . There were also coordination challenges since no agency was given overall responsibility for the project . Implementing agency performance was also moderately satisfactory . Project reporting by the Ministry of Finance improved once the AFMIS was operational . Commitment to the civil service component was strong when the Head of Civil Service, and Minister of Finance, were former Bank staff . When they left, their successors had less commitment, and the component languished . ARDS recruited adequate staff after some delay, and produced timely reports of good quality after standardized reporting formats were introduced . IARCSC worked well with the DFID specialist, but shortcomings included weak capacity for project management, monitoring and evaluation leading, inter alia, to serious procurement delays . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: Implementation Status and Results Reports were perfunctory . They could have been more informative with a more extensive discussion of key issues and actions for management attention, key pending actions, and by reporting progress on all key performance indicators provided in the PAD, rather than just one of them . Although there is anecdotal evidence that local capacity was developed, particularly among highly motivated contract staff, it is difficult to systematically measure the results of the training provided . For example, the ICR indicates that 150 procurement staff were trained and that trainees received certificates of completion, but that many weren't involved in actual procurements . There is no indication of whether trainees were tested for the knowledge and skills that they gained . It also isn't clear that the trained staff changed their workplace behavior as a result of the training, and thus could have contributed to improved procurement . ICR also states "140 senior civil servants trained in senior leadership and 1530 civil servants trained in managerial and common competency training.", but doesn't say what testing was done, whether trainees changed their behavior and worked smarter to deliver key tasks, and how many were retained in the civil service . For training of local vendors, significant progress is reported in bid preparation in the works category, presumably linked to the training. Better monitoring of other training results would have helped to evaluate training outcomes . It would also have been useful to see how a better understanding of the context through formal AAA produced as the project was underway was used to adjust project implementation . Also, the lack of any ISRs for the last two years of the project is a shortcoming . Weaknesses in supervisory reporting were also highlighted in a 2004 Quality of Supervision Assessment carried out by a QAG panel . There is no results framework in the PAD, although three outcome indicators are provided . A results framework with outcome indicator status first appeared in the ISR of October 31, 2005, but the presentation and status reporting is incomplete relative to the PAD. a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately The lack of measurable results from Satisfactory training provided by the project was a moderate shortcoming. Risk to Development High High Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: The ICR’s “lessons learned� are all supported by this review, including the need for simple and realistic objectives, separating support to critical operations from capacity building to ensure that both are performed, taking a long-term approach, closely monitoring consultants, carrying out priority actions within a comprehensive framework, mobilizing technical assistance in a timely manner, ensuring strong leadership from political leadership, and asking agencies like IARCSC to guide and oversee ministerial reforms rather than try to implement them. In fragile state settings like Afghanistan, it will often be expedient to hire relatively qualified (and loyal) contract staff who can meet expectations to have essential systems perform at a high standard, rather than to fill vacancies at civil service salaries . Capacity building interventions should be designed with a realistic, planned mix of contractual and regular civil service staff, balancing the need for immediate performance of essential tasks at a high standard with the goal of broader institutional development and sustainability . In addition, there should be heightened effort to monitor the results of capacity building support, including testing for the knowledge and skills gained, changes in workplace behavior, and improved task performance . 14. Assessment Recommended? Yes No Why? This is part of a program of three operations to improve public administration capacity that need to be assessed together. 15. Comments on Quality of ICR: The ICR is of satisfactory quality . It could have been improved with greater analysis of the tradeoffs between doing emergency work and building capacity to do the work . The EPAP I ICR made useful suggestions for improving the latter, and it isn't clear why the suggestions were not taken on board . Furthermore, the ICR rating of "substantial" for risk to development outcome is not in line with the OPCS -IEG Harmonized Evaluation Criteria. Also, there are small discrepancies in actual cost data . a.Quality of ICR Rating : Satisfactory