ICRR 14059 Report Number : ICRR14059 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 06/20/2013 Country : Rwanda Project ID : P066386 Appraisal Actual Project Name : Rw-public Sector US$M ): Project Costs (US$M): 20.8 unknown Capacity Building Project L/C Number : C3955 Loan/ US$M): Loan /Credit (US$M): 20 19.4 Sector Board : Public Sector Cofinancing (US$M): US$M ): Governance Cofinanciers : Board Approval Date : 07/08/2004 Closing Date : 12/31/2009 12/31/2011 Sector (s): Central government administration (83%); Vocational training (7%); Law and justice (5%); Information technology (5%) Theme (s): Public expenditure; financial management and procurement (29% - P); Administrative and civil service reform (29% - P); Other rule of law (14% - S); Other public sector governance (14% - S); Decentralization (14% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Iradj A. Alikhani Clay Wescott Ismail Arslan IEGPS2 2. Project Objectives and Components: a. Objectives: The Project Appraisal Document (PAD, p.6) states the Project Development Objectives (PDO) as follows: "Public sector entities have the capacity for more efficient, effective, transparent and accountable performance in their redefined roles and functions and for achievement of their strategic objectives contributing to the implementation of the Poverty Reduction Strategy Paper (PRSP)." There is no substantial difference between this statement and that of the Development Credit Agreement (DCA). The project was restructured on January 13, 2010. As part of narrowing the project, the PDO was restated in the Project Paper (PP, pp.2 and 7) as follows: “Public sector institutions supported by the project achieve increased capacity for efficient and accountable use of public resources .� While broadly in line with the original PDO, the effectiveness and transparency sub -objectives are no longer stated explicitly and the reference to contributions to the PRSP is also removed. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 01/13/2010 c. Components: The original project was composed of five components (PAD pp.8-12): 1. Coordination of the Government’s Multi-Sector Capacity Building Program (MSCBP) -- Cost at appraisal US$4.9 million. Main activity involved support to the establishment and operations of the Human Resource Capacity Building Agency (HRCBA), which subsequently became the Human Resources and Institutional Development Agency (HIDA). 2. Strategic Human Resource Development -- Cost at appraisal US$4.3 million. The project funded technical assistance to: (a) develop human resources and attract and retain talent; and (b) build the capacity of local training institutions. 3. Cross-Cutting Public Sector Reforms -- Cost at appraisal US$5.3 million. The component aimed to enhance the functioning of the public sector through : (a) improving public financial management (PFM) and procurement systems; (b) reforming the public service; and (c) strengthening the national M&E system . 4. Enhancing Agency Performance -- Cost at appraisal US$4.5 million. This component supported ministries and other public agencies. 5. Information and Communication Technology (ICT) -- Cost at appraisal US$1.8 million. The aim was to equip the public sector ICT effectively . Revised components were as follows (pp. 22-23 and ICR pp. 6-7): 1. Public Financial Management -- Cost at restructuring US$6.61 million. This component would continue and deepen PFM and procurement reforms and related capacity development, originally under subcomponent 3(a). 2. National Monitoring and Evaluation -- Cost at restructuring US$0.69 million. This consisted of a continuation of the original subcomponent 3(c) above. 3. Civil Service Reform -- Cost at restructuring US$4.31 million. The objective, in line with that of original subcomponent 2(b), was to improve capacity of public sector institutions to manage and develop civil servants. 4. Public Sector Reform Coordination -- Cost at restructuring US$1.32 million. This component strengthened the implementation capacity of the Public Sector Capacity Building Secretariat (PSCBS), which replaced HIDA. This is in line with original component 1. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The original project cost of US$20.8 million was initially financed by an IDA credit in the amount of SDR 13.7 million (equivalent to US$20 million). The remaining US$800,000 was from Government counterpart funding . Since the actual counterpart funding received is unknown, actual total project cost is unknown . According to project documents, 50.2 percent of the original Credit had been disbursed by December 31, 2009, at the time of restructuring. The restructuring involved expanding and allocating resources to two of the original components and terminating two others. Final IDA disbursements under the project amounted to US$ 19.4 million. The amount of government counterpart funding disbursed is unknown . As of the end of April 2012 a balance of US$2.6 million remained undisbursed and was scheduled to be cancelled -- the difference between the initial and final dollar amount is due to changes in the exchange rate with the SDR in which the Credit was denominated. Consequently about 50% of disbursements took place prior to restructuring . The Credit was approved by the Board on July 8, 2004 and became effective on March 30, 2005. The Credit was extended once by two years and closed on December 31, 2011. This extension reduced the project scope and provided the time needed to complete important activities . The revised components allocation /cost does not include all expenditures prior to restructuring, which are partially reported in the ICR . This review is unable to report component cost at completion as the figures presented in the ICR are not credible -- when compared to disbursement figures, ICR Annex 1(a) significantly understates costs prior to restructuring and overstates them post-restructuring. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Substantial pre - and post -restructuring – The project addressed an area that was highlighted as a priority in the last two Bank Country Assistance Strategies (FY03-08 and FY09-12). Both the original and revised objectives are relevant at present, as they were throughout project implementation . Specifically, capacity building is seen as a key risk mitigation measure in the current CAS (para. x, p. ii) and this area is identified as a cross -cutting filter for Bank interventions (paras. 75 and 78). The project was similarly consistent with Government strategy both at the time it was prepared and at present, as reflected by its alignment with the 2002 MSCBP and the strategic capacity building initiative launched recently (ICR pp.1 and 14). b. Relevance of Design: Modest pre -restructuring – Even though the objectives were relevant, there were significant shortcomings in initial design, over-ambitious timing and excessive complexity . More specifically, the project was motivated by broad objectives with partial connection with the results matrix . Project components contributed to project objectives, but activities were too dispersed for causal relationships to be established. Many of the outcome indicators were affected by factors outside the purview of the project . The link between objectives, outcome indicators and intermediate outcome is thus imperfect and unclear; for example: Many factors outside the project may affect how beneficiaries perceive improvements in Government operations, especially as the project focused on some Ministries but not all . Furthermore, it is unclear what intermediate result/component contributes to this outcome and how . Improvement in work environment can only be measured for those who receive training and /or equipment. A qualitative feedback may also be negatively affected by other reforms supported by the project . For instance, some civil servants in line ministries may perceive decentralization of their responsibilities as deterioration in work environment. Shortcoming in revenues and/or late budgetary releases (say due to delayed budget support ) can explain undershooting. The proportion of public expenditures decentralized suffers from a number of issues . In part, resources may not follow decentralization of functions, unless inter -governmental flows are also reformed -- say through mechanisms such as block grants to decentralized entities . Also, the problems affecting the previous indicator may also affect this one . Finally, Rwanda's capital expenditures have been growing rapidly and this may result in decreased relative allocation to decentralized functions, while respecting the spirit and letter of the policy objective . Substantial post -restructuring - Project restructuring was partly motivated by the need to address the above issues. It led to a much improved design, including narrowing of scope of activities and an implementation agency (PSCBS) focused largely on project activities . The restructured project’s results might have benefitted from being more closely linked to PEFA indicators, as well as tackling underlying challenges, especially dealing with institutional and human resource instability . 4. Achievement of Objectives (Efficacy): The original project objectives were to make public services more efficient, effective , transparent and accountable . The revised project aimed to increase capacity for efficient and accountable use of public resources – thus retaining the efficiency and accountability objectives and shifting the focus away from service delivery (an outcome) to how public resources are used (an output). The present review attempts to briefly reconstruct results prior to restructuring by examining progress largely based on original project indicators . Outcome indicators defined in project documents are mostly associated with one or more sub -objectives stated in the PAD and/or PP. However, neither of these documents nor the ICR attempt to explicitly link the intermediate indicators to project objectives -- the results listed correspond to components . The review here attempts such a mapping based on information available in project documents . However, some of this task is hindered by certain results contributing to a multitude of sub -objectives, while others are not easily associated with a particular development sub -goal or readily attributable to the project . For this reason and to allow for a clearer presentation, some aspects of the project that contribute to the overall PDO are highlighted under the heading “cross-cutting objectives.� Another practical difficulty encountered is that some of the actions initiated during the first half of the project are associated with output /outcome achieved two to four years later (some after the ICR was issued). Limited information on other complementary activities that led to these results makes attribution difficult (and reconstructing these is beyond the scope of the present review ). Pre -restructuring Achievement of Objectives Pre- 1. Effectiveness Modest . According to the results matrix in the PAD (Annex 3), the effectiveness sub-objective was to be measured by the percentage of surveyed government employees who confirm that the work environment is more enabling and they are better facilitated to work . The ICR (table on page 16) does not provide a baseline but cites a 2008 survey and an 89% figure for this indicator. The source of this survey is not cited anywhere else in the ICR, and its quality, coverage and reliability is not explained either – the December 2009 ISR characterizes this survey as “questionable�. Furthermore, in the absence of a baseline, the indicator as reported is of limited use. Its attribution to the project is uncertain given that it may be influenced by factors external to the project . Three intermediate outcomes (related to facilitation of employees ) might have shed more light on progress towards reaching this objective (component 3: percent of civil servants trained and percent of senior public servants attending in service training; and component 5: number of personnel trained in computer applications ). The ICR does not report on these indicators, but provides the following information on output (pp. 21-23): Training activities for various levels of public servants : change management for 22 staff; English for 200 public servants; payroll systems 282 human Resource Directors; and executive leadership for 78 senior staff. Provision of a ‘public sector coach’ for a period of one year, with limited results . Training of parliamentarians and their staff . Training of trainers in procurement management to 75 people, 72 of whom graduated, and 11 short courses on management to 153 people. A number of training activities could not be completed, including study tours on public service reform, and specific training. In the absence of targets, the above information is hard to assess, but realization appear to have been below targets set in the PAD and insufficient to lead to a significant improvement in effectiveness, especially once staff turnover is taken into account . There is no evidence provided on how the training made public service more effective, and whether beneficiaries of training remained in the civil service and used training received in the course of fulfilling their duties. The ICR also highlights the fact that institutional instability, which is directly linked to the work environment, was an issue throughout the project . The sub-objective has also been supported since 2008 by the Belgian Development Agency, through the "Support to Capacity Development in Rwanda Project, " which would share attribution of any results achieved . The project also supported an integrated personnel and payroll information, which is now fully operational and contributed to effectiveness, as well as efficiency . Progress by 2010 is reflected in PEFA rating on payroll control. 2. Transparency Modest . The information provided in the ICR and project documents on realization of this objective does not indicate significant progress in this area at the time of restructuring . The PAD indicator for transparency was : percentage of surveyed beneficiaries of central government entities who perceive improvements in transparency and accountability in Government operations – the indicator thus also covered the accountability objective . The ICR cites a figure of 47% for 2007 – the data has the same source and suffers from measurement problems already noted for the previous objective . The only intermediate outcome indicator related to transparency measures the number of Ministries, Departments and Agencies (MDAs) with websites. Project documents and recent information indicate some progress in this area, possibly linked to continued dialogue after the project was restructured and since closing . Specifically a new policy on communication and information initially developed under the project is now being rolled -out in public institutions following approval by Cabinet in 2012. The impact of this output cannot be measured at this stage . 3. Efficiency Modest . A measure of financial efficiency relates to procurement, where the December 2009 ISR notes limited progress (9% of audited institutions complied with procurement rules ). While this result reflects improved transparency, it is quite modest, even taking into account the low base the country started from . The project (ICR p.3) also contributed to enactment of the new procurement law and the production of related guidelines and rules, training in procurement and the new organic budget law and the implementation of comprehensive public procurement reforms. Another measure would correspond to improved institutional efficiency, including human resources. Evidence of progress in this area by end -2008 includes: (a) MDA’s annual plans – 7 ministries; and (b) ministries undergoing decentralization – all 25 ministries. However, no information is provided in the ICR on outcome of plans or decentralization status . Data from other sources indicates that transfer from the central government to districts increased from 1.4% of domestic revenues in 2002 to 33% in 2012. However, the relative attribution to the project vis a vis to that of others (such as the decentralization project http://www-wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2004/05/24/000160016_20040524 104950/Rendered/PDF/27632.pdf) is unclear. 4. Accountability Substantial . The shortcomings of the PAD indicator for transparency, which also covered accountability, has been explained above. The ICR does not provide any assessment of this objective, and mentions that the project was associated with the implementation of the Public Financial Management (PFM) strategy and subsequently the Public Financial Accountability Assessment (PEFA). The ICR states that the project “supported� implementation of the PFM plan and gives some examples : (a) these include the Organic Budget Law in 2006; (b) Financial Regulations in 2007; (c) Financial Policies and Procedures; (c) manuals for the Government in 2007; and (d) contributions to the development of the PFM strategy in 2008. While the project contributed to these objectives so did donor projects supporting the same areas . Two intermediate indicators would have helped provide indication of progress prior to restructuring . The first, which was maintained in the restructured project, relates to audits of MDAs, where coverage improved (between 60% and 70%; ICR p.19) but most, 97% according to the December 2009 ISR, were qualified. The last indicator relates to availability of MDA ’ s websites to facilitate access to information, which as noted above also contributed to improved transparency . Overall, while accountability improved thanks to better information, much remained to be done at restructuring and qualitative improvements were limited – e.g., high percentage of qualified audits . 5. Cross-cutting objectives Negligible . The project supported actions that contributed to most /all the above sub-objectives and are thus best reviewed is a consolidated manner . The centerpiece of the original project consisted of the establishment of the HRCBA (renamed HIDA), which was also responsible for project implementation . This institution was dissolved in 2008 and its main responsibilities in areas covered by the project were transferred to the PSCBS – there is no information on the extent to which capacity built in HIDA and other agencies that were affected by institutional changes was retained . As noted in the ICR (p.11) “numerous changes in implementing agencies and coordinating agencies, as well as the changes in staff of these agencies also impacted negatively upon implementation as it resulted in a loss of institutional memory and capacity �. Another important goal was to build capacity in the public sector, fill vacant staff and ultimately reduce staff turnover . As noted in the Government comments (ICR p. 47) staff turnover remains a continued challenge and at the time of restructuring (December 2009 ISR) 40% of positions were vacant or filled by foreigners – progress under the project appears minimal. Therefore, even though the project had funded capacity building and studies, setting the stage for subsequent capacity building, by end -2009 progress towards achieving cross -cutting objectives was limited. In line with the above summary, the ICR (p.3) recognizes that implementation was slow prior to project restructuring, which took place at the original closing date . Post- Post -restructuring Achievement of Objectives 1. Efficiency Modest . The main measure for efficiency under the restructured project was related to improvements in procurement and human resources . The project trained government procurement officers, and supported the establishment of a professional institute for procurement professionals and the Rwanda Public Procurement Authority. Improvements in procurement are reflected in the fact that by project end 81% of procurement was tendered competitively or justified, compared to a target of 85% – ICR p.iii. Neither the PP nor the ICR define the notion of "justified", which presumably refers to non -competitive, and possibly less efficient tenders subject to special procedures permitted under the procurement code – if so it would have been important to report the percentage of procurement following this approach . The other issue is that it is unclear whether the percentage is weighted by procurement value – presumably this is the case . Furthermore, the ICR (p.23) notes significant issues at one of the key institutions supported by the project (Rwanda Institute of Administration and Management) which ranged from staff turnover and delays, to allegations of corruption . In terms of achievement, the 2010 PEFA (ICR p.18) associates a “Very Satisfactory� (A) rating with the procurement indicator to which the project contributed through preparation of a procurement law and associated rules, as well as training. However, much remains to be done in the area of human resources in the face of staff turnover, unfilled positions and intuitional instability remain issues – the Government has very recently taken steps to improve this situation but results will not be observable for a while . 2. Accountability Substantial . The aim of the project was to improve accountability through enhanced public financial management. By the end of the project, 9% of the agencies’ audits were unqualified, compared to a modest project target of 7% -- in other words over 90% of the audits uncovered issues with financial management . Audits now cover 73% of government expenditure (ICR p.19) compared to 60% in 2007. As part of improving PFM functions, the project provided capacity building benefitting accountants, budget officers, Financial and administrative officers, and the Chief Internal Auditor and training for the cadre of accounting officers, internal auditors, audit committees members, director generals and others . Project support to the establishment IFMIS also led to significant results (ICR, p.17). Finally the ICR (pp. 18-19) associates the project with significant improvements in PEFA indicators. However, in addition to the fact that the changes to PEFA indicators were measured between 2007 and 2009, prior to project restructuring, many other donors were also supporting this area of reform and related capacity building, including : World Bank (PRSC7 and Multi-donor Trust Fund TF057314 funded by EU and DfID); DfID (http://projects.dfid.gov.uk/project.aspx?Project=201819); SIDA ( http://www.indevelop.se/ongoing-and-completed-assignments/public-financial-management-pfm-systems-asses sment-in-rwanda-for-sida/); and Belgium. Funding for technical assistance under donor projects and Trust Funds exceed US$ 10 million, which not only raises the issue of how to segregate project results, but also whether in the absence of project funding all activities would have been implemented with funding from other sources – the ICR does not shed any light on this. Notwithstanding these issues, the evidence presented in the ICR suggests that the sub -objective may have been substantially achieved, even if the project ’s contributions are not fully spelt -out. 3. Cross-cutting objectives Substantial . The project was responsible for the delivery of important output . It helped develop civil service reform strategies and policies, including improving teachers ’ wages and a Public Sector Management strategy, which have been submitted to cabinet but not yet implemented . The roll out of the citizen charters and service directories supported by the project aimed at improving accountability of Government to citizens and constitutes an achievement. Similarly, the first annual report on the state of capacity building issued in 2011 is cited as a hallmark achievement, even if the second one could not be issued . A national M&E framework was also developed with project funding . The project strengthened the PSCBS as well as local training capacity by supporting a number of institutions, but there is limited analysis in the ICR concerning the depth, quality and sustainability of this support . Furthermore, the indicator on human resources shows limited progress in this area, which is exacerbated by continued institutional instability, turnover of staff and unfilled positions (ICR pp. iii, 20-23). Finally, an independent assessment of the payroll and information system (not cited in the ICR) indicated that 75% of users were satisfied with the selected institutions performance . Despite the absence of baseline and information on other factors that contribute to the result, these results appear significant enough to justify a substantial rating for this objective . 5. Efficiency: Modest pre -restructuring . This was an institutional/capacity building TA project and no rates of return (economic or financial) were calculated – relevant analysis in PAD (pp. 19-20) is qualitative and summarized in the ICR. The ICR (section 3.3) provides potential examples of efficient use of project funds, including : (a) minimizing consultants’ costs; and (b) delivering training under budget . However, the ICR does not provide data to support these statements . The project performed poorly at first and better after restructuring . The Credit initially funded capacity building in some agencies, which were subsequently dropped from the project and /or were given a different mandate, contributing to inefficient use of resources : aside from HRCBA/HIDA a number of other institutions were supported during the first 5 years of the project, but dropped at restructuring . The ICR does not provide clear evidence whereby capacity building goals had been attained in those institutions, and /or whether functional reviews (ICR p.20) and project support resulted in improvement in their capacity to deliver better services in a sustained manner. Another important source of inefficiency (highlighted in the ICR p.24) is the drawn-out restructuring process, which was completed two years after the mid -term review had satisfactorily identified the thrust of what needed to be done . This factor together with moderate effectiveness delays resulted in project requiring to be extended by 2 years. The ICR does not provide information on implementation costs of the project. On the Bank side, the annual supervision budget (ICR p. 41) averaged US$181,000 per year between FY06-FY09 and US$73,000 thereafter which may point to inefficient use of Bank budget prior to project restructuring. Substantial post -restructuring . A combination of refocusing the project on fewer objectives and activities, together with task management from the field, resulted in a doubling in the pace of disbursements and improved efficiency. Better project outcomes during the last two years of the project are also consistent with greater efficiency. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project objectives before and after restructuring were substantially relevant as they were and remain aligned with the current CAS and Government strategy . However, the original project failed to realize many of its originals goals and to meet targets (about half of disbursements). Project outcome prior to restructuring is unsatisfactory (modest rating on all three criteria discussed above ). Performance improved after project restructuring (about half of disbursements). The shortcomings in design were generally overcome at restructuring. There remained unresolved structural challenges at project closing, such as institutional instability and staffing issues (high turnover and open positions ), as well as continuing delays in adoption and implementation of laws and regulations, and nine out of ten institutional audits being qualified . Despite these issues, recognizing that capacity building requires a sustained effort and not withstanding other weaknesses noted above, outcome for the project post -restructuring is rated Satisfactory . The project meets the minimum threshold for a Moderately Satisfactory rating for overall project outcome, based on the average of the two equally weighted outcomes This result is affected by the two years it took to restructure the project and disbursements that took place during that time period, and would have been more robust if restructuring had been completed earlier. a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The ICR notes that the original project risk was rated at the upper end of “moderate� as identified risks appeared adequately mitigated. It also states that the risk associated with the shifting institutional environment was not identified and continues to affect project outcome . It should be noted that the overall risk rating in the PAD did not integrate fiduciary and procurement risks (PAD Annexes 7 and 8) which were respectively rated substantial and high, and if taken into account could have affected the overall risk rating . The original and/or restructured project failed to identify other key risks, notably : 1. Project complexity. While complex projects may be implemented successfully, this requires a carefully designed work program, and unbundling of tasks and clear responsibility for their execution that were absent in this project. 2. Relying on a newly created institution HRCBA /HIDA with low initial capacity and no track record for project implementation. 3. Government staff once trained changing jobs or leaving the civil service altogether . This was discussed in the PAD but not monitored by the project – as seen below under discussions of M&E . The overall significant risk rating takes into account the fact that while some of the results – notably associated with PFM reforms – are likely to be sustained, institutional stability remains uncertain and high turnover of staff continues to be a persistent problem . Also, laws and regulations prepared by the project were adopted late and implemented with delay, and performance in areas such as percentage of qualified audits needs to be improved significantly in order to improve use of public resources . Successful implementation of the ongoing follow -up IDA-funded operation and implementation of measures in the area of Human Resources recently announced by Government may reduce many of these risks over time . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: Unsatisfactory . The project supported Rwanda ’s efforts to address human and capacity challenges affecting public service delivery, as it transitioned from the post -genocide period. Such an approach was broadly justified by the positive experience gained in other post -conflict countries, notably neighboring Uganda during the 1990s. However, this typically was done in smaller steps through a series of operations spanning over two decades – an approach initially not given sufficient consideration in the case of the present project . The ambitious project was supposed to be implemented over a period of about 5 years. However, its scope was unrealistic, excessive and overextended . The project aimed to strengthen quickly and on a broad front the capacity of many public sector entities for more efficient, effective, transparent, and accountable performance. It thus aimed to tackle a multitude of needs in a context where turnover of staff was and remains a significant problem, as was institutional instability . Limited funding alone might have called for a less ambitious approach. These issues were recognized at the time of project restructuring, at which point effectiveness and transparency goals were dropped from objectives, so was service delivery in favor of better use of public resources, thus putting in place a more modest and somewhat better sequenced approach. The ICR (p.27) states that the project was consistent with Government priorities and the CAS and that : (a) risks were identified and mitigated; and (b) lessons learnt had been incorporated in project design and choice of instrument. However, there were significant gaps in both latter areas (as noted in this review) as well as other shortcomings noted in the ICR, such as : (i) the need for broader consultations with stakeholders to strengthen and extend ownership; (ii) weaknesses in M&E that resulted in some indicators not being measurable or attributable; and (iii) insufficient coordination with other initiatives and more effective governance structure. There were additional design flaws that led to under performance under the project, which needed to be corrected through restructuring : the project unrealistically tackled too many issue at once; it supported too many institutions superficially and without sufficient outcome orientation; its design did not allow for the implementation of various activities in a way they would not be affected by and be largely insulated from problems with other components; and implementation arrangement were over -optimistic about how quickly and effectively a new institution (HIDA) with no track record would become operational – even abstracting from the fact that Government decided to undertake unexpected institutional changes during the project . These issues might have been alleviated through testing key approaches and kick -starting HIDA through a Project Preparation Facility (now called Project Preparation Advance ) funding and/or retroactive financing. at -Entry Rating : Quality -at- Unsatisfactory b. Quality of supervision: Moderately Satisfactory . The ICR (pp.12-13) notes that the 14 supervision missions left comprehensive aide-memoires and promptly brought issues to management attention . Furthermore, the Bank was active in the policy dialogue as co-chair of the two sector working groups, and as manager of the multi -donor Trust Fund. Following restructuring, which was quite well -conceived, the presence of a field -based Task Team Leader (TTL) resulted in proactive supervision and improved implementation, as evidenced by the relative turnaround in the project and its improved performance . Consequently, disbursement during the last 2 years equalled that of of the previous 5, and many of the project results were attained during that period . Improvements took place despite the high turnover of TTLs (four since mid-2009) and a decreased supervision budget. Nevertheless, given the difficulties being experienced during implementation and shortcomings in the achievement of objectives during the first five years, the largely satisfactory ISR ratings on progress towards achieving PDO prior to project restructuring did not correspond to realities on the ground, lacked candor and failed to adequately inform management of problems being encountered . This situation was exacerbated by the fact that project restructuring took two years, even though its scope and key modifications had been largely defined by mid-term review. This delay contributed to the weaknesses in project outcome and reflects a significant insufficiency in supervision and /or possible insufficient Bank management oversight of the project. Another aspect of supervision concerns the actions of the Bank in the 2 years prior to its restructuring during which the project operated without a project implementation unit (ICR p.11). It seems that this situation contravened at least one legal covenant (DCA Schedule 4, A.4.) and that immediate remedial action should have been taken at that time -- this could range from suspension of activities to amendment of the DCA to provide for a suitable interim arrangement . Significant shortcomings in supervision during the first 5 years of the project affect the overall rating for quality of supervision, which was unsatisfactory . The turnaround in the project reflects much more effective Bank supervision without which outcomes would have been more modest . The Moderately Satisfactory rating for supervision recognizes the satisfactory effort post -restructuring. Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: The project benefited from strong ownership at the conception stage as manifested by the Government's MSCBP and putting public service delivery as a priority -- e.g., PAD p.3. However, too many specific activities were agreed to by Government without sufficient consideration of prioritization and sequencing, and expectations turned-out to be unrealistic. There was a subsequent decline in its performance during much of project implementation attributable to turnover of public officials and insufficient /delayed project staffing. Periodic changes in the institutional set -up by the Government, notably HIDA, affected project implementation, but the reasons for these modifications are unclear . In addition, laws and regulations prepared by the project were not adopted and /or implemented. Furthermore, it appears that the Government's reluctant support contributed to delays in restructuring the project . The Government's performance improved following project restructuring, even though passage and implementation of laws continued to experience delay . The Moderately Satisfactory rating herein reflects the moderate shortcomings in Government performance, which contributed to delays i n restructuring and institutional instability . Government’s positive contributions during project preparation and the last two years of project implementation somewhat eclipsed the aforementioned issues, as well as late adoption of policies /reform prepared under the project -- especially in as some of the backlog has been addressed recently . Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: While there was variation in performance across agencies, according to the ICR coordination was a challenge and contributed to implementation delay . These problems were magnified by shortcomings in M&E. The project operated without an implementing agency for two years (2008 and 2009). Agencies' performance was also affected by turnover of officials and changes in their institutional mandate . While some of these problems persisted throughout the implementation period, the turnaround project performance after its restructuring reflects the better performance of agencies, notably PSCBS, after 2009. Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: Negligible . The PAD (Annex 3) included a detailed results matrix with 26 indicators (including 4 for outcome). The restructured project proposed a streamlined results matrix with 11 indicators (including 3 for outcome). The PP notes that changes involved “removing indicators that proved not to be measurable .� The ICR does not provide information on which were retained . In fact, the restructuring of indicators was more fundamental : (a) the outcome indicators were all new; and (b) only one intermediate outcome appears to be identical (PFM (i)) while some are clearly new (Civil service (vii)). The ICR (p.13) recognizes this “shift in indicators during restructuring has also made it difficult to assess impact evenly across pre - and post-restructuring periods.� The ICR’s candid assessment of the original M&E is as follows : “The M&E design of the project had weaknesses stemming from the selection of indicators at the design stage, the vast number of indicators and the range of indicators that could not be measured .� Other shortcomings of the M&E included the absence of baseline data, use of qualitative outcome indicators that were at best partially attributable to the project, and not incorporating indicators that would have more clearly been linked to project objectives . Specifically, high staff turnover is listed as a key issue (e.g., PAD p.2 and clearly stated in component 2) and a related outcome indicator should have been incorporated into the project (e.g., percentage of public staff trained by the project that continued to perform the job for which they received training and /or remained in the public sector). The restructured project also failed to include such a result . Similarly, the PAD (p.11) mentions an appropriate output indicator (six well-functioning agencies) which would have shed light on the performance of the project prior to its restructuring. This indicator is missing from the PAD’s results framework. The restructured project included an improved M&E process . The new baseline was from end-2008. Given that the PP was prepared in 2009 this corresponded to the latest available information . Aside from the above-mentioned absence of an indicator on staff turnover, the revised M&E system could have included more explicit reference to improving attributable PEFA scores which is presented in the ICR (p.18) as one of the project’s achievements, even if as discussed above its contributions may have been be partial . b. M&E Implementation: Negligible . Limited progress was made in implementing project M&E prior to restructuring and availability of baseline data remained limited (ICR p.12). Baseline surveys for original indicators had not been undertaken until project restructuring and this constitutes a significant shortcoming in implementation . Similarly, some of the indicators were apparently dropped after mid -term review, even though the project was formally restructured 2 years later. Furthermore, as noted in the ICR, throughout project implementation “the reorganization of MDAs various times throughout the implementation of the project makes it difficult to track progress on indicators .� The ICR (p.13) notes improvements in M&E implementation with the hiring of an M&E specialist after project restructuring. c. M&E Utilization: Modest . The ICR (p.13) notes that M&E use was modest and fed into work plans only to a limited extent . The ICR uses information largely outside the M&E framework, some only partially attributable to the project, to justify outcome ratings. The project M&E utilization was thus quite low . However, a positive aspect of the project was the development of a National M&E system (ICR p.4) which is overlooked in the analysis of M&E . This M&E was used to monitor Rwanda’s Economic Development and Poverty Reduction Strategy – which seems to be a substantial contribution . The overall rating of Negligible for M&E reflects the significant shortcomings in the above areas prior to project restructuring and imperfections thereafter, despite the development under the project of a national M&E . M&E Quality Rating : Negligible 11. Other Issues a. Safeguards: This Technical Assistance project is Environmental Category C, and Safeguard Classification S 3, and did not trigger any environmental or safeguard policies (Integrated Safeguards Data Sheet, p .1). This was reconfirmed at the time of restructuring (ICR p.13). b. Fiduciary Compliance: The PAD (Annexes 6 and 7) had classified project risk as substantial for financial management (FM) high for procurement. In both cases mitigation measures proved effective and the main problem encountered was recruitment and retention of qualified staff and delays in implementation of contracts . FM risk was assessed as moderate during the final year of implementation . The ICR (pp. 13-14) mentions that there were no issues with project audits or their quality . ISRs confirm that audits were prepared on time and were unqualified. But they also highlight periodic weaknesses in financial management that resulted in downgrading in 2009 of its rating to MU, largely because of use of special account as an advance for ineligible expenditures that required counterpart funding . The ICR makes references to other issues encountered, including the need to use standard reporting formats, maintaining proof of payment, and accounts being kept on spreadsheets without proper back -up. The ICR (p.14) does not mention any additional procurement specific issues . The ISR rating for procurement was satisfactory throughout the project and this suggests that the risk rating in the PAD may have been overstated or effectiveness of mitigation was better than anticipated . c. Unintended Impacts (positive or negative): d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Significant Risks not explicitly identified/mitigated Outcome : in the PAD and/or the ICR, notably continued institutional instability and staff turnover, and uncertainty over the implementation of legal and regulatory reforms. Bank Performance : Moderately Moderately The project scope was unrealistic, Satisfactory Unsatisfactory excessive and overextended . Important issues affected project design, many of which could have been anticipated . The project design did not incorporate prevailing good practices in public sector capacity building. The project received relatively high supervision resources prior to its restructuring . Limited supervision resources does not explain why the Bank could not be more responsive to the changing needs of the project and restructure it in a timely manner: As recognized by the ICR "when finally restructured in 2009, it was too late in the project life cycle ." Supervision improved after restructuring, but could not make up for earlier slippages. Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Unsatisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR (section 6) lists appropriate even if generic lessons learnt, including : (a) the importance of ensuring highest level political support in post -conflict countries with shifting institutional settings and high staff turnover; (b) the need for flexibility and rapid response capability being built into project design and utilized when needed; (c) the importance of focusing interventions on select agencies and not covering too many activities; and (d) the importance of field-based supervision in this project's turnaround . The Government's evaluation report includes other relevant lessons such as : (a) viability and sustainability of project depend on full participation of beneficiaries; (b) human capital constraints need to be identified and adequately mitigated; (c) once reforms are adopted resources are needed to implement them; (d) there should be greater reliance on national systems; (e) M&E should be undertaken at regular intervals; (f) complex projects should either be avoided or be managed by a single and empowered project manager; and (g) the basket funding mechanism is an effective way to support development initiatives . Government comments conclude with a recommendation whereby a Project Performance Assessment Review is prepared in two to three years . This recommendation is not repeated below in large part because such an assessment should be done once the follow -up IDA project is completed. In addition to the important lessons listed above, additional ones emerge from the present review : It is essential that a mid-term review takes place as scheduled or advanced if needed, especially when legal covenants concerning implementation arrangements are not respected . Subsequent restructuring should be completed in a timely manner -- no more than 6 months later. Complexity in TA operations is often inevitable . However, the risks associated with complexity need to be clearly identified and addressed in a proactive way, ideally for each major distinct activity . The benefit of a demand-driven approach towards capacity building is that it can provide rapid response that would not be feasible with more onerous preconditions and eligibility criteria . However, its shortcoming is that resulting interventions may be too superficial without resulting impact . Therefore, larger/more ambitious interventions require more up -front analysis in terms of an institutional reform plan and focusing on few priority institutions . For smaller interventions, better selectivity may be achieved through a matching funding approach and /or simple specification of results sought, which need to be monitored ex-post. It is extremely risky to depend on a newly created agency to both implement a project and to deliver key results. In cases when existing institutions do not provide suitable alternatives, capacity in the new institution needs to be adequate and preferably build during project preparation through a PPA . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR presents a good deal of information about the project and its implementation post -restructuring. Some of its assessment is quite candid . However, many aspects of the presentation fall short of expected standards both on substance and details . The text is missing important information and is not well organized, and its internal logic is at times unclear and contradictory . Major shortcomings, include the following : The presentation in the ICR (section 3.2) suffers from disconnect in logic and being built mostly around some output under project components and not PDO goals, and /or from referring to results that may not be fully attributable to the project . Furthermore, ICR coverage is mainly of the revised project with limited information on and assessment of sub -objectives and activities that were dropped . The ICR thus does not respect guidelines whereby pre - and post-restructuring achievement are assessed and weighted on the basis of disbursements. It reports indicators and results primarily for the restructured project . There is thus limited reporting of results prior to restructuring and the outcome rating is overstated . Furthermore, the ICR fails to fully take into account of remaining issues at closing, and inefficiencies caused by drawn -out project restructuring. The assessment of various aspects of the project is not well supported by judicious analysis in the ICR and is contradicted at times by the text, which leads to exaggerated ratings . Furthermore, the terminology for some of the ratings is non-standard. For example, the ICR rating for relevance is High (p.16), which would imply the project incorporates best practice . Even though there is no assessment of project design in the relevant section of the ICR (3.1) this rating is contradicted in much of the ICR write -up, as evidenced by issues that needed to be addressed through restructuring . Similarly, the ICR (p.15) associates Rwanda’s overall strong economic performance (including progress on MDGs and CPIA ratings ) with the project, even though the chain of logic and attribution is respectively unclear and partial at best . The ICR seems (text is unclear) to rate efficiency as “moderately satisfactory�, which is not part of the scale for efficiency, but appears to correspond to a “Substantial� rating. This rating does not reflect the long delay in project restructuring. Certain passages lack specificity and clarity (e.g., section 3.3) and are internally contradictory (e.g., section 5.2). The main text of the ICR is too long and the discussion should have been shorter and better organized. The ICR fails to fully reflect Borrower feedback (Annex 8 only briefly mentioned in section 3.6) in the main text, including: (a) unpredictability of donors; (b) impact of expected continued high staff turnover on sustainability; and (c) delays in replenishment and perception whereby internal Bank coordination was an issue. Similarly, the desirability of relying on basket funding mechanisms is highlighted in Borrower comments, but while mentioned this approach is not discussed in any depth in the ICR . The costing table (ICR Annex 1) contains erroneous figures . Project costs are inconsistent throughout the document due to apparent omissions . For example; country financing parameters were approved as part of the 2010 restructuring and prior to that counterpart funding should have been provided by government for some categories of expenditures (as per DCA Schedule 1); and total disbursements and at time of restructuring are stated incorrectly . Similarly, The date of the restructuring is misstated (2009 instead of 2010). The ICR falls well short of expected quality standards on both form and substance . It suffers from internal inconsistencies, fails to provide information correctly, and proposes ratings that are exaggerated and inadequately supported by its analysis . These issues affect its overall quality, which is rated Unsatisfactory . a.Quality of ICR Rating : Unsatisfactory