ICRR 13970 Report Number : ICRR13970 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 12/19/2012 Country : Solomon Islands Project ID : P097671 Appraisal Actual Project Name : Sb Health Sector US$M ): Project Costs (US$M): 1.5 1.2 Support Project (ta) L/C Number : CH360 Loan /Credit (US$M): Loan/ US$M ): 1.5 1.5 Sector Board : Health, Nutrition and US$M): Cofinancing (US$M ): 0 0 Population Cofinanciers : Board Approval Date : 03/20/2008 Closing Date : 03/31/2011 12/31/2011 Sector (s): Central government administration (63%); Sub-national government administration (27%); Health (10%) Theme (s): Health system performance (100% - P) Prepared by : Reviewed by : ICR Review Group : Coordinator : Judith Hahn Gaubatz Judyth L. Twigg Soniya Carvalho IEGPS1 2. Project Objectives and Components: a. Objectives: According to the Project Appraisal Document (PAD, page 6), the project objective was: to improve the institutional capacity of the Ministry of Health and Medical Services MHMS ) in the areas of (MHMS) public expenditure management and sector performance monitoring . The Grant Financing Agreement (page 4) articulated the objective in similar terms : to improve the institutional capacity of MHMS in public expenditure management and sector performance monitoring so as to contribute to the overall success of the Program . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: 1: Public Expenditure Management (Appraisal: US$0.55 million; Actual: US$0.58 million): This component aimed to strengthen national and provincial level MHMS planning and budgeting systems and procedures, so that priorities were set within a known resource envelope and linked to expected results . Activities included: completion of an initial version, and subsequent annual updates, of a health sector medium -term expenditure framework (MTEF); expenditure monitoring to provide feedback on planning; a health facility efficiency study; and development of operational models for effective engagement of communities and households in an effort to positively impact health seeking behavior. 2: Sector Performance Monitoring (Appraisal: US$0.38 million; Actual: US$0.37 million): This component aimed to strengthen monitoring and evaluation (M&E) of the national Health Sector Support Program (HSSP) and integrate M&E into sector management for better decision -making. Activities included: refinement of the M&E framework (including identifying indicators, data needs, and baseline figures )and development of a balanced scorecard for provincial level management. 3: Training and Capacity Building (Appraisal: US$0.52 million; Actual: US$0.29 million): This component aimed to strengthen management capacity of senior managers and provincial health directors in strategic planning and execution. Activities included training and continuing management education for MHMS staff . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project cost: The estimated final project cost was US$ 1.23 million, which was less than the appraised cost of US$ 1.5 million. According to the project team, the difference was in part due to a shortfall in implementing Component 3, as well as exchange rate fluctuations (XDR to US$). Financing: The project was entirely financed by the IDA grant, which according to the project team, disbursed about 82% of the estimated amount (in XDR). Borrower contribution: There was no planned Borrower contribution . Dates: The project closing date was extended from March 2011 to December 2011, in order to allow completion of activities, that had been delayed at the start of the project . 3. Relevance of Objectives & Design: a. Relevance of Objectives: Substantial . The country was emerging from a period of post -civil conflict and recovery, and thereby shifting from an emergency context to a longer -term development approach. In the health sector particularly, there was inadequate capacity to deliver health services effectively and efficiently both at the national and provincial levels . The Bank's Interim Strategy Note for FY10-11, the most recent strategy note, identifies support for improved public administration and management as one of three key pillars, including improving efficiency and planning in public spending . The government's National Health Sector Program (NHSP) for 2011-2015 identifies improved service provision and improved financial and planning performance as key objectives for the sector . Thus the capacity building objectives of this project, in the areas of public expenditure management and sector monitoring, were substantially relevant, given the need to establish more effective core government systems to improve health sector performance . b. Relevance of Design: Substantial . The project was designed as a technical assistance operation to address the lack of clarity in resource allocation (i.e. overall resource envelope, proportion of funding for national vs . provincial programs, recurrent vs . investment expenditures) as well as the lack of M&E capability . Thus the project was designed to establish the "building blocks" for health sector management, which would contribute to the overall effectiveness of the health sector program. As noted in the PAD (page 5), the main constraints to improving health sector performance lay more in resource allocation and management, rather than in the level of financing itself . Activities focused on the two specific areas of public expenditure management and sector performance monitoring, which would likely contribute to improved capacity at both the national and provincial level . 4. Achievement of Objectives (Efficacy): Objective : To improve the institutional capacity of the MHMS in public expenditure management Substantial , based on evidence that the project outputs increased capacity of the MHMS and contributed to policy dialogue and decision-making. Outputs The final version of the MTEF was completed in 2011. Budgeting and accounting processes were improved; for example, the budget process now allows disaggregation of hospital and primary health care expenditures at the provincial level . Key MHMS staff attended the World Bank Institute's Flagship Course in Health Systems, rating the course "very highly" and underpinning their "sense of usefulness of the MTEF ... and importance of planning within budget constraints" (ICR, page 27). Annual operational plans are being developed on a more routine basis at the provincial level, with the participation of provincial leaders and community groups . However, the medium-term planning process remains fragile and quantitative monitoring remains inadequate . The health facility survey was not completed as planned . The MHMS initially expressed concerns about potential overlap with another donor study on infrastructure needs . Although MHMS subsequently decided to proceed with the survey, the decision came in the last year of the project and therefore the survey could not be initiated on a timely basis. Outcomes The first annual update of the MTEF is being planned for the third quarter of 2012. The MTEF is recognized by the MHMS as the only source of detailed sector expenditures, as other existing budget processes do not provide a complete picture of sector expenditures . The Ministry of Finance is planning to introduce a MTEF process for all sectors in the coming months . There is increased transparency in health expenditures . The ICR reports (page 25) that details on FY11 budgets and expenditures, and FY 12 budgets, are now available and can provide more systematic "trend" data on resource flows. The draft MTEF in 2010 contributed to determining the overall resource envelope for the National Health Sector Plan for 2011-2015, as well as for the annual operational plans and budgets for FY 10 and FY11. The final MTEF in 2011 informed the FY12 operational plans and budget . The MTEF highlighted significant recurrent costs that will arise in the future from ongoing donor projects, such as maintaining new hospital buildings, a refurbished malaria building, and staffing returning overseas - trained doctors. The proportion of health sector expenditures on primary health services and provincial health programs increased from 17% in 2008 to 29% in 2011, surpassing the target of 22%. However, attribution to the project's activities is not clear. A strategic decision was made in the NHSP to decrease national program recurrent budget funding by 14% and increase provincial programs by 17% by 2015, based on MTEF data. Objective : To improve the institutional capacity of the MHMS in sector performance monitoring Modest , based on partial implementation (and uneven quality) of project outputs, and limited evidence of impact on policy and decision-making. Outputs A new NHSP for 2011-2015 was developed, including a comprehensive M&E framework with priority indicators, baseline and target figures, and data collection arrangements . The framework is being adopted by MHMS and donors; however, the existing health management information system (HMIS) remains inadequate for providing the required M&E data. Joint Annual Performance Reviews, with the participation of national and provincial MHMS directors, donors, and NGOs, were conducted in 2009, 2010, and 2011. Another joint review was scheduled for June 2012. However, according to the ICR (page 10), the quality of the review processes and data availability varied considerably over the project period . Provincial operational plans are developed with M&E indicators included . Balanced scorecards were not implemented as planned, due to the lack of prioritization placed on this activity by the incoming MHMS leadership. Outcomes: The ICR reports (page 14) that the "process" consultancy approach (vs. "task" consultancy) used for developing the new NHSP helped to build capacity through "learning by doing." The ICR (page 11) also notes that "the MHMS executive has been increasingly making informed evidence-based decisions based on a range of assessments including ... monitoring data generated from program and provincial operational plans " and that "the new NHSP was based on quantitative and qualitative data generated from situational assessments and from available M&E data ." However, no specific examples are provided. 5. Efficiency: Modest . While the ICR (page 12) points to potential gains in sector efficiency (i.e. allocative efficiency of health spending through increasing resources to primary health care and to provinces ), there is limited evidence provided in the ICR of efficient use of project resources, suggesting modest efficiency . There were significant delays in project implementation due to limited implementing agency capacity, with the first disbursement occurring 24 months after effectiveness and the initial two major staff positions being filled 16 months after effectiveness . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Moderately Satisfactory . Relevance of objectives and design are rated Substantial . Achievement of the objective to improve institutional capacity of the MHMS is rated Substantial for public expenditure management, but Modest for sector performance monitoring due to partial implementation (and uneven quality) of project outputs, and limited evidence of impact on policy and decision -making. The limited evidence on efficient use of project resources suggests efficiency is Modest . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The Bank, the MHMS, and donors are maintaining active discussion regarding support to the health sector, including improving the implementation and results focus of the HSSP . The Bank, specifically, plans to continue capacity building in health sector management through trust fund -supported activities. The "learning by doing" approach used for some aspects of the project contributed to sustainability of the capacity building activities, although frequent staff turnover remains an issue . The instability in the political environment may also affect political commitment to sustaining outcomes. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: The initial project approach was to develop a pooled financing model under a sector -wide approach (SWAp); therefore, extensive consultations with stakeholders and background analyses had been conducted, including a health sector strategy note, a qualitative study of health care decision making, a partial MTEF, a health expenditure review, a report on annual planning and budgeting, and reviews of hospital and pharmaceutical services. The project approach was significantly scaled down to a technical assistance operation upon determining that a SWAp-based operation was not feasible (due to concerns about capacity to manage the project's fiduciary aspects and the lack of agreement among donors on acceptable fiduciary processes ). The project's risk level was assessed as High, including potential risks related to fiscal crises, lack of action taken by government according to Bank team recommendations, low fiduciary capacity, and political instability . The last two risks subsequently materialized and led to significant project implementation delays . Measures to mitigate weak fiduciary capacity, in particular, were ineffective . The M&E framework also had shortcomings (see Section 10). at -Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: The project experienced significant delays in start -up due to inadequate capacity, particularly in procurement (see Section 11). As noted previously, the first disbursement occurred 24 months after effectiveness and the initial two major staff positions were filled 16 months after effectiveness . The ICR (page 8) reports limited Bank support in the area of financial management and infrequent supervision during the initial project period, particularly given the high level of need in the limited capacity environment . However, the Bank team demonstrated flexibility in adding project activities at the request of the Borrower, of which the addition of the senior health policy advisor and the financial management specialist were particularly helpful in moving project implementation forward after the delayed start . From 2010, supervision intensity improved, with increased Bank presence in the country and more active communication with MHMS and donors . The Mid-Term Review resulted in the addition of M&E indicators to more effectively capture project results, and there was active dialogue on sector performance during supervision missions; however, the overall M&E framework remained inadequate (see Section 10). Quality of Supervision Rating : Moderately Unsatisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: Political instability due to frequent changes in administration led to a state of uncertainty for senior MHMS staff, and during the second quarter of 2009, a new Health Minister, Permanent Secretary, and almost completely new MHMS senior management team were appointed . These turnovers led to significant delays and disruptions in project implementation. Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance: There were considerable delays in project implementation due to lack of understanding of project arrangements within the MHMS, as well as high turnover of senior staff in 2009. The new MHMS leadership eventually endorsed the project and also requested additional project activities, including management mentoring and development of the next NHSP (2011-2015). The addition of the senior health policy advisor and the financial management specialist "proved decisive and reenergized implementation " (ICR, page 5), in particular by linking expenditure management and M&E to the health sector priorities . There were significant difficulties in procurement, primarily due to weak capacity (see Section 11), along with shortcomings in financial management . Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: Project monitoring was to be undertaken within the M&E framework of the overall health sector program; however, this framework was developed late in the project period, and only in draft form . The HMIS was not functionally operational and thus could not provide the required data . M&E indicators had shortcomings, including lack of specificity ("increased community participation in planning " at the provincial level), lack of clear means to collect data ("executive decision-making increasingly based on M&E reports "), and difficulty in measurement due to inadequate accounting systems (separation of hospital and primary health care expenditures ). Evaluative activities were to include joint annual performance reviews with donors to assess sector performance . b. M&E Implementation: The M&E framework was significantly delayed, and only partially completed, while availability of data was constrained by the operational shortcomings of the HMIS . Joint reviews were undertaken as planned, although the quality of such reviews was variable . The Bank maintained active dialogue with the MHMS and donors, with the Bank aide-memoires being used as the "de facto" reports on sector performance . c. M&E Utilization: There are some specific examples provided in the ICR on the use of M&E to inform policy and decision - making (see Section 4). M&E Quality Rating : Modest 11. Other Issues a. Safeguards: As a Category "C" project, no safeguard policies were triggered . b. Fiduciary Compliance: Financial management: Initial financial management capacity was limited and constrained project performance . Project audits were unqualified, although they were often late due to incomplete or misplaced filing of paperwork . Interim financial reports were also submitted late and often needed further refinement to meet the Bank's financial management requirements. Financial management specialists eventually helped to improve performance and overall capacity. The ICR (page 8) notes that financial management support from the Bank's fiduciary implementation support team was limited and not well coordinated with broader Bank supervision missions . Financial management reviews (two formal and one informal) only took place in March 2011. Procurement: Procurement risk was rated "High" in the PAD (page 42) due to institutional and capacity constraints, absence of procurement planning tools, perception of corruption, and limited number of local contractors in -country. Although upfront actions to address these factors were planned, there were significant procurement delays from the outset. There was limited capacity and frequent staff turnover in the procurement unit, along with limited management understanding and direction on the procurement process . Although MHMS had agreed to recruit an international procurement advisor for the project, it was not able to attract and retain a qualified specialist in a timely manner. The advisor was appointed nine months after project effectiveness, but initial procurement actions were still delayed due to a misunderstanding about the advisor's responsibilities within MHMS . The Bank eventually reached a new agreement with MHMS and AusAID on continued procurement advisory support, although procurement delays persisted throughout the project period . c. Unintended Impacts (positive or negative): d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately There were significant implementation Satisfactory Unsatisfactory delays, as well as shortcomings in fiduciary performance. Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: Based on the findings of this review, it is important for a project that has as one of its objectives to improve performance monitoring to ensure that the proper monitoring systems are in place prior to, or very early on in, the project period. In the case of this project, the shortcomings in the MIS could not be overcome once the project became effective and thus the project activities intended on taking the monitoring "to the next level" could not be implemented and led to shortcomings in achieving the objective . According to the ICR lessons (page 18), this relatively small but high risk project was worth the high transaction costs because the Bank was able to add value across core domains in the sector . This work on core domains was then leveraged throughout the entire health sector, including in preparing the new health sector strategy and improving planning and budgeting systems . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR Quality is Satisfactory. The quality of the evidence was adequate for assessing the project outcomes . The quality of the analysis was satisfactory, although discussion of attribution would have further strengthened the analysis. The lessons were clearly drawn, and relevant particularly for considering Bank support to small countries with limited capacity. The ICR was concisely written and overall consistent with guidelines . a.Quality of ICR Rating : Satisfactory