Document of The World Bank Report No: ICR00002051 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-H3600) ON A GRANT IN THE AMOUNT OF SDR 1 MILLION (US$ 1.5 MILLION EQUIVALENT) TO SOLOMON ISLANDS FOR A HEALTH SECTOR SUPPORT PROGRAM TECHNICAL ASSISTANCE PROJECT June 29, 2012 Human Development Sector Unit East Asia & Pacific Region CURRENCY EQUIVALENTS (Exchange Rate Effective June 29, 2012) Currency Unit = Solomon Islands Dollars SB$ 1.00 = US$ 0.14 US$ 1.00 = SB$ 6.76 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS AusAID Australian Agency for International Development DHS Demographic and Health Survey DP Development Partner FM Financial Management HIV/AIDS Human Immunodeficiency Virus/Acquired Immune Deficiency Syndrome HMIS Health Management Information System HSSP Health Sector Support Program HSSP-TA Health Sector Support Program – Technical Assistance ICR Implementation Completion and Results Report IDA International Development Association IFR Interim Financial Report ISN Interim Strategy Note ISR Implementation Status and Results Report M&E Monitoring and Evaluation MDPAC Ministry of Development Planning and Aid Coordination MHMS Ministry of Health and Medical Services MOF&T Ministry of Finance and Treasury MTEF Medium Term Expenditure Framework NGO Non-government Organization NHSP National Health Strategic Plan PAD Project Appraisal Document PDO Project Development Objective PER Public Expenditure Review SIG Solomon Islands Government SPC Secretariat of the Pacific Community STI Sexually Transmitted Infection SWAp Sector Wide Approach TA Technical Assistance TAL Technical Assistance Loan TOR Terms of Reference UNFPA United Nations Population Fund UNICEF United Nations Child Fund WHO World Health Organization Vice President: Pamela Cox Country Director: Ferid Belhaj Sector Manager: Toomas Palu Project Team Leader: Susan Ivatts ICR Team Leader: Susan Ivatts   Solomon Islands Health Sector Support Program Technical Assistance Project Contents Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 3 3. Assessment of Outcomes ............................................................................................ 9 4. Assessment of Risk to Development Outcome......................................................... 14 5. Assessment of Bank and Borrower Performance ..................................................... 15 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 19 Annex 1. Project Costs and Financing .......................................................................... 21 Annex 2. Outputs by Component ................................................................................. 22 Annex 3. Economic and Financial Analysis ................................................................. 28 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 29 Annex 5. Beneficiary Survey Results ........................................................................... 31 Annex 6. Stakeholder Workshop Report and Results................................................... 32 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 33 Annex 8. Comments of Co-financiers and Other Partners/Stakeholders ...................... 38 Annex 9. List of Supporting Documents ...................................................................... 42 MAP IBRD 35742 ........................................................................................................ 43   A. Basic Information SB Health Sector Country: Solomon Islands Project Name: Support Project (TA) Project ID: P097671 L/C/TF Number(s): IDA-H3600 ICR Date: 06/28/2012 ICR Type: Core ICR Lending Instrument: TAL Borrower: SOLOMON ISLANDS Original Total XDR 1.00M Disbursed Amount: XDR 0.82M Commitment: Revised Amount: XDR 0.82M Environmental Category: C Implementing Agencies: Ministry of Health and Medical Services Cofinanciers and Other External Partners: B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 05/09/2006 Effectiveness: 07/14/2008 07/14/2008 Appraisal: 11/01/2007 Restructuring(s): 08/26/2010 Approval: 03/20/2008 Mid-term Review: 09/02/2010 09/02/2010 Closing: 03/31/2011 12/31/2011 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Moderate Bank Performance: Moderately Satisfactory Borrower Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Moderately Satisfactory Implementing Quality of Supervision: Moderately Satisfactory Moderately Satisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Moderately Satisfactory Performance: Performance: C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry Yes None at any time (Yes/No): (QEA): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Satisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration 63 63 Health 10 10 Sub-national government administration 27 27 Theme Code (as % of total Bank financing) Health system performance 100 100 E. Bank Staff Positions At ICR At Approval Vice President: Pamela Cox James W. Adams Country Director: Ferid Belhaj Nigel Roberts Sector Manager: Toomas Palu Fadia M. Saadah Project Team Leader: Susan Lynette Ivatts Muhammad Ali Pate ICR Team Leader: Susan Lynette Ivatts ICR Primary Author: Ian P. Morris Michelle Lee F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The Project's development objective is to improve the institutional capacity of MHMS in the areas of public expenditure management and sector performance monitoring. Such capacity development would be critical to the overall success of the Program in improving health services delivery and outcomes. Revised Project Development Objectives (as approved by original approving authority) (a) PDO Indicator(s) Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Indicator 1 : Completion of the first rolling medium term expenditure framework Completion of first rolling medium Value term sector Final revised quantitative or None expenditure version of MTEF Qualitative) program linking produced resources with priorities Date achieved 02/01/2008 12/31/2009 09/01/2011 Comments A substantial draft of the MTEF was developed in 2010, and a final MTEF was (incl. % produced in 2011. achievement) Increased proportion of health sector expenditure on primary health services and Indicator 2 : provincial health programs The proportion of health sector expenditure on Health expenditure primary health increased from services and Value 17% on primary provincial health quantitative or None health services and programs has Qualitative) provincial health increased from 17% programs (estimated base at appraisal) to 29% for FY11 (target 22%). Date achieved 02/01/2008 12/31/2011 12/31/2011 Comments (incl. % Source: Final MTEF 2011 achievement) Indicator 3 : Completion of Health Facility Survey Value Health Facility quantitative or None Not Applicable Survey completed Qualitative) Date achieved 02/01/2008 12/31/2008 09/15/2010 Comments (incl. % The MHMS decided not to go ahead with the health facility survey. achievement) Indicator 4 : Joint Performance Reviews Conducted Annually Value Less substantial reviews Joint Performance Joint Annual quantitative or were undertaken in 2007 Reviews Performance Qualitative) and 2008 conducted Reviews are annually conducted each year by the MHMS Date achieved 02/01/2008 12/31/2011 12/31/2011 Comments JAPRs are held annually with the participation of Provincial Health Directors, (incl. % program managers, DPs, non-government organizations (NGOs), and faith-based achievement) organizations (FBOs). (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Indicator 1 : Percentage of planned activities that are completed increased from 60% to 75% This indicator was Value Planned activities inadequately (quantitative None increased from specified and not or Qualitative) 60% to 75% monitored quantitatively Date achieved 02/01/2008 01/31/2011 12/31/2011 Comments Provincial health staff have indicated improved satisfaction with planning within (incl. % budget constraints and recognized some difficulties in fully implementing achievement) planned activities Indicator 2 : Increased community participation in health services planning at provincial level Increased community Value All health planning is Community participation in (quantitative conducted in a centralized involvement has health services or Qualitative) and top-down manner increased planning at provincial level Date achieved 02/01/2008 12/30/2011 12/31/2011 Comments There has been a gradual increase in community involvement in provincial health (incl. % service planning through the operational plan development process although achievement) capacity constraints at this level need to be appreciated Indicator 3 : Health facility survey completed and analyzed Value Health facility (quantitative None Not progressed survey completed or Qualitative) Date achieved 02/01/2008 07/15/2009 12/31/2011 Comments (incl. % The MHMS decided not to go ahead with the health facility survey achievement) Budget planning, development and accounting processes enhanced and Indicator 4 : institutionalized Value All financial systems are Budget Annual provincial (quantitative fragile. At the provincial planning, and national or Qualitative) level, the system is unable development operational plans to separate expenditures and with SIG and DP from the hospital and accounting budgets are now primary health care processes developed routinely programs. enhanced and and accounting of institutionalize expenditures on d. This MYOB now allows indicator was separation of added after the hospital and mid-term primary health care review to at the provincial better reflect level the specifics of the project Date achieved 02/01/2008 09/13/2010 12/31/2011 Comments (incl. % achievement) Indicator 5 : Comprehensive sector performance indicators and monitoring system developed M&E framework was developed as part of the new NHSP 2011-2015 Health systems are very and identifies Value Comprehensive fragile, unsystematic and agreed priority (quantitative M&E framework often subject to delays in indicators for each or Qualitative) developed processing program. It details the baseline, target, source and frequency of data to be collected Date achieved 02/01/2008 12/31/2009 10/19/2011 Comments (incl. % achievement) Indicator 6 : MHMS executive making decisions based on M&E reports The MHMS are The MHMS executive M&E framework making decisions Value place minimal priority on developed and based on a range of (quantitative the need for timely used by MHMS assessments or Qualitative) information to inform for decision including the M&E their strategic decisions. making framework Date achieved 02/01/2008 12/31/2010 12/31/2011 Comments The MHMS executive has been increasingly making informed evidence-based (incl. % decisions based on a range of assessments including the MTEF and data achievement) generated from program and provincial operational plans Indicator 7 : Balanced scorecards for provinces and national programs Value Balanced (quantitative None scorecards for Not progressed or Qualitative) provinces and national programs Date achieved 02/01/2008 12/31/2010 12/31/2011 Comments Balanced scorecards were not progressed because it was not deemed a priority (incl. % under the new leadership in the MHMS achievement) Provincial planning process including medium term plan and operational Indicator 8 : planning system adopted with strong M&E framework Provincial health plans with clear Enhanced role targets and The Ministry has a top- of provinces in indicators have Value down annual operational planning and been developed (quantitative planning system without implementing with the support or Qualitative) strong and timely M&E programs with from the planning framework a strong M&E team but the framework medium term plan process still remains fragile Date achieved 02/01/2008 12/31/2011 12/31/2011 Comments (incl. % This indicator was added to the results framework during the mid-term review achievement) Indicator 9 : Increased share of provincial health plans implemented Significant progress has been made in developing Provincial provincial plans and Provincial plan are top- plans budgets. However, down and often developed, Value quantitative comprised of a series of implemented (quantitative monitoring was in- activity lists without a and monitored or Qualitative) adequate. Some strategic program focus or for progress provinces have M&E framework against agreed reported difficulties targets with implementing all planned activities Date achieved 02/01/2008 09/13/2010 12/31/2011 Comments Progress reports to be submitted in 2nd quarter of 2012 will identify the (incl. % challenges faced by the provincial teams. achievement) Self-assessment by trained provincial health directors 3 months after completion Indicator 10 : of management training Formal training was limited due to high Value Self-assessment of Senior workloads, frequent (quantitative management skills prior management staff overseas travel, and or Qualitative) to training undertake training a significant number of alternate sponsorships became available. No formal self- assessments were made Date achieved 02/01/2008 12/31/2011 12/31/2011 Comments (incl. % achievement) G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 10/24/2008 Moderately Satisfactory Moderately Satisfactory 0.00 2 06/30/2009 Moderately Satisfactory Moderately Satisfactory 0.00 3 05/22/2010 Satisfactory Moderately Satisfactory 0.00 4 05/11/2011 Satisfactory Moderately Satisfactory 0.82 Moderately 5 03/11/2012 Moderately Satisfactory 1.04 Unsatisfactory H. Restructuring (if any) ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in USD millions There was considerable delay in project implementation which the Ministry acknowledged was mostly due to lack of understanding of the project arrangements within the 08/26/2010 N S MS 0.20 Ministry itself and to a high turnover of senior staff in 2009. Restructuring enabled the grant closing date to be extended from March 31, 2011 to December 31, 2011. I. Disbursement Profile 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. The project was developed as Solomon Islands emerged from three years of serious civil conflict and three years of “emergency recovery� to a medium-term development program. At the time of appraisal, the Solomon Islands Government (SIG) was paying increased attention to medium-term strategy and planning at both the national and sectoral levels. The Ministry of Health and Medical Services (MHMS) had prepared: (a) its strategy document – The National Health Strategic Plan 2006-2010 (NHSP) with eight priority areas 1; and subsequently, (b) the Health Sector Support Program (HSSP) 2 a more detailed program designed to operationalize the NHSP with support of Development Partners (DPs). 2. The initial intention for HSSP was to have a pooled funding arrangement under a sector wide approach (SWAp). Significant staff weeks were committed to shaping this design with SIG, AusAID and other DPs (refer Annex 4b). However, after two years of project preparation, agreement could not be reached on fiduciary assessments and related requirements so HSSP was designed as an over-arching program with complementary financing of a single sector program led by the Government and in coordination with several DPs. Challenges for getting the health sector back onto a sustainable medium- term development path included that DPs financed about 50 percent of both total and recurrent sector expenditures, and concerns about the technical quality of health services and capacity of the health system to deliver services effectively and efficiently. 3. Several aspects of HSSP required further development, specifically, the sector’s overall resource allocation picture was not clear. The initial partial Medium Term Expenditure Framework (MTEF) developed as part of preparation of the HSSP did not reflect the full sector envelope (government plus donors) and the government had yet to revise expenditure priorities to allocate health resources in alignment with its strategy. In addition, the possibilities for efficiency gains, equity concerns and the sustainability of sector financing had yet to be fully explored. There was an unclear division of responsibility between provinces and national programs. Finally, the monitoring and evaluation (M&E) framework needed to identify a core set of key performance indicators to measure the effectiveness, efficiency, equity, and sustainability dimensions of sector performance. 4. This context provided the rationale for Bank support through a Technical Assistance (TA) Grant to improve the capacity of the MHMS in the selected areas of public expenditure management, de-concentration of services to provincial managers, sector performance monitoring, and management training. It was agreed that such capacity development would be critical to the overall success of the HSSP and that 1 The Solomon Islands National Health Strategic areas were: (i) people focus; (ii) public health programs; (iii) malaria; (iv) common childhood diseases; (v) non-communicable diseases; (vi) HIV/AIDS and STIs; (vii) family planning and reproductive health; and (viii) health systems strengthening. 2 The HSSP focus areas were: (i) community focus; (ii) priority health programs; and (iii) health system strengthening. 1 assistance with the development of key building blocks for the medium-term development of the health sector was critical. Furthermore, the Bank’s sectoral expertise and experience in sector wide operations and fragile and post-conflict countries were considered of value by the client and the key DPs. 1.2 Original Project Development Objectives (PDO) and Key Indicators 5. The PDO was to improve the institutional capacity of MHMS in the areas of public expenditure management and sector performance monitoring. This was critical to the overall success of the HSSP in improving health services delivery and outcomes. The PDO indicators were: (i) completion of the first rolling MTEF; (ii) increase in the proportion of health sector budget expended at provincial level and on primary health care services; (iii) completion of a health facility survey and use of the results to improve health sector planning; and (iv) quality, participatory annual joint sector reviews conducted on schedule. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 6. No revisions were made to the PDO during the mid-term review. Two intermediate indicators were added to the results framework to better monitor PDO achievement. 1.4 Main Beneficiaries 7. The primary target groups were stakeholders at both the national and provincial level. Key beneficiaries at the national level were to be MHMS executive, finance and planning staff through enhancing their capacity in public expenditure management and monitoring of the health sector. Provincial level managers would be better equipped to plan ahead and to work within a more predictable resource envelope to more effectively allocate and manage resources to improve provincial level health outcomes. 8. Secondary beneficiaries, in the form of health service users, were expected to benefit through the positive impact of the project on the HSSP with its focus on: (i) communities; (ii) reorientation of public health resources away from the center to the provinces; and (iii) emphasis on improving the equity of health outcomes. 1.5 Original Components 9. Component 1: Public Expenditure Management (US$545,000) aimed to: (a) support strengthening of MHMS (national and provincial) health planning and budgeting systems and procedures; and (b) build on work done during preparation to ensure priorities were set within a known resource envelope and linked to expected results. The focus was on: (i) completing and updating the health sector MTEF annually; (ii) strengthening provincial planning and budgeting procedures and monitoring; (iii) improving understanding of the costs and efficiency of health facilities; and (iv) increasing community and household engagement in health care. 2 10. Component 2: Sector Performance Monitoring (US$382,000) aimed to support the MHMS to develop the M&E framework for the HSSP covering the key dimensions of sector performance (effectiveness, quality, efficiency, equity, and sustainability) and integrate M&E into sector management. The component aimed to enable the MHMS to better track achievement of NHSP objectives, and to provide feedback through the annual budget planning and review processes so adjustments in resource allocation were linked to priorities. This would enable improved sector policy and management decisions based on evidence. This information would facilitate the annual joint sector (HSSP) review process and meetings of MHMS and DPs. 11. Component 3: Training and Capacity Building (US$523,000) aimed to strengthen the management capacity of MHMS, including provincial managers to be more effective in work program planning and execution. 1.6 Revised Components 12. There were no formal revisions to the main Project components. 1.7 Other significant changes 13. Producing the new NHSP 2011-2015 (new NHSP) was a new activity which was achieved by combining existing elements from the project components. It was fully consistent with the objective of developing capacity to ensure expenditure priorities reflect desired efforts to improve health services delivery. 14. Closing Date Extensions. The closing date was formally extended for nineteen months in August 2010 from March 31, 2011 to December 31, 2011 primarily due to the significantly delayed start-up of the project. This enabled key project activities to be implemented and the PDO to be more fully achieved. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 15. The overall quality at entry is rated as Moderately Satisfactory based on the following assessment: (i) the soundness of the background analysis - initial project preparation activities focused on trying to forge an AusAID-International Development Association (IDA) pooled financing model for sector support with all DPs supporting and signing a Joint Partnership Agreement with MHMS. The joint donors, particularly the Bank and AusAID, supported a strong preparation effort to develop the key building blocks for pooled financing under a SWAp 3. While initial analysis had suggested the 3 Relevant project preparation activities included: (i) a health sector strategy note; (ii) a qualitative study of health care decision making in the Solomon islands; (iii) an initial but partial health MTEF; (iv) a health expenditure review; (v) a 3 key building blocks for a SWAp were present (i.e. sound sector priorities; SIG ownership; agreement on desirability of pooled financing; and initial relatively positive fiduciary and procurement assessments) experience gained during preparation indicated it was not possible to gain consensus within SIG or with DPs on how to proceed on all fronts. Specifically, project preparation had taken two years; leadership within the MHMS had changed and was less committed to pooled funding; the central agencies became concerned about their capacity to manage the financial/fiduciary aspects of the project; and there was inadequate DP support for procurement and financial management processes acceptable to the Bank. A pooled fund SWAp model for the project was finally rejected in favor of a TA Loan/Grant (TAL), where SIG and key DPs adopted a model of complementary (parallel) financing of a single sector program led by SIG. This was an appropriate decision and fully consistent with the lessons learnt from the previous Bank-supported SIG Health Sector Development Project and other sector projects throughout the region. The main constraints to improving the health sector lay in resource allocation, management capacity, and a lack of effective performance monitoring. The project was designed to specifically address these gaps in the existing HSSP; (ii) the suitability of project design - was appropriate to meet sector needs as identified during project preparation. The objectives and components of the project were clearly identified by MHMS and DPs as strategic building blocks for both health sector development and to enable a pooled SWAp to be developed in the future – a continuing objective of both SIG and DPs. Importantly, SIG remained committed to their leadership of one HSSP program supported by DPs and the project design supported this. The focus of the project was on the development of core government systems (particularly provincial budgeting, sector expenditure planning and budgeting, and an M&E framework) that would assist HSSP implementation, and make a full SWAp feasible. A SWAp, all parties agreed, held out the prospects of reducing sector transaction costs while still driving for immediate results, particularly reallocation of resources towards expressed priorities. The final design recognized that DPs needed to learn to work collectively in support of one program and MHMS would also need to lead a collective effort in support of the HSSP. The final TAL design significantly reduced the range of Bank safeguard policies involved compared to the initial approach (e.g. environmental and land acquisition/involuntary resettlement policies) and reduced the scale of financial expenditure and procurement oversight; (iii) the adequacy of participatory approaches and government commitment- the joint World Bank and AusAID preparation and design missions facilitated extensive consultations with ministers, senior SIG officials, national and provincial MHMS staff, key health sector stakeholders, and DPs. There was strong participation of MHMS program directors jointly with provincial directors and some of their staff. Church agencies providing health services and health-related Non-Government Organizations (NGOs) also participated in project planning sessions. These discussions confirmed ongoing support from SIG and key stakeholders for the HSSP report on annual planning and budgeting system; (vi) a review of hospital services and their management; and (vii) a review of pharmaceutical services, procurement and distribution arrangements 4 framework. The MHMS increasingly recognized the need to work more closely with provinces and NGOs – a positive achievement of the preparation process; and (iv) the approach to risk and mitigation - the overall project risk was rated as high given the country and sector contexts. The assessments were realistic ratings of the risks and the identified mitigation measures appropriate. 2.2 Implementation 16. Overall implementation progress is rated Moderately Satisfactory as SIG was able to make reasonable progress implementing most activities by the end of the Project (as outlined in more detail in Annex 2). This was despite major changes in the MHMS senior management team during the project lifecycle, and the ongoing capacity challenges of a small workforce in a resource constrained environment. 17. Project leadership and staffing had a significant impact on implementation. During the first year of the project senior MHMS management were in a state of uncertainty about their future and this reduced focus on the project in the initial start-up phase. In the second quarter of 2009, a new Health Minister, Permanent Secretary, and an almost completely new MHMS senior management ‘executive’ team were appointed. The new executive was initially unclear about the project, its objectives, and implementation arrangements and how this worked with the HSSP and a major new AusAID funded parallel malaria project. Once the arrangements were clarified the new MHMS executive endorsed the project and its objectives but also requested support for: (i) management mentoring; and (ii) developing the new NHSP 2011-2015. The Bank agreed to support this as it was fully consistent with project objectives. MHMS engaged a senior health policy advisor to support: (a) a participatory planning approach for the new NHSP; (b) development of an M&E framework for the sector; and (c) on-demand management mentoring advice to the executive. This, together with the work of the MHMS-appointed financial management specialist under the project, proved decisive and reenergized implementation. This work linked expenditure management and M&E to revised health sector priorities through a participatory process for strategic planning involving provincial and program managers, NGO partners and DPs. 18. Fiduciary capacity was the other major factor influencing project implementation; this is outlined in more detail in section 2.4. 19. Beyond immediate project fiduciary concerns, some technical aspects of provincial budget structures (chart of accounts) and processes, controlled by the central agencies were outside the control of MHMS, and proved more difficult than anticipated at appraisal. This made the separation of hospital and primary health care programs at the provincial level more difficult, adversely affecting some aspects of the desired budget reporting and provincial planning processes within MHMS. This remains a work in progress, but capacity developed under the project will enable MHMS to adjust to and reap the gains of these wider reforms when implemented. 20. The Mid Term Review reconfirmed the PDO and expanded the results framework to better monitor on-going activities. The MHMS executive indicated it had come to understand better the effective use of TA. TORs and Bank dialogue with the executive 5 emphasized the desirability of TA to undertake “process� rather than “task� consulting. Using this ‘learning by doing’ process approach enabled the staff of MHMS to undertake the work themselves within strategic frameworks/ government systems. The new NHSP support (including the M&E design) and financial management assistance used this approach and it is likely this will improve sustainability of the project post completion. Key staff sent on training, including degree level training for one officer, will also assist in this regard, although continued staff mobility within MHMS remains a problem. 2.3 Monitoring and Evaluation Design, Implementation and Utilization M&E Design 21. Project monitoring was designed to be undertaken within the M&E framework of the overall health sector which was to be developed further under the project. This was developed in draft only late in the project as part of the new NHSP and is now being refined after discussion at the September 2011 MHMS/DP sector meeting. Major efforts are now underway to reinvigorate the health management information system (HMIS) to supply the desired monitoring data. The MHMS decided not to seek support to design and implement a balanced score card for provincial and national health programs (a project activity) but rather to focus on the overall HSSP M&E framework. 22. Some project indicators relied on qualitative information which were not well specified and/or easily collected (e.g. increased community participation in planning at provincial levels, executive making decisions based on M&E reports, percent of provincial health plans implemented) and others turned out to be more difficult to measure because of chart of accounts issues (e.g. separation of hospitals from primary health care) which required considerable reworking of accounts which was achieved mid-project with considerable effort. M&E Implementation 23. The project envisioned that participatory joint annual reviews (with SIG and DPs) of the HSSP would take place to monitor sector performance. This was an agreed “building block� for the SWAp. To reduce transaction costs, it was agreed that MHMS would not prepare separate M&E systems or reports for individual DPs. The Joint Annual Performance Reviews would discuss the progress of the health system and health indicators and their quality. Notwithstanding the delays in finalization of the M&E framework, the existing (partial) framework was used for the ongoing sector monitoring. The biggest constraint was the availability of data from the HMIS which faced long standing technical constraints processing health facility data. This is now being fixed as proposed by the M&E framework and it is expected that real time data will become available in July 2012. A Demographic and Health Survey 2007 (DHS) and analytical work undertaken by the Bank on health financing options 4 drawing on the Household Income and Expenditure Survey provided key sector performance benchmarks and helped frame dialogue on the new NHSP and M&E framework. 4 World Bank, (2010). Solomon Islands Health Financing Options 6 24. The project results framework did not work as originally planned. The agreed benchmarks for a number of project indicators were not able to be monitored as intended. Nevertheless, the MHMS executive did monitor key aspects of the project and MHMS staff willingly undertook dialogue with the World Bank team and other DPs on the sector’s progress, including that of the project. A key project M&E process adopted was by reporting through review missions and documentation within relevant aides memoire. Bank aides memoire became the “de facto� DP report on sector performance and Bank project progress, and was used as such by the MHMS executive and DPs. M&E Utilization 25. The M&E draft framework developed for the new NHSP was only developed in late 2011 just prior to project completion. Nevertheless, the MHMS undertook annual sector reviews based on available data from project inception. Presentations by program managers and provincial managers at the Joint Annual Performance Reviews indicated they had at times, significant high quality data not reflected in the formal HMIS data bases, that: (a) underpinned their decision making on service priorities; and (b) informed dialogue between MHMS and DPs. 26. Available HMIS data, the World Bank Health Financing Options Note, the DHS 2007, and a situational analysis prepared by MHMS and DP/MHMS dialogue underpinned the decision making on the new NHSP and many HSSP decisions. The voice given to provincial managers through their participation in these processes also led to demands for change of national program priorities. This process also made decision making more transparent and contestable. Provincial and national program managers believe it also positively influenced annual planning and budgeting processes, an intended outcome of the project. While it is hard to demonstrate the extent of all the program priority changes as yet, the proportion of expenditures on primary health services increased from 17 percent to 29 percent (target 22 percent) and the portion of provincial spending is planned to increase while national programs decrease. The new NHSP outlines both significant reforms on program priorities and of organizational and management priorities. Increased commitment by MHMS to evidence-informed policy making, reinforced and buttressed by a range of project activities and agreements, is a significant outcome of the project. The MHMS executive fully recognizes this is both a new approach and that the processes will need to continue to be reinforced by DPs under the HSSP. 2.4 Safeguard and Fiduciary Compliance Procurement 27. This was rated a “high risk� at appraisal and an action program was agreed, most of which were upfront actions (procurement plan, high levels of prior review, on-going procurement advisory support, and a procurement manual). Provision was also made for upfront and on-going procurement training. Nevertheless, significant procurement delays emerged from the outset of the project and overall procurement performance was rated as “moderately unsatisfactory�. 28. Limited experience and capacity in a very small team was the major constraint within the procurement unit (the two staff involved had changed since the previous Bank 7 project). Management understanding and direction on the procurement processes, particularly in the early years of implementation, was limited. Continued staff mobility within the procurement unit and within the executive exacerbated these capacity constraints. Although MHMS had agreed to recruit an international procurement advisor for the period of the project (financed by AusAID under HSSP), MHMS was not able to attract and retain a suitably qualified specialist for the required timeframe. When a procurement advisor was appointed nine months after effectiveness, there was a misunderstanding about the advisor’s role in assisting the MHMS with Bank-financed project procurement. This significantly constrained the ability of MHMS to complete on a timely basis the initial procurement actions, including for two key advisors. The Bank subsequently responded to a MHMS request for assistance by initiating more intensive: (a) implementation support, including for procurement; and (b) joint dialogue with DPs and MHMS on implementation and performance monitoring. The Bank also reached a new agreement with MHMS and AusAID on continued procurement advisory support, and one member of the executive was made responsible for procurement to prevent continued procurement delays. Nonetheless, procurement delays throughout the project were significant, resulting in the initial two major TA positions being filled 16 months after project effectiveness, and the first disbursement occurring 24 months after effectiveness (although some expenditure had been pre-financed by MHMS). Financial Management and Auditing 29. Financial management of project accounts was rated as “moderately unsatisfactory� or “unsatisfactory� throughout its life, with incomplete or typically misplaced filing of paperwork on project files. Again, limited capacity in a small team was the main constraint. Project accounts were audited by the Government’s Auditor General. The audit for the period ending 31 December 2010 was over nine months late and was qualified by exception; the qualification did not identify a material misstatement of the financial statements. The final audit is due in June 2012. Interim Financial Reports were submitted late and then often needed further work to satisfy World Bank financial management requirements. Withdrawal applications were often delayed even though expenditures had been incurred (the government pre-financed project expenditures). A key staff member took long-term study leave and this further delayed withdrawal applications at one point. MHMS appointed a part time financial management specialist to assist with broader sectoral budget and accounting work under the project, and this also helped to improve the project disbursement process and capacity through training and technical support. Financial management support from the Bank’s fiduciary implementation support team was limited, and not well co-ordinated with broader Bank implementation support missions. During the last quarter of the project, the MHMS hired a new experienced Finance Controller to buttress the MHMS finance team; this has been an important step to improving management capacity and oversight in this core area. 30. The project did not trigger any safeguards policies. 2.5 Post-completion Operation/Next Phase 31. The Bank, MHMS and AusAID together with other DPs are actively discussing an ongoing work program to support the health sector. There is a consensus and commitment to improving the implementation and results focus of the HSSP with an 8 emphasis on how best to implement the strategic directions of the new NHSP. MHMS has committed to a work program to further improve sector governance and organizational reform. This includes coordination of technical cooperation, finalization of a role delineation policy to define the range of services across the country, improvements to financial and procurement management, including at the provincial level, and strengthening sector M&E. 32. In the immediate future the World Bank team plans to support MHMS through a Bank executed Trust Fund arrangement for the Pacific. Initially, this will finance further health expenditure analysis and assist with updating the MTEF, including setting out and tracking provincial funding allocations and expenditure in line with the directions set out in the new NHSP. Other areas, including health workforce issues, costing and efficiency analysis of national and provincial hospitals, options for public-private partnerships, and a health services plan for Honiara are under discussion. There will also be continued support to improve budget planning, preparation and execution at provincial and national levels. The Bank is currently recruiting an experienced health economist to be based in the Bank’s Honiara office. Although the position will have regional responsibilities, this will be a significant additional commitment by the Bank to sustain its work on the health sector in the Solomon Islands. 33. These efforts are fully consistent with the proposed focus on service delivery under the new Country Partnership Strategy being developed by the WB and SIG. The Bank and other DPs are also assisting SIG with a financial management reform program. This will assist with improving the relationship between central agencies and sectors and potentially set the basis for a health sector focus under a possible next phase of budget support provided under a SIG/WB Development Policy Grant. There may also be scope with these reforms, to explore a future health specific grant using the WB ‘Programming for Results’ instrument focused on improving service delivery at the provincial level. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 34. The PDO: to improve the institutional capacity of MHMS in the areas of public expenditure management and sector performance monitoring continue to be relevant to both current sector and country priorities and remain consistent with the Bank’s Interim Strategy Note FY 2010-2012 (ISN) for Solomon Islands – a strategic pillar of which includes: “supporting improved public administration and management.� The new NHSP 2011-2015, developed with project support, reaffirms the need to focus on: (a) health priorities based on evidence; (b) further de-concentration of health program expenditures and management, including provincial planning and budgeting, to get resources to where they are needed most; (c) overall financial management; (d) M&E; and (e) continued updates of the MTEF to underpin improved focus on expenditure priorities and efficiencies. The MHMS Corporate Plan 2011-2015 reiterates these points. This policy focus is fully consistent with the PDO. Project components remained relevant to the needs of the MHMS and targeted clearly identified priorities in the overarching HSSP. Sector stakeholders supported HSSP objectives, particularly as refined and included in the new NHSP. 9 35. The project design, including its framing within the context of joint efforts to establish a health sector SWAp, remained consistent with Bank priorities through both the World Bank’s Regional Engagement Framework FY 2006-2009 (which highlighted the importance of improving the effectiveness of public expenditure in the social sectors through strategic partnerships with key DPs); and in the ISN, which recognizes that changes in administrative arrangements and management of public resources are essential to improved service delivery. DPs, as outlined in section 7b, also strongly supported the focus and continued relevance of the key objectives and activities of the project. 3.2 Achievement of Project Development Objectives ICR Rating: Achievement of PDO is Moderately Satisfactory. 36. Project Outcome Indicator 1: Completion of Medium Term Expenditure Framework. A substantial draft was completed in 2010 and a final initial MTEF was produced by MHMS in 2011. Although the MTEF was significantly delayed it is now completed and informing dialogue on: (i) strategic resource allocation priorities; and (ii) incremental recurrent costs of sector investments and hence the fiscal sustainability of the program. The first annual update is planned for third quarter 2012.The MHMS recognizes this is the only source of detailed “whole of sector� health expenditure and its financing as the development budget processes are weak and do not provide complete pictures of sector expenditures. Thus the MTEF also greatly assists policy dialogue and dramatically improves the transparency of health expenditures and their financing. The central agencies are now beginning to develop a “whole of government� MTEF process so there is increased reason to believe the MTEF process will become institutionalized. 37. Project Outcome Indicator 2: Increase from 17%, the proportion of health sector expenditure on primary health services and provincial health programs. The proportion of health sector expenditure on primary health services and provincial health programs has increased from 17% (estimated base at appraisal) to 29% for FY11 (target 22%). (Source: Final MTEF 2011) 38. Project Outcome Indicator 3: Completion of Health Facility Survey. This survey was not undertaken as planned (see Annex 2). The MHMS delayed its initiation due to: (i) concerns about sequencing of other work and their capacity constraints; (ii) overlap with another AusAID study of infrastructure needs; and (iii) proposals from UNICEF to undertake a costing study using the tool Marginal Budgeting for Bottlenecks (later rejected by MHMS). Finally, MHMS decided, as a result of policy dialogue, to proceed with hospital and facility costing/survey work in the last year of the project. However, delayed procurement actions meant this work was not commissioned on a timely basis. Confirming the on-going relevance of this activity the MHMS is pursing options with DPs to finance a similar survey(s) now (see 2.5 above). 39. Project Outcome Indicator 4: Joint Annual Performance Reviews. Joint Annual Sector Performance Reviews are now held annually with the participation of Provincial Health Directors, program managers, DPs, NGOs, and faith-based organizations. Although the quality of the review processes and data availability varied considerably over the project period, the trend was in the right direction with the 10 2011review enabling MHMS and DPs to jointly reflect on the gains of the health sector and to identify specific areas for improvement (also supported by the first MTEF). The operationalization of the HMIS will also greatly improve the quality of this process in the future. 40. Component 1: Public Expenditure Management. This component focused on improving sector resource allocation planning and expenditure management and efforts to de-concentrate to the provincial level consistent with sector priorities. The aim was for: (i) the percentage of planned activities completed increased from 60% to 75% - this indicator was inadequately specified and not monitored quantitatively. However provincial health staff have indicated improved satisfaction with planning within budget constraints and recognized some difficulties in fully implementing planned activities; (ii) increased community participation in provincial health services planning - the level of beneficiary/stakeholder participation in the development of both the new NHSP and of provincial operational plans has been significant. Efforts intended to extend this down to facility level has been problematic and remain a challenge – constrained by over stretched provincial managers. Nevertheless, the achievements on stakeholder participation should not be underestimated, including getting Provincial Premiers and community groups involved in planning and monitoring discussions; (iii) the health facility survey to be completed and analyzed - this did not proceed as discussed in paragraph 38 although alternative data sources (including the health financing note) provided some important insights; and (iv) (added at mid-term review) budget planning, development and accounting processes enhanced and institutionalized - annual provincial and national operational plans with SIG and DP budgets are now developed routinely, and accounting of expenditures on MYOB now allows separation of hospital and primary health care at the provincial level. 41. Component 2: Sector Performance Monitoring. This component focused on improving the sector performance monitoring system. To this end the component envisaged: (i) development of a comprehensive M&E framework - this was developed as part of the new NHSP and identifies agreed priority indicators for key programs and for organizational and management reforms and details the baseline, target, source and frequency of data to be collected. Financial management, macroeconomic and microeconomic indicators are detailed in operational plans and the MTEF; (ii) MHMS executive would make decisions based on M&E reports - the MHMS executive has been increasingly making informed evidence-based decisions based on a range of assessments including the MTEF and monitoring data generated from program and provincial operational plans. The new NHSP was based on quantitative and qualitative data generated from a situational assessment and from available M&E data. The evidence based process will be improved when the HMIS and the M&E framework is fully operational in 2012. Evaluative work, including the MTEF and the Bank generated Health Financing Options Paper, also informed decision making. The NHSP Planning and Evaluation Advisor provided strategic mentoring advice and guidance to the MHMS executive team on the critical importance of M&E; (iii) the use of balanced scorecards for provinces and national programs - these were not progressed as it was not deemed a priority under the new leadership of the MHMS; and (iv) (added at mid-term review) provincial planning process including medium term plan and operational planning system adopted with strong M&E framework - provincial health plans with clear targets 11 and indicators have been developed with support from the planning team but the medium term plan process still remains fragile. 42. Component 3: Training and Capacity Building. This component focused on improving the human capacity in the health sector. It was envisaged that this would be monitored by: (i) self-assessment by trained provincial health directors 3 months after completion of management training - formal training was limited due to high workloads, frequent overseas travel, and the availability of a significant number of alternate sponsorships. No formal self-assessments were made although those attending the Bank sponsored Health Systems Flagship course indicated it underpinned their sense of the usefulness of the MTEF (because it set out total expenditures and how they were financed), setting priorities, M&E and the importance of planning within a budget constraint; and (ii) the increased share of provincial health plans implemented - significant progress has been made in developing provincial plans and budgets, however quantitative monitoring was inadequate. Some provinces have reported difficulties with implementing all planned activities particularly when they were developed without reference to budget constraints. Progress reports to be submitted in second quarter of 2012 will better identify the challenges faced by the provincial teams. 43. While performance is mixed, outputs under these components have contributed towards achieving the PDO. The budgeting system, particularly at provincial level, is now more firmly established and disaggregated to show separate primary health care and hospital expenditures. Managers now plan within a relatively stable and known resource envelope and budget execution is monitored on a regular basis. On a number of occasions, project outputs helped MHMS to argue with some success an evidence-based case when SIG proposed budget reductions due to broader fiscal issues. MHMS has noted that it could not have made these achievements without the project. 3.3 Efficiency 44. The project appraisal did not undertake a cost-benefit or other formal analysis of cost-effectiveness. On this basis, the ICR has not undertaken a cost benefit analysis. Nevertheless, the appraisal did assert that the project would: (a) assist to improve allocative efficiency of health expenditure by aligning allocations to priority needs; (b) improve the technical efficiency of health expenditures by providing the tools and improved information; and (c) improve equity in resource use by providing information to form the basis of reallocating resources to improve regional equality. 45. The project has been partially successful in this (increased resources were allocated to primary health care and to provinces) and the new NHSP strongly commits the MHMS to improved allocative efficiency. The MTEF has demonstrated the scale of the challenge to redirect both domestic and DP sector financing towards agreed priorities. The MTEF, for the first time documents the “whole of sector� expenditure program and how it is financed. Sector analytical work undertaken by the Bank has shown that access to health is remarkably equal in Solomon Islands compared to most countries with a similar income. However, work at the provincial level shows that on a per capita basis, health resources are unequally distributed. Analytical work generated by the project was 12 used by MHMS to reallocate provincial grants on a more equitable basis. The challenge will be to reallocate staffing according to need. 46. The health facility work to estimate unit costs and service indicators would have added to the richness of the information available to inform decision making. However, this activity was not progressed as the new NHSP was given precedence by MHMS. It is clear that MHMS and all DPs now see the benefit of this costing information and some of this will be included under future support. 3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 47. The overall rating is based on the continued relevance of the objectives and the extent to which they were achieved. Project objectives remain highly relevant to the current strategic priorities of Solomon Islands and the World Bank. MHMS used the project to set a sound foundation for ongoing sector development through (a) the new NHSP which defines MHMS’ overall aims and M&E arrangements for the next five years; (b) completion of the first MTEF; and (c) strengthening of the provincial planning and budgeting. These elements have helped identify and improve allocative efficiency of health expenditures (in a situation where details of total “whole of sector� expenditures were not known) and increased the overall policy dialogue between SIG and DPs on core sector priority issues. 48. The specific outcomes and outputs of the project are documented in Annex 2. Despite delays the MTEF was finally completed and will be continued on a rolling basis. It will continue to support planning within realistic resource envelops at national and provincial level. The new NHSP with its strategic priorities and M&E framework has very strong ownership within MHMS and with SIG more generally. Provincial planning and budgeting capacity has been developed considerably – including finding a way to meaningfully disaggregate expenditures between primary health and hospital care at the provincial level. Provincial primary health care expenditures have reached 29 percent of expenditures up from 17 percent and over the target of 22 percent. Formal joint MHMS/DP annual performance reviews are now conducted routinely. The MHMS and DPs will, in the future be able to use the M&E framework, supported by the improved HMIS data, about to come on line. Considerable training, including planning workshops with a “learning by doing� approach as with the new NHSP and the development of budget and accounting systems were very successful. Planning and budget execution have been strengthened – albeit from a very low base. 49. Given the thin capacity in this small post conflict country these achievements are considerable. Nonetheless, it is recognized that more could have been done to prevent and manage delays. MHMS recognizes they need additional support moving forward and this will be provided through new World Bank and other DPs assistance. 13 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 50. Bank analytical work showed access to health services by the poor was quite equitable compared to other low income country. Work under the project led to important adjustments on provincial equity grounds in the distribution of national grants to provinces as a result of work undertaken under the project. The new NHSP has a reasonably strong gender focus with its increased focus on mother and child health and family planning. Disaggregated data by sex is a key feature of the M&E framework and MHMS actively supports training opportunities for women. The process of producing the new NHSP involved provincial premiers, community and NGO stakeholders which was a noted improvement on previous efforts. Annual provincial planning is increasingly involving communities although this needs further strengthening – including additional support to overstretched provincial directors. (b) Institutional Change/Strengthening 51. Strengthening the capacity of the MHMS to effectively manage and monitor health expenditure and outcomes is the main aspect of the PDO. Important progress has been made on this but all parties recognize that the institutional development and strengthening has been moderately satisfactory as it takes a long time in fragile states with weak systems to develop capacity. It has to be recognized that the existing inadequate development budget processes mean sectors (and SIG) do not have an accurate picture of resources planned to be spent and how they are financed. The MTEF documented this for the first time. Using a “process� consultancy approach for the development of the new NHSP and with the financial management work in the second phase of this work program clearly helped build capacity through its “learning by doing� approach. As a result ownership of MHMS has clearly improved as a consequence. The new NHSP sets out a significant organizational and management reform agenda which is forming the basis for ongoing dialogue and support from DPs. (c) Other Unintended Outcomes and Impacts (positive or negative) 52. Not Applicable 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 53. Not Applicable 4. Assessment of Risk to Development Outcome Rating: Moderate 54. The risk that development outcomes will not be sustained and deepened in the future is assessed as moderate. There are real countervailing considerations in this assessment. On the negative side there are risks that: (a) leadership of the MHMS will change resulting in a reduced commitment to the project (particularly decentralization) objectives; (b) the strength of the national program directors to resist de-concentration of 14 health programs to provincial managers will prevail; and (c) that the broader government does not continue with its financial management reform agenda or does not maintain a focus on financial and planning reforms. 55. On the positive side there is considerable evidence that most of the reforms will be adequately sustained. First, the core of the reforms supported by the project was clearly identified in the new NHSP. Second, as discussed in section 2.5, the MHMS has requested ongoing support from the Bank (in cooperation/partnership with AusAID) to support the ongoing implementation and deepening of these reforms. MHMS and other DPs (including AusAID) are in the process of agreeing new financing support for the health sector and while these are not yet finalized the ongoing dialogue is premised on the basis that the key building blocks for further development of the HSSP will require on-going work in these areas, including decentralization of health programs (provincial planning, budgeting and accounting of funds for service delivery at the provincial level). Third, decentralization is a strong theme of the current government and in many ways health is showing the way forward for other sectors. Finally, the central agencies have recently decided: (a) to formally create and implement a chart of accounts which will allow for significant decentralization of the financial management of service delivery including health; and (b) to introduce a “whole of government� MTEF process. For these reasons, notwithstanding the challenges that remain, the risks to development outcomes in the future are assessed as moderate. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 56. The preparation was supported by a strong well qualified Bank team. A situational assessment of the health sector was prepared and this was supported by initial information from a health module in the Household Income and Expenditure Survey undertaken by SIG with support of the preparation team. The final project design, as appraised, was focused on developing strategic building blocks necessary to: (a) increasingly devolve management of health services delivery to provincial managers within the MHMS; and (b) implement a results focused program of support for the HSSP jointly with DPs and government/MHMS in the future. This was a sound approach with continued strategic and policy relevance. The final project outcomes suggest the institutional development targets set for the project were ambitious but focused on the right issues, and as discussed, require continued support. The alternate approach of going immediately to a pooled funding SWAp (considered during preparation) without adequate building blocks in place, and a strong consensus with DPs on the approach, clearly would not have worked. However the work done under the project now makes this more feasible. In this regard DPs are now working with MHMS to work through the compacts needed to enable increased flexibility in the manner DP funding and TA is used to support service delivery programs. The government’s continued constrained fiscal position makes the focus of financial management reform and the MTEF appropriate. The MTEF focus on: (i) highlighting the misalignment of expenditures between HSSP 15 objectives; (ii) gaps in support for priority programs (by DPs and government); and (iii) the incremental recurrent costs arising from policy and investment decisions, underline that the project focus was relevant technically, financially and economically. 57. The risk assessments for the project at appraisal were accurate and reflected the considerable issues expected to be faced during implementation. Weak fiduciary arrangements were identified up front and a wide range of risk mitigations measures were adopted, including upfront actions. Nevertheless, reliance on another DP to maintain a suitable procurement advisor to support the MHMS procurement unit which was assessed as weak proved problematic. Financial management capacity, particularly maintaining project accounts was also weak and the mitigation measures also underestimated the problems to be faced by the project. This, while very important, relates to project accounts only and should not be interpreted to mean that the strengthening of the financial management of the MHMS and of provincial divisions overall were not well designed and prepared. The M&E system for the project was reasonably designed but was seen as a small part of the overall efforts until its relevance was highlighted as part of the development of the new NHSP. (b) Quality of Supervision Rating: Moderately Satisfactory 58. The quantity and quality of Bank supervision increased over the life of the project. During the first year, the Bank did not provide the level of implementation support that is needed in a limited capacity environment such as the Solomon Islands. Infrequent initial supervision (both technical and fiduciary) meant that the full extent of capacity and implementation constraints, particularly of procurement, was only fully realized one or more years into project implementation. This was exacerbated by turn-over of almost the entire senior management team of MHMS responsible for Project activities. From early 2010 supervision intensity increased and additional consultant support was added to the team to work with MHMS management and the TA engaged under the project, particularly for the initial provincial budgeting and planning support and the MTEF work. Increased Bank presence in the Solomon Islands and ongoing communication with the MHMS executive and DPs has meant the Bank managed to stay more fully engaged and increasingly responsive to implementation constraints over the final two years of project implementation. By participating in all main sector meetings with MHMS and DPs(at least three missions per year), the Bank has been able to engage effectively in a more coordinated support to the sector assisted by a period of strong leadership of the MHMS. It should be noted here that Bank supervision had to judiciously mix focus on the overall HSSP (working with MHMS and other DPs on the building blocks of the HSSP- an intended outcome of the project) and the specific fiduciary issues and institutional developments which were needed to manage the more narrow implementation aspects of the Bank project. Bank aides memoire were seen as very important monitoring documents for the overall HSSP by both MHMS and DPs. 59. It is evident that, in the final analysis, there was inadequate FM and Procurement support by Bank staff. Regional procurement and FM staff saw this project as relatively small and it was only on rare occasions that their supervision missions were coordinated with the main implementation support missions. This was seen as less important for 16 procurement because there was a very high level of prior review. There were only two formal and one informal FM reviews, all from March 2011. Thus very little was done to establish the basic system for project accounts at the outset, and as there was little or no expenditure (in part caused by implementation delays and subsequently limited capacity to prepare withdrawal applications), FM did not give the project a high priority. The first withdrawal was only made in mid 2010. In the final analysis the delays were primarily due to capacity constraints. The core problems associated with procurement and FM derived from these capacity constraints not from any desire to avoid the fiduciary protocols of the Bank. The technical task team provided considerable implementation support during missions and through email/phone to help compensate for this constraint. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 60. The overall rating of Bank performance is moderately satisfactory reflecting the two dimensions: quality at entry, and supervision/implementation support. While initial technical and fiduciary implementation support was inadequate for the limited capacity environment, the Bank task team increased its engagement as implementation issues emerged. From late 2009 this involved at least three missions per year as part of broader sectoral meetings, plus substantial email, phone and video conference communications. Implementation support arrangements facilitated the transfer of global knowledge through policy dialogue on the HSSP, the development of the new NHSP, and the establishment of core building blocks for development of the HSSP. Overall, this made a significant contribution to the health sector policy and management focus. 5.2 Borrower Performance Government/Implementing Agency Performance (on basis that government and implementation agency are inseparable for practical purposes). Rating: Moderately Satisfactory 61. At the time of writing the final ISR (March 2012) the 2010 audit was overdue by nine months, the resubmission of a number of IFRs were outstanding, and only 66 percent of the grant had been disbursed. For this reason the rating for Implementation Progress in the final ISR was Moderately Unsatisfactory. However, by the end of the disbursement period (April 2012) final project expenditure was 82 percent. The outstanding IFRs have now been submitted and accepted, and the audit completed qualification by exception; the qualification did not identify a material misstatement of the financial statements. While compliance with these project specific fiduciary requirements was inadequate, it is recognized that the small fiduciary team within MHMS was stretched managing their own Government budget and other much larger DP funding arrangements. The World Bank reporting and management requirements were an additional layer of work for a comparatively small funding allocation. However, when looking at project performance more broadly, the MHMS maintained a significant commitment to the objectives of the Project and the HSSP, and made good overall progress with its key project outcomes: the new NHSP with its M&E framework, the MTEF, and the achievements on provincial planning and budgeting. The MHMS has also delivered on many of its agreements including holding regular HSSP meetings that have supported high level policy dialogue for improved management of the sector. 17 MHMS staff have always made themselves available to meet with Bank Staff and other DPs. Based on this assessment the overall borrower performance is rated Moderately Satisfactory. 6. Lessons Learned 62. The MHMS at the time of project preparation, was trying to build consensus on a medium term health program, having relatively successfully stabilized service provision following the “troubles.� There were many DP projects financing different programs and one large AUSAID project managed by a contractor. Virtually all DP projects were managed outside of the SIG budget framework and there was no standard system (budget framework) for MHMS (or SIG) documenting planned expenditures and their outcomes. MHMS recognized they had no system for oversight and that the executive had no system to manage and monitor implementation of the overall health programs. DPs agreed and also recognized that the transactions costs for DPs and SIG were very high under the current arrangements. Governments and DPs had a growing commitment in the Pacific, consistent with global trends, to the development of SWAps as a way to reduce sector transaction costs, improve focus on outcomes and to support one agreed national strategy for the health sector. At face value, it seemed a SWAp, could address the key issues facing Solomon Islands. To this end, a significant part of preparation activities for the project focused on establishing the building blocks for a health SWAp. A key lesson learned is that the transaction costs to establish the preconditions for a pooled SWAp are very difficult in a fragile and low capacity environment. It requires a strong commitment of DPs to work together with a collective understanding of what this means as a way of doing business, including government execution of programs, use of government systems, and appropriate fiduciary oversight. There was significant conflict between DPs on these issues during preparation. The MHMS, understandably, was unable to exercise strong leadership on these issues as it was unsure how to proceed. There were also weak relationships with the central agencies of Government and they did not fully appreciate the issues either. Thus, many preconditions to establish a SWAp were absent. An agreement on upfront criteria to assess readiness of government systems for pooled financing would have been helpful. 63. The project team took the right decision to shift to a TAL focused on the core building blocks for the HSSP – support for one sector program; enhanced DP coordination; and joint monitoring and strengthening of MHMS budget and planning systems. This perhaps took too long because of the collective regional commitment to a SWAp without being clear on what the up-front readiness conditions should be. 64. A key question is whether a small project with high transaction costs is worth it? The transaction costs to both SIG and the Bank were significant. The view of this ICR is that the project was high risk with potentially high benefits and the Bank team took the right decision to prepare a TAL. The Bank was able to add value across the core domains of the project– a view expressed by the client and DPs (see annexes 7 and 8). This impacted positively on the wider HSSP – thus the small Bank TA grant leveraged the whole of the health program through its work on planning and budgeting systems, the MTEF, and most importantly the new NHSP and M&E. DPs indicated that the project 18 and Bank team greatly assisted MHMS and DPs to stay focused on the “big picture� i.e. how resources were being deployed, key gaps, and how outcomes were evolving. 65. Another lesson is that enhanced supervision and implementation support is needed, particularly in fiduciary management for projects, in countries such as the Solomon Islands, no matter what the scale of the project. Procurement and particularly financial management support from specialized staff in the Bank was inadequate and typically not coordinated with review missions. The separation of team responsibilities within the Bank for different parts of the supervision process in the recent past has reduced the effectiveness of the Bank in supervision of projects. A key lesson is that the Bank needs to develop a better business model for undertaking fiduciary supervision in small countries where it cannot justify locating fiduciary specialists in-country. 66. Another lesson is that persistence on the need for core Bank requirements sometimes pays off. The executing agency often complained strongly about the Bank’s insistence, under the project, to develop a procurement plan and follow it. The MHMS now openly recognize that this is precisely what they need for their own government development budget and are now planning one for their 2013 development budget because of significant delays in 2012 (and earlier) due to a lack of planning. 67. Finally, a core lesson of this project has been that the focus on the development of systems is “in and of itself� a capacity development strategy. In the latter part of the project TORs for TA, support was defined as “process� consulting i.e., providing tools and decision making frameworks for MHMS staff to make strategic decisions collectively. For the development of the new NHSP and for the finance advisor the TA support was specified as process support. This meant that outcomes were controlled by MHMS staff with the consequence that ownership of decisions and positions taken were clearly owned – a situation that is not always evident with a “task� consulting approach. It also means that much larger numbers of MHMS staff are involved and understand objectives and processes – a significant achievement. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 68. The MHMS believe the development objectives remained relevant and focused on key aspects of the health systems development. The new executive grew to better understand the benefits of the project as it was implemented. While small, the project was an integral part of the overall support to the HSSP. The MHMS had lost all institutional memory of the past project and the design, without a project management unit, was thus left to the executive to manage. This led to an implementation hiatus when the executive was changed. As an inexperienced executive there was limited understanding of the Bank’s focus on the procurement plan. This is now much better understood by the MHMS (other DPs undertook most major procurements) and it plans to use such an approach for the development budget. 69. The MHMS believe the project outcomes were significant. Highlighted is the new NHSP with its key programmatic priorities and organizational and management 19 reforms required to achieve the programmatic objectives. The M&E framework is also now well owned and its benefits better understood. The focus on budget systems development has been beneficial in developing capacity and allowing better understanding of how resources are being used. The support to the provincial planning process has been very significant and will enable further de-concentration of service delivery. The MTEF for the first time brought together “whole of sector� analysis of expenditures and how they are financed. The new executive was initially not sure of its benefits but now see the importance of completing it annually. The new NHSP and the budget and planning processes have involved extensive stakeholder and provincial involvement in health planning, budgeting and policy dialogue. This is an important development. 70. The MHMS recognizes that staff turnover and limited initial understanding of the project by the new executive led to project delays. Project delays were the core issue due to capacity constraints - not intent. Limited experience with recipient execution meant that implementation capacity needed to be developed. The overall rating of implementation agency performance is understood. 71. The Bank was very supportive on implementation issues during support missions and contributed to policy dialogue. This kept the focus on the strategic issues – particularly those related to overall expenditures and expenditure priorities. Its sector work provided timely strategic advice not otherwise available. Banks aides memoires were important mechanisms for the overall monitoring of the HSSP and the project. The MHMS also believes the support on financial management of the project from the Bank was inadequate. 72. With respect to the future the MHMS believes the MTEF, provincial planning and budget system reforms and the M&E framework are now well established and will be continued as a result of project support. It is recognized that additional and continued support will be needed for public financial management to deepen the reforms already in place. The MHMS is also keen to explore efforts to build on project achievements and consider results based financing at the provincial level. (b) Co-financiers 73. The DPs have all recognized the positive contribution of the project to the HSSP and of the Bank’s strong contribution through implementation support missions to the overall policy dialogue on health sector issues. It is clear that the DP commitment to a HSSP has grown and that trust between all parties, including of MHMS with DPs, has grown significantly over the past 2-3 years – trust that was not always there previously (see Annex 8). 20 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Appraisal Actual/Latest Percentage of Components Estimate Estimate Appraisal (USD millions) (USD millions) a Public Expenditure Management 0.545 0.578 1.06 Sector Performance Monitoring 0.382 0.366 0.96 Training and Capacity Building 0.523 0.286 0.55 Total Baseline Cost 1.450 1.230 Physical Contingencies/Unallocated 0.050 0.000 Price Contingencies 0.000 Total Project Costs 1.500 1.230 Project Preparation Facility (PPF) 0.000 Front-end fee IBRD 0.000 Total Financing Required 1.500 1.230 82% (b) Financing Appraisal Actual/Latest Type of Percentage Source of Funds Estimate Estimate Cofinancing of Appraisal (USD millions) (USD millions) Borrower 0.00 0.00 0.00 IDA Grant 1.50 1.52 1.01 TOTAL 1.50 1.52 1.01 a The final disbursed amount totaled SDR 817,666.94 which represents a disbursed amount of 82% of the original grant amount of SDR 1,000,000. 21 Annex 2. Outputs by Component PDO Project Outcome Indicators Comments The project development (i) Completion of the first rolling medium The MTEFs were delayed: an initial draft was objective is to improve term sector expenditure program linking completed in 2010 and a final version in 2011. the institutional capacity resources with priorities. of MHMS in the areas of The draft MTEF in 2010 formed the basis for the public expenditure The criteria to indicate successful completion definition of the overall resource envelope for the management and sector are: new NHSP and for the annual planning and performance monitoring. the activities are aligned with clear budgeting process in FY10 and FY11. The final objectives; the activities are budgeted in a MTEF in 2011informed the FY12 operational plans realistic manner; and and budget. Importantly the MTEF created and informed dialogue on (i) sector expenditure priorities The MTEF is indeed used and reflected in and formed the basis for some change – government the budget discussion, as evidenced by the and DP financed; and (ii) highlighted major future reallocation of resources towards priority significant and unplanned imposts on future recurrent areas from one year to the next. budget arising from both DP financed project investments (e.g. a new hospital and refurbished malaria building) and of policy decisions (e.g. returning Cuban trained doctors). A strategic decision of the new NHSP, based on the MTEF work, was to decrease national program recurrent budget funding by 14% and increase provincial programs by 17% by 2015. MHMS wants to periodically update the MTEF to inform planning and budget processes and ensure improved transparency of resource allocations. 22 Further, MOF&T is now planning to introduce a rolling national MTEF for all sectors over the coming months as part of the budgeting processes. (ii) Increased from 17% the proportion of The proportion of health sector expenditure on health sector expenditures on primary health primary health services and provincial health services and provincial health programs programs has increased from 17% (estimated base at appraisal) to 29% for FY11 (target 22%). (Source Final 2011 MTEF) (iii) Results of completed health facility This survey was not undertaken as planned. Initial survey used in health planning; inefficiencies priority, given limited management capacity in arising from inappropriate mix of inputs MHMS was to undertake the MTEFT and provincial would be identified and corrected by planning work. In the initial project hiatus AusAID reallocation of inputs; variations in unit costs went ahead with a facility survey focused primarily and occupancy rates used to identify but not exclusively on infrastructure. This finally inefficiencies in production and reallocate provided some important information for the resources appropriately. provincial planning processes and the MTEF. UNICEF entered into discussions to undertake a costing study using the tool Marginal Budgeting for Bottlenecks. Finally, the MHMS also decided not to proceed with the study and to focus on completion of the new NHSP. MHMS decided to proceed with hospital and facility costing/survey work in the last year of the project. Sector dialogue generated a demand for this work. However, time constraints, arising from international travel of the executive, meant selection of consultants were not finalized and the work lapsed. 23 (iv) Participatory annual joint reviews are Formal Joint Annual Performance Reviews were conducted on schedule and clearly identify conducted in 2009, 2010, and 2011 and will continue the main issues and formulate actionable annually – next one scheduled for June 2012. The recommendations MHMS executive and DPs continue to see value in this process and jointly agree the process is becoming more productive and focused, and should continue, as all parties learn to work in new ways. Decisions made are monitored and discussed in joint dialogue established scheduled for three times a year. Improved data from the HMIS should further enrich this process. Intermediate Outcomes Intermediate Outcome Indicators Results Component 1: Public Expenditure Management Improve resource Percentage of planned activities that are This indicator was inadequately specified and the allocation and completed increased from 60% to 75% expected source of data was not clear. The final expenditure management MTEF used FY2011 data as base – so this can’t reflecting sector priorities measure base and future change. Trend data from FY11 to be generated latter in FY12 will enable base data to be systematically documented and reviewed from June 2012 (see below). Provincial Directors do report available data at annual conferences. Increased community participation in health There has been a gradual increase in community services planning at provincial level involvement in health service planning at the provincial level – through the operational plan development process. While there is significantly more stakeholder involvement in the national – more effort is required at provincial level – although capacity constraints at this level need to be 24 appreciated. It is policy to establish health facility management committees – this will enable increase voice but outcomes still embryonic and will required future external support. Health facility survey completed and Did not proceed as discussed above. Alternative data analyzed sources provided some important insights. Budget planning, development and Provincial Operational plans – with SIG and DP accounting processes enhanced and budgets – now developed annually. Accounting of institutionalized* expenditures on MYOB now allows separation of hospital and primary health care at provincial level and accounts for SIG and DP on budget expenditures by program i.e. provides more detail than MOF&T budget system can. Base year for whole of sector established for FY11 and use in MTEF. This was initially supported by provincial planning TA and subsequently by a senior financial management specialist – the latter institutionalizing MYOB accounting package across MHMS. Local consultants were hired to provide MYOB training in the provinces. The next performance review conference will have data on FY11 budgets, expenditure and FY12 budgets – generating systematic trend data for the first time – and enabling dialogue of expenditure outcomes. Given the fragmentation of sector financing this is a major achievement and for first time resource flows are transparent. 25 Component 2: Sector Performance Monitoring Improved performance Comprehensive sector performance The new NHSP has an embodied comprehensive monitoring system indicators and monitoring system developed M&E framework. Work is underway to ensure HMIS can generate the required data. The framework is being adopted by MHMS and DPs for the anticipated new sector support for the HSSP. MHMS executive making decisions based on The MHMS executive has been making evidence- M&E reports informed decisions based on a range of assessments including the MTEF and monitoring data – mainly generated from program and provincial operational plans. Process will be improved when HMIS and M&E framework is operational in 2012. Evaluative work including the MTEF and the Bank generated Health Financing Options Paper informed decision making. The NHSP Advisor provided strategic mentoring advice and guidance to the MHMS executive team on the critical importance of M&E. Balanced scorecards for provinces and These were not progressed as it was not deemed a National Program priority under the new leadership of the MHMS. Provincial planning process including Provincial health plans with clear targets and medium term plan and operational planning indicators have been developed with the support system adopted with strong M&E from the planning team. The medium term plan framework* process remains fragile – with changing time table for annual budgets imposed by F&T and capacity at provincial levels limited confounding 26 implementation efforts. Component 3: Training and Capacity Building Improved human Increased share of provincial health plans Significant progress has been made in developing capability in the sector implemented provincial plans and budgets. Provincial health through training, systems teams have reported some difficulties with and better procedures implementing all planned activities – particularly when they were developed without reference to budget constraints. Progress reports to be submitted in 2nd quarter of 2012 will identify the challenges faced by the provincial teams. Self-assessment by trained provincial health The high level of workload, frequent overseas travel, directors 3 months after completion of and ongoing management ofDP-sponsored programs management training led to the withdrawal of several senior staff from planned training programs. Key MHMS staff attended the World Bank Institute Flagship Course in Health Systems in Suva, Fiji. This was rated very highly by attending staff and in particular underpinned their sense of the usefulness of the MTEF, setting priorities, M&E and the importance of planning within a budget constraint. *These items have been added to the results framework at the mid-term review to reinforce and reflect the directions of specific activities as they have developed 27 Annex 3. Economic and Financial Analysis An economic rate of return was not estimated for the program at the time of appraisal, nor was it calculated after its closure. 28 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Names Title Unit Lending Muhammad Pate Task Team Leader EASHD Al Picardi Consultant, Environment EASHD Aparnaa Somanathan Health Economist EASHH Cristiano Costa e Silva Nunes Procurement Specialist EAPPR David Michael Chandler Senior Financial Management Specialist EAPCO David Evans Health Specialist EASHD Edward Daoud Senior Finance Officer LOAFC Hope C. Philips Volker Senior Operations Officer EASHD Ian Morris Consultant, Public Expenditure Management EASHD Juliana Williams Senior Program Assistant EASHD Lingzhi Zu Senior Procurement Specialist EASHD Reem Hafez Junior Professional Associate EASHD Sheila Braka Musiime Legal Counsel LEGES Supervision/ICR Stephen David Close Task Team Leader EASHE Susan Ivatts Task Team Leader EASHD Al Picardi Environmental Management Specialist EASHD Amber Wilink Junior Professional Associate EASHD Anna Pigazzini Junior Professional Associate EASHD Aparnaa Somanathan Health Economist EASHH Chris Chamberlin Consultant, Health Economist EASHD Chris Scarf Consultant, Hospitals Specialist EASHD Cristiano Costa e Silva Nunes Procurement Specialist EAPPR David Michael Chandler Senior Financial Management Specialist EAPCO David Evans Health Specialist EASHD David Whitehead Financial Management Specialist EAPFM Hope C. Philips Volker Senior Operations Officer EASHD Ian Morris Consultant, Public Expenditure Management EASHD John Nyaga Senior Financial Management Specialist EAPFM Kylie Coulson Senior Financial Management Specialist EAPFM Lingzhi Zu Senior Procurement Specialist EASHD Michelle Lee Program Assistant EACNF Miriam Witana Procurement Specialist EAPPR Muhammad Pate Country Sector Coordinator EASHD Philip O’Keefe Country Sector Coordinator EASHD Truman Packard Country Sector Coordinator EASHD Stephen Paul Hartung Financial Management Specialist EAPFM 29 (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) USD Thousands Stage of Project Cycle No. of staff weeks (including travel and consultant costs) Lending FY06 22.63 121.75 FY07 34.86 144.57 FY08 28.57 128.12 Total: 86.06 394.44 Supervision/ICR FY08 0.20 0.63 FY09 6.64 90.00 FY10 6.48 118.58 FY11 4.18 68.07 FY12 1.44 47.96 18.94 (+37.38) Total: 325.24 See non-BB table below (c) Staff Time (Trust Fund and non-BB) Supervision/ICR No. of Staff Weeks FY10 8 FY11 9 FY12 3 Total: 20 Total Staff Time Supervision/ICR 37.38 FY08-12 (BB, TF & non-BB) Table 4 (c) was added to provide a more accurate reflection of staff time spent on supervision of this project. Two of the TTLs during the supervision phase were paid directly out of a trust fund, and the fiduciary team did not charge to the BB code; thus the staff time charged to BB for FY 10-12 in Table 4 (b) is lower than actual staff time spent on supervision/implementation support. 30 Annex 5. Beneficiary Survey Results A beneficiary survey was not conducted 31 Annex 6. Stakeholder Workshop Report and Results A stakeholder workshop was not conducted 32 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR MHMS were interviewed for their feedback and their comments recorded and submitted for clearance by the MHMS Permanent Secretary as per email below. From: Lester Ross To: "sivatts@worldbank.org" Cc: mlee2@worldbank.org, iandcmorris@bigpond.com, sdalipada@moh.gov.sb Date: 06/04/2012 08:45 AM Subject: RE: Follow up on World Bank Health Project Implementation Report Dear Susan, I write to endorse the ICR on behalf of the MHMS. Thank you for your assistance and support towards its completion. We certainly look forward to an ongoing partnership with the World Bank. Look forward to your visit. Kind regards, Lester. Development Objectives and Project Design. The MHMS believes the project development objective was relevant and remains relevant through the completion of the Project and beyond. The Project design was focused on key aspects of system development required to establish the capacity to implement the HSSP/SWAp the MHMS was very keen to operationalize. It is fair to indicate that the MHMS more fully appreciated the relevance of the design one we had produced the new NHSP and could see how the pillars of the Project were assisting to put in place systems that would make a SWAp more feasible to manage. We increasingly realized that the Project, while a small stand-alone Technical Assistance Program, was in fact an integral part of and supported implementation of the HSSP. Implementation Arrangements. It has to be acknowledged that the MHMS had lost most of its institutional memory, in part because of the “troubles� which led to significant staff turnover throughout the Ministry, about how a World Bank project needed to be organized and managed within MHMS. Presumably, because the project was relatively small, a decision was taken not to establish a Project Management Unit but rather to work through existing units of MHMS (the Planning Unit for policy issues supporting provincial planning and related issues; the Finance Section for financial record keeping, and financial management reform support; and the Procurement Unit to undertake all procurement activities). The consequence was that no one in the MHMS was directly responsible for the Project’s implementation except for the Health Executive. This meant that there was a hiatus in project implementation in the early years as a consequence of the appointment in 2009 of both a new Minister and almost totally new executive within MHMS. Frankly, it also took some time for the new executive to fully realize what the project was about and its strategic objectives. Initially, we were focused on the on-going HSSP financing by AusAID and then the new very large malaria project which took up much of our management focus. The Project did not take off and get the due management attention it fully deserved until one member of the MHMS Executive took ownership of the Project. Even, then 33 implementation was slow because of limited knowledge of Bank procedures. One important constraint was clearly the capacity of the procurement unit. AusAID was financing a full time internal procurement advisor as part of their program of support for the HSSP but it proved difficult to keep the position filled. There were a number of turnovers of advisors with significant gaps. Additionally, there was also a period when AusAID decided that the Procurement Advisor should only work on AusAID procurement activities. This proved difficult to work through and as the new Executive was not party to the agreements it was more difficult. When this was resolved things worked more smoothly. It does however need to be observed that the MHMS were not clear on the procurement arrangements and did not understand the continual focus of Bank mission on the procurement plan and its associated steps for each procurement action. We grew to appreciate that the procurement plan was a very valuable tool and we are now working to develop a procurement plan for major procurement actions we have throughout the entire Ministry. This has been an important lesson for the Ministry. It is only recently that we have had to undertake significant procurement actions – under the initial support to the MHMS from AusAID a managing contractor arranged for most procurement and it is only recently that the SIG financed development budget has grown to any significant size. In the end, procurement planning has gone from an imposed tool by the Bank to one which we now recognize needs to be and will be used within the Ministry. Assessment of Outcomes. The view of MHMS is that outcomes supported by the Project were significant. Specifically the following were achieved: • the most important outcome was the support for the development of the new NHSP which was developed with strong stakeholder ownership (both within MHMS, SIG and non-government health related organizations). This was developed in a participatory manner and was a sound example of “process� consulting which enabled participants to make the strategic decisions thus establishing strong ownership of the key outcomes which included: (a) agreement on the core programmatic outcomes being sought by the MHMS programs; and (b) agreement on key organizational and management reforms needed to better implement the agreed priorities. An important aspect of the new NHSP was that it did not just focus on part of the programs that have priority (e.g., the MDGs) but also on how management of MHMS have to manage all health services and try to ensure they all use resources cost-effectively; • establishment of a draft monitoring and evaluation framework for the health sector as part of the new NHSP. This draft framework is being finanalized and significant effort, as agreed under the new NHSP, is focused on operationalization of the Health Management Information System so that it is expected we will be able to review key health information systematically from July/August this year; • re-enforcement of the central and provincial capacity through enhancement of government systems to budget and managed expenditures was central to the efforts to 34 improve health services delivery and is a strategic achievement. This included separation of primary health care programs from hospital services at the provincial level so that MHMS could for the first time monitor SIG financed expenditures at the provincial level by program. The importance of this should not be underestimated; • re-enforcement of provincial and national program planning through the process of annual operational plans – increasingly linked to the budget process. These are important tools for Provincial managers and have the potential to better assist as the processes are bedded down and improved over time; • the production of both the draft and final MTEF, while late due to a need to recruit new consultant support to complete the exercise, is also a major mile stone. One theme of the new HNSP was that there was a clear need to achieve significant efficiencies from within existing budget allocations. The MTEF, for the first time, generate a “whole of ministry� picture of the allocation of all resources available to health (SIG financed recurrent and development and development partner financed resources) for the first time in a consistent format (with meaningful identification of input and program allocations). This is the first time this had been done for several years in any agency in the Solomon Islands. It required significant work to collect the data (particularly from development partners) which in Solomon Islands has not been systematically collected. This work highlighted a number of important issues with respect to resource allocation in the health sector relative to the decisions on health priorities taken under the new NHSP: (a) revealed expenditure priorities, including of development partner financing do not always accord with national priorities in that many programs are over financed relative to others; (b) there are significant gaps in funding of important programs; (c) there are significant quality enhancing recurrent budget funding from development partners (including of pharmaceuticals and key medical supplies) which will eventually have to be financed by SIG signally that the sector faces significant sustainability risks; and (d) there are a number of areas where there is a need to document future costs/sector needs to redress the identified gaps in the current expenditure program including: (i) rural health facility refurbishments and up-grades; (ii) primary health care equipment needs to bring facilities up to agreed standards; (iii) strategic hospital refurbishment and upgrading and associated medical equipment needs; (iv) future staffing needs – including ensuring adequate numbers and mix of cadres are produced and provision made for their employment within forward estimates for the sector. The MHMS sees the need to regularly update the health sector MTEF to inform policy dialogue with both SIG and health sector stakeholders including development partners. We are also encouraged that the MOFT is now planning to introduce a national MTEF to support the budget processes of government. • one aspect of the objectives of the Project that has recorded considerable progress is increased community and stakeholder involvement in the planning of health services. Production of the new NHSP was very participatory and was a real milestone in the involvement of stakeholders in health planning. In fact, the really strong participation of provincial managers in the plan was very significant. Previous planning – 35 including for the NHSP which was in place when this project started --was almost completely a top down exercise with little participation of provincial level staff. There is no doubt this has created strong ownership of the new NHSP within the Ministry and it will be very difficult to reverse this trend if a future Executive tried to operate in the old ways. It is interesting to observe that there is now increased community participation at the provincial level in health services planning. This will require sustained encouragement and we have to recognize provincial managers are not well resourced with staff or money to allocate time to this. More generally, we note that SIG has started opening up the whole budget process to a community consultation process – so we are proud to be in the vanguard of these types of efforts. Evaluation of MHMS Performance and Lessons Learned. The MHMS recognizes the senior staff turn-over and an inadequately staffed planning unit within the organization contributed delays in implementation – particularly in the first 18 months of the Project. An almost completely new management team was appointed to MHMS and it took time for the team to both get on top of the HSSP/SWAp process being established and even longer to appreciate the contribution the Project could make to this effort. For the new management team, working with development partners, and particularly with the Bank, which insisted on recipient execution of all aspects of Project implementation, there was a very steep learning curve. Further, at about this point, key staff left the planning unit and new staff had to be recruited. It was not until we had a new and very good head of the planning unit and had reached agreement within the Ministry Executive that one member of the Executive would be responsible for the Project that implementation picked up. The current Executive is unable to really assess the performance of MHMS in preparation of the Project as none participated in it. As discussed, the current Executive grew to more fully understand the rationale of the Project and recognize its importance in supporting the establishment of a SWAp and of building capacity at the provincial level and in PFM more generally. The fact that the Project was relatively flexible and the Bank team was able to respond to MHMS needs was very appreciated – particularly to generate support for the new NHSP was extremely valuable. Implementation was mixed –outlined above – but in truth we feel the core issue were delays. We had planned to undertake the MTEF on a more regular basis under the project and make it part of the rolling budget planning process. The relatively poor performance of the initial consultant undertaking the MTEF work is however part of this story. We are planning to repeat that this year as noted. We had planned to undertake more survey work and use of the training parts of the TA were perhaps less than optimal. On the other hand we did request training support for the new management team (on how to be managers and how to work as a team) which was not approved by the Bank team at the time. The Performance of the World Bank. The Bank has been very active during review missions raising the need to focus on systemic issues and processes and in making positive suggestions on how they may be addressed. Examples include: (i) the focus on 36 financial management issues and practical suggestion on how the Ministry could de- concentrate service delivery to managers; and (ii) support to the provincial planning processes. This work was particularly clearly recognized as critical by MHMS upon completion of the new NHSP which had built a consensus with the Ministry on the need to de-concentrate service planning, resourcing and delivery. We also need to acknowledge that World Bank aides memoire were very important documents for the MHMS Executive. They consistently documented the issues discussed during joint development partner missions with all partners and agreements reach with respect to both the HSSP and on the Project specifically. These were very important monitoring tools for both MHMS and other development partners. We often used them to confirm the history of evolving issues and agreements. They also focus MHMS on the next strategic steps and analysis required for management decision making. The MHMS recognizes the responsiveness of the Bank to policy and most implementation issues we were dealing with, particularly over the last three years, was very important to us. The support given to the MHMS with the MTEF and the Finance Advisor who also under took the revamped MTEF work was valued. This work was an important underpinning to the last major meeting of the health sector with development partners. The MHMS also recognizes the support given to it by the Bank through its analytical work – some included in aides memoires and most importantly the sector report on Health Financing Options. This report has been influential in giving both strategic direction and options to the Ministry. It has also helped in our policy dialogue with Cabinet. We do note however that the support on the financial management front was somewhat inadequate. Specifically, there was limited technical support through supervision, to the finance team on accounting procedures and how to make withdrawals. The fiancé staff had to go to the World Bank office to gain access to the Project Portal – a less than satisfactory situation. We do however acknowledge that Project filing was not always as complete or systematic as desired. We also acknowledge the lateness of the 2011 audits but these are now complete and the final audits for the close of the Proejct are under preparation. On Sustaining Project Activities. The core elements of the Project – the MTEF, provincial budgeting and planning, improvement to PFM within MHMS more generally and the monitoring and evaluation framework – are now well institutionalized within the Ministry. It is recognized that each aspect needs improvement and nurturing but these are lasting legacies – processes – which will support us into the future. They are all core to the implementation of the on-going SWAp and of significant importance to implementation of the HSSP. The MHMS recognizes it needs continued support in the PFM area and with each odd these elements of the Project activities as there are always opportunities to improve. But in our view the Project has helped establish sound practices and systems. In recognition 37 of this the MHMS is sending three senior staff on a PFM course shortly – focused on the health sector -- organized by the Bank for the region. Colleagues from the MOFT will also attend so this will assist in establishing a joint understanding of the issues we face as a sector. Finally, we sincerely hope we can continue our productive relationship with the Bank into the future. We have initiated discussions about continued support to PFM and provincial planning and options to explore results based financing of provincial services. This latter option would not have been remotely feasible without the core system developments assisted by the Project. Annex 8. Comments of Co-financiers and Other Partners/Stakeholders Introductory Note: The Project was a TAL and while it was in that sense a stand-alone project it was nevertheless implemented within the framework of an emerging SWAp where major DPs had agreed with MHMS through the signing of a Partnership Framework agreement to work cooperatively on health system issues and to increasingly move toward “pooled� or at least more flexible funding of the health sector by DPs. Further, as discussed, these objectives underpinned much of the objectives of the TAL in that they were focused on developing some key building blocks that were both necessary for MHMS to better focus on improved management of resources to achieve improved service delivery and to give DPs increased confidence that appropriate structures were in place to enable increased “pooled� or more flexible funding. In this context the major DPs involved in this – AusAID, WHO and SPC/Global Fund – were interviewed by the ICR author at their suggestion in lieu of requesting formal written submissions. In this sense they are strategic partners/stakeholders jointly funding the overall HSSP without formal co-financing agreements. AusAID through its agreement with MHMS to continuously support the procurement unit was more directly linked to project implementation. Comments by AusAID. • The Project helped put a number of key building blocks in place for the health sector and for DPs ability to forge ahead with a SWAP with increased programmatic and more flexible financial support. Specifically the Project: (a) strategically supported the development of new NHSP and its associated M&E framework; (b) significantly improved provincial planning, budgeting and budget execution capability within MHMS particularly at the provincial level; and (c) ensured the completion of the initial MTEF which highlighted key mismatches between resource allocations and stated health priorities and the scale of unplanned for incremental recurrent costs arising from DP investments and MHMS policy decisions. • The Bank focused and cared about the use of all health resources and through the policy dialogue on the Project and the HSSP kept both MHMS and DPs focused on 38 these issues (the work on provincial planning and budgeting; new NHSP; and the MTEF were particularly important for this as was the work of the Bank supervisions/implementation support teams). The Bank contribution to the policy also helped keep the focus consistently on development outcomes. The Financial Management Road Map for the future development of systems with MHMS was a significant success on which we can all now build. • DPs have an obligation to bring international experience and best practice to the Policy Table -- in this regard the focus on joint reviews by DPs of the HSSP were very important and the Bank has tried to do this consistently and quite well. The Bank has been an honest broker with DPs and MHMS and consistently supported enabling processes to resolve issues and ensure ownership within MHMS.AusAID saw the Bank and WHO as key advisors to MHMS on content (cfAusAID) in their areas of competence and recognized that much has been learnt about how DPs can work together to support MHMS – the increased ownership and leadership of MHMS has also helped in this respect. • The Project clearly helped MHMS and AusAID maximize the benefits on AusAID support to the sector (the major financier of the HSSP). • Institutional development requires a long term effort and it is important to remember MHMS and SIG were coming out of a difficult period of conflict. AusAID believes the processes set up for the HSSP will make it more likely that the key building blocks – many key ones of which were supported by the Project -- will become sustainable. The new support from AusAID is building on the efforts achieved through the project and the wider HSSP. • We all acknowledge capacity building is key. The processes the project has supported, and the process of preparation of the new NHSP 2011-2015 in particular,have, in and of themselves helped build both capacity and ownership within MHMS. • The Bank’s regular Project implementation support aides memoire(together with mission Statement of Objectives) became a very important monitoring tool for DPs and MHMS and added significant rigor to HSSP review processes – although some times its need to focus on details of the Project in addition to the focus on the overall HSSP meant they sometimes dealt with details not always seen as relevant to the strategic requirements of the HSSP. Sometimes there was a perception that a parallel agenda emerged (see comments in section 7). Nevertheless, the aides memoire provided a key record of deliberations, they were open to comment before finalization and no other DP consistently shared all documentation with other DPs and MHMS. • AusAID felt that a lack of in-country support for the health sector was a constraint on the effectiveness of the project. This put additional pressure on AusAID in country. 39 Comments by WHO • The partnership forged under HSSP between DPs and MHMS changed the way WHO works with sector – it is now much more collaborative and some funds, because of the agreements and work done under the Bank project, are now pooled and more will be in the future. • It is hard to change collective behavior around program financing and support for joint dialogue and but the Project assisted this. Specifically the partnership of DPs with MHMS and working with provinces has created a new structure to give voice to provinces and their needs which would not have been achieved without the project. There was significant work done with the establishment of the accounting package (MYOB) at the provincial level but, as we all know, this needs sustained follow-up to make it fully institutionalized. It would be good if the Bank could continue to support this effort. • The new NHSP, was a significant achievement – both for its clear statements of health sector priorities and the identified organizational and management changes required to better implement the agreed health strategy but there remains a need to put new structures in place in MHMS to allow effective management – particularly at the provincial level and below to the facility level. • Challenges remain on the M&E front – the new NHSP incorporated a sound framework for M&E – but management information systems still need to be effectively operationalized. While this is now being worked on the reality is that a lack of data has constrained the effectiveness of M&E. • It was recognized that the Bank did some qualitative work on the demand side for health services as part of preparation and did plan under the project to do more. This did not happen largely because MHMS did not see this as an immediate priority but WHO feels this remains an important area for future work. The Project was successful in giving increased voice to provinces and their stakeholders in the planning and budgeting processes but more needs to be done to move down to the facility level and understand the dynamics at the coal face. • Key lessons learned include that staff turnover – both of MHMS and DPs – in a situation where there are not strong management structures means that transaction costs can be high and ownership waxes and wanes. Another is that DPs need to work with the management team and not just key individuals – often small health systems particularly with fragile country systems become person dependent rather that system dependent. Comments by SPC/Global Fund • The development of the new NHSP, with support of the Project, by MHMS was a major achievement. Project support drove the process and the framework within 40 which MHMS staff and health stakeholders (including development partners) made the strategic decisions on the new plan. The great benefit of the process was the level of ownership within the MHMS of the new strategy. • The Project also drove important financial management and planning reforms within MHMS – particularly at the provincial level. The acid test will be the determination of the MHMS to continue to devolve decision making and support down to provincial managers – a key recommendation of the new strategic plan. • The MTEF, while late, is a major milestone for MHMS on expenditure monitoring and evaluation, and provided strategic information on current expenditure allocations not previously available. It clearly showed that current expenditure allocations were not always in accord with expenditure priorities expressed in the new NHSP. It is very good that MHMS is keen to continue to up-date the MTEF. • While the Project has really helped develop key building blocks for the HSSP/SWAp the MHMS will continue to need strategic support to sustain and enhance the gains made. • The aides memoire of the Bank were very important monitoring tools of overall developments in the health sector for both development partners and the MHMS. • The Bank played a crucial role in continually providing strategic leadership on both process and priorities for reform. This was recognized by other development partners and by MHMS. 41 Annex 9. List of Supporting Documents AusAID Commissioned Report (2007). Pharmaceutical and Medical Supply Assessment, Solomon Islands SWAp. AusAID/Ministry of Health and Medical Services Commissioned Report (2006). Health Sector Support Program: Interim Progress Report. Government of Solomon Islands/Health Sector Improvement Project, (2005). Health Status Assessment Report. Government of Solomon Islands, (2006). Ministry of Health and Medical Services, National Health Strategic Plan 2006-2010 Government of Solomon Islands/AusAID, (2006). Solomon Islands Health Expenditure Review. Government of Solomon Islands/Health Sector Improvement Project, (2006). Solomon Government of Solomon Islands/Health Sector Improvement Project, (2006). Solomon Islands National Health Review. Government of Solomon Islands (2006) National Pharmacy Services Strategic Directions, 2005 – 2010, (R.F. Skinner). Government of Solomon Islands/Ministry of Health and medical Services (2010). Malaria Action Plan. Government of Solomon Islands, (2011).Ministry of Health and Medical Services, National Health Strategic Plan 2011-2015 Government of Solomon Islands, (2011). Ministry of Health and Medical Services, Corporate Plan 2011-2015 Knowles, James C,( 2006). Note on Health Sector Strategy, Solomon Islands (World Bank). Ministry of Health and Medical Services, (2011). Medium Term Expenditure Framework United Nations Development Program, (2007). Poverty Report: Analysis of Solomon Islands Household Income and Expenditure Survey. World Bank, (2006). Pacific Regional Strategy 2006-2009 World Bank, (2006). Interim Procurement Capacity Assessment (Health) World Bank, (June 2006; December 2006; October 2008; May 2009; February 2010; April 2010; September 2010; December 2010; July 2011; September 2011; December 2011). HSSP-TA Aide-Memoire World Bank (2007). Making Health Care Decisions in the Solomon Islands: A qualitative Study (Draft Mimeo) World Bank (2007) Hospital Services in the Solomon Islands – A Commentary on management and Development Issues (Draft Mimeo) World Bank, (2008). Project Appraisal Document World Bank (2009). Do Sector Wide Approaches Achieve Results; Independent Evaluation Group. World Bank, (December 2009; May 2010). HSSP-TA Record of Mission World Bank, (2010). Solomon Islands Interim Strategy Note FY10-FY11 World Bank, (2010). Solomon Islands Health Financing Options World Bank, (2010). HSSP-TA Restructure Paper World Bank, (October 2008; May 2009; June 2009; December 2010; April 2011; January 2012). HSSP-TA Implementation Status and Results Report 42 156°E 158°E 160°E 162°E 164°E 166°E 168°E 170°E Ontong Java Atoll PA P U A NEW GUINEA 6°S 6°S Roncador Reef CHOISEUL Taro Island Sasamungga Choiseul SOUTH PACI FI C SOLOMON Shortla nd I s. Luti M A L A I TA OCEAN ISLANDS Ne Mono w Kia Ge Vaghena ISABEL Vella Lavella Sosolo or gi Kolombangara a Santa Isabel 8°S So Dai 8°S Kundu un Buala Ranongga Gizo New Georgia d Dadale Hapai Tatamba New Georgia Vangunu San Jorge Daringali Rendova Seghe Group Auki Tetepare Nggatokae CENTRAL Florida Is. Malaita WESTERN Russell Is. Tulagi Su'u Maravova Tarapaina Tutumu Guadalcanal HONIARA Maramasike Mt. Makarakomburu So l o mo n (2,447 m) Avu Avu Paruru Ulawa Duff Is. 10°S 10°S Sea GUADALCANAL Heuru MAKIRA Reef Is. Kirakira Tinakula Apaora San Cristobal Noka Lata Nendo Mwaniwowo Santa Cruz Islands TEMOTU 0 50 100 150 Kilometers Bellona RENNELL AND Utupua Tigoa 0 50 100 150 Miles BELLONA Vana Vanikolo Anuta Rennell Tinggoa Fatutaka 12°S 156°E 158°E Tikopia Indespensable Reefs SOLOMON ISLANDS HEALTH SECTOR SUPPORT PROJECT Coral Sea SELECTED CITIES AND TOWNS VA N U AT U 14°S 14°S PROVINCE CAPITALS NATIONAL CAPITAL MAIN ROADS OCTOBER 2007 IBRD 35742 This map was produced by the Map Design Unit of The World Bank. PROVINCE BOUNDARIES The boundaries, colors, denominations and any other information shown on this map do not imply, on the part of The World Bank INTERNATIONAL BOUNDARIES Group, any judgment on the legal status of any territory, or any endorsement or acceptance of such boundaries. 164°E 166°E 168°E 170°E