Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) Report Number : ICRR0023070 1. Project Data Project ID Program Name P148017 MA: Health Sector Support Country Practice Area(Lead) Morocco Health, Nutrition & Population L/C/TF Number(s) Closing Date (Original) Total Program Cost (USD) IBRD-85070,IBRD-91470 31-Dec-2019 135,000,000.00 Bank Approval Date Closing Date (Actual) 24-Apr-2015 30-Sep-2021 IBRD/IDA (USD) Grants (USD) Original Commitment 100,000,000.00 0.00 Revised Commitment 135,000,000.00 0.00 Actual 135,000,000.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Joy Antoinette De Beyer Judyth L. Twigg Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Program Context and Development Objectives DEVOBJ_TBL a. Objectives At appraisal, the program development objective (PDO) was “to expand access to primary healthcare in targeted rural areas in the Program Area.” The wording was the same in the original Loan Agreement (LA, p. 5) and the Project Appraisal Document (PAD, p. 7). The PDO was expanded (as indicated by the part in italics) in the fifth year of implementation as follows: “The objective of the Program is to expand access to primary health care in targeted rural areas and to strengthen Page 1 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) detection and case management to respond to the COVID-19 pandemic in the Program Area (LA 2020, p. 5, and Third Amendment to the Original Loan Agreement letter, July 2, 2020). Because an objective was added at third restructuring (in June 2020), a split evaluation will be done to take this into account. Although one disbursement-linked indicator (DLI) target was also reduced, no additional disbursements were made against that target, and further applying the split rating methodology based on the revised target would not impact the assessment of achievement of objectives. b. Were the program objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 16-Jun-2020 c. Will a split evaluation be undertaken? Yes d. Components The program focus was primarily in the rural areas of nine regions targeted because of their low health indicators. Some activities (mobile outreach, human resources, and quality assessment budgets) had national components, mostly for strategy design. (In this report, the World Bank-financed support will be referred to as the “Program-for-Results” (PforR) or Program, while the broader national government program on maternal and child health that it supported will be referred to as “the program." Results Area 1: Increase equitable access to primary care in rural areas 1.1: Accelerate reduction of maternal, neonatal, and under-five mortality Guided by the government’s “Action Plan to Reduce Maternal and Neonatal Mortality, 2012-2016” and “Action Plan to Reduce Maternal, Neonatal and Child Mortality, 2013-2015," the Program would upgrade equipment and infrastructure, train health workers on best practices, involve communities in health prevention and promoting use of health services, put accountability mechanisms in place at local and regional levels, improve transport systems including to enable pregnant women to travel safely between home and hospital and to refer deliveries with complications for needed Caesarean sections, and provide selected services for free. 1.2: Strengthen the detection and management of non-transmissible diseases (NTDs) To enhance prevention, the Ministry of Health (MoH) would organize: (i) awareness campaigns for the general population, schools, and health professionals in rural areas on the dangers of smoking and benefits Page 2 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) of a healthy and balanced diet and physical activity; (ii) local activities to promote healthy lifestyles through mosques, colleges, schools, public primary health centers (ESSPs), mobile teams, local radio broadcasts, souks, etc.; and (iii) training on smoking cessation support for health workers. To improve screening, the MoH would: (i) equip each health center and mobile team with screening equipment (electronic and wall sphygmomanometers, glucometers, test strips, gynecological tables, and kits for cervical cancer screening); (ii) train ESSP health staff on screening techniques and whom to target for screening; and (iii) conduct screening activities, including for diabetes, hypertension, and breast cancer, among the target populations. In addition, the MoH would: (i) disseminate standards of care to all ESSPs and mobile teams; (ii) train professionals on standards of care; (iii) provide health facilities with basic essential drugs; (iv) establish a therapeutic education program on diabetes; and (v) establish a chain of specialized care. 1.3: Strengthen mobile health coverage The Program would support: (i) purchase of vehicles, equipment, and medical supplies for 57 mobile units to be attached to ESSPs in the nine target regions; (ii) upgrading of the equipment and staffing in mobile units and ensuring they did outreach on regular and more frequent schedules; and (iii) training for health care providers in managing mobile medical units. Results Area 2: Improve governance of the health system at the primary level 2.1: Define an incentive system to improve the performance of human resources in health, based on a situation analysis The MoH wanted to develop an incentive mechanism that would improve productivity and quality of services provided by medical and paramedical staff. To support this, the PforR would fund a diagnostic study and design of an incentive mechanism, including measurable objectives, and performance criteria for allocating bonus payments to various categories of staff in different health facilities. 2.2: Improve accountability by developing the main quality control tool (quality contest) and establishing a comprehensive grievance handling mechanism (GRM) The main quality assessment tool in the health sector consists of an annual competition (concours qualité, CQ) among health facilities. Staff at participating health facilities do a self-assessment, completing standardized questionnaires that measure a range of quality indicators (user satisfaction, care accessibility and availability, rationalized use of resources, security and responsiveness, leadership, community participation, and institutional functionality), and there is an audit by peers. In return for enrolling in CQ and highlighting their strengths and weaknesses, employees can access supplemental training programs, and centers may receive new equipment or funds for renovation. The PforR aimed to increase low health center participation in the CQ, especially in the nine targeted regions, by supporting: annual reviews of the quality assessment tool (based on feedback by the peers doing the audits and personnel at facilities that were audited in the previous round); training, including for the personnel of all facilities that enroll in the CQ; self- assessments; audits (including a new module about implementation of previous recommendations); and recognition and dissemination events to include more primary healthcare facilities, in particular rural ones. Grievance handling mechanism (GRM). At appraisal, the MoH had several ways for grievances to be expressed: (i) a new Division of the Inspection Générale staffed by four professionals; (ii) a hotline (cellule d’écoute) to the Secretary General’s office staffed by four telephone operators trained in how to respond to Page 3 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) phone calls and register complaints using Excel spreadsheets but without procedures on how to analyze or respond to complaints; and (iii) at the decentralized level, multiple informal channels of complaints (e.g., directly at the facilities or to regional officials from the MoH and other departments, including the Ministry of Interior). The MoH planned to develop an effective GRM easily accessible to all, including through reviving the national hotline. The PforR would support the MoH to: (i) clarify institutional structures, roles, and responsibilities in order to streamline the GRM; (ii) develop a preliminary implementation manual to guide the process of addressing grievances, including a technical workshop with key MoH and other staff at central, regional, and facility levels; (iii) pilot the new GRM for one year; (iv) review the implementation manual; and (v) roll out the GRM. These measures would comply with national policy (the inter-ministerial decree) for complaints handling and include procurement and fiduciary complaints handling systems. The activities were expected to provide the MoH with a better understanding of patient and population needs and satisfaction with service delivery, and improve MoH’s responsiveness to complainants. 2.3: Develop an integrated, computerized and accessible health management information system (HMIS) and implement it in one region World Bank financing would support laying the strategic, legal, institutional, governance, and structural groundwork for a national, integrated computerized HMIS, and implementing it in one region. This would entail setting up six provincial systems and one regional system and integrating them into the central system. Specifically, the PforR would support: (a) planning and analytical work, including an analysis of the coverage and status of the existing HMIS, and formulating a data dictionary and coding standards, a model of data flows and data types across the health system, and national charts of accounts and statistics; (b) updating the HMIS Master Plan; (c) developing integrated computerized systems in one region, including modules for surveillance, referral and counter-referral, facility management (finance, billing, materials management, etc.), clinical management and patient flow, and patient management (patient electronic records); (d) developing central support structures and functions (e.g., data warehouse, decision support, etc.); and (e) establishing adequate and instant access to information for all stakeholders, including various branches of the MoH, other government departments, civil society, and the public, including using new technologies to facilitate information sharing, transparency and accountability. Results Area 3: Emergency response to COVID-19 (added when AF and third restructuring were approved in 2020) Within three months of the detection of the first COVID-19 case in Morocco, the PforR helped the government to mobilize financing to address its impact. An additional loan of US$35 million was approved under the Fast-Track COVID-19 Facility, and US$12.2 million was reallocated from DLI 2 (Skilled Birth Attendance) and US$0.795 million from DL6 (Quality Assessment Competition) to COVID-19-related activities. This funding was to be used to procure essential clinical inputs for COVID-19 case detection and management (diagnostics, drugs, and other medical supplies) to scale up testing and laboratory capacity and set up 55 resuscitation units in facilities across the country; provide technical expertise and support; and improve epidemiological information systems to inform decision making and enable contact tracing. e. Comments on Program Cost, Financing, Borrower Contribution, and Dates Page 4 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) Project Costs, Financing, and Borrower Contribution: Total Program costs were estimated at appraisal at US$226.2 million (all sums are in dollar equivalents using exchange rates at appraisal), to be financed through a US$100 million World Bank loan, a US$15.2 million contribution from the Borrower through annual MoH budget allocations, US$107 million from the European Union (EU), and US$4 million from the United Nations (UN; PAD, p. 18). Additional World Bank loan financing (AF) of US$35 million was approved in June 2020. The ICR (p. 53) shows actual World Bank disbursements as the full US$135 million, no contribution from the UN, and far more than expected from the EU (US$356 million parallel financing) and from the Borrower (US$420 million), as well as US$329 million from the African Development Bank that was not mentioned in the PAD. Dates: The program was approved on April 24, 2015, signed on July 2, 2015, and became effective on September 21, 2015 (ICR, p. 2). A Mid-Term Review was held in September 2017. The project closed on September 30, 2021, 21 months after the original closing date, after two extensions. There were three restructurings, the first two approved by the Country Director, and the third approved by the Board. The first restructuring on July 3, 2017, nearly two years into implementation with no disbursements other than an initial advance of US$25 million, had been anticipated at negotiations, pending the availability of 2014 data; the most recently available data at the time of appraisal and negotiations were for 2012. The Program Area was changed to reflect Morocco’s new regional boundaries and number and names of regions, matching as far as possible the original Program Area; however, the redefined Area meant that outcomes before and after the first restructuring could not be compared (ICR, p. 11, footnote 9). All baseline data were updated to 2014 actual data for the Program Area (instead of 2012 rural averages), and the original target percentage increases were applied to the revised baseline numbers to derive revised targets expressed as absolute numbers. The second restructuring on December 25, 2019, one week before the original closing date, extended the closing date by one year, and extended the target end date for the intermediate outcome indicator to define a human resources (HR) incentive mechanism in rural ESSPs to June 30, 2020. It also revised the baseline value for DLI 5 from 3.75 million to 3.79 million to include mobile clinic consultations; modified the name of DLI 6 to reflect that the quality assessment exercise took place every two years and not each year (although in fact the two cycles that took place during the program began three years apart; i.e. in 2014- 2015 and 2017-2018), changed participation rates for health facilities from per year to biennially, and changed the baseline participation rate from 11.5 to 13; reformulated DLI 7 to cover the whole country and not only one region, and changed several related (sub) indicators to reflect the changes in strategy for the HMIS from a complex system including patient-level records that would be implemented in one region to a national system that would include facility-level data; and revised the Program Action Plan to reflect the changes in the indicator for the HR strategy. The third restructuring and AF approved on June 16, 2020 expanded the scope of the program to include actions to respond to COVID-19; added an additional objective to the PDO to reflect this as well as two new DLIs (also added as PDO indicators) and two intermediate results indicators; extended the closing date by another nine months; and reallocated US$13.015 million of undisbursed funds for COVID activities. Additional changes reduced substantially the target for the corporate indicator for number of people who received essential health, nutrition, and population services, which had been defined to be the same as DLI 2 (attended births); changed the wording and targets for DLI 7 to reflect the change in strategy for developing the HMIS; and revised the Program Action Plan to reflect these changes. Page 5 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) 3. Relevance TBL a. Relevance of Objectives New Rationale The Program addressed pressing development problems: health outcomes and inequalities that were generally worse than those for other countries in the region and of similar income levels, especially for maternal, newborn, and child health (MNCH) and noncommunicable diseases (NCDs), as well as increased citizen dissatisfaction and expectations expressed during the Arab Spring. NCDs accounted for over 80 percent of the disease burden in 2011. Maternal mortality in rural areas was more than twice the rate in urban areas, partly the result of lower access to and use of health services. There were large discrepancies in other health outcomes, notably malnutrition and infant and child mortality (ICR, p. 5). Slow economic growth had constrained increases in health spending, and government expenditure on health per capita had declined. Health system organization and delivery was fragmented, and although there was a strong network of urban and rural primary health care centers, health care service delivery and utilization were affected - especially in rural areas - by scarce human resources, shortages of basic supplies and drugs, a poor referral system, and absence of an integrated information system (ICR, p. 6). When the COVID-19 epidemic erupted, the PforR was quickly restructured and extended (and AF provided) to help the country respond quickly. The objective aligned with citizen demands and government goals for improving health outcomes, with a particular focus on strengthening primary health care in rural areas. A 2013 health sector white paper and the 2012–2016 health sector strategy had outlined a comprehensive reform agenda. This agenda informed the design of the government’s Health Sector Program (the program), which sought to kickstart health sector reform by improving financing, service delivery, physical and human resources, and governance. The World Bank loan financed a part of the government program, targeting rural areas in regions with mostly below- average health outcomes and above-average poverty, where 83 percent of the national rural population lived (PAD, p. 31). The development objective also aligned well with a 2020 strategic program for the health sector focused on patient-centered health reforms (ICR, p. 13, p. 19). The Program was consistent throughout with the World Bank’s assistance strategy in Morocco. The PforR was fully aligned with the World Bank’s Country Partnership Strategy (CPS) for FY2014–2017 (Report No. 86518-MA) and the World Bank’s 2013–2018 Middle East and North Africa Health, Nutrition, and Population Strategy. It supported the third pillar of the CPS on strengthening governance and institutions for improved service delivery to all citizens in both results areas. It also aligned with two of the CPS’ cross- cutting themes: gender, and voice and participation. The focus on improving maternal health outcomes contributed to gender, and the focus on the establishment of a GRM and on quality competition contributed to voice and participation. The PforR remained well aligned with the Country Partnership Framework for FY19-FY24 (Report No. 131039-MA), which emphasized quality and efficiency of health delivery systems, through its focus on improving access to quality rural primary health services. The PDO is also fully aligned with the latest World Bank Health, Nutrition, and Population Strategy, which prioritizes equitable and high-quality access to primary health care. Page 6 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) Relevance of PforR instrument The performance-linked disbursement instrument was appropriate given Morocco’s country capacity and systems, and that the World Bank and parallel financers (especially the EU) were supporting a broad reform effort in the sector. The PforR instrument enabled common DLIs to be developed by the Bank and EU, as requested by the Ministries of Health and Finance (MOF) (PAD, p. 13). The instrument encouraged a constructive focus on results rather than activities or outputs. The ICR stated that the PforR contributed to the “comprehensive and ambitious” health reform program announced in 2020 by involving a wide range of stakeholders and helping to articulate the priorities and challenges faced by the health sector (ICR, p. 19). The PAD noted that "the PforR will help the Government focus on strengthening the institutional capacity for transparent and efficient health services” (p. 13). This was illustrated by the availability of health sector data: at appraisal, and for the initial years of the PforR, the required data were available only with a two-year time lag, which delayed verification and disbursement. The PforR provided an incentive to improve the timeliness of data availability, also useful for providing more current data for MoH monitoring and decision-making. Rating Relevance TBL Rating High b. Relevance of DLIs New DLI RELEVANCE TBL DLI 1 DLI Increase in number of pregnant women receiving antenatal care during a visit to a rural primary healthcare facility (ESSP) in the Program Area Rationale Overview for all DLIs: The DLIs aligned fully with the PDO and reflected government priorities in expanding access to quality primary health care. They were consistent with the objectives and activities included in the National Health Financing Strategy finalized in 2019 and the health sector reform priorities announced in 2020, which aimed to expand access to primary health care through scaling up service coverage and strengthening governance. The addition of PDO indicators and DLIs on strengthening detection and case management to respond to the COVID-19 pandemic ensured continued relevance. The DLIs expressed the PDO as concrete, measurable outcomes. The first 5 DLIs related to Results Area 1: Expanding equitable access to primary care in rural areas, and focused on access to rural primary health care facilities for prenatal care, attended deliveries, new visits for under 5 children for curative care, diabetes, and total numbers of visits. DLIs 1 through 5 targets were estimated as percentage increases in the baseline, but the targets were expressed as absolute numbers rather than as changes in coverage rates. This meant that these DLIs did not capture relative improvements in service coverage (the percent of the target population accessing the service). This was problematic, Page 7 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) particularly for indicators affected by declines in the rural population resulting from urbanization and falling birth rates. DLIs 6 and 7 related to the second results area: Improving health system governance at the primary level, and measured rural health centers’ participation in the annual quality assessment, and establishment of an HMIS. HR strengthening could usefully have been included as a DLI rather than only an intermediate indicator; a DLI would have provided a stronger incentive for improvements. When a third results area for the emergency COVID-19 response was added in July 2020, DLIs 8 and 9 were added, measuring expanded COVID-19 testing and case management capacity. DLI 1: Increase in number of pregnant women receiving antenatal care during a visit to a rural ESSP in the Program Area This DLI relates to a service that is important for improving maternal and newborn mortality and morbidity, and the government’s goal of reducing the large disparities between urban and rural access to essential health services. This is highly relevant to the PDO. However, as noted earlier, it was a shortcoming that the target was expressed as an absolute number rather than as a change in the coverage rate. Rating Substantial DLI 2 DLI Increase in number of deliveries of rural women attended by skilled health personnel in public health facilities in the Program Area Rationale This DLI relates to a service that is important for improving maternal and newborn mortality and morbidity, and the government’s goal of reducing the large disparities between urban and rural access to essential health services. Defining the DLI and related targets in absolute numbers, derived from target percentage increases from the base value number, was especially problematic for DLI 2. The absolute numbers of skilled deliveries attended by skilled health personnel in public health facilities in the Program Area increased far less than the increase in the percentage of deliveries that took place in these facilities, because the expected number of births in rural areas was declining as a consequence of falling birth rates and urbanization. In addition, DLI 2 counted only deliveries in public health facilities, omitting deliveries in private facilities, which were increasing during implementation. Women who chose to travel to the cities to deliver in hospitals (where they were better equipped to deal with possible complications) were receiving skilled attendance but were also not captured in this indicator. Rating Substantial DLI 3 DLI Increase in number of new visits of children under 5 to a rural ESSP in the Program Area for curative care Page 8 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) Rationale This DLI also captures an important aspect of increased access to essential health services that can improve health outcomes among young children, an important goal for the MOH. However, as noted earlier, it was a shortcoming that the target was expressed as an absolute number rather than as a change in the coverage rate. In addition, neither the PAD nor the ICR explain why this indicator focused only on curative services, rather than including preventive care visits as well. Rating Substantial DLI 4 DLI Increase in number of patients with diabetes diagnosed and treated at a rural ESSP in the Program Area Rationale This DLI relates to the important new impetus of the MoH to detect, prevent and treat non-communicable diseases, which were especially under-diagnosed in rural areas. Rating High DLI 5 DLI DLI 5: Increase in number of visits to rural ESSPs in the Program Area Rationale This DLI was a useful summary indicator of improved access to health services in Program rural areas, relating directly to the PDO. However, as noted earlier, it was a shortcoming that the target was expressed as an absolute number rather than as a change in the coverage rate. Rating Substantial DLI 6 DLI Percent of rural health centers with delivery services (CSCAs) in the Program Area that participate in the main annual quality assessment (concours qualité) Rationale Page 9 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) In 2013, 86 urban hospitals and 173 urban health centers participated in the quality assessment competition, but only 82 rural health centers did so, with participation rates of 61, 22, and 4 percent, respectively. Increasing participation by rural facilities was expected to encourage a culture of evaluation and quality improvement through self-assessment, external peer audit, and implementation of improvement plans. DLI 6 was a relevant indicator, but adding a measure of the extent to which the results of the quality competition were used to make improvements would have been useful. Rating Substantial DLI 6 REVISION 1 Revised DLI Percent of rural health centers with delivery services (CSCAs) in the Program Area that participate in the main biennual quality assessment (concours qualité) Revised Rationale The January 2020 (2nd restructuring) changed the word annual to biennial in the DLI name and all disbursement-linked results (DLR), and changed the DLRs as follows: DLR 6.2 was changed from 40 percent participation to “The Borrower has updated the guide of self-evaluation of CSCAs in a manner acceptable to the Bank”; DLR 6.3 was changed from 50 percent participation to “The Borrower, through its MoH, has approved/adopted the guide of self-evaluation updated under DLR 6.2”; and DLR 6.4 was changed to reduce the target participation rate from 60 percent to 44 percent. The quality competition had originally been funded by the Japan International Cooperation Agency (JICA). When JICA funding ended, the MoH decided to run the competition every two years. In the first year, facilities would sign up to participate. In the second year, the assessment and peer audits would be done, and the results announced. Revised Rating Substantial DLI 7 DLI Establishment of the HMIS in one region within the Program Area Rationale The establishment of a computerized HMIS that would enable data to be entered, compiled, and more quickly made available to health care decision-makers was an important priority in order to enable increased efficiency, monitor the results of new initiatives and investments, identify areas that were particularly successful in achieving increased access and utilization, share good practices, make better-informed Page 10 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) decisions. Developing an HMIS often poses difficult challenges; this DLI was expected to increase the incentive to make timely progress. Rating High DLI 7 REVISION 1 Revised DLI Establishment of the HMIS in one region within the Program Area (no change to the DLI title) Revised Rationale The first (July 2017) Restructuring changed the indicator for DLI 7 first disbursement from “Urbanisation process, data dictionary and Data Model complete” to “Urbanisation process complete," which was judged a more realistic interim goal. Revised Rating High DLI 7 REVISION 2 Revised DLI Establishment of an HMIS in the Program Area Revised Rationale The MoH changed its strategy for the HMIS. The original plan was for an HMIS that included patient-level electronic records as well as facility-level data, and was piloted in one region. Recognizing the complexity of the original plan, and the barriers to small rural facilities with poor or no internet connectivity entering their data, the MoH changed its strategy to leave patient-level electronic records to a second phase and begin with facility-level data for the most important services and health conditions. A simpler system would have the great advantage of being able to be implemented across the country far more quickly. At the January 2020 second restructuring, three of the DLRs were changed, as follows: DLR 7.1 reverted to the original, i.e “Urbanisation process, data dictionary and Data Model complete” DLR 7.3 changed from “Data center is operationalized, regional and provincial reporting systems (in one region) are functional” to “The national digitized health management information system on maternal and child health, family planning and curative care (SMIPF-SC) has been operationalized in 100 percent of the public health centers of at least 4 regions within the Program Area” DLR 7.4 changed from “Legal documents drafted, HMIS operational in >80 percent of facilities in one region” to “The SMIPF-SC has been operationalized in 100 percent of the public health centers of all the regions within the Program Area” Page 11 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) Revised Rating High DLI 8 DLI Number of Polymerase Chain Reaction (PCR) tests conducted to diagnose COVID-19 (added July 2020 in Third Restructuring) Rationale The third restructuring and AF expanded the scope of the PDO and added a third results area related to quickly establishing capacity to respond to COVID-19. Widespread capacity for COVID-19 testing was a crucial part of the response, DLI 8 had two DLRs: a first disbursement could take place once 229,000 PCR tests had been conducted, and a final disbursement required an additional 400,000 PCR tests. No explanation is offered in the ICR or restructuring paper for why the targets were set at these particular levels. Rating High DLI 9 DLI Number of health care facilities designated COVID-19 facilities and equipped as per MoH guidelines (added July 2020 in Third Restructuring) Rationale Capacity to treat COVID-19 cases was also a core part of the emergency response; DLI 9 related to this vital goal. There were two DLRs: The Borrower’s MoH has adopted the guidelines for the designation of a health care facility as “COVID-19 facility” in a manner acceptable to the Bank, and at least 50 health care facilities have been designated COVID-19 facilities and equipped as per MoH Guidelines, with disbursement of US$ 120,000 per facility up to a total of US$ 6 million allocated. Rating High OVERALL RELEVANCE DLI TBL OVERALL RELEVANCE RATING Rationale Page 12 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) The overall relevance of the PforR is Substantial, reflecting the High relevance of the PDO and High or Substantial relevance of the DLIs. DLIs 1 through 5 were very closely aligned with the PDO (as were all DLIs) but would have been better expressed as changes in coverage rates rather than absolute numbers. The DLIs could have included more explicit focus on the systemwide binding constraints on access to rural primary care, human resources for health and the quality of care provided. Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Expand access to primary health care in targeted rural areas in the Program Area Rationale The theory of change was that procuring essential medical supplies, equipment, diagnostics, and drugs, and providing clinical guidelines for health workers as well as training and supportive supervision, would improve the quality and utilization of primary health care services at rural health facilities. Purchasing, equipping, and deploying more mobile medical units staffed with well-trained providers would increase access to medical care in rural areas, especially in areas that were far from fixed facilities, and for people unable or unwilling to travel to facilities. In addition, behavior change campaigns were expected to increase demand for and use of available services, and result in greater uptake of antenatal care, attended deliveries, care for sick children, screening, and diagnosis and treatment for hypertension and diabetes. With regard to governance improvements, incentivizing and encouraging more facilities – especially rural primary care facilities – to participate in the annual quality assessment exercise and develop action plans to improve areas of weakness, as well as establishing a grievance redress mechanism and training Ministry of Health staff in addressing complaints, would contribute to improvements in the quality of rural health services and improve user satisfaction and utilization of those services. Establishing an integrated digital health information system was expected to provide data that could be used to identify areas needing improvement and attention, and enhance transparency, accountability, and efficiency in the way health services were managed. The Program defined two Results Areas that would contribute to the overall objective. The first was to expand equitable access to primary care in rural areas. Outputs  The national list of essential drugs was revised in 2016 and 2019. Essential medicines were supplied to facilities to prevent stock-outs. Page 13 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017)  More mobile medical units were bought, equipped, and deployed to rural areas, especially areas far from health care facilities.  The MoH developed new clinical guidelines and trained providers in their use. Maternal, Newborn and Child Health  Training to improve provider skills and knowledge was conducted, including in early essential newborn care and neonatal screening for congenital hypothyroidism and childhood deafness.  Facilities were upgraded, including with additional new medical equipment to improve the quality of care in maternity, neonatology, and other units. This included ultrasounds in rural health facilities.  Health workers based in health centers gave priority to outreach to offer home-based care for a set of services and to screen for health conditions.  Technical assistance (TA) was provided to regions to help define action plans to eliminate preventable maternal and newborn deaths. The quality of care and services in maternity hospitals and neonatology units in the 12 participating regional hospitals was evaluated, response plans were developed to improve the quality of care, and implementation of the response plans was monitored and followed up on.  Health professionals were trained on the guidelines of the maternal and neonatal health program at the provincial level.  A maternal and neonatal death surveillance and audit and response system was implemented to inform improvements to quality of care, including training of regional trainers.  A public awareness campaign provided information on the services offered to pregnant women in the context of COVID-19.  Immunization campaigns were held under the National Immunization Program, including for COVID- 19 vaccinations in 2021.  A National Integrated Child Health Policy covering health goals for children up to 18 years by 2030 was defined and disseminated.  A national plan of response against viral acute bronchiolitis in infants under 24 months of age was institutionalized.  To improve nutrition, a screening system for obesity and nutritional disorders was developed, more than 200 health professionals were trained, and regional awareness campaigns were held to promote and motivate exclusive breastfeeding for the first six months and adequate infant and young child feeding. During COVID-19, the campaigns were done using virtual workshops.  Advocacy activities were implemented to promote healthy eating and physical activity in targeted schools. NCDs  The National Multisectoral Strategy for the Prevention and Control of Non-Transmissible Diseases 2019–2029 made diabetes a priority. A national charter for the prevention and control of NCDs was signed by 17 national partners.  Four public awareness campaigns on diabetes emphasized the importance of screening and informed diabetic patients about the services offered at health facilities. A national campaign in 2015 promoted healthy lifestyles, including awareness on the harms of salt consumption and the importance of a healthy and balanced diet and physical activity. Page 14 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017)  A Stepwise Survey, a national survey on risk factors of non-transmissible diseases, was done in 2017–2018.  A diabetes information system was set up to support ESSPs, and an integrated diabetes-health technology assessment information system was improved. A diabetic care network was institutionalized in 2015. Screening and treatment flowcharts for type 2 diabetes and hypertension were updated to reflect new international recommendations, and all facilities were supplied with electronic and wall-mounted blood pressure monitors for screening and monitoring hypertensive patients.  Between 2017 and 2019, an average of 12,000 NCD screening and diagnosis campaigns were conducted each year in rural areas through mobile clinics.  1,300 general practitioners were trained in 2016 on management of high blood pressure, and level 2 health centers were provided with drugs and electrocardiogram machines. Intermediate Results  More rural areas were regularly visited by mobile medical units, which increased access to health services especially for people in remote areas and others with constrained ability to travel to a health facility. The NCD screening and diagnosis campaigns by mobile units in rural areas resulted in an average of over 500,000 medical consultations, 200,000 specialized consultations, and 77,000 laboratory and radiology consultations every year. All NCD prevention and treatment services were offered free of charge at all mobile units and public primary facilities. The Borrower’s completion report (ICR, p. 57) reported that 992,671 diabetics were screened in public entities, covering 37 percent of the estimated national cases and nearly 40 percent of estimated rural cases. The number of hypertensive patients monitored at the ESSP level increased by 68 percent, accounting for nearly half of all hypertensives monitored nationally. Outcomes  All indicators were assessed using the baselines revised in 2017 during the first restructuring, which replaced the rural averages estimated using 2012 data with 2014 actual data from the PforR regions, aligned with the change in Morocco’s regional boundaries. The target for DLI / PDO indicator 1 - the number of pregnant women receiving antenatal care at a rural ESSP - was exceeded: an actual increase of 17 percent compared to a target of 6 percent. The target for DLI / PDO indicator 2 - deliveries in facilities by skilled birth attendants - was not met. The modest increases in numbers during implementation triggered disbursement of less than 13 percent of the allocated financing. Although the 2020 Restructuring Paper (RP) noted that “a review did not find conclusive evidence that the PforR played a role in incentivizing an increase in deliveries at public facilities” (paragraph 13), the ICR noted that deliveries of rural women attended by skilled personnel increased faster in the PforR area than in other parts of the country (ICR, p. 16). Household survey data (available for 2011 and 2018) showed substantial increases in coverage of antenatal care visits and skilled birth attendance (ICR Annex 5, p.68) especially in rural areas, about halving the gap between rural and urban areas. The 2020 RP noted that skilled attendance at childbirth for rural women increased from 53 percent in 2011 to 69 percent in 2018, according to household surveys, with increases especially in public hospitals (where 50 percent of deliveries for rural women took place) and private clinics (4.5 percent of deliveries for rural women). The stagnated number of Page 15 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) deliveries among rural women at public facilities was attributable to increased urbanization, declining fertility rates (which fell in rural areas from 3.2 to 2.8 between 2011 and 2018), and increased deliveries in private facilities. The ICR noted that the PforR “was instrumental in ensuring the scale-up of skilled birth attendance in rural areas” (ICR, p.16), noting that if the indicator had tracked the percent of births taking place in facilities attended by a skilled attendant rather than absolute numbers, it would have shown a clear improvement. Improved access to primary health care in the targeted areas was reflected in a substantial increase in utilization of rural health services: outpatient visits per person in (all) rural areas increased from 0.4 in 2014 to 0.6 in 2019. DLI / PDO indicator 3 – the number of new visits by children under 5 to a rural ESSP in the PforR area for curative care – increased by 6.4 percent, compared to a target increase of 5 percent. DLI 5 (also an IRI) – the number of all visits to rural ESSPs in the PforR area – increased 8 percent from the baseline compared to a target of 4.3 percent. DLI /PDO Indicator 4 – the number of patients with diabetes diagnosed and treated at a rural ESSP in the PforR area – increased by 65.5 percent, well above the target of 17.6 percent. The ICR's Annex Table 1A records almost 90,000 people with diabetes as diagnosed and initiated on treatment at ESSPs in the PforR areas during implementation (ICR, p. 34). The number of people with hypertension diagnosed and treated in rural ESSPs increased from 180,000 to 312,326, exceeding the IRI target of 287,000. The second results area was to improve health system governance at the primary level. Better quality of rural facilities, better responsiveness to patients including through an accessible grievance redress mechanism, and a more efficient health information system to support decision making were expected to lead to increased access to and use of rural health services. Outputs  The National Plan for Medical and Pharmaceutical Waste Management was updated to incorporate new procedures. Responsible persons in regions in the PforR area were tasked to ensure compliance. A budget line was created in the regional directorates for contracting out medical and pharmaceutical waste management at the ESSP levels.  An assessment of the existing GRM was completed in 2015, and a new procedure manual drafted. The new system was piloted and evaluated. The new comprehensive GRM system consisting of an online service and call center (“Chikaya health”) was rolled out at national and regional levels.  The Borrower reported several measures to improve working conditions, morale, and satisfaction among providers: completion of a detailed assessment and Action Plan; increased allowances; upgrading of directors of hospitals to Heads of Division and Central Services; upgrading the status of State-certified nurses; institutionalization of motivational conditions; improved management of transfers and access to basic and continuous training; and COVID-19 bonus payments.  The guide for Self-Evaluation of Quality by rural health centers was updated.  A situation analysis was completed, the Master Plan/roadmap for developing a computerized HMIS was updated and validated in 2018, and a data center was set up. In 2019, the government changed its approach: instead of a system built on patient-level electronic medical records to be implemented first in one region, a decision was taken to implement a national integrated, digital HMIS incorporating detailed data on service utilization from all facilities. Agreement was reached to include Page 16 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) the following data in the new computerized HMIS: maternal health (National Pregnancy and Childbirth Monitoring Program), including prenatal care, delivery, and postnatal care; child health, including preventive vaccination and nutrition services for children under 5 years of age and integrated management of sick infants (0–2 months) and children (2–59 months); family planning; breast and cervical cancers, including screening and diagnostic and therapeutic management; sexually transmitted infections/AIDS; and curative care for hypertension and diabetes (collectively known as SMIPF-SC). The system was implemented in public health centers in the PforR area.  In 2017, the formal decision was published to set up an audit department, with 12 regional coordinators directly attached to the Inspector General to become responsible for audits (instead of the Ministry General Inspection). Audit terms of reference were developed and agreed. In 2018 and 2019, training seminars and courses were held at the central level, and audit managers participated in audit mission teams of the General Inspection and then independently. (These capacity building activities were put on hold during 2020 because of COVID.)  Three-year framework contracts were signed to reduce medicine supply shortages. All health personnel involved in medicine supply chain management were trained on stock management and quality control through an initiative implemented by the Supply Division of the MoH. Intermediate Results The GRM system processed around 10,000 grievances between its launch and the writing of the ICR. The Borrower reported that complaints related to care, making appointments, and corruption decreased over time, but those related to absenteeism of health professionals, availability of medicines, and hospital stays increased. Participation of health centers in the quality competition increased. In the most recent round, 421 health centers took part, including 180 level 2 rural health centers. DLI 6 showed a large increase in participation by CSCAs in the PforR areas, from 13 percent at baseline to 43.4 percent in 2018. This was considerably below the original target of 60 percent, but close to the revised target (as of June 2020) of 44 percent. By the end of 2019, the SMIPF-SC was operational in every public health center in all regions. Rural facilities (many of which had no connectivity) continued to submit paper-based service data to the province, which then entered the data into the HMIS. Data were aggregated through dashboards at provincial, regional, and central levels. Internal Audit Department (CRAI) recruitment and capacity-building enabled a gradual increase in control and audit activities. In 2019, CRAI executives conducted 172 independent missions, and in 2020, before being affected by the COVID-19 pandemic, they completed 173 independent missions, mostly to audit procurement contracts (purchase of dialysis services, outsourced services at hospital level, and so on) and to carry out organizational audits of hospitals and ESSPs. Outcomes The ICR stated that the comprehensive GRM fostered increased accountability and trust between the population and health sector stakeholders (ICR, p. 17), although no supporting evidence was provided. Page 17 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) The ICR reported that results from the seventh and last round of the quality competition showed that most of the 191 health centers that also participated in an earlier round of the competition had significantly improved, but no supporting data were provided. Best practices were shared across regions and facilities (ICR, p.17). The new HMIS was noted to have improved the availability of on-time data, including the data points used for verification of the PforR, and formed the basis for the information system that was scaled up for diagnosis and treatment of COVID-19. No supporting data on timeliness of availability of data was provided. The intended outcome of a GRM system is that complaints would be addressed, leading to improvements that would result in fewer complaints. The data cited above on trends in complaints suggests mixed success. Changes in health outcomes: The PDO was defined at the level of access to services, and measured by utilization data. The ultimate goal of a health system is to improve health outcomes. Reducing maternal mortality was a top priority in the Government’s 2012–2016 Health Strategy, and remained a national strategic priority in the 2016–2021 strategy. Analysis of household survey data confirms substantial improvements in maternal health service utilization and outcomes during implementation, with narrowing inequalities between urban and rural areas. Between 2010 and 2017, maternal mortality declined by over one-third, from 112 to 73 maternal deaths per 100,000 live births and rural maternal mortality declined by one quarter, from 148 to 111 maternal deaths per 100,000 live births. Overall neonatal mortality fell by almost 40 percent, from 22 to 14 newborn deaths per 1,000 live births. Rating Substantial OBJECTIVE 2 Objective The original project did not contain a second objective. Rationale The original project did not contain a second objective. Rating Not Rated/Not Applicable OBJECTIVE 2 REVISION 1 Revised Objective Strengthen detection and case management to respond to the COVID-19 pandemic Page 18 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) Revised Rationale The theory of change for the original PDO remained unaltered. For the addition to the PDO ("and to strengthen detection and case management to respond to the COVID-19 pandemic"), the theory of change was that procuring diagnostics, medical supplies, drugs, and equipment, and designating facilities and laboratories around the country to deal with COVID-19, would enable the country to diagnose and treat cases more quickly and efficiently. Outputs  Biomedical and analysis equipment and drugs were procured rapidly to equip laboratories to process tests for COVID-19 and to equip hospitals to manage cases.  An oxygen generator was installed at each of the 34 main hospitals in the country.  An epidemiological surveillance system for SARS-CoV-2/COVID-19 infections was put in place. A response plan and a communication and awareness plan were developed.  Rapid intervention teams were set up at the provincial/prefectural level, and the MoH issued circulars (19/2020 and 64/2020) on the management of confirmed COVID-19 cases at hospital, ambulatory, and primary health care facilities.  COVID-19 vaccines were procured, and a widespread national vaccination campaign was carried out. Intermediate Results Widespread testing capacity was set up quickly and 111 laboratories equipped with COVID-19 diagnostic capacities, exceeding the intermediate outcome target of 15. The ICR noted that of the 111 laboratories, 31 are public and 80 are private sector. Across the country, 72 hospitals were designated and equipped as COVID-19 facilities, including with oxygen generators. This exceeded the DLI 9 and PDO indicator target of 50. The ICR (p. 18) stated that “the establishment of 55 resuscitation units according to MoH guidelines was supported by the PforR,” but Annex Table 1A (p. 40) shows 0 resuscitation units and a target of 55 units (p. 40). The project team later explained that the units were in fact established and that the table likely reproduced an error in the last Implementation Status and Results Report (conversation with TTL on November 2, 2022). Outcomes By project closing, 2,364,454 Polymerase Chain Reaction (PCR) tests to diagnose COVID-19 had been conducted, greatly exceeding the DLI 8 and PDO indicator target of 629,000. The ICR reported that more than 11.4 million tests had been conducted since the beginning of the pandemic, and that the test positivity rate was below 10 percent during most of the pandemic, indicating that test capacity had been scaled up enough to ensure routine detection of cases. (Since an average of 90 percent of people tested were not positive for COVID-19, there were clearly large numbers of people being tested; when testing capacity is constrained, then tests tend to be rationed, and far fewer negative tests would be expected.) Page 19 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) Rapid designation and equipment of hospitals as COVID-19 facilities provided physical and human resources at health facilities to cope with the increased volume of patients. Facilities were able to triage and manage severe cases, reflected in Morocco’s cumulative confirmed rate of COVID-19 deaths of 430 per million people being substantially lower than that of almost every middle-income country in the Middle East and North Africa region, and lower than most middle- or high-income countries with substantial testing capacity globally. Revised Rating High OVERALL EFFICACY Rationale Contribution of the Program to Results The ICR used data from annual health sector reports from 2016 through 2019 (the first year after the new regions were defined and the latest year available) to compare changes in indicators across urban areas, rural areas in the PforR area, and rural areas outside the PforR area. Rural PforR areas had substantially higher rates of increase than urban areas and other rural areas for deliveries in facilities (DLI 2), patients with diabetes diagnosed and treated (DLI 4), patient visits to rural ESSPs (DLI 5), number of total outpatient visits at rural health centers (IRI 1), and patients with hypertension diagnosed and treated in rural ESSPs (IRI 2). The changes were about the same both in and outside the PforR area for antenatal visits (DLI 1), and changes in PforR rural areas were lower than other rural areas for new visits to rural ESSPs for curative care for children under 5 (DLI 3). This suggests that the PforR had a clear impact on expanding access to primary health care in targeted rural areas, and increased utilization of many key healthcare services. The ICR noted that the highly centralized Moroccan health system allows very limited space for regional health directorates or other health sector stakeholders in PforR areas to implement activities that differ significantly from non- PforR areas. Validity of Indicators The validity of the DLIs – which were also the PDO and some intermediate results indicators -- was discussed above. The indicators were all consistent with SMART criteria. The fiduciary performance indicators provided well-chosen additions to the Results Framework. Annual targets, data sources, and responsibility for data collection were all clearly documented in the PAD. Overall rationale The program fully disbursed the allocated amounts against all DLIs except for the poorly specified indicator for attended deliveries in facilities. However, there was substantial improvement in attended delivery coverage in the PforR area and in the related health outcome of maternal mortality. Strong results were also achieved in expanding screening, detection, and treatment for selected NCDs. The Program achieved the overall development goal of expanding access to primary health care in the targeted rural areas. Page 20 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) The program disbursed most of the allocated amounts for the two DLIs relating to improvements in health system governance at the primary level: US$ 23.75 million of US$24 million allocated for the HMIS, and US$ 7.204 million of US$ 8 million allocated for participation in the quality competition. The change in HMIS strategy was well-considered and enabled the MoH to develop and implement a simple and robust HMIS system across the whole country quickly, greatly improving the availability of current (and trend) data on a wide range of services and conditions. Participation in the quality competition by rural facilities was notable, although lower than the original target, and scores increased between rounds, indicating that facilities had made quality improvements. The GRM system was successfully rolled out. A diagnostic study was completed but not used to define an HR incentive mechanism for staff in rural ESSPs, as intended. Rating Substantial OVERALL EFFICACY REVISION 1 Revised Rationale The results area added in the third restructuring to establish capacity to respond to COVID-19 exceeded its targets and fully disbursed the allocated amounts under the DLIs, achieving the goal of strengthening detection and case management in response to COVID-19 in the Program Area. High achievement of this objective, combined with Substantial achievement of the original objective, produce a Substantial overall efficacy rating for the restructured project. Revised Rating Substantial 5. Outcome The rating of Substantial for Relevance and Substantial for Efficacy (under both the original objective and the revised objectives, after the additional of the COVID-19 objective) warrant an outcome rating of Satisfactory, indicating that there were only minor shortcomings in the project's preparation, implementation, and achievement. Outcome Rating Satisfactory 6. Risk to Development Outcome The ICR assessed the risk that the program achievements would not be maintained as negligible, because all aspects of the PforR are to be scaled up as part of an ambitious and comprehensive government five-year reform program announced by the King of Morocco in 2020, and codified in the Government Program of 2021–2026, Social Protection Framework Law (Loi Cadre 09-21), and New Development Model. This Page 21 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) program includes demand-side reforms to extend health insurance coverage to 11 million currently uninsured people, which will increase coverage from 70 percent to universal coverage. On the supply side, the new program proposes to build and upgrade health infrastructure, create a family and community medicine system, establish a new public health service, deconcentrate service delivery to the regions and territories, and implement an integrated HMIS. All the initiatives that were included in the PforR – the focus on reducing inequities between rural and urban areas by strengthened primary care, strengthening human resources for health, improving accountability and quality of care, and strengthening the HIS – are included in the new program's priorities and are planned to be scaled up. A new family health model is expected to facilitate referrals and coordination of care and especially improve rural health outcomes. Development partners are on board with the reforms, which will be supported by a series of planned World Bank programmatic Development Policy Loans. The government is discussing a follow-up PforR with the World Bank, which would support strengthening health care services with a focus on improving quality of care. These factors substantially reduce the risk that the achieved outcomes will not be maintained and provide optimism that the improvements supported by the PforR will be consolidated and extended. 7. Assessment of Bank Performance a. Quality-at-Entry The program addressed important development challenges in seeking to improve health outcomes, reduce inequities in access, and improve the efficiency and quality of public health services. The PAD assessed the challenges faced by the health system; the binding constraints and lagging indicators on MNCH and service utilization and growing problem of undiagnosed NCDs were correctly identified. The strategic relevance and approach of the PforR was sound. The combined focus on improving access to primary care in rural areas and strengthening governance was essential for improving health outcomes. It was the World Bank’s first deep engagement in the health sector in Morocco and launched an effective policy dialogue. The project's objective, targets, and indicators were well chosen to focus clearly on key health outcomes and important sector reforms. The logic of the results chain was clear, and activities were well chosen to achieve the objective. The program was appropriately selective in its geographic focus, with an important concern for reducing health inequities. The PAD clearly described monitoring, evaluation, and DLI verification arrangements. Project appraisal was rigorous, and the detailed, clear, and comprehensive PAD references relevant global evidence and good practice. Coordination with other development partners was reflected in the development of common DLIs with the EU, and use of the PforR instrument which is similar to the EU budget support process. (This was one of the first PforRs in the health sector and in Morocco.) Implementation arrangements were clearly described. The appraisal process helped the government finalize its Program. Technical, poverty, gender, environmental, social, and fiduciary assessments, as well as risk assessments, were rigorously conducted. The team effectively identified binding constraints and risks at entry (summarized in the PAD's Annex 7) and identified and planned reasonable measures to mitigate them. Page 22 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) The minor shortcomings in World Bank performance during program design relate to the decision to specify the DLI for attended deliveries in absolute numbers instead of coverage rates noted above, and omission of DLIs on human resources incentives, or on MoH budget growth. Also, the resources mobilized for TA proved inadequate; an investment component with funding for TA would have been helpful for facilitating some aspects of Program implementation. Quality-at-Entry Rating Satisfactory b. Quality of supervision The project became effective five months after signing, and a constructive mid-term review (MTR) took place two years into the original expected four-year implementation period. Supervision was rigorous technically, and from a fiduciary and safeguards perspective, but more proactive supervision would have reduced delays in disbursements and restructurings. Support was limited during the first three years of implementation, when substantial assistance was needed to help accelerate implementation and address critical issues given the novelty of the instrument. The MTR in September 2017 pointed out ways to accelerate program implementation, but these were not acted upon until the restructuring in late 2019, after multiple rounds of deliberations within the World Bank to decide on the full scope of restructuring. The second restructuring of December 2019 and more active supervision substantially accelerated disbursements and implementation, and the momentum continued despite the emergence of the COVID-19 pandemic, when the PforR and AF were effectively mobilized to provide surge capacity. Despite four rounds of handovers between task team leaders during implementation, the MoH appreciated the constant engagement, particularly following the MTR when implementation support was substantially improved. Greater in-country presence and additional TA resources would have strengthened policy dialogue with the government, development partners, and beneficiaries. Almost the entire technical team, including task team leaders, were based in headquarters throughout implementation, which hindered continuous and deep technical policy dialogue. Increased World Bank presence could also have enabled a more comprehensive assessment of Program impact through engagement with implementers and beneficiaries in rural areas. The opportunities provided by several restructurings to improve the specification of DLI / PDO 2 (or make its numeric targets more realistic) were missed. The COVID-19 pandemic slowed down verification of various DLIs due to movement restrictions. However, the task team’s rapid action in processing a third restructuring and AF enabled Morocco to respond swiftly and decisively to the COVID-19 pandemic Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory Page 23 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) 8. M&E Design, Implementation, & Utilization a. M&E Design M&E design had many strengths. The PDO was well focused on a key development challenge and consistent with the design of the program. The theory of change was sound. The results framework linked sectoral challenges to the PforR results areas, activities, outputs, and outcomes, and was detailed, complete, and well aligned with the PforR goals. The DLIs identified most key PforR results. DLIs and the PDO and intermediate outcome indicators were generally well designed and covered the main services delivered in rural primary health centers. The number of indicators was manageable and generally met the “SMART” criteria. Most targets were well selected to be both achievable and challenging, balancing ambition, feasibility, and the level of financial risk attached to each DLI. The DLIs were strongly aligned with government priorities and also were aligned with EU-financed Budget Support targets as far as possible. Alignment between the DLIs, PDO indicators, and intermediate outcome indicators simplified M&E. Verification protocols for the DLIs were broadly defined during design, although the ICR noted that because the PforR was a new instrument, details needed to be finalized during PforR implementation (ICR, p. 24). The ICR noted the constraints on M&E design: the absence of any data on health service quality, which is an important variable affecting access to and utilization of health services; infrequent household surveys; and the limited, fragmented, and paper-based HMIS that meant that at appraisal, baseline data were extrapolated from national rural average data for 2012. Necessary revisions to the baselines and targets using data from the PforR regions were only able to be done in 2017, two years after effectiveness. The main shortcoming in the M&E design was discussed above: targets – especially DLI 2 – specified in absolute numbers rather than coverage rates. An indicator for the percent rather than number of rural births in facilities with skilled attendants would have recognized the improvements achieved in access to and utilization of this service. b. M&E Implementation Implementation of the M&E framework improved substantially over time. Significant delays in data availability resulted in many DLIs being validated two years after accomplishment. With new leadership and attention in the Ministry and World Bank team after 2018, M&E implementation accelerated rapidly, and DLIs for 2019 and 2020 were validated in a much shorter time, resulting in rapid and full disbursements. This was also facilitated by the operationalization of the SMIPF-SC, which made the relevant data available much more efficiently and quickly. The government was able to build on the success of the SMIPF-SC and quickly develop a fully digital information system for COVID-19. Although verification of DLIs was initially a key bottleneck, sustained engagement resulted in institutionalization and acceleration of rigorous processes. Independent verification and auditing were implemented consistently by the General Inspection Directorate of the MoH, and verification protocols were finalized during the MTR with technical support from the World Bank. Proactive implementation support by the World Bank strengthened M&E implementation, by supporting the design of an HMIS under DLI 7 and ensuring accelerated implementation of verification procedures. Policy dialogue during and following the MTR facilitated increased capacity for DLI verification. Page 24 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) c. M&E Utilization The task team encouraged the MOH to set up technical committees through which the regional directorates and sub-directorates met at least twice each year to discuss program progress and data, challenges, solutions, and plans for the coming months. As noted, PforR data on several DLIs were also used as indicators for the EU-funded program, and the EU was able to rely on the PforR verification procedures. The MoH also used the M&E data to showcase regions that were doing especially well and used the quality competition self-assessment results to identify well-performing facilities and to disseminate best practices. As noted, the delays in data collection and verification, particularly for DLIs 1 through 5, were significant bottlenecks in the ability to use the M&E data. Although the relevant data were collected and collated within a year, internal approval and disclosure processes substantially delayed data availability. In addition, since the Results Framework was defined at the level of the whole PforR area, there was limited use of data at decentralized levels, also driven by the highly centralized health system. This constrained impact, because regions that were lagging and slowing implementation were not readily identified, held accountable, and empowered to improve. Best practices across regions, particularly for the overperforming indicators on hypertension and diabetes diagnosis and management, could have been more rigorously documented. Data from the quality competition could have been used more proactively. The multiple rounds of quality competition implemented during the PforR generated substantial data. However, these data were only presented in aggregated reports or in facility-level improvement plans, and there was no way to access the data in a way that could have been more useful to empower stakeholders to learn from and emulate good practice, and to track and celebrate improvements. More proactive use of these data, including for a quality improvement-related DLI, would have incentivized and supported transition toward a high- quality health system. Despite the constrained data availability noted above, the M&E system enabled validation of DLIs (albeit with substantial initial delays), informed Program management and decision-making, and improved substantially over time. The focus of the M&E system on outcomes was a positive point, as are the continued MoH focus and achievements in improving the HMIS. M&E Quality Rating Substantial 9. Other Issues a. Safeguards An Environmental and Social Systems Assessment (ESSA) was done at appraisal and updated substantially in June 2020 with the third restructuring and AF. The assessment of the ESSA action plan found that a long delay in designating environmental and social focal points at the national and regional levels had limited dissemination of environmental and social management procedures and actions to (a) Page 25 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) develop regional medical waste management plans, (b) conduct a diagnostic study of the waste treatment systems, and (c) conduct an analysis of waste treatment centers and wastewater management systems. Implementation of corrective actions accelerated in 2019, and an urgent action plan was defined at the end of 2019. While most of this action plan was implemented, focal points at the regions for environmental and social aspects were not designated, which constrained implementation. However, audits and assessments, as well as multiple supervision missions, did not identify any challenges. Medical waste management improved in hospitals in urban and semi-urban areas, but not in rural health centers. A budget line for health waste transport and elimination in the annual budget of the MoH regional directorate and MoH delegations enabled contracting for medical and pharmaceutical waste management at hospitals and ESSPs, but the budget was not enough to cover all provinces in each region. Many regions have regularly reported on their waste management capacity; however, no audits were done in three of the seven regions in the PforR Area. Establishing a GRM was one of the PforR intermediate results indicators, and the GRM led to some improvement in social accountability. The quality competition also encouraged citizen engagement and social inclusion. The ICR noted (p. 27) that providers have incorporated the results and feedback from patients through both mechanisms in their practice. The MoH reported that 22 percent of submissions to the GRM were requests for information, assistance, or guidance, which were answered by the center’s telephone operators or through the web portal by the heads of the entities involved, and 70 percent of eligible GRM claims were successfully addressed. Complaints sent using the chikayasante.ma link were sent to relevant managers for response. The PforR helped strengthen Morocco’s system for addressing grievances, improving accountability and responsiveness to patients in some areas (with mixed results in other areas, as noted in Section 4), and also improved health waste management. b. Fiduciary Compliance All gaps identified in the fiduciary assessment done at appraisal were addressed. Actions included establishing functional internal audit and control systems at the central and regional levels; audits of contract management, governance, and fiduciary performance; and instituting a programmatic budget and passing the Organic Finance Law. The 2020 audit (for the AF) found a lack of progress in implementing internal control measures, and corrective actions were taken. By the PforR closing, all required fiduciary actions had been completed. The PforR implementation built Public financial management (PFM) capacity. An Internal Audit Framework guided operationalization of internal audits, with over 250 audit missions each year focusing on procurement contracts, data verification for PforR DLIs, and the organization of hospitals and health centers. Institutionalizing auditing capacity helped accelerate disbursements for DLIs that were met and built capacity for implementing similar programs. PFM systems were also strengthened, especially through implementing the new PFM framework and Organic Law on Finance, including preparation of a programmatic budget with multi-year programming and a performance-based budget. These initiatives provided stronger monitoring of government resources allocated to the Program. Page 26 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) The ICR noted that the expenditure framework was not rigorously defined at design, with identified budget lines aligned with Program activities (probably because this was one of the first PforRs in health and in Morocco). This was addressed during implementation. Procurement performance also improved substantially during implementation after a framework agreement to centralize procurement for essential medicines and medical supplies dispensed at rural health centers was adopted. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 10. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 11. Lessons These lessons are drawn from the ICR (with editing). 1. When PforR funding is channeled as budget support to the Ministry of Finance, achievement of DLI targets is facilitated by program design and government budgetary mechanisms (or coordination with development partners) that ensure that the Ministry of Health has sufficient budget, capacity, and incentive to make every effort to do so. In this Program, initial shortfalls in MoH budget, buy-in, and motivation, coupled with inadequate support and attention from the World Bank team, resulted in a slow start. 2. Support for the development of a national health information system requires attention to ensuring that data are available and used for decision making. In this case, delays in availability of the data needed to measure and verify DLI targets delayed PforR disbursement and inhibited implementation. The PforR instrument can be a useful motivator and vehicle for actions to improve sector information systems. Timely and reliable data are needed for a PforR, but even more importantly, the instrument can encourage their routine use and dissemination to improve health system accountability, responsiveness, and outcomes. The ICR offered the example of Morocco’s National Initiative for Human Development (INDH), which routinely collects and uses data on the performance of its early childhood development Page 27 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) programs, and noted that the ongoing PforR operation on early childhood development had contributed to strengthening this culture in the INDH. 3. Facilities can achieve significant improvements in service quality and develop a culture of quality enhancement through periodic self-evaluations using a detailed and thoughtful quality checklist, with audit and feedback, announcement of results, and support for following through on action plans that address areas that need strengthening. The quality competition in Morocco used a comprehensive checklist that covered most aspects of quality of care (ICR, p. 24). This was the first time that health service quality was measured in Morocco. Staff at participating facilities and peer reviewers were asked for suggestions for improving the instrument for successive rounds. Most facilities that participated in two rounds of the competition achieved significant performance improvements between rounds, and best practices were shared across regions and facilities. 4. When falling birth rates, urbanization, or other causes of population migration are changing population numbers in program areas, it is inadvisable to set service targets for maternal services as absolute numbers. Although it is much easier to collect data on absolute numbers, these data can easily miss changes in coverage rates that are a much better measure of access, coverage, and utilization of services. 12. Assessment Recommended? No 13. Comments on Quality of ICR The ICR was clear, thorough, and candid. It gave a comprehensive account of the project's context, design, restructurings, implementation, and results. The theory of change was logical and well articulated. The ICR was forthright in describing the shortcomings in the project and presented details and data to support comments. Especially useful were the data from two household surveys showing trends in health outcomes related to the program indicators. The ICR mostly followed guidelines. The lessons and recommendations were thoughtfully selected and based on evidence provided in the ICR. There are some minor omissions and shortcomings, mostly relating to Annex tables. There was no explicit discussion of the reliability of the data on the program results. The ICR reproduced Annex 1C, the Program Action Plan, from an Aide-Memoire, without editing to remove comments such as “The team will follow up on the progress of this action during upcoming mission” (p. 47), or to correct the incomplete text (p. 49, first column). A consolidated table showing the original and revised DLIs, indicators, baselines and targets, and actual values would have made the achievements of the PforR against the targets much clearer. ICR Table 2 lists the DLIs (p. 10) but gives no details other than “baseline and target revised." Annex Table 1A has some units of measure missing, shows the original targets in the line for actual values (pp. 40 ff), and for some indicators the same number is listed for the original target and formally revised target; it would have been clearer if only the changed date were shown, if the target date but not the target number was revised. Several comments noted the actual percentage change without comparing it to the targeted percentage change, some repeated the indicator name instead of commenting on achievement against targets, and some comment areas were left blank. In Annex 1B, instead of actual values, target increases were shown, and the line for allocated amounts showed the disbursed amounts instead of the allocated amounts. Annex 1C did not Page 28 of 29 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MA: Health Sector Support (P148017) show due dates for actions, and therefore it was not possible to tell which actions were completed on time and which were delayed or by how long. a. Quality of ICR Rating Substantial Page 29 of 29