P R O C E S S E VA L U AT I O N N O T E Regulation for Safety and Quality of Care A Process Evaluation of the Health Inspection Pilots of the Kenya Patient Safety Impact Evaluation1 Key Findings and the counties, with contributions from multiple stakeholders; (iii) an innovative, adaptive learning 7 The largest randomized control trial on patient process which led to data- and evidence-informed mid- safety standards in any low- and middle-income course corrections; (iv) a remarkable effort to enforce country, conducted in Kenya, shows that regulation warnings and sanctions beyond the provision of on minimum patient safety standards for all types information and feedback to facilities; and (v) important of facilities improved the regulatory safety score by external support, capacity-building, and facilitation. 15% (0.49 standard deviations) in treated facilities 7 The cost of routine inspections (once the system is relative to control facilities, without increasing patients’ set up) for the pilot was between US$95–US$165 out-of-pocket payments or decreasing facility use. per visit, much lower than the cost of private 7 The process evaluation shows that the “at-scale” supervision services. pilot was successful with high compliance across 7 Critical risk factors for sustainability of the operation at most intervention components. This is remarkable, a larger scale without external support require attention, considering that nearly the entire system was such as: inadequate governance arrangements and developed from the ground up, including a new institutional coordination at all levels; insufficient regulatory framework and a system for monitoring capacity to meet conditions on the ground; failure to and enforcement. enforce warnings and sanctions on time; and exclusion 7 Facilities report positive perceptions and experiences of unlicensed facilities from government systems. with the new regulation, in contrast with the previous 7 The study demonstrates how a strong accountability system that was perceived as less transparent, and system can improve patient safety in contexts with more discretionary and punitive. underdeveloped systems, if the key elements of 7 External factors such as presidential elections and accountability are aligned in the design of the regulation health workers’ strikes did not seriously affect as well as during its implementation. It also highlights operations and outcomes due to a strong institutional the investment required for such systems and the setting and high-level commitment to the pilot. capacity-building efforts required for effective regulation. 7 Several critical mechanisms are at work: (i) a strong 7 The intervention has influenced policy­making in multiple authorizing environment and institutional arrangements; areas including the scale-up of the implementation of (ii) strong leadership from the Ministry of Health (MOH) the regulatory framework to all 47 counties in Kenya. 1 This technical note is based on the forthcoming working paper “Regulation as a Policy Lever to Improve Patient Safety and Quality of Care: A Process Evaluation of the Health Inspection Pilots of the Kenya Patient Safety Impact Evaluation” by Guadalupe Bedoya, Jishnu Das, Amy Dolinger, Rebecca de Guttry, Yoon Sun Hur and Ju Young Lee. Context A “Patient safety is the absence s the global community works given the low starting point of only 3% of to achieve universal health facilities complying with minimum patient coverage (UHC), there is growing consensus regarding the of preventable harm to a safety standards prior to the trial. Overall, the average facility before the trial was in the importance of quality health care services patient during the process ” lowest category of “minimally compliant, to achieve this goal. Nonetheless, there is limited knowledge on how to assess and of health care and reduction well below the benchmark established by of risk of unnecessary harm the government for full compliance (scoring monitor quality, beginning with patient above 60% of the maximum score). The safety. The first tenet of medical care is associated with health care to intervention improved patient safety for all an acceptable minimum.” preventing adverse effects to patients and types of facilities without increasing patients’ health care workers during health care out-of-pocket payments or reducing provision. Estimates suggest that 4 out of demand for health care, both among richer — WHO (2020) — 10 patients are harmed in ambulatory care and poorer patients, demonstrating the and that 2.6 million deaths occur in inpatient potential of regulatory-based accountability services alone in low- and middle-income countries (LMICs) due at a reasonable cost (Bedoya et al., 2020). Second, KePSIE covered to unsafe care (WHO, 2020). Most of the global disease burden the entire universe of public and private facilities (formal and informal) caused by adverse events (82%) is estimated to fall on LMICs with in the study counties, or 10% of all facilities and population in the the cost of safety violations exceeding by far the cost of prevention country (4.5 million catchment population). Third, the intervention (WHO, 2018; IHME, 2015). However, robust evidence is lacking to was a country-led initiative, with all stakeholders deeply committed inform policymakers looking to strengthen their accountability to the process and using an adaptive learning process. Conceptually, systems through external inspections (Flodgren et al., 2016; Flodgren stronger regulation and implementation were developed and et al., 2011; Campbell et al., 1998) or even through health service adopted by the government for this trial after an assessment of accreditation and certification systems (Brubakk et al., 2015; Hinchcliff the limitations of the system at the time. Close consideration and et al., 2012; Greenfield et al., 2012). Closing this knowledge gap is adaptation to local conditions took place and, by and large, the particularly consequential in Africa, where only a few countries intervention operated under government rules and constraints. have established national policies on safe health care practices and Finally, the impact evaluation was designed with a comprehensive corresponding monitoring systems (WHO, 2014). monitoring system, including process indicators, outputs and intermediate outcomes to monitor fidelity to the intervention design In this context, the Kenyan Ministry of Health and the World Bank and potential mechanisms at play. All in all, these elements make Group started a partnership, the Kenya Patient Safety Impact this intervention a rare opportunity to shed light on the process of Evaluation (KePSIE), to strengthen the regulatory framework for high-stakes inspections, as they would work at scale, and help us inspections and evaluate its impact at scale. As part of this effort, identify the next critical elements for advancing the agenda on how in 2016, Kenya gazetted a new high-stakes regulatory framework on government regulation systems can support quality improvements minimum patient safety standards including a scoring system with in health care provision. warnings and sanctions for all public and private health facilities. In 2017, a pilot to implement this framework at scale was conducted in Process Evaluation Objectives three Kenyan regions to evaluate the impact of the new regulation. KePSIE is the largest randomized control trial on patient safety and Dimensions conducted in any low- and middle-income country, and the first This process evaluation assesses the implementation of the health experiment to assess the impact of regulatory health inspections, makers inspections system piloted in KePSIE. It aims to inform policy­ and the focus of this analysis. and practitioners looking to implement similar systems at scale and particularly in contexts with underdeveloped systems to measure, At least four elements make this case a unique learning opportunity. monitor, and improve quality standards of health service providers. First, KePSIE’s high-quality impact assessment indicates the The study describes the main decisions and activities undertaken in intervention significantly improved patient safety, improving the developing such a system, assesses the extent to which the pilots regulatory safety score by 15% (0.49 standard deviations) and were implemented as planned, sheds light on potential mechanisms moving the average treated facility up a compliance category from at work, and identifies the contextual factors that acted as barriers “minimally compliant” to “partially compliant. ” This is significant or facilitators in the implementation of such systems. 2 REGULATION FOR SAFETY AND QUALITY OF CARE Process evaluation dimensions and questions 4) governance and institutional arrangements influencing the implementation; 5) critical resources and efficiency determinants, and Implementation 6) sustainability and risk factors for replicability and scalability. These • What is implemented and how? dimensions are assessed across KePSIE’s intervention components, the intervention’s theory of change, and the hypothesized links Mechanisms of Impact between the intervention components and outcomes. • How does the delivered intervention produce change? Context KePSIE Inspection Pilots • How does context affect implementation and outcomes? and Theory of Change Governance and Institutional Arrangements The KePSIE health inspections were expected to improve patient • What is the authorizing environment for the inspections pilots? safety by strengthening accountability in the health system to align • To what extent are the institutional and governance arrangements designed for the intervention effectively used, and how may they incentives for compliance with minimum patient safety standards. affect the implementation and results? There are three broad components across this intervention: Resources and Efficiency (1) a regulatory framework accompanied by clear guidelines on • What are the resources allocated by the different stakeholders? the minimum patient safety standards that facilities are expected • What are the fundamental factors affecting implementation efficiency? to comply with; (2) a monitoring system to track compliance with minimum quality of care and patient safety standards, and enforce Sustainability and Risks warnings and sanctions over time; and (3) a scoring and information • What are the critical elements for the sustainability and scalability of the inspection system and threats to implementation fidelity? scorecard system to publicize health facilities’ compliance with minimum patient safety standards. The figure below presents a simplified theory of change behind the intervention, including its Six dimensions are assessed in this study: 1) implementation main components/inputs, activities, outputs, and the hypothesized including fidelity, dose (the quantity of the intervention delivered causal chain to select outcomes of interest. and received), coverage and adaptations; 2) mechanisms of impact such as participant responses and potential mediator The main assumptions behind the theory of change of this processes explaining subsequent changes; 3) contextual factors intervention is that the combination of two or more of these potentially acting as barriers to or facilitators of the intended effects; components leads to an inspection system that provides incentives REGULATION FOR SAFETY AND QUALITY OF CARE 3 KePSIE Theory of Change Components Activities Outputs Short-and intermediate-term outcomes Long-term outcomes Develop standards and New regulatory framework 1 guidelines and guidelines Regulatory Knowledge of framework Develop warning and System of warnings and HFs about PSS and clear rules sanctions for different levels sanctions for different levels improves of non-compliance of non-compliance on PSS Health Develop dissemination plan Dissemination materials for Indicators for HF HFs Improve Develop inspection Implementation guidelines, 2 parameters (# inspectors, Compliance inspections planned, System to frequency) inspectors trained with PSS PSS delivered check for Develop enforcement plan for improves improves warning and sanction Functioning MIS compliance and to enforce Develop a monitoring system Inspections conducted warnings and and its protocols Warnings and sanctions sanctions Conduct inspections and enforced enforcement 3 Scorecard Develop HF signaling strategy Scorecards posted and (e.g., scorecards) Knowledge and grading and checked attitudes of patients information Develop plan for signaling regarding HFs and checks Dissemination of scorecards system performed (# of visits, # of its quality change PSS: Patient Safety Standards for HFs Develop dissemination plan materials disseminated) HF: Health Facility of signaling for patients MIS: Management and Information System Source: KePSIE project documents for health facilities to comply with patient safety standards. For facility. This intervention, therefore, falls within the definition of instance, activities and outputs from component (1) are expected complex interventions by the UK Medical Research Council (Craig to affect the knowledge of minimum patient safety standards by et al., 2008). It involves multiple interacting components, spanning facility in-charges, which is a necessary (although not sufficient) from a regulatory reform to the development of a system to manage, condition to improve compliance. Activities and outputs in monitor and enforce it. Its implementation relies on the interaction component (2) are expected to directly affect compliance by of several organizational levels, including the Ministry of Health and creating incentives (and costs for noncompliance) through regulatory boards and councils at the national level, and the county feedback and enforcement. These two components aim to create health teams and health facilities at the local level. top-down accountability. Finally, the third component is expected to affect consumer demand, through the provision of information to The target population of the intervention is all public and private patients (bottom-up accountability), causing a reallocation of facilities in three counties of Kenya—Kakamega, Kilifi, and demand to facilities with higher patient safety scores, which in Meru—ranging from Level 2 primary clinics to Level 5 hospitals.2 turn may induce changes in provider behavior. In the long-term, KePSIE uses a randomized design to assess impact and, therefore, better compliance with minimum patient safety standards among the census of health facilities in these counties is randomly divided treated health facilities contributes to improvements in the health into three groups. One treatment group (T1) receives components outcomes of the population they serve. (1) and (2) to test the impact of top-down accountability. A second group (T2) receives components (1) and (3) to test the additional The intervention required a new regulatory framework with warnings impact of bottom-up accountability. A third group only receives and sanctions that are enforced (weak sanctions and enforcement, the regulatory framework in (1), which applies to all facilities at except for extreme cases of malpractice, were previously the norm), the national level, but no high-intensity inspections except for a new system to check for compliance and to enforce warnings cases of malpractice. and sanctions at scale (around 4% of facilities were inspected in a 2 According to Kenya’s Essential Package for Health (KEPH) classification. The given year previous to the intervention), and the development of intervention does not include stand-alone laboratories, pharmacies, or other facilities a scorecard system to inform patients of the performance of the providing only specialized services. 4 REGULATION FOR SAFETY AND QUALITY OF CARE Process Evaluation Data Sources Data Source Analysis Examples of Indicators Sample / Respondents Project Documents Qualitative Regulation gazette published; inspection protocols established; scorecards NA validated; dissemination plans developed Administrative Data Quantitative Licensing status of facilities and departments; facility inclusion in government All treated and not treated facilities records Management and Quantitative Proportion of HFs inspected; proportion of warnings and sanctions enforced, All treated facilities Information Systems proportion of closed facilities found non-operational during quality checks Survey Data Quantitative Proportion of in-charges that are aware of the regulation; proportion of All treated and not treated facilities patients that have been affected by closures; proportion of HFs that report receiving full JHIC report Semi-structured Interviews Qualitative Implementation Coordinator (WBG); Actors’ roles and responsibilities; perceived obstacles to implementation; Logistics Firm (Medical Board); Inspectors; overall assessment of the intervention components and their long-term County authorities Focus Group Discussions Qualitative sustainability; recommendations for improvement Inspector Training Expert Group (ITEG); World Bank Quality Team Methods Legal Notice No. 46 in the Public Health Act (Cap. 242) on March 21, 2016 to be applied at the national level. The elements of the This process evaluation uses a mixed-method assessment that framework included (a) a refined Joint Inspection Health Checklist combines quantitative and qualitative instruments and methods. The with itemized minimum patient safety indicators to make it leaner overarching framework closely follows the UK Medical Research and less discretionary, easier to deploy and further focused on the Council (MRC) Process Evaluation of Complex Interventions fundamentals of patient safety; (b) a scoring system that allows Guidance and is complemented by other sources including Wholey, facilities to be categorized according to the level of risk presented Hatry, and Newcomer (2004) and Rossi, Freeman, and Lipsey (2013). to patients; (c) warnings and sanctions to be enforced according to Quantitative sources include survey data from KePSIE’s baseline a facility’s level of risk. Additionally, the new regulatory framework and endline, and implementation data on all treated facilities from KePSIE’s management and information system (MIS). We use a combination of observational analyses and take advantage of KePSIE’s experimental design to assess impact on relevant intermediate outcomes, when possible. In addition, we complement these with qualitative analyses of project documents, administrative data, semi-structured interviews and focus group discussions with the main implementation actors. Results Implementation: What was implemented and how? The implementation of the pilots was a multi-year effort because it required setting up an entire system. The implementation can be divided into two phases: the preparation (2013–2016) and the implementation of the one-year inspections pilot (November 2016– December 2017). Preparation included the reform to the regulatory framework of the inspection system for minimum patient safety standards, and the development of the institutional framework, systems, parameters, nomination and training of inspectors and overall capacity building required to operate the pilot inspections at scale. The enhanced regulatory framework was gazetted in Kenya Supplement No. 31 (Legislative Supplement No. 25) as part of REGULATION FOR SAFETY AND QUALITY OF CARE 5 Large operation to deliver the intervention The World Bank Group The study counties were provided technical assistance selected by the health and facilitation throughout 2,523 total visits to management representatives the process, developed the treatment health facilities of the 47 Kenyan counties. MIS system and monitored the implementation. 385 visits to enforce closures of facilities and departments The MOH and regulatory The MOH led the boards and councils seconded enforcement of closures of the inspectors, who became health facilities and the first cadre of full-time departments. joint inspectors. 468 visits that did not end up in inspection The county governments were The Medical Practitioners and a focal point in Dentists Board conducted the communication with health logistics for the daily facilities and the implementation 3 visits and 2 completed operations of the teams, and dedicated resources inspections per treated facility implementation. for inspectors. was extended to public facilities (only private facilities were covered Multiple stakeholders cooperated to develop and deliver the by the previous legislation). intervention. All elements of the intervention were designed and implemented through a participatory approach over a five- A large and complex operation took place in the study year process. counties. The implementation of the pilot in the 3 counties included 2,523 visits to health facilities in the treatment arms. These visits The various components of the KePSIE intervention were included all successful inspections, as well as some visits that did delivered successfully overall, including the development of not lead to inspections (for instance, if the facility was closed at the a regulatory framework with clear rules of the game, a strong time of the first visit, triggering the need for additional visits) and a system to check for compliance, and a scorecard system large number of visits by the MOH and the county team to enforce to disclose facilities’ performance, reflecting high fidelity to the closure of facilities and departments due to the widespread elements of the intervention related to plans, rules, and the licensing issues. development of systems. KePSIE plans, rules, and systems developed as planned Standards and guidelines on patients safety (JHIC) Regulatory Top-down accountability Scoring system 1 framework and clear Warnings and sanctions for different levels of non-compliance rules of the game Dissemination plan on standards for facilities Operational guidelines, plans, and Management and Information System (MIS) for carrying out electronic inspections, tracking progress, and assessing and ensuring quality System to check for Protocols for informing facilities about the new regulation 2 compliance Protocols for enforcing closures Inspectors with standardized training accountability Bottom-up Scorecards Scorecard system to Dissemination plan on scorecards for facilities 3 disclose HF Dissemination activities performance Texting system for verification of scores by patients 6 REGULATION FOR SAFETY AND QUALITY OF CARE Regulatory Compliance Categories, and Follow-Up Actions facilities reported for closure and that had not solved their license issues by the time of the closure visits. Checklist Score Compliance Category Follow-up Action ≤ 10% or no license Non-compliant Immediate closure Facilities’ compliance with the implementation varied. Re-inspection in 3 months. Compliance with scorecards was high: during quality checks, Facility will be closed if it scorecards were still found displayed in 89% of treatment facilities 11% – 40% Minimally compliant does not score over 40% of (on average 3 months after the inspection). Compliance with the maximum score in the 3rd inspection. closure was much lower: 52% of facilities where closure was physically enforced were found inactive (on average 2 months Re-inspection in 6 months. Facility will be closed if it after the physical closure), while the remaining 48% had reopened. 41% – 60% Partially compliant does not score over 60% of the maximum score in the Widespread licensing issues and enforcement of closures 3rd inspection. imposed a significant cost to the system, while signaling to 61% – 75% Substantially compliant Re-inspection in 12 months facilities that enforcement was a credible threat. Across the > 75% Fully compliant Re-inspection in 24 months three counties, 64% of facilities were private. A majority of them (61%) and the departments within these facilities (89%) reported Source: Kenya Gazette Supplement No. 31. 21st March, 2016 (Legislative Supplement No. 25). Legal Notice No. 46. The Public Health Act (Cap. 242). at least one licensing issue. This ranged from having an expired license, which implied a 90-day grace period and a re-visit to verify, to having no license available, which resulted in a report for immediate The new regulation has strong “sticks” but under a supportive principle. The government decided to design a regulatory framework that includes warnings and sanctions while providing sufficient KePSIE inspections had high fidelity to intervention components time (6–12 months) and feedback to facilities to help them meet minimum standards (and resort to closure only when all else fails). HFs received a copy of the JHIC 99% Under the new regime, facilities that score less than 60% of the Inspections HFs received at least maximum score are visited frequently (the lower the score, the more one inspection 100% frequent the visits). They have 3 visits to improve to the next category or face closure. Once they are above 60%, they are inspected every HFs did not have any pending 96% routine follow-up inspection 12 months or 24 months (if they score above 75%) without facing risk of closure. Only facilities with no license or scores below 10% Summary reports were delivered 100% at the end of inspections face immediate closure. In practice, closures were mostly due to lack of licenses. Scorecards were posted in Scorecards 100% A high level of delivery of the inspection components is Scorecard treatment HFs reflected across multiple indicators. Almost all (99%) of the Scorecard treatment HFs received 97% treatment facilities reported receiving a copy of the JHIC before or scorecard dissemination during the first inspection. All (100%) were inspected at least once, and received a summary report outlining the inspection results, Grace periods were followed Warnings and 96% Sanctions findings, and recommendations. At the end of the implementation, by license verification visits almost all (96%) follow-up inspections that were due (as determined Reports for closure of HFs 94% by the regulation based on the results of previous inspections) had were physically enforced taken place. In all scorecard treatment facilities, a scorecard was posted at the end of each inspection (100%) and dissemination visits HFs complied with Compliance 52% were conducted to raise awareness among patients about their physical closures Facility meaning (97%). Finally, the majority of warnings and sanctions were executed: grace periods given to facilities and departments HFs left scorecards displayed 89% to comply with licensing requirements were followed by license verification visits in 96% of the cases and, as of the end of the Source: KePSIE Management Information System (MIS) Facility compliance with physical closures is based on quality checks on average implementation, closure of facilities and departments was physically 3 months after the inspection. Facility compliance with scorecards displayed is based enforced by the MOH and the county authorities in 94% of the on quality checks on average 2 months after closure. REGULATION FOR SAFETY AND QUALITY OF CARE 7 61% of private facilities and 89% of private departments were found The county government did not perform closure visits without with at least one licensing issue during the implementation the leadership of the MOH.  The closure visits took place only Facilities Departments when the MOH team travelled to the counties to carry out the enforcement in closure rounds. In focus group discussions and None interviews with stakeholders, it was mentioned several times 11% that because of the county permanent presence in the areas None Reported and the familiarity of the county officials with the communities, 39% for closure Grace the closure visits were problematic and county officers may be 43% period Reported 28% conflicted. Therefore, the presence of an “external” government for closure 61% body, such as the MOH, allowed the closures to take place, Grace period although at a much lower frequency than considered previously. 18% Given the importance of enforcement, this is an area for consideration in the scale-up of inspections. Source: KePSIE MIS Notes. Indicates most severe license-related sanction ever applied for all private facilities that received an inspection. Excludes 6 facility closure reports that were By the end of the implementation (December 2017), a large not due to licenses. proportion of licensing issues related to grace periods had been resolved, and a lower—but nonetheless—important share of the closure.3 Given the extent of these issues, new protocols were lack-of-license issues had also been resolved. 61% of facilities with developed so that inspectors provided facilities with information grace periods had obtained a license verified by an inspection, and about how to renew their licenses. Detailed license information for 45% of the departments with grace periods had done so as well by facilities and departments, including contact information and GPS December 2017. This is an important achievement and mobilization coordinates for facilities, was captured in the MIS and shared with by the facilities and boards and councils. Of the facilities and the boards and councils to facilitate the licensing process. Overall, departments reported for closure due to lack of license, 29% and the operation required 385 visits to facilities for the enforcement 11% respectively had obtained a license that had been verified by of closures. The majority of these physical closures were due to the end of the implementation. licensing issues.4 With delays in the rollout of the implementation, most facilities were not inspected more than twice and, therefore, The year after the intervention, during KePSIE’s endline data closures for reasons related to JHIC performance, such as scoring collection, 65% of the facilities that were physically closed less than 60% and not improving to the next highest compliance due to licensing issues, and had not resolved their issues by category by the third inspection, were rare. December 2017, were found operational and offering their services. A large majority of these facilities did not have a license A team from the MOH and the county offices conducted closure by the end of 2018 as per administrative records from the regulatory visits in four waves during the year of implementation. The boards and councils (B&Cs). Furthermore, the majority of all operating process involved posting of a closure scorecard and notifying the facilities (97%), including the unlicensed facilities, consented to the in-charge of the decision and process to obtain a license. The closure visits happened on average 70 days after the closure report (vs. a 1-day Licensing issues status by the end of the implementation pilots protocol). Therefore, a large number of facilities (around one third of % of cases unresolved/resolved those that received closure reports in the inspection) were able to obtain the licenses before the closure visit, and avoid physical closure. 11% 29% The closure visits were visible events, in many occasions involving 45% 61% the participation of the community. The participation from the MOH was particularly important as the team and MOH coordinator explained in detail the reasons for the closures and implications for 89% 71% the patients of receiving care from unlicensed providers. 55% 39% 3 As per protocols for the implementation, inspectors did not enforce closures in the pilot system. To separate the role of inspections and closures, inspectors administered closure reports after an inspection and the county government and MOH were respon- Reported for Grace period Reported for Grace period sible to enforce closures of facilities and departments based on the inspection reports. closure closure 4 A few closures were due to performance on the JHIC, such as scoring below 10% of the maximum score (the non-compliant category) or scoring less than 60% of the Facility level Department level maximum score and not improving to the next highest compliance category by the third inspection visit. % Unresolved % Resolved 8 REGULATION FOR SAFETY AND QUALITY OF CARE survey. Therefore, the impact evaluation reports that these facilities, intervention and the results. Some illustrations from the project while also improving their score, have much lower patient safety, are as follows: which lowers the floor of patient safety. 7 The enhanced regulatory framework was developed by a The number of inspections to facilities was smaller than Technical Working Group (TWG) based on the assessment originally envisioned, due to delays in the implementation of by all stakeholders. The draft regulatory tool was tested in inspections. 42 facilities in Nairobi and results were used to simulate different scoring systems and helped show that most facilities would 7 It took over 7 months to complete the first inspection in 90% not comply with the new regulation. Based on these findings, of facilities, which led to facilities receiving less visits than the the new regulatory framework was developed to give facilities one-year schedule based on the regulation. time to improve. Significant adaptation also took place during 7 One third of follow-up inspections were conducted with an implementation. Many scenarios arose in the implementation average delay of 81 days. of the inspections that were not anticipated or clearly defined in 7 Licenses (for grace periods) were verified after 120 days (vs. a the regulation or its guidelines. These scenarios led to different 90-day protocol). paths of actions for which protocols were developed for 7 Physical closures were executed 70 days after the report for KePSIE’s implementation. For instance, given the significance closure (vs. a 1-day protocol). of the licensing issues in the private sector, 3-month grace periods for facilities and departments with expired licenses The implementation of the intervention was modified were implemented to give them time to renew their licenses. considerably to adapt to the conditions on the ground. This required a new protocol and an additional visit to verify they Adaptations included increased participation of the government had obtained the new licenses after 90 days. in the delivery and logistical management of inspections, 7 A large number of visits did not result in an inspection because customization of warnings and sanctions to address widespread the in-charges were absent or had left when the inspector informality in the private sector, centralization of their enforcement, arrived (likely due to the lack of a license). The government, and a larger operational role of the management and information therefore, decided to establish a new protocol that would system managed by the WBG. Overall, most adaptations contributed lead to closure reports for multiple unsuccessful visits. The to the pilots closely resembling what the government would face protocol includes a notice letter which states that an inspector in a scale-up. The table below presents the elements that help has visited the premise to conduct an inspection, waited up to fit the intervention to the context and the ones that threaten the 30 minutes, and was unable to carry out the inspection. The fidelity of the intervention. letter provides contact information and notifies the facility staff that a second inspection will be attempted in the next weeks Overall, an adaptive learning process embedded throughout and, should the staff not be available at the next attempt, the the process seems to contribute to the success of the facility will be reported for closure. Adaptations and elements that helped or threatened intervention fidelity Adaptation Elements helping the intervention fit the context Elements threating fidelity Medical Board assigned by the MOH as the logistics 7 The logistics benefited from greater knowledge of 7 Confusion in line of command led to inefficiencies organization (vs. private provision planned) reality on the ground and more authority/credibility and delays Inspectors were selected among government staff 7 Government inspectors lent inspections more 7 B&C’s capacity led to 6 available inspectors on nominated by the B&Cs (vs. call for government and authority and credibility average (vs. 12 planned) private sector candidates) 7 Misaligned incentives and accountability contrib- uting to absences and other HR issues Grace periods (90 days) were introduced for HFs and 7 Better fit to widespread informality and sudden 7 Weakened enforcement or sanctions departments with expired licenses demand to B&Cs to manage license applications 7 Additional inspectors’ workload (due to the license 7 Differential treatment of informal (unregistered) verification visits required after the grace periods) versus expired license Closures enforcement changed from immediate 7 Enabled physical enforcement as closures by 7 Weakened enforcement of sanctions physical closure executed by the counties to a few county officials as per the original protocols were closure waves executed by MOH and county not being executed at all Management and information system scope 7 Facilitated standardization, management and 7 Dependence on external team increased substantially monitoring in a constrained environment 7 Facilitated accountability through transparency and sharing of data REGULATION FOR SAFETY AND QUALITY OF CARE 9 BOX 1.  How Did KePSIE’s MIS Support the Implementation? The KePSIE management and infor- IllustraƟon: KePSIE MIS Summary Figures mation system (MIS) was a pilot system consisting of an application to conduct inspections electronically and a platform to manage inspec- tions planning and monitor progress on inspections. This customized solu- tion made available timely and action- able information for all stakeholders, including the facilities, inspectors, MOH, B&Cs and counties for plan- ning, monitoring, adaptation, and accountability. IllustraƟon: KePSIE Facility InspecƟon Report To illustrate how the system worked, it is useful to follow the journey of different actors as they interact with the system. First, an inspector con- ducted the inspection in a facility with the eJHIC in a tablet, using a software that calculates the facili- ty’s score, compliance category, and related follow-up actions in real time. The records were uploaded to a web- based system, where they were available to officials from the Ministry of Health, the Boards and Councils, the counties, and the implementation team. Facility in-charges would also receive a system-automated email (if an email was provided) with a full report of compliance with each JHIC standard. Within two weeks, the inspector would print and deliver copies of the full inspection reports to the local government health office. Next, the inspector planned for upcoming inspections over the next days and weeks, and for which the system provided a list of assigned facilities, precise locations on maps, and due dates for follow-up inspections based on each facility’s inspection history with daily updates. When the government closure teams were ready to enforce closures of facilities and departments (e.g., pharmacies and laboratories within a facility), the system automatically produced a list based on inspector closure reports including the history of previous actions and inspection reports for each facility and department. Finally, when the Ministry of Health coordinators, the Boards and Councils, and the counties wanted to use data for planning and policy, the system provided easy- to-read, nearly live reports on the progress and results of inspections. The system underwent extensive development and testing. Before the implementation of inspections, field-testing and fine-tuning activities were carried out for more than six months to verify measurability and relevance of the standards included in the JHIC, and the eJHIC was extensively tested in the KePSIE baseline. For the pilot, the MIS was designed to be highly adaptive as it reflected an entirely new inspection system being implemented for the first time at scale in Kenya. During the pilot, many scenarios arose that were not anticipated or clearly defined in the regulation. These scenarios led to different actions that inspectors may take in the field, for which protocols were developed with the MOH based on the current regulations for KePSIE’s implementation. The electronic inspection tool in the tablets underwent more than 40 rounds of revision due to this learning-by-doing process, and the web-based system expanded in scope to include many additional elements for the management of inspections. Over 4 years, including before and during the pilot, the standardization of the inspections and detailed protocols were fine-tuned by a multidisciplinary team. Continuous learning with the MOH, B&Cs, and counties led to a comprehensive measurement framework of indicators for timely and actionable information for managing inspections across counties. The MIS package, including the electronic JHIC (eJHIC), web- based system source codes, and other implementation support tools, were shared with the MOH and B&Cs along with training sessions and workshops to support the transfer of knowledge for the national scale-up. 10 REGULATION FOR SAFETY AND QUALITY OF CARE 7 The electronic inspections and management systems helped Facility in-charge knowledge of regulation this process by reporting progress, performance, and challenges % of health facility in-charges in real time.The system included: (i) data on planning and progress 61% of the inspection pilots (e.g., are inspections taking place? ); Control (ii) inspection results at the facility and aggregate levels for each Treatment pilot (e.g., how are facilities performing in each intervention? ); 31% (iii) up-to-date history at the facility and department level of 26% previous actions and due dates to support enforcement, and (iv) third-party monitoring indicators to assess intervention 12% 9% quality and protocol adherence (e.g., what is the quality of 3% the inspection delivered?). As Box 1 describes, the system Familiar with the Knows new regulation Knows number of was critical to integrate multiple stakeholders and assess new regulation is stricter compliance categories the progress and issues with the system, and for mid-course corrections. Having more inspections and being in the scorecard treatment arm correlates with better knowledge of the regulation and Mechanisms of Impact: How might the of the facility’s performance. In-charges of facilities with 2 or delivered intervention produce change? more inspections are significantly more likely to be aware of the Knowledge, feedback and enforcement are key intermediate regulation than those with one inspection (72% vs. 53%), and to outcomes that the intervention aimed to affect to improve know the number of possible compliance categories (30% vs. compliance. Merging survey and monitoring data helps shed 19%) and their own compliance category (66% vs. 46%). Similarly, light on what mechanisms may be stronger for the intervention in-charges in the scorecard arm are more familiar with the new to produce impact. regulation (70% vs. 64%), are more likely to know the number of compliance categories (35% vs. 18%), and are more likely to know Knowledge of the regulation improved significantly. At endline, the compliance category they belong to as per the last inspection the percentage of in-charges in treatment facilities who report (73% vs. 48%), than facilities in the inspections-only arm. being familiar with the regulation doubled with respect to control. However, many still do not recognize important features of the Enforcement seems to be a stronger mechanism than new regulation: 26% of in-charges know the new regulation is information or feedback working to produce impact. Facilities stricter, and only 9% know the number of compliance categories. with a higher number of inspections reported a higher impact, and In-charge familiarity with regulation (treatment facilities) % of health facility in-charges - Endline reports and MIS data Familiar with new regulation Knows number of compliance categories Knows own compliance category 73% 72% 66% 70% 64% 53% 46% 48% 35% 30% 19% 18% 1 2 or more No Yes 1 2 or more No Yes 1 2 or more No Yes Inspections Scorecard Inspections Scorecard Inspections Scorecard REGULATION FOR SAFETY AND QUALITY OF CARE 11 the scorecards and no scorecard arms had similar impacts. These Changes Implemented results, together with higher knowledge due to more inspections Facilities that report implementing changes—% of health facilities and scorecards suggest that enforcement (larger number of visits Minor equipment 73% to enforce warnings and sanctions) is a stronger channel than knowledge or information/feedback alone (scorecards acted as an Renewing/obtaining licenses 63% additional feedback/information loop affecting knowledge further Infrastructure upgrade 63% with no additional impact). New SOPs 50% Plan with goals 40% In-charges of facilities that were inspected rated favorably the Joint Health Inspections system across multiple dimensions. Help from county office 39% Elements such as clarity of content, professionalism of inspectors, Reporting tools 31% and clarity of recommendations were rated at 4 points out 5, Management structure 20% while considerations of fairness with the scoring and closures Additional temporary staff 17% were rated lower at between 3.5 and 3.7. The lowest element Additional permanent staff 12% was the possibility to report issues (3.4), which suggests a limited response for questions or queries from the facilities. These results are in contrast to the previous perception of a In-charges and decision-making bodies at treated facilities punitive and harassment system reported by the private sector reported multiple elements of focus for compliance, including and stakeholders.5 acquiring minor equipment (73%), followed by addressing licensing issues (63%) and upgrading their infrastructure (63%). Therefore, Positive perceptions and experiences of in-charges with the new the intervention had differential responses by type of item in the system may contribute to the success of the system regulation. The largest costs were reported in infrastructure. % of health facility in-charges The ranking system was easy for patients to understand, but Beneficial most of them did not notice the scorecards; therefore, the 92% for health facility bottom-up accountability could not be tested properly. A better dissemination campaign would be necessary to further understand Beneficial 91% the potential of informing patients about facility performance. for patients Patient Intermediate Outcomes by Arm In-charge rating of Joint Health Inspections (if inspected) (% patients) On scale from 1 to 5, 1 being “very bad” and 5 being “very good” 73% 75% 75% Clarity of the content 4.0 Control Professionalism of T1 inspectors 4.0 T2 Clarity of recommendations 4.0 21% 13% 14% Overall rating of inspection experience 3.9 Fairness of scoring system 3.7 Knows scorecard letter rankings Ever noticed a scorecard Fairness of closing facilities when not improved over time 3.6 Patients are not significantly affected by the inactivity or Fairness of closing facilities closure of facilities. A patient exit survey with 11,100 patients when needed 3.5 shows that when issues are reported, they are mostly related to Possibilities to report issues 3.4 the need to travel farther to receive health care. This may explain why the government did not face important challenges in closing facilities. Most facilities that were closed were also located in 5   See further work: “What lies behind successful regulation? A qualitative evaluation of Kenya’s health facility inspection reforms, ” by Eric Tama, Irene Khayoni, Catherine highly dense markets where patients had other alternatives. Goodman, Dosila Ogira, Timothy Chege, Njeri Gitau, and Francis Wafula, forthcoming. Protocols were also established by the government to ensure 12 REGULATION FOR SAFETY AND QUALITY OF CARE Patient has been affected by facility closure(s) % of patients 1% 1% Control Treatment that catchment areas would not be without the provision of health care as a consequence of the intervention. These included: (i) assessing markets following government closures based on the number of health facilities in the market; (ii) the level (size) In-charge HF Awareness of regulation of the health facilities in the market; and (iii) problems that may % of facilities at baseline be out of the control of the health facility to fix in the proposed time frame and the number of points that these represent in the 75% overall JHIC score (only in cases where a closure was due to score, which accounts for <1% of closures) (World Bank, 2016). Context: How does context affect 23% implementation and outcomes? The regulation had a significant impact on patient safety in spite of multiple unfavorable contextual factors, including low level awareness In-charge knows about In-charge has ever seen of the regulation, multiple actors visiting the facilities, and other joint inspections the JHIC negative external shocks. At baseline, facilities reported low awareness of the regulation. health inspection teams (22%). Some private facilities reported 75% of the facility in-charges reported knowing about the previous belonging to franchises (17%) or undergoing some accreditation joint inspections that had been operating for 2 years by the time process (9%). Visits varied in frequency, duration and enquiries. There of the survey. Only 23% reported having seen the Joint Health is some suggestive evidence that these multiple actors and visits Inspection Checklist. created confusion. For instance, the JHIC teams at that moment had not conducted 22% of inspections in these 3 counties. Facilities also reported multiple groups visiting them for diverse Therefore, these visits may include other government officials types of supervision. Facilities reported a high number of actors different from the joint inspections team, which indicates confusion visiting them in the previous year, including the national and local about the teams that are visiting the facility. governments and private organizations. The county and subcounty officials were the most present (73% of facilities reported visits), In addition, the implementation of the inspection pilots interacted followed by individual boards and councils (29%), and the joint with three important external factors: (i) turnover of high-level government officials at the national and local level; (ii) two presidential election rounds; and (iii) nurses’ and doctors’ strikes. Government supervision visits in last 12 months % of facilities at baseline The nurses’ strike lasted for 5 months and created some delays in 73% inspections in public facilities. 13% of public facilities could only be inspected for the first time in the last quarter of the year of implementation. No major delays occurred in the private sector 29% where the implementation was focused. Overall, private and 22% public facilities received on average the same number of inspections. Against predictions from all stakeholders, minimal problems were reported during physical closures even in the Joint health Individual board or County/subcounty inspections (national) councils (national) (local) middle of two presidential elections and with government turnover REGULATION FOR SAFETY AND QUALITY OF CARE 13 at the national and county levels. The resilience of the pilots to Examples of government actions: high level these factors seems to be in part explained by the high-level commitment to the project, the institutional arrangements, and Agreement and commitment of stakeholders (Windsor Agreement) the fact that most closures happened in highly dense markets, so patient choice was not affected (see mechanisms section). Appointment of KePSIE Task Force (KTF) bringing together national and county levels, and public and private sectors Appointment of technical working group to draft new Governance and Institutional Arrangements: regulatory framework What is the basis of the authorizing environment for the inspection pilots? Nomination of Inspector Expert Training Group (ITEG) to train inspectors and provide technical oversight of implementation To what extent are the institutional and governance arrangements designed for the intervention effectively used and how may they affect the implementation and results? The nature of the intervention required participation of multiple institutions and layers of government, and new arrangements and procedures. The high-level commitment and decisions taken by the MOH and the county health offices created the authorizing environment for public and private actors alike. The government facilitated a series of agreements and appointments that became the institutional framework for the different components of the intervention. The outputs of these groups were then enacted as regulation (including the gazettement of new regulation and the first gazettement of authorized sole joint inspectors under the Public Health Act), as well as process documents that defined how the interventions would be implemented Examples of government actions: operational level (protocols for how to apply the new regulation). This is a critical factor to consider when developing similar initiatives in Appointment of MOH Coordinator to manage inspectors and oversee comparable contexts. implementation Secondment and gazettement of first cadre of full-time inspectors from regulatory boards and councils Multiple organizations committed their staff and Nomination of county-level focal points to facilitate implementation expertise, with strong leadership from the MOH with health facilities to make the operations work under government Nomination of Kenya Medical Practitioners and Dentists Board to conditions, while building their capacity. oversee logistics of implementation The teams and organization operated under structures that were designed for the pilot and required substantial support from the MIS developed by the research team to manage and monitor the inspections, an area in which government capacity was low. The facilitation of a multidisciplinary team from the World Bank Group supported the implementation and monitoring of intervention fidelity. Elements that worked well include the partnership and responsibility from each stakeholder, especially at the high level. There was a foundational commitment of stakeholders and general guidelines defined at the Windsor Agreement in October 2013. In 14 REGULATION FOR SAFETY AND QUALITY OF CARE the development of the institutional framework, the mandates and Resources and efficiency: What are the responsibilities of each stakeholder were clearly defined in detail resources allocated by the different at the oversight and technical levels. stakeholders? What are the fundamental factors affecting implementation efficiency? Government capacity building, planning, and communications have room for improvement. Due to the large scale of inspections KePSIE inspections were possible due to the effort and and follow-up actions and, in particular, the high proportion of resources from multiple stakeholders. The Ministry of Health, licensing issues, the government lagged behind in capacity to the county governments, and the World Bank Group joined efforts deal with the high level of requests. Issues in logistics and to fund this intervention in a unique partnership. There are two communications were reported most by inspectors in surveys important components of the cost of the intervention. First, the conducted during the year of the implementation, in spite of the investment to set up the system. Second, once the system is in additional support from the World Bank. place, the cost of conducting routine operations. We focus on the cost of routine operation activities in this section, which helps us assess the average cost per visit to make the system work as “There are substantial delays in B&Cs issuing intended by the regulation. licenses. There should be regular updates of the staff list from various boards and councils to The pilot followed a particular model of inspections where all inspectors were located in the county headquarters and used minimize calls during inspections.” vehicles provided by central and regional governments to visit — Feedback survey of Inspectors at Feb 2018 health facilities. Inspectors were seconded by different government institutions, and most transferred from other regions. Facility Some select responsibilities and roles of key institutional closure visits required staff from the central government to groups, such as the county governments, were not performed travel to the regions. Additional external support was required for as planned. This is an area requiring better assessment to address implementation and monitoring of the operation. This is a poor the risks of participation and feasibility when multiple actors and model for costs in a fully scaled-up version, where the number potential conflicts of interest are involved. In particular, the main and location of inspectors can be flexibly determined and external responsibility of the county government for enforcement of closures support is minimized. Nevertheless, the routine pilot costs help was not conducted as originally planned, but was led by the national provide a benchmark that can be improved upon using standard MOH team. tools from operational research. On average, a visit to a health facility during the pilot cost around $165 in operational costs. “The counties should have a clearer role both at high Of this, $54 (33%) were inspector costs, including salaries, level and technical level (e.g. logistics, closures).” allowances and compensations for being outside of their duty — Interview of County Director of Health station; $17 (10%) were transportation costs to visit each facility; and $13 (8%) included other costs related to office, supplies and technology. The remaining $81 (49%) of the total, included “On closures, the County Public Officers might government management ($22 per visit) and external World Bank have a conflict of interest and/or it was not clearly support for implementation, MIS management and inspection understood their role in closures.” quality assurance ($59 per visit). However, several factors — FGD of Inspectors complicate the interpretation of this cost. First, for 28% of visits the inspector could not start the inspection and the facility required In summary, the institutional and governance arrangements designed multiple visits.6 Second, there were days when vehicles were used for the inspection pilots at the national and local level were, overall, for other government activities or were not functioning. Third, conducted as planned. However, the government had limited capacity there were days when vehicles were available, but inspectors were to address the scale of issues reported, the number and wide range of actions required by the regulation schedule, and the level of informality of the private sector. These areas require special attention when 6   Visits did not result in an inspection because the in-charges were absent or had left developing similar systems as they affected the implementation and when the inspector arrived (likely due to lack of a license). Due to vehicle constraints, inspectors in these cases were required to wait for a shared vehicle to come back are likely to have affected the results (e.g., decreasing the potential after taking other inspectors to separate (sometimes distant) facilities, before pro- impact of the intervention). ceeding to the next facility. Waiting times could be up to several hours. REGULATION FOR SAFETY AND QUALITY OF CARE 15 Implementation Resources Resources contributed by stakeholders for the implementation of the pilots Ministry of Health County Department of Health Standards, World Bank Group Regulatory Boards and Councils Governments Quality Assurance, and Regulation (B&Cs) (DHSQAR) Ⅲ MOH coordinator of inspections Ⅲ Secondment of inspectors Ⅲ Secondment of inspectors Ⅲ Stipends, top-ups, and per Ⅲ MOH high-level staff to lead and Ⅲ ITEG member time in training Ⅲ Inspection offices diems for inspectors support the process and technical support Ⅲ Inspection vehicles Ⅲ Transportation (fuel, drivers) Ⅲ Staff to support development of Ⅲ In-kind resources to support Ⅲ County focal points Ⅲ Inspection materials and regulation workshops and training Ⅲ Public health officers to equipment Ⅲ Staff to support development of support closures Ⅲ Implementation and regulation management support Ⅲ Staff to support development of regulation, tools, protocols, MIS, and quality monitoring absent. Therefore, we view this cost per visit as an upper-bound, cycle in the year of inspections. Inspectors were also not available since at least three of these problems–unsuccessful visits, non- 100% of their time (22% of the time they were out of duty or in functioning vehicles and inspector absence (during which we paid other activities, not including standard leave and an 18-day election for the vehicles)–can be sharply reduced in subsequent years with period), which also affected the flow of inspections. Therefore, how more experience. For instance, at best, a team of two inspectors 7 the inspections are planned and the time availability of inspectors could complete 6 inspection visits in a day (versus 3.5 during the are flagged as risk elements for the efficiency and sustainability of pilot) with variation across regions based on market structures. the system. Additionally, the World Bank support management valued at local government costs would be reduced considerably. These two A second type of constraint includes transportation and logistics- actions would imply a per-facility cost of $95 per visit. Further, related issues that were identified as important factors for alternate models where (for instance) inspectors are either located determining efficiency. Given the number of vehicles, inspectors in multiple cities in the county or have multiple bases within which reported spending 50% or more of their time either waiting for a car they travel will further decrease transport costs, and a larger scale to pick them up or in transit. In addition, 11% of the time on average of inspections will also decrease costs per visit related to office, (and around 35% in some months) vehicles had problems causing supplies, technology and management. Next we discuss details 8 delays, including breakdowns, maintenance, fuel not being available on select items where efficiency gains can be produced and costs due to payment delays, and, in some cases, because the vehicles could be reduced. were being used for other duties. In a short 1-week exercise, the research team found that an additional vehicle and improvements There are opportunities identified for efficiency improvements. in the logistics planning could double the number of visits and As we expect when building a complex system at scale, there was a completed inspections. This suggests that greater efficiency could diverse set of challenges. A first type of constraint was the efficient result from different combinations of resources. However, this use of inspectors. Due to the dynamic nature of the intervention, depends on the inspection load, number of unlicensed providers, the requirements for follow-up actions and visits depended on vehicle costs, and inspector salaries. Therefore, this was identified the performance of facilities according to the inspection cycle. In as a constraint that needs to be analyzed according to the local combination with a lower number of inspectors seconded (with conditions and inspection cycle. respect to what was planned), this limited the completion of the Building a system with inspectors based in the region and strategic areas within the counties (versus inspectors transferred from 7   For example, the government established a new protocol that would lead to closure reports for multiple unsuccessful visits. These visits are expected to decrease further other regions) will improve the cost-efficiency of the operation, as the system matures. by reducing extra stipends and allowances due to working 8  We exclude from these costs the fixed costs of building the inspection system, which included the development of the enhanced regulatory framework, implemen- outside of the duty station, and minimizing commuting time and tation protocols, training materials, and the electronic inspection system. These costs transportation costs, helping address the two constraints described are detailed in a forthcoming study by Chege et al. (2020) and may be useful for setting up similar systems in other countries. above. 16 REGULATION FOR SAFETY AND QUALITY OF CARE Inspection visits by day and select events 5-month nurses strike (June–Nov) 1-week short test exercise: KePSIE Inspections start in team hires extra Kakamega for 1 week vehicle in each launch with all county inspectors. Includes 35 training in the field and first learning and setup No inspections of inspections workflow National due to multiple 30 elections vehicle issues 25 No inspections 5-day inspector due to vehicle strike due to HR Inspections stopped breakdowns in issues 20 and delays due to HR all counties National issues. Inspectors not elections yet entirely released run-off 15 from their duties 10 5 0 Nov-16 Dec-16 Jan-17 Feb-17 Mar-17 Apr-17 May-17 Jun-17 Jul-17 Aug-17 Sep-17 Oct-17 Nov-17 Notes. Vehicle issues include breakdowns/maintenance, no fuel due to payment delays, and vehicles being used by county government. “Inspectors in Meru spent half of the time in a given Inadequate capacity remains a risk for the system to work or to work day, waiting for a vehicle to pick them up, when they at the lowest cost possible. In Kenya, the scale-up of this model is being implemented through the county governments and a new only have one vehicle at their disposal.” institution at the national level is taking leadership in inspections, — Bi-weekly Monitoring Report the Kenya Health Professions Oversight Authority. Given the high- level government commitment and county government teams that In general, rolling out such a large and complex operation are established and experienced with inspections, the country has implied limited capacity. Inspectors rated logistics and great leverage for the organizational structure necessary for the communication the lowest in surveys on the implementation. scale-up. However, the decentralization also imposes some risks. Inadequate governance and communication systems across the “There were delays on responses from [MOH and multiple agencies and levels of institutions required for making logistics coordinators] due to their excessive work.” the inspection system work is a high risk. The World Bank Group — Interview with Inspectors in March 2018 facilitated the governance and coordination of these elements in the pilot, and considerably supported the communication across different actors and the implementation of the pilot interventions. Sustainability and risks: What are the critical Appropriate support and capacity-building to meet the adequate elements for the sustainability of the inspection levels of institutional coordination, governance, and communication system and threats to implementation fidelity? is critical. A critical element for the sustainability of the implementation of There are a few additional areas where risks are identified based such an inspection system at scale is the necessary infrastructure on the lessons from the pilot. These risks stem from threats to and institutions to support the components of the intervention implementation fidelity, that is, that the intervention is implemented consistently and reliably in the long term. Planning would benefit in the way that is intended by the regulation. A few areas are from data systems and continuous analyses of the data that take flagged including threats to sufficient tracking or monitoring due into consideration the health market conditions in each county. to the complexity of the intervention and its dynamic nature REGULATION FOR SAFETY AND QUALITY OF CARE 17 (multiple potential outcomes of the inspections and follow-up required to take advantage of the returns to digital development, visits). Multiple threats to service delivery are flagged including is an important question for scale-up and replication. the risk of failure to comply with warnings and sanctions on time, and to include remote, distant facilities or an important number of Despite the expected challenges when developing a new system facilities that are not in the government records, which have on at scale, the inspections were successful and the lessons from this average the lowest patient safety in the system. Failure to monitor enforcement with closures, and limited capacity of the boards Threats to implementation fidelity and councils to provide required licenses are also identified as key areas to consider as risks. Threats to intervention design Complexity of the intervention Lack of standardization of implementation guidelines for inspectors Finally, due to the role of the MIS, the complexity of the intervention, Inadequate governance, institutional coordination, and communication and the reliance on an external team for this function, the risks strategies related to quality should be assessed and addressed when Threats to intervention training, supervision and support implementing the intervention at large scales. Building the pilot MIS Insufficient tracking or monitoring of service delivery/dosage took a multi-disciplinary team beyond ICT to leverage technology Inconsistent supervision and/or inadequate communication while making sure the system responded to the needs of the different actors and activities. Substantial efforts were dedicated Threats to the service delivery to verifying and improving the adherence to inspection protocols Inspector motivation, competing activities Failure to comply with the warnings and sanctions on time and quality of implementation. Quality officers conducted back- Failure to respond to facilities’ requests check visits in which they administered a subset of the JHIC to Failure to plan logistics adequately check the quality of the inspections data. All inspection reports Inspector caseload too low (or too high) Failure to visit faraway facilities were monitored with automated data flags, and some were Failure to include facilities that are not in the government records also manually checked either visually or through double entry to ensure that they reflected the correct results of the inspections. Threats to service take-up or compliance Quality officers also conducted return visits to more than half of Failure to monitor and enforce compliance with sanctions the facilities reported for closure to verify whether they complied Failure of boards and councils to provide necessary licenses Failure to provide support for compliance (e.g., counties to public facilities) with the closure report. What conditions and capacity building is 18 REGULATION FOR SAFETY AND QUALITY OF CARE endeavor are expected to shed light on critical elements to build ” BMC Health Services research underpinning their development and impact. inspections systems at scale. KePSIE’s inspections proved that Research (2012) 12: 329. DOI: 10.1186/1472-6963-12-329. building and successfully operating strong accountability systems Hinchcliff R, Greenfield D, Moldovan M, Westbrook JI, Pawsey M, Mumford is possible in Kenya and similar contexts. V, and J Braithwaite. Narrative synthesis of health service accreditation literature. BMJ Quality & Safety 2012; 21:979–991. References Institute for Health Metrics and Evaluation (2015). GBD Compare. Seattle, Bedoya G, Das J, and Dolinger A. “Randomized Regulation: The Impact of WA: IHME, University of Washington, 2015. Accessed March 2020: Inspections and Quality Standards on Health Markets. ” 2020. Forthcoming http://vizhub.healthdata.org/gbd-compare. Working Paper. Longo DR, Hewett JE, Ge B, and S Schubert. “The Long Road to Patient Bedoya G, Das J, Dolinger A, de Guttry R, Hur YS, and Lee JY. “Regulation ” 2005. JAMA; 294(22): Safety. A Status Report on Patient Safety Systems. as a Policy Lever to Improve Patient Safety and Quality of Care: A Process 2858-2865. Evaluation of the Health Inspection Pilots of the Kenya Patient Safety Impact Ministry of Health, Kenya, The Joint Health Inspections Technical Working Evaluation.” 2020. Forthcoming Working Paper. Group, “Implementation Guidelines for the Joint Health Inspections Bedoya G, Dolinger A, Rogo K, Mwaura N, Wafula F , Coarasa J, Goicoechea Checklist: Checklist for Singular or Joint Inspections for Public and Private A, and J Das. “Observations of infection prevention and control practices ” 2015. Providers by Health Regulatory Bodies under the Ministry of Health , in primary health care, Kenya” . Bulletin of the World Health Organization 2017;95:503-516. doi: http://dx.doi.org/10.2471/BLT.16.179499. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, Moore L, O’Cathain A, Tinati T, Wright D, and J Baird. “Process evaluation of complex Brubakk K, Vist G, Bukholm G, Barach P , and O Tjomsland. “A systematic ” Electronic interventions: UK Medical Research Council (MRC) Guidance. review of hospital accreditation: the challenges of measuring complex document, retrieved on August 18, 2019 from: https://mrc.ukri.org/ ” BMC Health Services Research (2015) 15: 280. DOI: intervention effects. documents/pdf/mrc-phsrn-process-evaluation-guidance-final/. 10.1186/s12913-015-0933-x. Rossi P, Freeman H, and M Lipsey. Evaluation: A Systematic Approach. Campbell ME, Gardner CE, Dwyer JJ, Isaacs SM, Krueger PD, and J Ying. Thousand Oaks, CA: SAGE, 2004. “Effectiveness of Public Health Interventions in Food Safety: A Systematic Review” . Can J Public Health (1998) 89: 197. https://doi.org/10.1007/ Tama E, Khayoni I, Goodman C, Ogira D, Chege T, Gitau N, and F Wafula. BF03404474. “What lies behind successful regulation? A qualitative evaluation of Chege T, Wafula F, Tama E, Khayoni I, Ogira D, Mwaura N, and C Goodman. ” 2020. Forthcoming. Kenya’s health facility inspection reforms. “How much does effective health facility inspection cost? An analysis of Wachter RM. 2010. “Patient Safety at Ten: Unmistakable Progress, Troubling ” 2020. the economic costs of Kenya’s Joint Health Inspection innovations. ” Health Affairs; 29: 165-73. Gaps. Forthcoming. Wholey J, Hatry H, and K Newcomer. Handbook of Practical Program Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, and M Petticrew. Evaluation. San Francisco, CA: Jossey-Bass, 2010. “Developing and evaluating complex interventions: the new Medical Research Council guidance.” BMJ 2008; 337:a1655. DOI: 10.1136/ World Bank Group and Ministry of Health, Kenya, “Implementation Manual bmj.a1655. for Pilots of Intensified Government Inspections: Kenya Patient Safety Flodgren G, Gonçalves-Bradley DC, Pomey MP . “External inspection of ” 2016. Impact Evaluation (KePSIE), compliance with standards for improved healthcare outcomes. ” Cochrane World Health Organization, Organisation for Economic Co-operation and Database of Systematic Reviews 2016, Issue 12. Art. No.: CD008992. Development, and The World Bank. “Delivering quality health services: a DOI: 10.1002/14651858.CD008992.pub3. ” 2018. License: CC BY-NC- global imperative for universal health coverage. Flodgren G, Pomey MP , Taber SA, Eccles MP. “Effectiveness of external SA 3.0 IGO. inspection of compliance with standards in improving healthcare organisation behaviour, healthcare professional behaviour or patient World Health Organization. “Guide for Developing National Patient Safety outcomes. ” Cochrane Database of Systematic Reviews 2011, Issue 11. Art. ” 2014. Electronic document, retrieved on August Policy and Strategic Plan. No.: CD008992. DOI: 10.1002/14651858.CD008992.pub2. 18, 2019 from: https://apps.who.int/iris/handle/10665/148352. Greenfield D, Pawsey M, Hinchcliff R, Moldovan M, and J Braithwaite. World Health Organization. Patient Safety. 2020. Accessed October 1, 2020. “The standard of healthcare accreditation standards: a review of empirical https://www.who.int/patientsafety/en/. REGULATION FOR SAFETY AND QUALITY OF CARE 19 Process Evaluation Team and Funding The process evaluation research team is comprised by Guadalupe Bedoya, Jishnu Das, Amy Dolinger, Rebecca de Guttry, Yoon Sun Hur and Ju Young Lee. Thomas Escande, Seungmin Lee, and Chex Yu provided research assistance throughout the project. The KePSIE implementation teams provided helpful inputs throughout the monitoring of the intervention, interviews and focus group discussions. We especially thank Jorge Coarasa, Ana Goicoechea, Njeri Gitau, Khama Rogo, and Frank Wafula. We also thank Jay Bhattacharya, Paolo Belli, Mickey Chopra, Aidan Coville, Arianna Legovini, and Edit Velenyi for providing valuable comments. Funding was provided by the DIME Impact Evaluation to Development Impact (i2i) fund, the Korea World Bank Group Partnership Facility (KWPF), the Trade and Competitiveness Impact program (Compel), the Knowledge for Change Program (KCP), the Development Research Group (DECRG) at The World Bank, and the Korea Institute for International Economic Policy (KIEP). 20 REGULATION FOR SAFETY AND QUALITY OF CARE