Policy Research Working Paper 8782 The Three-Gap Model of Health Worker Performance Fabliha Ibnat Kenneth L Leonard Luke Bawo Rianna Mohammed-Roberts Health, Nutrition and Population Global Practice March 2019 Policy Research Working Paper 8782 Abstract The Three-Gap Model examines the determinants of these gaps across health workers in a sample can be used low-quality health care by examining the patterns and to diagnose failures in the system as well as evaluate the determinants of three gaps. Using four measures of perfor- outcomes of policy experiments. Using data on pediatric mance—target performance, actual performance, capacity care from hospitals in Liberia, the paper illustrates how the to perform, and knowledge to perform—this paper defines model can be used to investigate the potential for improve- three gaps for each health worker: the gap between target ments in the quality of care from several possible policy performance and what they have the knowledge to do interventions. The analysis of the relationships between (the know gap), the gap between their knowledge and these gaps across health workers in a health system help to their capacity to perform (the know-can gap), and the paint a better picture of the determinants of performance gap between their capacity and what they actually do (the and can assist policy makers in choosing relevant policies can-do gap). The paper demonstrates how the patterns of to improve health worker performance. This paper is a product of the Health, Nutrition and Population Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/research. The authors may be contacted at ibnatf@terpmail.umd.edu, kleonard@arec.umd.edu, and rmohammed@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team The Three-Gap Model of Health Worker Performance∗ , Kenneth L Leonard‡ Fabliha Ibnat† , Rianna Mohammed-Roberts¶ , Luke Bawo§ JEL Classification: C93, I1, J41, O1 Key Words: Pay for Quality, Pay for Performance, Health Care Evaluation, Liberia ∗We acknowledge the funding support from the Health Results Innovation Trust Fund at the World Bank and thank Gil Shapira for his comments on earlier drafts. † Department of Agricultural and Resource Economics, 2200 Symons Hall, University of Maryland, College Park 20742 (email:ibnatf@terpmail.umd.edu). ‡ Department of Agricultural and Resource Economics, 2200 Symons Hall, University of Maryland, College Park 20742 (email: kleonard@arec.umd.edu) §Ministry of Health and Social Welfare, Monrovia, Liberia ¶The World Bank, Washington DC 1 Introduction Despite significant increases in funding directed at improving access to health care in developing and transition countries, important obstacles to quality care remain. Although overwhelming majorities in all countries now have access to a health facility, in many coun- tries and settings, the quality of health care services is low, reducing the value of access. A primary symptom of this low quality is low adherence to medical protocols (Berendes et al., 2011; Das, 2011; Das & Gertler, 2007; Das & Hammer, 2007; Das et al., 2008; Holloway et al., 2013, 2015; Lange et al., 2014; Leonard & Masatu, 2010; Peabody et al., 2017; Rao et al., 2011). Given the significant evidence that increased adherence is one of the most effective ways to improve outcomes and prevent childhood deaths (Chopra et al., 2013; Jaribu et al., 2016; Manzi et al., 2018; Peabody et al., 2014), a continued focus on process quality seems appropriate. An important element of understanding the determinants of low performance has been the documentation of a know-do gap in health care quality: health worker knowledge of correct protocol is often found to be well above performance, suggesting that lack of knowledge is not the only constraint for improved performance (Kabongo et al., 2017; Lange et al., 2014; Mohanan et al., 2015). This paper demonstrates that this focus on the size of the gap does not fully explain the scope of the problem, nor does it shed light on how to best increase performance. Figure 1 graphically illustrates why. The y-axis shows performance and the x-axis shows level of knowledge for a sample of hypothetical health care workers, plotted in blue. Performance is measured as the percentage of tasks the health worker has performed correctly and knowledge is measured as the corresponding percentage of vignette questions the health worker answered correctly. The dashed line shows the line of best fit for this sample of health workers and the solid 45-degree line represents the ideal level of 2 performance for each level of knowledge (ideally, knowledge = performance). The red point marks the average level of knowledge and the average level of performance. The know-do gap is the vertical distance between average performance and average knowledge. Focusing on the size of the average gap comes at the expense of the full picture; it tells us nothing about how the size of the gap changes for different levels of knowledge. Looking instead at the relationship between knowledge and performance (the dashed line) allows us to consider the rate at which health workers are able to convert their knowledge into performance. Figure 1: Performance as a Function of Knowledge Expanding on this insight, we propose a new methodology for examining the determinants of performance and the role that inputs play in improving it. Our model moves away from the one-dimensional approach of measuring gaps towards a multi-dimensional approach of exploring the relationships between the determinants of performance. We break the well- documented know-do gap into two components, the know-can gap and the can-do gap, where “can”, the capacity of the health worker, is measured as the performance of a health worker when they have access to all necessary equipment and resources. We then show the 3 relationship between capacity, knowledge, and performance by modeling can as a function of know and do as a function of can. With the focus now on the relationships between these factors rather than the size of the gaps, our model allows us to ask and answer two key questions. First, what are the slopes? Knowing the slope tells us the gain in performance we can achieve if we increase knowledge or capacity — through training or improving inputs — by one unit. Second, what are the determinants of these slopes? We argue that health worker motivation and improvements in equipment quality are two key inputs that can help increase the slopes and move them closer to the ideal 45-degree line. In this sense, we are modeling the know- can and can-do relationships as nonlinear functions of inputs by interacting inputs with baseline levels of knowledge and capacity. This allows us to measure how well policy levers like improvements in infrastructure or incentives that increase motivation can improve the relationships between knowledge and capacity, and capacity and performance respectively. In other words, it allows us to examine which policies increase the rate at which health workers can translate knowledge into capacity and capacity into performance. These inputs can change the know-can and can-do relationships in different ways, leading to different policy implications. Some inputs may improve outcomes for all health workers, causing the pattern to simply shift upwards without changing its slope. In this paper we show that positive worker behavior, a motivation score measuring how proactive and positive a health worker is while on the job, is an input that causes the linear can-do function to shift upwards. Increasing positive worker behavior increases performance for health workers at all levels of capacity. Other inputs may cause the slope to increase by rotating the function, improving outcomes for some health workers and decreasing it for others. We show that feeling valued by the facility, a motivation score measuring the worker’s perception of whether she is valued by the hospital, is an input that causes the can-do function to rotate. 4 The two different ways an input can affect the know-can or can-do relationships have different policy implications. If an input simply shifts the function upwards, it can be implemented as a stand-alone intervention for all health workers meaning it does not need to be targeted to certain types of health workers or implemented alongside complimentary policies. Our empirical results show that a policy that increases positive worker behavior will improve performance uniformly for workers at all levels of capacity. Conversely, if an input increases the slope of a function by rotating the function, then it can be most effectively implemented as either a targeted intervention or alongside a complimentary intervention. We show that a policy that increases feeling valued by the facility will only increase performance for workers above a certain capacity threshold and will decrease it for workers below the threshold. Therefore, policy makers should either increase feeling valued by the facility for workers above this threshold only, or increase feeling valued by the facility and capacity through two interventions simultaneously. In section 2 we introduce the Three-Gap model and outline three specifications that demonstrate how the role each input plays in improving performance can be empirically examined. In section 3 we use data from hospitals in Liberia to illustrate how this model can paint a fuller picture and provide policy makers with better information about how to improve the quality of care. Section 4 concludes. 2 The Three-Gap Model of Quality of Care The Three-Gap model starts with three measures of performance — performance, capac- ity and knowledge — and three gaps: the gap between what a health worker should be doing and what they have the knowledge to do (the know gap); the gap between what they have 5 the knowledge to do and their capacity to perform (the know-can gap) and the gap between what they have the capacity to do and what they do (the can-do gap). In this section, we show that these measures are not simply summary statistics for a health worker or system but can be used to analyze the determinants of quality and the possibilities for effective interventions. 2.1 A graphical illustration of the model To show the overall relationship between knowledge, capacity, and performance and these three gaps, we introduce the Three-Gap model illustrated with four interconnected graphs in Figure 2. Graph 1 in the upper left-hand corner displays capacity (C) and performance (D) and represents the can-do gap. Graph 2 on the upper right displays knowledge (K) and performance (D) and is used to show the know gap. Graph 4 in the lower right displays knowledge and capacity and represents the know-can gap. Graph 3 in the bottom left displays capacity (C) on both axes, which simply reflects capacity between the vertical axis in Graph 4 and the horizontal axis in Graph 1. The characteristics of each health worker can then be represented by three sets of co- ordinates, where (K, C ) is the knowledge-capacity pair, (C, D) is the capacity-performance pair, and (K, D) is the knowledge-performance pair. In order to interpret Figure 2, first focus on Graph 1, and consider a single representative health worker with capacity C1 and performance D1. Using the 45-degree line, we can project (represented by blue arrows) the health worker’s capacity C1 onto the Y-axis to determine her “capacity to perform” — the level of performance if actual performance were equal to capacity. The can-do gap is represented by the distance labeled G3. Note that this gap is also the difference between the capacity performance pair (C1, D1) and the 45-degree line. The same health worker’s 6 Figure 2: The Three-Gap Framework knowledge-capacity pair is represented in Graph 4, at point (K1, C1). Note that Graph 3 reflects capacity C1 on the horizontal axis in Graph 1 onto the vertical axis in Graph 4. Knowledge is also represented, directly, on the horizontal axis of Graph 2 and the 45-degree line in that graph projects (represented by the green arrows) the worker’s knowledge onto the Y-axis of Graph 1 to determine her “knowledge to perform” — the level of performance if actual performance were equal to knowledge. The know-can gap is represented by the distance labeled G2, which can be seen both as the gap between (K1, C1) and the 45-degree line in Graph 4 and as the difference between the capacity to perform and knowledge to perform on the Y-axis of Graph 1. Finally, Graph 2 shows the worker’s capacity-performance pair at (K1, D1). The know gap is represented by the distance labeled G1, which can be seen both as the gap between the 45-degree line and the target level (the horizontal line at the top) and as the difference 7 between knowledge to perform and target performance in the Y-axis of Graph 1. Although quantifying the sizes of the average gaps G1, G2, and G3 provides a snapshot of quality for a health worker or system, it is less useful as tool for exploring the possible effectiveness of policy interventions. By modeling capacity as a function of knowledge and performance as a function of capacity, Figure 2 illustrates how we can move away from one-dimensional measures of the gaps to a multi-dimensional approach of exploring the relationships between knowledge, capacity, and performance. Instead of asking how to reduce the size of the gaps of the average health worker, we instead ask how we can improve the relationships between these three measures. 2.2 The relationships between performance, capacity and knowledge Figure 3 shows how analyzing relationships rather than gap sizes can give a more complete picture of the drivers of poor performance. In this figure, the dashed lines show the lines of best fit for a hypothetical sample of health care workers. Knowledge and capacity (Graph 4) have a strong positive one-to-one relationship (a one-unit increase in K increases C by one unit) and capacity and performance (Graph 1) have a weak positive relationship (a one- unit increase in C increases D by half a unit). This is illustrated by the dashed lines on Graphs 4 and 1, which show slopes of 1 and 12 respectively. Note that just as the points representing a health worker are connected across the graphs, the slopes are related as well. Given the relationships between know, can, and do, what is the most effective way of increasing performance? Consider a policy maker with two possible levers: increasing the knowledge of all health workers or increasing their motivation (an input that affects the 8 Figure 3: The effect of increasing knowledge on performance can-do relationship in Graph 1). What happens if we use training to increase performance? Since capacity is a function of knowledge and performance is a function of capacity, any gains in performance from knowledge are only as good as the existing know-can and can-do relationships. Training increases K from K1 to K2, and we see a similar increase in C from C1 to C2. However, the weak relationship between capacity and performance results in a small increase in D from D1 to D2. Although performance is not constrained by knowledge in this example, it is constrained by capacity. Therefore, any increase in knowledge translates into a 1-to-1 increase in capacity, but the increase in capacity does not translate into a similar increase in performance. Therefore, a motivation policy that improves the can-do relationship directly may be more useful. What happens if we improve motivation? Figure 4 shows two possible outcomes of using 9 motivation to increase performance: a shift versus a rotation. On one hand, an increase in motivation may cause the line F1 to shift upwards to F2, increasing performance for health care workers at every level of capacity. The health care worker that was originally at point (C1, D1) is now at (C3, D3), a performance level higher than D2. On the other hand, an increase in motivation may cause F1 to rotate to F3. In this case, health workers who have capacity levels below the rotation point, Cthreshold, see a decrease in performance in response to the motivation policy. The health worker who was originally at point (C1, D1), which is below the threshold capacity, now has a lower performance level of D4. However, health workers above Cthreshold have improved performance. Policy makers should target such motivation interventions at health workers above the threshold value, or implement the intervention in conjunction with a policy that also increases capacity. Figure 4: Two possible effects of increasing motivation on performance An empirical illustration of the know-do gap from Leonard et al. (2007) compares the 10 performance outcomes of a motivation policy that shifts the line versus a training policy that simply causes improvements along the line. Figure 5 shows multiple health workers’ knowledge and performance in outpatient clinics in Tanzania. This graph corresponds to Graph 2 in Figure 3. The know-do gap for each health worker is the vertical distance from the point to the 45-degree line. For all but four health workers, knowledge exceeds or equals performance and for a significant number of health workers, performance is much lower than knowledge: the know-do gap is usually greater than or equal to zero. Average knowledge is about 50 percent (the knowledge to adhere to protocol) and average performance (adherence to protocol) is about 34 percent, which means the average know-do gap is about 16 points and the know gap is about 50 points. Figure 5: Empirical Example of the Know-Do Quadrant Sources: Data from Leonard & Masatu (2010); Leonard et al. (2007) A focus on the average size of these gaps does not indicate how to best improve perfor- mance. Instead, analyzing the determinants of the relationship between know and do is more telling. Therefore, we add a dashed line representing the predicted relationship between per- 11 formance and knowledge given the observed distribution assuming zero competence leads to zero performance. If we assume that the average distribution of health workers is the best possible approximation of what would happen as a result of a policy intervention, then the dashed line predicts the result of an increase in knowledge for the average health worker. Consider a training intervention that increases knowledge by 20 percentage points (from 50% to 70%): we can see that the result would be an increase in performance of only 5 percentage points. On the other hand, an intervention that increases effort by shifting the line closer to the ideal 45-degree line can achieve an increase in performance of 15 percentage points. In this setting, it makes more sense to focus first on effort and motivation than on knowledge. In Figure 5, the slope of the relationship was assumed to be constant. In the real world, this may be too strong an assumption. Alternatively, it is possible to introduce a motivation tool that alters the slope between capacity and performance, as shown in Figure 4. For example, if policy makers implement a reward for achieving a fixed level of performance, it might encourage low performing health workers to improve performance while having no effect on the performance of high capacity workers who already achieved the target. Figure 5 also highlights the key shortcoming of improving performance via knowledge rather than effort in the current literature (also highlighted by Bucknall, 2012; Kabongo et al., 2017; Mohanan et al., 2015). A systematic review of the effects of a broad range of knowledge and training interventions finds that increasing health workers’ knowledge often has little or ambiguous effects on performance (Pantoja et al., 2017). The small increase in performance from knowledge in Figure 5 reflects these findings, and indicates that health facilities may see larger gains by increasing effort instead. Our model allows us to see this by breaking up the well–documented know-do gap into two components and focusing on changing the slopes that characterize the relationship between knowledge and performance in order to improve performance. 12 2.3 A mathematical representation of the Three-Gap model The relationships shown graphically in Figure 3 can be represented mathematically: D = η 1 + µ 1 C + E3 (1) C = η 2 + µ 2 K + E2 (2) K = µ31 + E1, (3) where equation 1 shows the relationship in Graph 1; Equation 2, that of Graph 4; and Equation 3 the relationship in Graph 2. The coefficients µ1, µ2, and µ3 represent the slopes as seen in Figure 3: how well health workers translate capacity into performance, knowledge into capacity, and how far knowledge is from the target level. One of the advantages of this representation is that these relationships can be more easily understood in the more nuanced space where multiple measures are considered. Equations 4 - 6 expand equations 1 - 3 to show the determinants of both the constants (η) and the slopes (µ). The policy levers include training (T ), equipment and infrastructure (I) and motivation (M ) which interacts with the model through the unmeasured effort of health workers (E(M )). D =α1+β1E(M ) + ν1C(K, I ) + γ1C(K, I ) ∗ E(M ) + E1 (4) C = α 2+ β2 I + ν2K(T ) + γ2K(T ) ∗ I + E2 (5) K = β3T + ν31 + E3 (6) 13 We can analyze Graph 4 of figure 3 using equation 5. We rewrite equation 5 as: C = α2 + (ν2 + γ2I )K(T ) + β2I + E2. (7) This expression shows that the slope of the dashed line in Graph 4 is ν2 + γ 2 I, and depends on infrastructure, I . If this slope is close to 1, then the know-can gap is small, and improving knowledge will have a significant effect on improving capacity. This is the case in Figure 3. Similarly, we can analyze Graph 1 using equation 4. We rewrite equation 4 as: D = α1 + (ν1 + γ1E(M ))C(K, I ) + β1E(M ) + E1, (8) This expression shows that the overall slope ν1 + γ1E(M ) depends on motivation, M . In Figure 3 this slope is not close to 1, indicating that the improvement in capacity implied by equation 7 does not translate into a significant increase in performance. However, we see that the slope depends not only on ν1, but also on E(M ). In order to increase performance, the more effective policy is to increase effort through motivation. As discussed in Section 2.2, an increase in E(M ) may affect the can-do relationship in one of two ways. If β1, the effect of E(M ) on the intercept, is positive and γ1, its effect on the slope, is insignificant, motivation may cause the intercept to increase, shifting the function upwards as illustrated by the movement of function F1 to F2 in Figure 4. If γ1 is positive and β1 is negative, an increase in motivation may cause the slope to increase, as illustrated by the rotation of function F1 to F3 in Figure 4. 14 3 An Empirical Example We illustrate the uses of this model using data from pediatric care units for a baseline evaluation of hospitals in Liberia (Bawo et al., 2015). In this evaluation, performance is measured with actual patients (do) as well as on vignettes designed to measure both knowl- edge and capacity. In the vignettes, knowledge (know) is measured by asking health workers to demonstrate how they would treat a patient “assuming you have all the necessary equip- ment and supplies.” The capacity (can) of health workers is measured by evaluating whether the equipment and supplies the health worker proposed to use in the vignette were actually present. For example, if a health worker knows that protocol requires the use of a particular medicine, but that medicine is out of stock, capacity will be lower than knowledge. The instruments and data collection are discussed in more detail in Bawo et al. (2015) and in Appendix A. Figure 6 shows the average relationship among the three measures of quality for each health worker in the sample.1 In addition, we plot the average performance, capacity, and knowledge as a green point in each graph and use them to show the average value of each gap. The different interpretations of the average gaps and the underlying relationships between the input measures are informative. The average know gap is about 55%, the average know-can gap is about 2.5%, and the average can-do gap is 8.9%. The imbalance in the gaps might suggest that increasing knowledge is the most important way to improve performance. In addition, the small can-do and know-can gaps might suggest that improvements in equipment and motivation are not important. However, the relationships between knowledge and capacity and capacity and perfor- 1The relationship shown as dashed lines is derived from non-parametric representation of each health worker in the respective two-dimensional spaces. We used locally weighted kernel regression with a bandwidth of 10 percentage points and the Epanechnikov kernel. 15 Figure 6: The Three-Gap model in pediatric hospital care, Liberia The relationship shown as dashed lines is derived from non-parametric representation of each health worker in the respective two-dimensional spaces. We used locally weighted kernel regression with a bandwidth of 10 percentage points and the Epanechnikov kernel. mance suggest a different story. Three important patterns are immediately visible. First, there is essentially no know-can gap at any point in the distribution in these data. Health workers at all levels of knowledge are able to use their training and there is no difference between their knowledge to perform and their capacity to perform. Second, for many health workers there is a large can-do gap and for the whole distribution, performance does not improve significantly when capacity improves (the slope is relatively flat).2 2Note that in Graph 2 of Figure 6, health workers with low levels of knowledge and capacity actually perform better than their knowledge or capacity. Part of the reason for this may be that health workers 16 Table 1 describes each relationship linearly. The three columns correspond to equations 1-3 outlined in Section 2. Column 1 is given by K = µ31 + E 3, column 2 by C = η2 + µ2K +E 2, and column 3 by D = η1 + µ1C + E1. Column 1 describes the know gap, which is explained only by a constant because the target level of performance is unchanging. Columns 2 and 3 describe the know-can and can-do relationships, which are graphically represented by Graph 1 and Graph 4 in Figure 6. Table 1: The know, know-can, and can-do gaps (1) (2) (3) Know Can Do Target 0 (.) Know 0.980∗∗∗ (0.0300) Can 0.157∗ (0.0850) Constant 0.451∗∗∗ -0.0156 0.270∗∗∗ (0.0336) (0.0157) (0.0407) Observations 831 831 831 Adjusted R 2 0.000 0.937 0.023 Clustered standard errors in parentheses. ‡Target is a constant, so the regression produces only a constant: there cannot be a systematic relationship between knowledge and the target. ∗ p < 0.1, ∗∗ p < 0.05, ∗∗∗ p < 0.01 Ideally, these coefficients would estimate the variance in performance that resulted from exogenous variation in motivation or capacity from, for example, a randomized control trial3 This study, as a baseline, includes only cross-sectional variation, so we cannot make causal in hospitals work in teams so a clinician with low levels of knowledge might perform better when part of a team that includes more competent providers. Indeed, when we reanalyze the data focusing only on cases in which health workers worked as individuals, this area of the figure disappears. We retain this particular specification in the analysis because teams matter in this context. 3Variation in performance (not volume of services delivered) driven by exogenous variation in capacity and motivation is seen in Basinga et al. (2011); Brock et al. (2016, 2018) and there are many studies of the link between performance and exogenous variation in training exposure (Jamtvedt et al., 2013). 17 statements. However, the model is designed to be used with exogenous variation, so we demonstrate its value in this context despite the danger of assuming these results are causal. Taking the coefficients at face value, a one-unit increase in knowledge should lead to a .98 increase in capacity, but this one-unit increase in capacity will lead to only a .16 increase in performance. On the other hand, the 0.84 unit gap in performance for each unit of capacity suggests that significant increases in performance are possible even without any increase in knowledge or capacity. What can we learn about the value of increased knowledge versus that of motivation? 3.1 Investigating the role of policy levers In this study, we have three sources of data that we can use to examine the role of potential policy inputs on performance: cadre or level of training of each health worker, infrastructure and equipment levels in each hospital and the responses of each health worker to a survey on their attitudes and motivation. Table 2 shows the effect of the relevant policy levers on the know gap. This regression corresponds to equation 6, K = β3T + ν31 + E3, where T is a vector of dummy variables for each cadre. Note that the target (a proper evaluation of children in the outpatient and emergency wards of a hospital) does not vary with cadre. The know gap does not appear to be strongly affected by cadre: the results show that, compared to the omitted category — physician’s assistants — only certified midwives increase the know gap, meaning that their levels of knowledge are further from the target. Since certified midwives are plausibly the only group whose training does not include pediatric care, this result is not particularly surprising. Using this narrow measure of training, we conclude that, with the possible 18 exception of certified midwives, all clinicians in this setting, are equally qualified to provide high-quality care. Although we already know that capacity has a one-to-one relationship with knowledge in this setting, we can still examine the role of equipment in the know-can gap in Table 3. This regression corresponds to equation 5, C = α2 + β2I + ν2K(T ) + γ2K(T ) ∗ I + E2. In the analysis of the relationships, it is important to include both the inputs of knowledge K and equipment I and the interaction between the two inputs. The level of equipment might help to explain the intercept on Graph 4 of the Three-Gap model: do increased levels of equipment improve the overall capacity of health workers? The interaction of know and the measure of infrastructure can help to explain the slope: as knowledge increases, does it make a difference if the equipment score is higher? In this case, unsurprisingly, the level of equipment available does not significantly alter either the intercept or the slope. This does not mean every facility has adequate equipment, only that their performance is not explained by the level of equipment. To examine the determinants of the can-do relationship, we summarize the results of 54 questions on the attitudes and motivation survey, coming up with five summary measures of motivation: self-satisfaction, feeling valued by the facility, job satisfaction, positive facility characteristics and positive worker behavior.4 The individual questions that make up the summary statistics are listed in Appendix B and each score is described briefly here: 4When the survey was conducted, the answers to the motivation questions were scored on a scale of 1 to 5, where 1 corresponds to “strongly disagree” and 5 corresponds to “strongly agree.” Afterwards, we flipped the scores of all negatively worded motivation statements for which answering “strongly disagree” indicated high motivation and “strongly agree” indicated low motivation. For example, the score for the negatively worded statement, “I do not get feedback from my superior so it is hard to improve my performance,” has been flipped so that a 1 means that the worker strongly feels that she does not get feedback from her superior and a 5 means that she feels she does. Therefore, all motivation questions used in the analysis in this paper are measured on a scale of 1 to 5 where 1 indicates low motivation and 5 indicates high motivation. 19 Table 2: Policy levers that affect the know gap (1) Know Target ‡ 0 (.) Certified Midwife -0.191∗ (0.101) Registered Nurse -0.120 (0.0974) Associate Degree Nurse -0.0394 (0.148) Bachelors of Science Nurse -0.0700 (0.0879) Medical Doctor -0.0270 (0.0839) Other -0.101 (0.146) Constant 0.508∗∗∗ (0.0839) Observations 767 Adjusted R2 0.057 Clustered standard errors in parentheses. ∗ p < 0.1, ∗∗ p < 0.05, ∗∗∗ p < 0.01 Note that the category “Physician Assistant” was chosen as the reference category for cadre in column 1 because it has the highest number of observations. It requires the second most years of education and training, after MDs. ‡Target is a constant, so the regression produces only a constant. 20 Table 3: Policy levers that affect the know-can relationship (1) Can Know 0.921∗∗∗ (0.114) Average of necessary equipment working -0.0485 (0.106) Know × Average of necessary equipment working 0.103 (0.199) Constant 0.0119 (0.0583) Observations 831 Adjusted R2 0.937 Clustered standard errors in parentheses. ∗ p < 0.1, ∗∗ p < 0.05, ∗∗∗ p < 0.01 Self-satisfaction: Eight questions related to the worker’s perception of herself and not directly related to the facility or her role in it. Most of them measure her feelings of self- worth. For example, “I feel that I have a number of good qualities.” Feeling valued by the facility: Nine questions related to the worker’s perception of whether she is a valued and useful employee in the facility. For example, “In this organization, I am valuable.” Job satisfaction: Six questions related to the worker’s satisfaction with the facility and feelings of fulfillment in her job or profession. For example, “I would recommend to my children that they choose the health profession” and “I am satisfied with the opportunity to use my abilities in my job.” Positive facility characteristics: Nineteen questions related to the facility’s processes, resources, and coworkers. For example, “This facility has a fair system for rewarding staff” 21 and “Too often the referral system does not work efficiently.” Positive worker behavior : Twelve questions related to the worker’s behavior on the job. For example, “I am keen to use any new tools to improve my performance” and “When I am not sure how to treat a patient’s condition I look for information or ask for advice.” Note that the unifying theme for categories 1-3 is that they are subjective questions and capture the health worker’s perception of herself and her work environment. In contrast, categories 4 and 5 are more objective and give information about the characteristics of the facility and the worker’s role in it. All questions are coded so that positive/good responses get higher scores by reversing the score for negatively worded questions such as “Too often the referral system does not work efficiently.” We use confirmatory factor analysis (CFA) to determine the extent to which these five motivation dimensions, defined a priori, are reflected in the data. Following the CFA best practices outlined by Borghi et al. (2018), we use structural equation modeling to assess the standardized factor loading values for each observed motivation variable onto the five latent factors we have identified above. Overall, we find that the factor loadings for most variables were significant at the p < .001 level. Only the factor loading for one variable (“facility provides access to relevant trainings”) onto the fourth latent construct (positive facility characteristics) is not significant. Goodness of fit measures suggest acceptable fit for the model. Although the chi-square test indicates a poor model fit to the data (chi-square = 14784.18 with p < .001), this measure of fit is sensitive to sample size and it is possible that the test will result in a rejection of an appropriate model when the sample size is large. The sample size of 783 in this analysis is considered to be large in the context of CFA. 5 Therefore, we use the root mean square error of approximation (RMSEA) to assess 22 goodness of fit, as 5The minimum sample size for CFA is 200 observations (Borghi et al., 2018). The sample in this analysis is almost four times this size. 23 recommended by Borghi et al. (2018). The RMSEA measure of .08 indicates acceptable model fit. After confirming that each of the five motivation dimensions are reflected in the data, we use factor analysis with varimax rotation to construct a latent variable for each category.6 We then normalize each factor to take on values in the range 0 to 1 in order to make the regression results easier to interpret. In order to demonstrate the Three-Gap model approach, we use columns 1-3 of Table 4 to show the results of three less nuanced analyses of the effect of motivation on performance, and compare the implications of these results to the Three-Gap model analysis in column 4. Column 1 shows the simplest analysis of the effect of motivation on performance, ignoring the role of capacity: is performance linked to measures of health worker motivation? This analysis suggests that self-satisfaction, job satisfaction and positive facility characteristics have no effect on performance. In contrast, increasing feeling valued by the facility leads to a decrease in performance and increasing positive worker behavior leads to an increase in performance. Why would a worker who feels more valued by the facility have lower performance? How should policy makers interpret this result? In Column 2 we examine the effect of motivation on the size of the can-do gap directly, calculated as the health worker’s performance score subtracted from their capacity score. The results show that neither positive worker behavior nor facility values the worker now significantly changes the size of the can-do gap. Column 3 examines performance after controlling for capacity and finds similar results to Column 1. Finally, we examine the effect of motivation using the Three-Gap framework in Column 22 6See Appendix C for rotated tables showing factor loading values. 22 4, by interacting each motivation factor with capacity, thereby allowing the relationship be- tween capacity and performance to change with each measure of motivation. This regression corresponds to equation 4, D = α1 + β1E(M ) + ν1C(K, I) + γ1C(K, I) ∗ E(M ) + E1, where M is a vector of the five motivation factors outlined above. While columns 1-3 confound the role of motivation in improving performance, the results in Column 4 clarify the role of motivation and help explain the counter-intuitive results of the simple analysis. We see that allowing feeling valued by the facility to interact with Can causes the y-intercept to decrease but that the slope of the overall relationship between capacity and performance to significantly increase. This indicates that an increase in feeling valued by the facility allows health workers to translate capacity into performance at a higher rate. However, the decrease in the intercept indicates that the can-do gap itself only decreases for workers with high capacity. For workers with low capacity, being more valued by the facility actually decreases performance. In other words, increasing feeling valued by the facility causes the can-do function to rotate. The intercept for positive worker behavior is now no longer significant, however, the p-value of the coefficient is 0.114 suggesting some possible positive effect. The coefficient for the slope (the interaction between the factor and capacity) is not significant. This could (cautiously) be interpreted to mean that workers with high levels of positive worker behavior provide higher performance independent of their level of capacity. In other words, increasing positive worker behavior causes the can-do function to shift upwards. This has two policy implications for how to use the motivation policy lever and capacity inputs to improve performance. First, if a health facility implements a program that improves how the facility values the health worker, a simultaneous intervention that increases capacity for low capacity workers at baseline is necessary for workers to benefit from the increase in performance from the motivation intervention. If, instead, the policy maker does not have 23 Table 4: Policy levers that affect the can-do relationship (1) (2) (3) (4) Do Can-do Gap Do Do self-satisfaction -0.0375 0.110 -0.0527 0.248 (0.0834) (0.168) (0.0853) (0.310) feeling valued by the facility -0.378∗∗ 0.167 -0.334∗∗ -1.326∗∗ (0.154) (0.251) (0.138) (0.514) job satisfaction 0.0119 0.234 -0.0394 -0.0112 (0.183) (0.231) (0.173) (0.535) positive facility characteristics 0.0782 -0.167 0.0968 -0.0261 (0.0979) (0.141) (0.0872) (0.257) positive worker behavior 0.395∗∗∗ -0.264 0.368∗∗ 0.874 (0.146) (0.298) (0.149) (0.548) Can 0.209∗∗ -0.422 (0.0864) (0.570) Can × self-satisfaction -0.698 (0.747) Can × feeling valued by the facility 2.235∗∗ (1.016) Can × job satisfaction -0.195 (1.443) Can × positive facility characteristics 0.279 (0.645) Can × positive worker behavior -0.867 (1.172) Constant 0.297∗∗∗ -0.00902 0.237∗∗∗ 0.469∗∗ (0.0844) (0.115) (0.0878) (0.226) Observations 616 616 616 616 Adjusted R2 0.043 0.022 0.074 0.092 Clustered standard errors in parentheses ∗ p < 0.1, ∗∗ p < 0.05, ∗∗∗ p < 0.01 24 the ability to increase feeling valued by the facility, then this analysis implies that it would also be effective to identify workers who work in facilities where the feeling valued by the facility score is high and increase their capacity. Thus, the feeling valued by the facility motivation factor works not by increasing the level of health worker performance, but by decreasing the can-do gap for high capacity workers. On the other hand, if a policy maker knows how to change the way a health worker approaches her job, improving the positive worker behavior score, there is no reason to specifically target particular facilities or to pair the intervention with another intervention. Improving this factor will cause increases in performance at every level of capacity, closing the can-do gap for all workers. 4 Conclusion When the goal of policy makers is to increase the performance of health workers, it is important to understand what can and cannot be done to improve it. Policy makers are beginning to learn that, even though knowledge of proper protocol is low, improving knowledge often fails to have any significant impact on performance. Advancing from that position, we suggest that this model allows for even more nuanced understanding of the ways that policy makers can improve performance. Using an example from a hospital setting in Liberia to illustrate the model, we show the value of measuring two intermediate inputs into health — knowledge to perform and capacity to perform — and comparing these to target performance as well as actual performance. In this setting, we show that knowledge, capacity and performance are all low, but that performance is only limited (currently) by knowledge and motivation. In other words, in this 25 setting, improvements in capacity are very unlikely to lead to improvements in performance. At first glance it seems that knowledge will not have an important impact on performance because there is a very weak link between knowledge and performance. However, when we use the model to take a closer look, we can see that knowledge might interact with a measure of motivation — the degree to which a health worker feels valued by their facility. As knowledge (and therefore capacity) increases, health workers who feel valued by their facility significantly improve their performance. By deliberately examining performance through other models that are less nuanced than the Three-Gap model, we can show that this potentially important interpretation of the impact of policy could have been missed. In fact, if we look at performance in isolation, it looks as if feeling valued by one’s health facility significantly decreases performance on average. This demonstrates one of the important advantages of a more nuanced model. This paper examines baseline data for a pay-for-quality intervention in hospitals in Liberia. The project is providing payments to facilities that improve performance as mea- sured by quality (not quantity) and is paired with a plan to improve both training and equipment and facilities in hospitals. This baseline analysis suggests that the improvements in equipment are unlikely to lead to improvements in quality, but that the improvements in training might improve performance specifically because they are paired with an increase in motivation. If the project succeeds in motivating hospital administrators to pay atten- tion to (value) the health workers within their facilities, then our analysis suggests that improvements in training will lead to improvements in performance. 26 References Basinga, P., Gertler, P. J., Agnes, S., & Sturdy, J. (2011). Effect on maternal and child health services in Rwanda of payment to primary health-care providers for performance: an impact evaluation. Lancet, 377(9775), 1421 – 1428. Bawo, L., Leonard, K. L., & Mohammed, R. (2015). Protocol for the evaluation of a quality- based pay for performance scheme in Liberia. Implementation Science : IS, 10. Berendes, S., Heywood, P., Oliver, S., & Garner, P. (2011). Quality of Private and Public Ambulatory Health Care in Low and Middle Income Countries: Systematic Review of Comparative Studies. PLoS Medicine, 8(4). Borghi, J., Lohmann, J., Dale, E., Meheus, F., Goudge, J., Oboirien, K., & Kuwawenaruwa, A. (2018). How to do (or not to do). . . Measuring health worker motivation in surveys in low- and middle-income countries. Health Policy and Planning, 33(2), 192–203. Brock, J. M., Lange, A., & Leonard, K. L. (2016). Generosity and Prosocial Behavior in Healthcare Provision Evidence from the Laboratory and Field. Journal of Human Resources, 51(1), 133–162. Brock, J. M., Lange, A., & Leonard, K. L. (2018). Giving and promising gifts: Experimental evidence on reciprocity from the field. Journal of Health Economics, 58, 188–201. Bucknall, T. (2012). Bridging the Know-Do Gap in Health Care through Integrated Knowl- edge Translation. Worldviews on Evidence-Based Nursing, 9(4), 193–194. Chopra, M., Mason, E., Borrazzo, J., Campbell, H., Rudan, I., Liu, L., Black, R. E., & Bhutta, Z. A. (2013). Ending of preventable deaths from pneumonia and diarrhoea: an achievable goal. Lancet (London, England), 381(9876), 1499–1506. 27 Das, J. (2011). The quality of medical care in low-income countries: from providers to markets. PLoS medicine, 8(4), e1000432. Das, J. & Gertler, P. (2007). Practice-Quality Variation in Five Low-Income Countries: A Conceptual Overview. Health Affairs, 26(3). Das, J. & Hammer, J. (2007). Location, location, location: Residence, Wealth and the Quality of Medical Care in Delhi, India. Health Affairs, 26(3). Das, J., Hammer, J. S., & Leonard, K. L. (2008). The Quality of Medical Advice in Low- Income Countries. Journal of Economic Perspectives, 22(2), 93–114. Holloway, K. A., Ivanovska, V., Wagner, A. K., Vialle-Valentin, C., & Ross-Degnan, D. (2013). Have we improved use of medicines in developing and transitional countries and do we know how to? Two decades of evidence. Tropical Medicine & International Health, 18(6), 656–664. Holloway, K. A., Ivanovska, V., Wagner, A. K., Vialle-Valentin, C., & Ross-Degnan, D. (2015). Prescribing for acute childhood infections in developing and transitional countries, 1990–2009. Paediatrics and International Child Health, 35(1), 5–13. Jamtvedt, G., Young, J., Kristoffersen, D., Thomson O’Brien, M., & Oxman, A. (2013). Audit and feedback: effects on professional practice and health care outcomes (Review). The Cochrane Database of Systematic Reviews, (6). Jaribu, J., Penfold, S., Manzi, F., Schellenberg, J., & Pfeiffer, C. (2016). Improving insti- tutional childbirth services in rural Southern Tanzania: a qualitative study of healthcare workers’ perspective. BMJ Open, 6(9), e010317. Kabongo, L., Gass, J., Kivondo, B., Kara, N., Semrau, K., & Hirschhorn, L. R. (2017). Implementing the WHO Safe Childbirth Checklist: lessons learnt on a quality improvement 28 initiative to improve mother and newborn care at Gobabis District Hospital, Namibia. BMJ Open Qual, 6(2), e000145. Lange, S., Mwisongo, A., & Mæstad, O. (2014). Why don’t clinicians adhere more consis- tently to guidelines for the Integrated Management of Childhood Illness (IMCI)? Social Science & Medicine, 104, 56–63. Leonard, K. L. & Masatu, M. C. (2005). The use of direct clinician observation and vignettes for health services quality evaluation in developing countries. Social Science & Medicine, 61(9), 1944–1951. Leonard, K. L. & Masatu, M. C. (2010). Professionalism and the know-do gap: exploring intrinsic motivation among health workers in Tanzania. Health Economics, 19(12), 1461– 1477. Leonard, K. L., Masatu, M. C., & Vialou, A. (2007). Getting Doctors to do their best: the roles of ability and motivation in health care. Journal of Human Resources, 42(3), 682–700. Manzi, A., Nyirazinyoye, L., Ntaganira, J., Magge, H., Bigirimana, E., Mukanzabikeshimana, L., Hirschhorn, L. R., & Hedt-Gauthier, B. (2018). Beyond coverage: improving the quality of antenatal care delivery through integrated mentorship and quality improvement at health centers in rural Rwanda. BMC Health Services Research, 18, 136. andez, M., Das, V., Giardili, S., Goldhaber-Fiebert, J. D., Rabin, Mohanan, M., Vera-Hern´ T. L., Raj, S. S., Schwartz, J. I., & Seth, A. (2015). The Know-Do Gap in Quality of Health Care for Childhood Diarrhea and Pneumonia in Rural India. JAMA Pediatrics, 169(4), 349–357. Pantoja, T., Opiyo, N., Lewin, S., Paulsen, E., Ciapponi, A., Wiysonge, C. S., Herrera, ˜aloza, B., Dudley, L., Gagnon, M.-P., Garcia Marti, S., & Oxman, C. A., Rada, G., Pen 29 A. D. (2017). Implementation strategies for health systems in low-income countries: an overview of systematic reviews. In The Cochrane Library. John Wiley & Sons, Ltd. Peabody, J. W., DeMaria, L., Smith, O., Hoth, A., Dragoti, E., & Luck, J. (2017). Large- Scale Evaluation of Quality of Care in 6 Countries of Eastern Europe and Central Asia Using Clinical Performance and Value Vignettes. Global Health: Science and Practice, 5(3), 412–429. Peabody, J. W., Shimkhada, R., Quimbo, S., Solon, O., Javier, X., & McCulloch, C. (2014). The impact of performance incentives on child health outcomes: results from a cluster randomized controlled trial in the Philippines. Health Policy and Planning, 29(5), 615– 621. Rao, M., Rao, K. D., Kumar, A. K. S., Chatterjee, M., & Sundararaman, T. (2011). Human resources for health in India. Lancet (London, England), 377(9765), 587–598. Appendices A Instruments The instruments used to measure knowledge, capacity, and performance are available at https://sites.google.com/site/hfqualityassessment, on the page titled Hospital Quality Assessment. The measurement of these three elements comes from a series of yes or no questions on each instrument. For each condition, there is something that the health worker is supposed to do (take the patient’s temperature or check that the patient has signed a consent form, for example) and the member of the research team administering the case 30 study, vignette or direct observation vignette will indicate whether that thing was done. The knowledge, capacity and performance score for each health worker is the percentage of items required or suggested by protocol that are actually administered. This follows the standard process for scoring direct observation and vignettes as outlined in Leonard & Masatu (2005). Knowledge and capacity are measured for identical procedures and differentiated by the use of specific questions about the availability of equipment. For example, when assessing newborn care with the Newborn Health Simulation vignette, the health worker is asked “Please tell me, when a healthy baby is delivered, what care is important to give them immediately after birth and the first few hours thereafter?” One of the items that indicate knowledge is that they administer vitamin K. If the health worker indicates this procedure, the enumerator is instructed to verify that vitamin K is immediately available to the health worker. Saying they would administer vitamin K indicates knowledge, saying they would administer vitamin K together with having vitamin K present indicates capacity. Of course, the final test comes during direct observation when we indicate whether or not the health worker actually did administer vitamin K. In other cases, at the end of the case study we ask “what equipment or materials would you have used if it had been available to you?” B Motivation survey questions 31 Table 5: Motivation survey questions Question Self satis- Job satis- Facility val- Facility Worker faction faction ues worker character- behavior istic Good performance is recognized by our x superiors This facility has a fair system for re- x warding staff My performance is appraised regularly x Some of the team members work well, x yet others do not and so this facility doesn’t perform well overall We do not know how our facility is per- x 32 forming compared to others in the dis- trict Our facility has clear goals that we are x working towards I understand how my work contributes x to the facility’s overall goals I am keen use any new tools to improve x my performance This facility has a good reputation in x the community This facility provides everything I need x to perform well at work There are enough health providers to x do the work in this facility Too often the referral system does not x work efficiently Table 5 (continued): Motivation survey questions Question Self satis- Job satis- Facility val- Facility Worker faction faction ues worker character- behavior istic Maintenance of broken equipment at x this facility is prompt and reliable I do not get feedback from my superi- x ors so it is hard to improve my perfor- mance My job duties and responsibilities are x clear and specific Relevant guidelines are easy to access x at this facility 33 I often feel left alone when I have to x make difficult decisions about a pa- tient’s care I regularly have access to relevant x trainings to keep my skills up to date It makes me feel appreciated when pa- x tients are grateful I usually cope well with changes at x work It is difficult for me to speak openly x to my superiors about how things are really going at work Suggestions made by health workers on x how to improve the facility are gener- ally ignored I intend to leave this facility as soon as x I can find another position Table 5 (continued): Motivation survey questions Question Self satis- Job satis- Facility val- Facility Worker faction faction ues worker character- behavior istic I would recommend to my children x that they choose the health profession I am willing to put in a great deal of x effort to make this facility successful I am proud to be working for this x health facility This hospital inspires me to do my best x on the job I am proud to tell others that I work x 34 in this ward / part of the hospital I am glad that I work for this facility x rather than other facilities in the coun- try These days I feel motivated to work as x hard as I can. My profession helps me to achieve my x goals in life Overall, I am very satisfied with my x work in this ward /part of the hospital I am very satisfied to have a position x where I can work closely with the com- munity I am satisfied with the opportunity to x use my abilities in my job. I am punctual about coming to work x I am a hard worker x Table 5 (continued): Motivation survey questions Question Self satis- Job satis- Facility val- Facility Worker faction faction ues worker character- behavior istic I work hard to make sure that no pa- x tient has to wait a long time before be- ing seen I always complete my tasks efficiently x and correctly. When I am not sure how to treat a pa- x tient’s condition I look for information or ask for advice I try to get on well with the other x health staff because it makes the work 35 run more smoothly I get along well with my superiors at x work I feel that I am a person of worth, at x least on an equal plane with others. I feel that I have a number of good x qualities. All in all, I am inclined to feel that I x am a failure. I am able to do things as well as most x other people. I feel I do not have much to be proud x of. I take a positive attitude toward my- x self. On the whole, I am satisfied with my- x self. Table 5 (continued): Motivation survey questions Question Self satis- Job satis- Facility val- Facility Worker faction faction ues worker character- behavior istic In this organization, I am taken seri- x ously. 36 In this organization, I am trusted. x In this organization, I am important. x In this organization, I can make a dif- x ference. In this organization, I am valuable. x In this organization, I am helpful. x C Defining motivation categories using factor analysis We use factor analysis to construct measures for five motivation variables. Below are the rotated factor loading tables for factors with eigenvalue > 1, as well as un-rotated tables showing the eigenvalues of each factor. We use a factor loading value cutoff of .40 to determine which variables are weighted more heavily for each factor. Each of the five categories load the majority of the corresponding variables determined via confirmatory factor analysis. Figure 7: Rotated factor loadings for facility characteristics 37 Figure 8: Un-rotated eigenvalues for facility characteristics 38 Figure 9: Rotated factor loadings for worker behavior Figure 10: Un-rotated eigenvalues for worker behavior 39 Figure 11: Rotated factor loadings for worker characteristic Figure 12: Un-rotated eigenvalues for worker characteristic 40 Figure 13: Rotated factor loadings for job satisfaction Figure 14: Un-rotated eigenvalues for job satisfaction 41 Figure 15: Rotated factor loadings for facility values worker Figure 16: Un-rotated eigenvalues for facility values worker 42