52656 THE WORLD BANK GROUP Washington, D.C. M a k i n g Se r v i c e s W o r k Indicators, Assessments, and Benchmarking of the Quality and Governance of Public Service Delivery in the Human Development Sectors Ariel Fiszbein, Dena Ringold, and Halsey Rogers December 2009 This note benefitted from comments by and conversations with many colleagues, including Peter Berman, Eduard Bos, Mukesh Chawla, Elizabeth King, Margaret Koziol, Sebastian Martinez, Harry Patrinos, Harry Reid, Emiliana Vegas, Marko Vujcic, Andreas Seiter, Stephanie Trapnell. Please send comments to afiszbein@worldbank.org, dringold@worldbank.org, and hrogers@worldbank.org. 1 I. INTRODUCTION Improving governance is central to improving results in human development. At the country and international level, there is a growing recognition that money is not enough: improved outcomes from service delivery require better governance, including incentives for performance and mechanisms for holding service providers accountable. At the same time, the World Bank is increasingly in the business of supporting innovative governance reforms through operational and analytical work that aim to strengthen the governance of service delivery. This has led to a growing demand for indicators to measure how and whether these processes work, and how they affect results.1 Indicators for measuring governance and service delivery can serve three different purposes. First, service delivery data can serve as a call to action for governments if the data reveal outcomes that fall short of expectations. Second, data can serve a diagnostic function. While there is increased recognition that the quality of governance and service delivery affects human development outcomes, the empirical evidence on those links remains limited. Increased data availability can allow for better measurement of policies and outcomes and for impact evaluation. Finally, within the development community, project teams need indicators to track the progress of country programs and strategies. As lending operations are increasingly moving to results-based approaches, better data on the performance of services are needed. This note makes the case for measuring governance and the quality of service delivery in the HD sectors and develops a framework for the selection and measurement of a set of health and education indicators. It proposes the adoption of a more systematic approach that will both facilitate the work of operational teams and allow for cross-country comparisons and benchmarking. This note is part of a broader effort by the HD Chief Economist`s Office to provide a conceptual framework for the measurement of service delivery and governance and identify a set of indicators that could be used by client countries and project teams alike. The first section of the paper introduces the conceptual framework; the second section discusses a potential long list of indicators, and the final section discusses data collection instruments, methodologies, and next steps. II. BACKGROUND Improving the quality of service delivery in health and education is at the center of the World Bank`s development agenda. In both low- and middle-income countries the Bank is increasingly engaged--through operational work, policy advice and analytical activities--in supporting efforts to reform service delivery systems. An important aspect of those efforts involves governance reforms aimed at strengthening incentives for service providers and increasing accountability for results. 1The interest in indicators of service delivery is Bank-wide. The Bank recently introduced a requirement that all IDA operations include a set of common indicators, and there is also an initiative to build a Bank-wide database of "Actionable Governance Indicators" as part of the GAC strategy. 2 It has now been six years since the launch of the World Development Report 2004: Making Services Work for Poor People (World Bank 2003). The report helped spark a flurry of Bank operational and analytical work on governance and service delivery in health and education, at a time when academic researchers were also taking a greater interest in the issue. That work has focused on measuring service delivery at the point of contact between provider and client. By identifying gaps in the quality and quantity of street-level service delivery, this measurement complements the essential work of measuring outcomes such as learning, educational attainment, and health status. Significant efforts have also been focused on evaluating the impact of health and education interventions that seek to improve incentives to providers and accountability mechanisms as a means of improving service delivery and human development outcomes. In education, these interventions include school-based management, teacher incentives, and the provision of information to users. Similarly, in health, a range of pay- for-performance schemes (and the associated accountability mechanisms) are being evaluated in a variety of countries.2 From the perspective of service delivery, governance can be understood as the set of incentives and accountabilities that affect the way provider organizations, their managers, and staffs behave, as well as the quality and efficiency with which they deliver services. From this vantage point, what is of interest is how providers are selected, paid, monitored, and held accountable for their performance. The link between governance and service delivery is a key element of the Bank`s Governance and Anti-Corruption (GAC) agenda (World Bank 2007). The HD Network has made making services work for all` the central theme of its ongoing efforts to mainstream GAC in the sectors. This theme reflects the increasing emphasis on these issues at the country level and the growing demand from Bank task teams for guidance and support on this issue. An area of particular importance within the GAC agenda is the identification and measurement of actionable governance indicators (AGIs) across sectors. The AGI initiative involves more systematic measurement of service delivery results, as well as of the critical elements of sector governance that are expected to influence those results. III. RATIONALES Why should we adopt a more structured approach to these HD governance indicators? Beyond the general AGI rationale ­ the desire to monitor and understand better these key governance elements in HD service provision ­ we see at least three important reasons: to allow more international (and intranational) benchmarking, to make impact evaluations 2See http://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726-1239047988859/5995659- 1239048041095/Basic_Ed_Cluster_HDNCE.pdf and http://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726-1239047988859/5995659- 1239048041095/P4P_in_Health_Cluster_HDNCE.pdf. 3 more feasible and informative for policy, and to improve monitoring of development projects. Better and More Standardized Measurement Allows International Benchmarking An old maxim holds that you cannot improve what you do not measure. Good measurement allows policymakers to see where service delivery and governance are falling short, allowing them to focus on the key problem areas. If the indicators are disseminated to civil society, measurement can help build societal consensus for reform of ineffective governance structures and promote accountability for better service delivery. In the context of specific projects, governance and service delivery indicators constitute key elements of frameworks for monitoring results. In this process, international, or at least intra-national, comparability can be very helpful. In many of these service delivery areas, measurement without context may not be enough to make it clear whether the service delivery is falling short. Take, for example, the case of leakage of public funds. If 87 percent of central-government funds intended for schools make it through various layers of government down to the school level, does that suggest good governance or poor governance? If 25 percent of doctors are absent from rural health centers at the time of surprise visits, does that suggest relatively strong or weak accountability for performance? If at least a core group of indicators are measured in a standardized way across countries, it is easier to answer these questions. In the case of fund leakage, for example, comparison with the findings of the original Uganda Public Expenditure Tracking Survey (PETS) ­ where only around 20 percent of cash transfers made it to the school level ­ might suggest that the country is not doing too badly (Reinikka and Svensson 2005) . Ditto with doctor absence, at least compared to the 74 percent absence rate found in small rural clinics in Bangladesh several years ago (Chaudhury and Hammer 2004). These examples are simplistic, but suggestive. Benchmarking intra-nationally can be even more effective in rousing action, because it allows the states or regions with the most effective service delivery to set the standard for government and the public, while making it harder to make the case that poor service delivery or governance are endemic given local culture or political environment. Two illustrations of how this intra-national comparative approach has been shown to be effective include the case of Papua New Guinea, where HDN worked with the Health Metrics Network to strengthen collection of infant mortality data in a way that allowed comparison among parliamentarians` districts, and India, where collection of teacher absence data that was representative at the state level increased media attention to gaps in performance among them (Kremer et al. 2005). For this purpose, it is necessary to standardize measurement only at the national level, but achieving even this level of coordination and standardization may require conscious attention. Particularly in large federal countries with a large degree of autonomy at the state or provincial level, there is a risk of state-specific studies that use non-compatible measures of performance. In 4 addition, having international comparators will strengthen the value of this intra-national measurement. Better Measurement Makes Impact Evaluation Possible Ultimately, we want to know not only how effective governance and service delivery are, but also what to do to improve them. This requires evaluating the impact of different interventions, at each stage along the results chain. Good measurement of governance and service delivery are essential to tracing these impacts and identifying where the intervention is or is not working. Consider, as an example, the case of an intervention aimed at transferring more managerial authority to the school level. In this case, the intervention is intended to improve such elements of service delivery as teaching quality and teacher attendance, with the ultimate goal of improving student outcomes. Better measurement and evaluation are necessary all along this envisioned results chain. Of all of these elements, several indicators of student outcomes ­ enrollment, persistence, and completion ­ are probably measured most regularly and accurately, but even in these cases we often must rely on household surveys to supplement the administrative data. Student learning is typically poorly measured, and so improving measurement of learning outcomes is a major thrust of the education-sector benchmarking exercise that the World Bank has launched. The other elements in the results chain ­ the governance intervention and the quality of service delivery ­ are generally not measured on a regular basis. With this initiative, our objective is to help increase the frequency and the consistency of measurement of those indicators. In the case of school-based management, for example, this would mean collecting data and doing qualitative assessments to determine to what extent and in what areas decision-making authority had been devolved to the school level, the extent of actual participation, the extent of teacher presence and teaching quality, and the extent of improvements in student learning outcomes. All of this applies equally in the health sector. Consider, for example, the case of an intervention directed at paying provider organizations according to their performance and giving them autonomy to manage inputs. As with education, we have better measures of final outcomes than of governance and quality of service delivery. For example, governance measurement might include the extent to which the process of contracting of health services was competitive and free of corruption. Measures of service delivery quality could include the quality of advice provided by medical providers, perhaps through direct observation of interactions between providers and patients. All these indicators are central to ongoing efforts in the area of health systems strengthening the HNP sector is pursuing. 5 Better Measurement Can Improve Project Monitoring Finally, better indicators for measuring service delivery can improve the monitoring of development projects and, ultimately, project quality. There are two main reasons why this is important. First, clients and the donor community are increasingly demanding tools to improve the results of development projects. And second, the design of projects is becoming more results-focused, with a growing number of projects linking disbursements to project outcomes. Development partners increasingly recognize that improving project monitoring requires better measurement. Tracking inputs and outputs is not sufficient for improving human development outcomes. Instead, better measurement of service delivery and the policies that affect service quality are necessary to look inside the black box of what actually happens at the point of delivery. For example, stock-out rates for pharmaceuticals can indicate whether essential medicines are reaching health clinics. Such indicators can be incorporated into project monitoring arrangements for investment projects and can also be used as targets in adjustment operations. The growing popularity of results-based projects also requires careful attention to indicators.3 These operations link disbursements to project outputs and outcomes, instead of inputs. For example, in education, a recent loan in Pakistan links disbursements to the extent of merit-based recruitment of teachers. In health, results-based projects link payments to coverage and health status outcomes. These examples require monitoring of intermediate policy indicators to ensure that the project is on-track, as well as of the outcome indicators linked to disbursements. The investment lending reform that is now underway at the World Bank aims to make results-based instruments more accessible and feasible. Finally, the use of governance and service delivery indicators for project monitoring can have the additional benefit of building the measurement capacity of countries. The inclusion of these indicators into results frameworks means that countries will have to collect this data over time. This provides a window of opportunity for incorporating governance indicators into administrative data and on-going surveys that can exist beyond the life of the project. IV. A FRAMEWORK FOR MEASURING GOVERNANCE AND QUALITY OF SERVICE DELIVERY While there is great interest in defining and measuring the role that governance and quality of service delivery play in human development outcomes, there is limited consensus on how this complex relationship works in practice. Ultimately service 3 For more on results-based financing approaches in health, see: www.rbfhealth.org. 6 delivery is only part of the story. Health status is influenced by outside factors, such as the quality of the water supply and infrastructure, while educational outcomes are influenced in part by family characteristics such as the level of parents` education. Behaviors of service users (e.g., adherence to medical treatment or time spent on homework assignments) are also critical in determining final outcomes. Similarly, governance is only one factor influencing the quality of service delivery. For example, the quality of medical care or teaching is influenced by factors as varied as the availability of technologies and the overall functioning of the labor market which, for the most part, we would consider as independent of governance arrangements ­ at least as defined in this paper. The discussion in this paper acknowledges this context and focuses in on two interrelated factors driving outcomes ­ namely, the role of governance and the quality of service delivery. As shown in Figure 1, governance can influence the quality of service delivery which in turn affects human development outcomes. Exogenous factors and other inputs contribute along the way. Figure 1: Governance, Service Delivery and Outcomes Other Other Other Factors Factors Factors Governance Governance: Quality of Human Policies Performance Service Development Delivery Outcomes In this context, governance has two dimensions that can be measured. First, there is the set of policies, or rules of the game, that influence the ways that service providers function. These are the incentive and accountability arrangements. For example, human resource policies in health establish the rules and regulations for hiring, firing, and rewarding doctors; the same is true in the case of teachers in education. Second, governance can be described in terms of performance ­ the ways in which policies actually influence the behaviors of providers. Governance performance is 7 measured by the extent to which the governance policies work in practice.4 In the personnel example, whether doctors or teachers show up regularly for work (e.g. attendance rates) are a measure of governance performance.5 Governance policies and performance in turn influence the quality of service delivery. This refers to the quality at the actual point of contact between provider (the teacher, doctor, or other medical provider) and client (the student or patient). Measurement of quality is often elusive; however, recent research has made progress in measuring dimension of quality such as the quality of medical advice and the classroom time use of teachers. Because of the difficulty of measuring quality, indicators of quantity are sometimes uses as proxies. For example, in education, the amount of time teachers spend in classrooms may be used as an indicator of quality, if the actual quality of teaching cannot be measured. This distinction between policies, performance, and quality of service delivery can help in thinking through the role of governance along the service delivery chain. For example, in health care, governance policies include personnel rules that regulate whether a local government can hire and fire workers, or pharmaceutical management and procurement policies that determine the way in which drugs reach facilities. Whether or not these policies work in practice can be measured by performance measures such as absenteeism rates for personnel, or the availability of drugs. We can then ask whether the increased availability of staff and drugs have improved the quality of service delivery, which could be measured through the number of health consultations following acceptable standards, or the number of patients receiving prescribed medications. Similarly in education a policy that provides bonuses for teachers who work in remote rural areas (governance policy) may influence the presence or absence of teachers in rural areas (governance performance) as well as the amount of time teachers spend teaching in the classroom (an indicator of service quality). These measures of service quality in turn influence outcomes. 4 It is important to note that governance performance indicators aim to go beyond simple "in practice" application of governance policies. Instead they capture evidence on the extent to which policies are followed. For example, a governance policy could require advertising of posts for recruitment of teachers. Whether posts are advertised is an "in practice" indicator, but it does not provide a great indicator of the performance of a recruitment and selection process in ensuring meritocratic hiring decisions. It captures whether the "in law" requirements are being followed. A better indicator of the "performance" of such a selection process would be the average number of qualified applicants per advertised vacancy. This is because an advertising requirement may be formally followed, but its purpose compromised if, for instance, it is widely believed by potential applicants that the selection process is rigged, and that an already identified candidate is all but assured of winning the competition. 5 This distinction between policies and performance builds on the AGI framework of indicators of governance determinants (e.g. policies, referred to in AGI parlance as the "r" indicators) and performance (the "Y" indicators). 8 This paper uses this distinction between governance policies, governance performance, and quality of service delivery as a framework for discussing indicators for measuring service delivery in health and education. The distinction raises a number of conceptual and measurement challenges, including the difficulty of defining the scope of governance policies that are relevant for influencing provider behavior and the complexity of distinguishing between measures of governance performance and the quality of service delivery. Box 1: Governance Performance vs. Quality of Service Delivery While there is considerable discussion among experts about whether to measure governance performance or the quality of services, in our view, as indicated in the discussion above, good measurement of both of these elements ­ as well as of education and health outcomes ­ is crucial to improving service delivery. Two examples make it clear that we cannot choose just one or the other: Is driven in part by: Which in turn depends on: Outcome Service Quality/Performance Governance Policy Health status of patients Quality of medical advice from Regulatory environment for providers medical sector Student cognitive Classroom attendance rate of Extent & quality of school- achievement teachers and time-on-task based management In the first case, the subsequent health status of those who seek medical care from providers depends in part on the quality of advice offered by those providers. It is important to have some direct measurements of that quality, in order to know whether health-care spending is able to lead to better health status. Research from several countries has revealed great shortcomings in the quality of care, so much so that in a significant share of cases, following the doctor's advice would actually harm the patient. But once we have identified shortcomings, in order to remedy them it will be important to track down their source. This means measuring the quality of upstream governance that directly affects the quality of care. For example, one might ask why the regulatory environment is such that large numbers of unqualified doctors are able to operate with impunity. In the second example, the evidence suggests that a teacher's skills and motivation are the most important school-based factor determining students' learning. But these factors have until recently been poorly measured: education officials and researchers have had to make do with measures like the education level and experience of teachers, which have been shown to be poor proxies for classroom effectiveness. It is important to get inside the black box of teacher factors by measuring more directly the quality of teaching actually provided in the classroom. As with health, it is also necessary to measure the quality of upstream governance factors that affect teacher behavior ­for example, the extent and effectiveness of school-based management, which is hypothesized to improve teacher effort and performance. Another important consideration for the discussion of measurement and indicators is the need to specify the level and/or unit of analysis. Measurement of governance can occur at the level of systems, as well as at the local provider level. For example, in some systems schools or hospitals may have their own human resource policies. Thus, a critical aspect of assessing governance systems is developing a clear understanding of the institutional arrangements, including roles, responsibilities, and the authority of key actors. This is particularly important in decentralized systems, where a mismatch in authority between levels of government can lead to poor governance outcomes ­ for 9 example if local governments are given responsibility for financing services, but no authority for holding providers accountable.6 V. MEASURING GOVERNANCE: POLICY AND PERFORMANCE INDICATORS Governance systems are multi-dimensional, and the rules of the game influencing the behavior of providers are often complex. To focus our work, we are proposing to concentrate within the governance arena on five dimensions of the service delivery process: human resources, financing systems, critical inputs, information, and provider entry. Together, these dimensions can help describe a governance system through its policies and performance. Measurement, in this context, implies both assessing the rules of the game (e.g. the policies ­ both formal/de jure and informal/de facto) in each of the five dimensions and capturing the effects of those rules on actual performance. As noted in Savedoff (2009), not all of the governance policies will have scores that are easily rankable, since we cannot be sure a priori which direction (that is, more or less of the variable) will be associated with better service delivery and outcomes. However, all are hypothesized to be related to the performance of governance in some model of service delivery. Indicators of governance performance, on the other hand, should capture the behavior change brought about by the governance policy. These are generally indicators for which there is a widely shared sense of directionality (that is, research has established whether more or less of the variable is desirable). The following sections discuss a potential long list of indicators in each of the five subareas of service delivery, covering both indicators of policy and performance. The goal of this discussion is to provide a set of indicators that teams can draw from to incorporate into surveys and monitoring frameworks. The HD Chief Economist`s Office is also coordinating the collection of some of these indicators. This effort is described in the final section of the paper. 1. Human Resources Health and education are labor-intensive services that involve many, often small, transactions between workers and users. As such, the rules of the game that define recruiting, hiring, compensating, assessing, rewarding, placing and firing public health and education workers; and the associated incentives that affect their behaviors at the point of service delivery, constitute a critical dimension in our framework. Assessing governance upstream involves a range of areas including policies and practices that regulate recruitment, retention (including tenure), and assignment; monitoring and evaluation of performance; employment status and job security; salary structure and other 6 Refer to Fiszbein (2001) for a framework for institutional analysis. 10 benefits (including retirement); workload, duties and autonomy; etc. Potential indicators include: a. Degree of meritocracy in civil-service hiring procedures. This can be assessed by asking what factors weigh in the decision to grant a public school teacher or a doctor an open-ended appointment. Options include: years of experience, educational qualifications, performance on the job, and area of expertise. The degree of meritocracy can also be assessed by examining the transparency of hiring procedures and the availability of credible redress mechanisms to ensure that hiring requirements are actually practiced. b. Share of pay of typical provider based on performance. This indicator examines all of the elements of the typical teacher`s or doctor`s compensation package ­ such as base salary, bonus for working in remote areas, performance bonus, benefits, etc. ­ and assesses how much of that compensation depends on the provider`s performance. c. Extent of decentralization. This examines the degree to which local governments and/or facility level administration are involved in hiring, firing, transfer, and salary/bonus decisions related to public school teachers, or health workers/school principal/health facility director. This indicator should also ideally capture the degree to which actors are held accountable for exercising discretion. d. Level of autonomy/regulation of professional work. For education this examines the degree of autonomy granted to teachers, based on existing laws and regulations, in such service delivery areas as curriculum content, choice of teaching methods, materials and textbooks, criteria for assessing students, and decisions on student grade repetition. Similarly, in health, this indicator measures the level of regulation in service delivery of health workers, including treatment protocols. e. Existence and application of performance evaluation systems. The policy indicator examines whether the performance of teachers or doctors is evaluated regularly (whether by facility directors, by a national authority, by a sub- national authority, or by a local authority). Beyond the existence of evaluation systems, indicators could look at: (i) whether the ratings are based on ex ante agreed performance targets, or, instead, on compliance-based criteria; (ii) whether the performance evaluation system includes checks on the assignment of ratings (to weed out favoritism and sloppiness in assignment of ratings); and (iii) whether the performance evaluation system includes a redress mechanism. 11 f. Role of unions. In many countries unions have an important role in the institutional and policy arrangements of the sector. Indicators could look at the strength of unions by assessing the share of providers who are union members, and importantly the roles and functions of unions, such as wage setting, teacher allocation, strike activity, as well as political roles. Evaluating the effects of these governance arrangements will typically involve measures of abuse and poor work effort on the part of health and education workers, as well as of the ability of provider organizations to attract and retain qualified staff. Potential indicators of governance performance in this dimension include: g. Corruption in the allocation of posts (initial posts/transfers) ­ e.g. frequency and amount of bribe-paying to influence hiring decisions. h. Share of ghost providers (those who are on the central administrative records but not on the rosters of facilities). i. Provider attendance/absence. Absenteeism can be measured through surprise visits and spot checks to determine the share of personnel contracted/on payroll who are not on-site during the observation. j. Amount of out-of-pocket/informal payments to public education personnel for private classes or tutoring, or to public health personnel for services provided outside of clinic hours. k. Level of skills of providers. Skill levels can reflect how successful the governance arrangements are at attracting and retaining providers with the expected level of skills. As indicators of skill levels, we can use the outcomes of assessments of medical providers using vignettes, and assessments of teachers using subject tests as well as information from performance evaluations. 2. Financing and Resource Management The second essential element is financing and resource management, including the rules by which budgets are defined, resources are transferred across agencies and levels in the service delivery process, the extent to which their use is monitored/reported; and the influence they have on resource availability and use at the point of service delivery. As a start, governance indicators for financing and resource management can build on the PEFA (Public Expenditure and Financial Accountability) indicators that are part of the Performance Measurement Framework. PEFA surveys are regularly conducted across countries and review public finance performance across a set of 28 composite indicators which rank countries on aspects of public financial management including: budget 12 credibility, transparency, and performance the budget cycle. PEFA studies have been conducted in approximately 100 countries. PEFA indicators include a combination of policy and performance indicators. For example, they incorporate de facto elements, such as the existence of a clear budget calendar, as well as de jure elements, such as whether the legislature has followed the budget calendar in the preceding year. PEFA indicators are scored ­ similar to the CPIA index ­ on a four-point scale (PEFA Secretariat 2005; Lewis and Pettersson 2009). PEFA indicators can be a useful entry point for assessing the public finance framework, but from the perspective of service delivery more disaggregation is needed to understand policies and performance at the sectoral level. There may be substantial variation in rules and their application across sectors, and the situation may be further complicated by the sectoral institutional arrangements and context. For example, in the health sector, resources may flow to local health facilities from central and sub-national governments as well as a health insurance fund, each of which may have its own rules, institutional culture, and effectiveness. As a result, comprehensive analyses such as public expenditure reviews (PERs) and public expenditure tracking surveys (PETS) may provide the best assessment of the financial aspects of service delivery. These instruments are discussed in the next section of this note. However, some governance policy and performance indicators can provide a useful snapshot. Governance Policy Indicators Can Include: a. Existence of clear rules and procedures for the budget cycle (PEFA). The PEFA indicator set includes a number of measures that look at the credibility and predictability of budgeting processes, including the existence and use of a robust classification for monitoring and tracking spending, the comprehensiveness of budget documentation and reporting arrangements, and the existence of a fixed budget calendar and approval process. b. Existence of clear rules on intergovernmental fiscal relations (PEFA). This indicator examines whether there are: transparent rules-based systems for allocating transfers from the central to sub-national governments; regular reporting of reliable information on budget allocations from the central to sub-national governments; and the extent to which fiscal data are collected and reported according to sectoral categories. This indicator could also be used to look at these policies between sub-national governments and facilities. c. Existence of clear rules on budgeting, accounting, and reporting at the facility level. This indicator reflects the extent to which schools and health clinics are expected to collect and maintain records on revenues and expenditures, arrangements for reporting, including to whom and when. It also identifies what kinds of rules exist to budget and transfer resources to facilities ­ for example, in education, whether allocations are made based on formulas or on a per capita basis. 13 d. Effectiveness of payroll controls (PEFA). This indicator captures the effectiveness of the payroll for public sector workers. This is important because weaknesses in personnel recording can lead to inefficiencies in service delivery and even leakages if there are duplicate records, or ghost workers who appear on the payroll, but do not exist as real personnel. Components of this indicator include whether personnel records and payroll data are integrated and reconciled; internal controls of changes to records and the payroll, and the existence of audits to identify problems such as ghost workers. Governance Performance Indicators Can Include: e. Aggregate expenditures compared to original approved budget (PEFA). This measure looks at the difference between actual and originally budgeted expenditures and reflects the government`s capacity to deliver planned services. f. Proportion of government funds that reach district-level facilities as intended. This is a measure of leakage of funds allocated for service delivery. PETS surveys can provide data for this measure. Detailed PETS can also provide information on leakage of funds at the facility level. g. Availability of information on resources received by the service delivery unit (PEFA). This indicator reflects implementation of budgeting and transparency policies. It captures whether facilities (such as schools and health clinics) can obtain information about the resources allocated to them. h. Frequency and extent of wage arrears for providers. This performance measure looks at the effectiveness of the financing system to pay workers.7 i. The effectiveness of payroll controls. Possible indicators could be the number and types of errors identified, including the number of ghost workers and payment duplications. 3. Procurement and Management of Critical Inputs8 While there are many different inputs that enter in the production of health and education services, textbooks and, particularly, pharmaceuticals are often considered critical for quality of service. The rules regarding what critical inputs are procured, how and by whom they are procured, and the associated results in terms of processes regulating availability, quality and cost constitute another important dimension of governance systems. 7 Note, it is unclear how to distinguish how much of this indicates poor governance and how much broader fiscal issues. 8 Health indicators draw from WHO, 2008. 14 Indicators of Governance Policy for Critical Inputs Can Include: a. Existence of an essential medicines and textbooks list. This indicator assesses whether there is a list of essential drugs mandated for national (or sub-national) procurement. The equivalent for education is the existence of clear criteria for the selection of textbooks. b. Existence of policies on drug and textbook procurement. This policy indicator captures whether policies ensure the procurement of cost-effective pharmaceuticals at the required quantities, how competitive the bidding process is, and whether suppliers are regulated to ensure quality. There would be a parallel indicator for textbooks. c. Transparency of procurement policies for textbooks and pharmaceuticals. This indicator reviews the level of openness provided in the marketplace through the regulatory framework. d. Openness of the market for inputs. This reflects whether there are restrictions on who can provide textbooks or pharmaceuticals for public services ­ for example, whether there is a restricted list of providers or a public monopoly. Indicators of Governance Performance Can Include: e. Stock-out rates (absence) of essential drugs in health facilities. This indicator measures the availability of essential drugs in health facilities, ideally over enough time to assess whether absence of drugs reflects a supply or a demand issue. f. Share of schools/classrooms with required textbooks in compulsory grades. Similar to the stock-out indicator for health, this indicator measures whether required textbooks (or other educational materials) reach schools. g. Share of pharmaceutical sales that consist of counterfeit drugs. This indicator reflects the quality of the drug supply and the extent to which quality assurance policies, such as inspections, are effective. h. Price differentials for critical inputs (e.g. drugs, textbooks). This indicator reflects the extent to which there is a competitive market by looking at the differences in prices across facilities and/or sub-national governments. 4. Information Decisions and behaviors by various actors in the service delivery chain are based on the nature of information that they have on inputs, outputs, and outcomes. The rules regarding which information is collected and made available, as well as the availability, reliability, and timeliness of that information, can influence governance performance. In addition, the extent to which the views of current and potential users are captured and considered in the design and implementation of services (e.g. through participation 15 mechanisms such as school committees; or grievance processes) and the level of access that beneficiaries have to information about services (e.g. through publication of test scores of individual schools) is also important. In this regard, indicators involving information look at the quantity and quality of information available; which stakeholders have access; and feedback mechanisms. Indicators of Governance Policies Include: a. Existence of systems for monitoring inputs, outputs, and outcomes. This indicator reflects whether information systems are in place, as well as rules for quality control and access to information. In practice this is a set of indicators defined for different subareas of policy (e.g. human resource management systems, public financial management systems) and for different levels of government (e.g. central, sub-national and facility; as well as by area of service delivery). For inputs, this includes systems on budgeting, personnel, and critical inputs [overlapping with the discussions above]; outputs include coverage rates (e.g. school enrollments) and numbers and types of services provided (e.g. visits to health clinics); and outcomes include student learning assessments and health status data, among others. The indicators should look at both the quality and the existence of information systems. b. Existence of mechanisms (such as beneficiary satisfaction surveys and grievance mechanisms) for obtaining client input on the existence of appropriate, timely, and effective access to health and education services. c. Existence of right to information (RTI) acts. This assesses whether specific requirements in health and education exist for public disclosure of information.9 Indicators of Governance Performance Can Include: d. Availability, use, quality and accessibility of information, outputs and outcomes by actors and stakeholders. This indicator captures whether information is actually made available and its quality and accessibility at various levels of the system ­ including to policy makers, providers and citizens. For example, it would reflect whether learning assessment results are published frequently and in sufficiently disaggregated form to be useful to stakeholders. e. Numbers of grievances and complaints. This reflects whether feedback mechanisms are functioning. More detailed data may be available on the types of complaints and how many were resolved. 9 Data on RTIs are currently being collected as part the Public Accountability Mechanisms (PAM) module of the AGI database. 16 f. Public access to key fiscal information (PEFA indicator). This indicator looks at whether the public has access to fiscal information, including annual budgets, reports of budget execution, year-end financial statements, external audit reports, and information on contract awards. 5. Provider Entry The ability of new providers to enter the market for health and education services can be an important determinant of outcomes in the sectors. Private non-profit and for-profit providers, as well as new entrants in the public sector (such as charter schools), can affect the quality of service delivery in two ways: by providing care directly, and by spurring behavioral change in public providers. These effects can be either positive or negative and reflect a trade-off between quality control and competition. On the one hand, requirements for entry can ensure that only qualified providers practice, while on the other hand, requirements can be poorly designed or applied, such that qualified doctors are teachers are not allowed to practice.10 Indicators of Governance Policies Can Include: a. Extent of legal restrictions on entry of private providers into the education and health sectors (as graded subjectively by experts), including those that apply specifically to nonprofits (as suggested in Harding and Preker (2003)). Policy indicators in this area could reflect whether there is a certification process for provider entry and whether the criteria used are related to quality. Indicators of Governance Performance Can Include: b. Qualitative assessment of the ease of starting up a school. This indicator was collected for a recent African Private Schools Investment Index (School Ventures and Economist Intelligence Unit 2008), and it might be feasible to update the indicator. This is a de facto and not a de jure indicator. c. Qualitative assessment of the conditions for private-sector involvement in health delivery. An example can be found among the indicators for USAID's PSP-One program in health (Private Sector Partnerships-One Project 2005).11 d. More general indicators of ease of starting and running a business. These indicators can be taken from the World Bank Doing Business survey; they would capture some of the general business climate variables that could affect whether it is feasible and potentially profitable to run a (for-profit) school. 10This section will be expanded drawing from the Investment Climate Team's flagship survey of private health care in Africa. 11Their specific indicators would probably not meet our needs, however. They focus on changes rather than levels, as in "number of barriers removed", so they do not give an assessment of the level of barriers. 17 Box 2: Governance Indicators for Social Protection While this paper focuses on indicators for education and health, a similar approach is needed to measure service delivery and governance in social protection. Social protection encompasses a range of different types of policies and programs, including social insurance, labor markets, and social safety nets. Some social protection programs take the form of services, provided directly to clients. Programs such as care for the elderly and disabled, and employment services for the unemployed, have parallels to education and health in that there are facilities and providers who deliver services to clients. In these cases, many of the indicators identified for education and health could be applied directly to social protection services. A more challenging exercise is to think through the indicators for social protection cash benefits. The large share of social protection resources take the form of cash benefits, such as old age pensions, unemployment benefits, and social assistance benefits for the poor. The framework developed in this paper can be applied to cash benefits, although some areas (such as information) become more critical than others (such as critical inputs). The following are some considerations for cash benefits based on three of the subareas of service delivery discussed in this paper ­ human resources, budgeting, and information. This is a starting point that will be developed more fully in subsequent versions of this work. Human resources: Cash benefits require front-line staff to assess eligibility, work with clients, and pay benefits. Governance policy indicators for recruiting, hiring, retaining, and evaluating these staff are relevant. Performance indicators on absenteeism and skills would also apply. In addition, OECD countries are increasingly putting emphasis on responsiveness and client-orientation in the delivery of social assistance ­ including providing more hospitable offices, less paperwork and bureaucracy, and more integration of services (through "one-stop shops"). In this area, a performance indicator could include waiting times for applying and receiving benefits. Budgeting: Policy and performance indicators in this area are very relevant as social protection programs may not have the same legal status as health and education systems. As a result, their funding base may be vulnerable to changes in government. In this regard it is important to identify the legal and administrative mechanisms that ensure that budgeting is available for the program, both at the national and the sub-national levels. In many countries sub-national governments have a significant role in financing social assistance benefits. Incentives and accountability mechanisms for local governments to deliver safety net programs are critical, since safety net programs lack influential constituencies who can pressure governments to pay benefits and there is a high risk that these expenditures can get crowded out. There is also a risk that the poorest local governments lack resources to pay for social assistance. Policy indicators could reflect the extent to which expenditures on cash benefits are mandated. Performance indicators could assess the extent to which local governments meet their obligations. Information: This is a critical area for cash benefits as the delivery of cash benefits requires strong information to ensure that the right people get the right benefit at the right time. The indicators discussed above for monitoring inputs, outputs, and outcomes are all relevant. The successful delivery of cash benefits also requires substantial coordination of information systems, database cross-checks, and other types of control and accountability mechanisms. Policy indicators would assess the existence of these mechanisms at various levels of the system, while performance indicators could look at the outcomes of these controls, such as the numbers and types of errors identified. Grievance mechanisms and monitoring of feedback are also important. VI. MEASURING SERVICE QUALITY: A PARALLEL AND RELATED AGENDA At the same time, governments and donors will need indicators of the quality of service delivery. Conceptually, what we want is a set of indicators that let us know how effective 18 the delivery of services is ­ in effect, how much the services can be expected to improve the health or education of the client. These quality measures can differ from the governance performance measures described in the previous section because other non- governance factors also affect outcomes. The line separating these indicators from the governance performance indicators is a blurry one, but it is useful to try to distinguish the two. For example, an important determinant of the quality of schooling could be whether or not there is a roof over the students` heads. While the availability of school buildings ultimately depends on governance, it would be a stretch to consider this one of the short- or medium-term governance performance variables. Our quality measure could implicitly into account the effects of having or not having a roof, whereas the governance performance indictors would not. Service-quality indicators that have been used so far are of two types--objective and subjective. 1. Objective Indicators Measuring the quality of service delivery is challenging. Even measures of governance performance, though only recently developed, have received more attention than quality measures. What do we know about how to measure service quality objectively? Health: In health, a set of recent studies has tried to measure the quality of care through observation of doctor behavior (summarized in Das, Hammer, and Leonard 2008). These studies focus on sets of symptoms for which the appropriate treatment is clearly known, so that they can assess doctors` performance against an objective standard. This allows comparison of quality across different sectors and environments ­ public vs. private clinics, for example, or rural vs. urban. Governance factors affect quality of care, but so could other dimensions like the clinic location and environment. An important contribution of these studies is their ability to distinguish the twin determinants of quality of care by providers: skill and effort. Specifically, the studies use medical vignettes (or hypothetical cases) to measure the doctors` knowledge of appropriate treatments, but then directly observe those doctors` dealings with patients to see whether the doctors apply this knowledge. They take the gap between knowledge and practice to indicate the level of effort made by the providers, which is a crucial dimension of quality of care. Effort can also be considered an indicator of governance performance, as it is influenced by incentive arrangements (e.g. fees for service or performance bonuses). However, it can also be influenced by other factors including the intrinsic motivation of individual providers. Indicators Could Include: a. Assessment of the quality of health care delivered. A number of recent studies have used direct observation of patient interactions to assess the quality of health care. While observed quality may be better than 19 typical unobserved quality, the evidence suggests that this observation effect dissipates over time. Education: In education, there are corresponding measures of the quality of teaching, gathered through classroom observation studies. One such indicator is time on task: the total amount of time that teachers are actively engaged in teaching and students are actively learning (for example, Abadzi 2006). Most observers will agree that schooling is likely to be more effective when teachers are able to devote less time to maintaining discipline or carrying out administrative tasks and more time to teaching. However, a concern with time on task studies is the amount of variation by enumerator. It is difficult for researchers to agree on how to categorize how teachers spend their time in the classroom. Beyond time on task, if we are willing to make assumptions about the most effective pedagogy (for example, by prioritizing active learning), then the time- on-task metrics could be made more detailed by measuring the amount of time spent on good pedagogy. Indicators Could Include b. Time on task (as described above). c. Quality of teaching: as assessed by trained and experience observers or by external teacher evaluation systems, another approach would be to ask school principals their views and validate with assessment data. A concern with this type of classroom observation is the variation across enumerators, if different observers assess classroom teaching in different ways. 2. Subjective Quality Measures A second type of indicators is measures of satisfaction of the clients or recipients of health and education services. These subjective measures can both be a proxy for measuring actual quality of services, as well as serve as an indicator of the extent to which services are responsive to the needs and preferences of clients. In the case of health services, we can survey patients about their experience, for example through exit interviews at clinics. In the case of education, one could survey the students, but it also makes sense to survey parents and other stakeholders (including employers) for their opinions of how well schools are functioning. Such indicators need to be interpreted with caution. One reason for caution is that (to take the education example) parents may have different objectives for their children`s education than society as a whole does. Especially given that public provision of services is often justified by the presence of externalities, it would be theoretically inconsistent to argue that the user`s subjective measures capture all desired outcomes. For example, parents may be happy with a teacher who reinforces traditional gender roles even as the government tries to open up opportunities for girls (or vice versa). A second reason is that even if parents and society share the same goal ­ such as more rapid student learning 20 ­ parents may have been conditioned to have low expectations, leading them to report high satisfaction despite poor schooling. A third reason for caution with subjective measures is information asymmetries. This is particularly evident in health services where providers have technical information and expertise that patients lack. As a result, patients` perceptions of whether they are receiving quality care may be based on factors not relevant to technical quality. For example they may believe they are receiving quality care if the clinic is clean and the doctor is friendly, even if the wrong medications are prescribed. That said, there is value in measuring users` opinions as a complement to the objective measurements. Ultimately, users should know at least what increases their utility better than the government or researchers will, as long as we take account of the possible biases noted above. Inconsistencies between the two types of indicators could be a useful warning flag about possible measurement problems in the objective measures, or at least a prompt for deeper investigation. Indicators Could Include: a. Results of international surveys of individual respondents on the quality of service delivery. One possibility is to use survey questions like those fielded by Gallup in a sample of countries in 2003-2006, which asked about the incidence of problems including poor teaching, overcrowded classrooms, school facilities in poor condition, lack of drugs, and disrespectful staff.12 Because they are included in general-purpose surveys, these questions will typically not yield detailed information, but may serve as a general barometer of quality of services. b. Results of more detailed household survey modules on availability and quality of services. Household surveys could go into more depth than the polling-style surveys, and could produce more rigorously quantitative measures of performance. VII. HOW TO MEASURE? The next question is how to gather these indicators of governance and service quality. One reason that they have received less attention from policymakers and researchers until recently is that in some cases, they require new instruments. Governments are generally better positioned to measure resource allocation, because of the natural attention to budgets, and so data on inputs (at least at some central and aggregated level) may be better. And with the proliferation and standardization of household surveys over the past generation, we have much better measurement of health and education outcomes (along at least some dimensions) than in the past. The challenge, as noted above, is to monitor 12The survey also included questions on provider absenteeism, which could be used under the "governance performance" indicators. 21 and measure what is happening between inputs and final outcomes. This section discusses briefly how best to approach that challenge. 1. Instruments for Data Collection To measure service delivery and its governance determinants well, it will be necessary to make use of several types of instruments. This is because governance involves a complex set of processes which requires different measurement approaches, as well as because much can be gained from corroborating, or cross-checking sources. For example, national databases on enrollments may not reflect actual attendance measured through household surveys. Two recent surveys ­ Lindelow and Wagstaff (2008) and Amin and Chaudhury (2008) ­ offer excellent detailed discussions of instruments and methodologies appropriate for the health and education sectors, respectively. The edited volume by Amin, Das, and Goldstein (2008) also reviews the instruments and lessons from their application to various cases. A brief overview of the main options follows here: a. Facility surveys (PETS/QSDS): The first line of data collection should be the facility survey, which is an instrument administered at the level of the school or health facility (Reinikka and Smith 2004). This instrument collects data on the quality and quantity of public services delivered and the resources used to deliver them at the facility level. The Public Expenditure Tracking Surveys (PETS) measure the flow of financial and other resources through the system and assess whether they result in service delivery. The Quantitative Service Delivery Surveys (QSDS) focus more on the quantity and quality of services and the behavior of providers. Specific data that can be collected using these survey instruments includes: Facility usage by clients (students or patients) Quality of infrastructure and equipment Attendance and activity level of providers (such as teachers, doctors, and nurses) Degree of supervision and involvement by community and officials Extent of bribe-paying and other corruption in service delivery Sources and flow of funds b. Expert surveys on system variables: It may be difficult to measure some of the variables of governance policies through facility surveys. One example is the degree of meritocracy in human resources practices: are teachers and doctors hired primarily because of their qualifications, or because of connections or side payments? This can be assessed in part through interviews in the facility surveys, by asking providers how they or other providers got their positions. 22 But there are obvious disincentives for the interviewee to answer truthfully, especially in the workplace. In such cases, governance data may be collected more easily through other instruments. One such instrument is expert surveys, such as those on which the Bank`s Doing Business studies are partly based (World Bank 2008) or the PEFA surveys mentioned in the previous section. Careful, well-specified questions about standard practices should make it possible to come up with comparable indicators of (in this example) hiring practices throughout the system. The Education Anchor is currently undertaking expert surveys of human resource policies in the sector. Similarly, the Investment Climate Team`s work on the private sector in the health sector in Africa also involves expert interviews. c. Household surveys: Household surveys can provide other insights into the performance of the education or health system. Respondents can be asked about the quality of services they receive, the typical wait times at facilities, out-of-pocket spending, the behavior and performance of providers, and their engagement in facility management, among other dimensions. They can also be asked how they typically responded to failings in service delivery, as a means of gauging how much client power operates in the system. Much of the data that has been collected this way is subjective ­ for example, How responsive is school management to your concerns? ­ but these instruments can be used to collect quantitative information as well. For example, parents could be asked How many times last week was your child`s main teacher absent from school? Household surveys may be especially helpful in measuring behavior that is easily changed or concealed when the enumerators arrive at the facility. For example, one indicator of poor school governance might be illegal physical or sexual abuse of students by teachers. Assuming that teachers do not feel complete immunity, they will not carry out such punishment in sight of enumerators. But household surveys can include questions about abuse, such as Was your child beaten by a teacher within the past month? or even Do you know of any cases of sexual abuse of students in your community within the past year? d. Secondary sources: Improving measurement and benchmarking does not mean drawing only on new primary data. Existing secondary sources can also include data that will be useful in constructing indicators for the measurement and benchmarking exercise. This can include data collected in the context of projects supported by the World Bank and other development partners. For example, recent projects supporting results-based financing in health incorporate the collection of outcome and performance indicators against which disbursements can be made (e.g. number of mothers receiving pre-natal care). 23 e. Administrative data: Existing government databases can be a rich data source, particularly for information on system inputs (e.g., numbers of facilities, personnel, equipment) and coverage of services (e.g., enrollments, medical procedures performed, etc.). Administrative data are also needed for making sense of governance system performance. For example, administrative data on the number of teachers assigned to schools are needed as a base for comparing the results of absenteeism surveys, e.g. to cross-check the number of teachers present with the number of teachers who should be there on a given day. f. Qualitative data: Qualitative data can complement the types of surveys and quantitative data sources mentioned above. Data collected through interviews with providers and beneficiaries can be particularly useful for understanding how governance policies on paper (de jure) are actually implemented in practice (de facto). For example, while human resource policies may specify provisions for competitive hiring processes, in practice, positions may be sold or procedures may be circumvented in other ways. Case studies can provide helpful insights into why governance policies may not have the expected impacts. 2. Data Collection: By Whom and How Often? Another question is who should carry out the data collection on which the indicators are based: the government, or an NGO or other outside actor? On the one side, there are good arguments for having the government take charge of data collection. Policymakers may feel greater ownership of the indicators if they trust that the data has not been collected with the aim of highlighting the government`s flaws, and they may therefore be more inclined to act on the indicators. Government ministries and statistical agencies have a geographical reach and command human resources (at least in terms of numbers) that exceed those of any NGO or even survey firm. Examples of good indicators of service delivery based on government data collection can include data on human resources, including the numbers of teachers and doctors, their qualifications, and levels of wages and benefits. On the other hand, an independent agency ­ whether an NGO or private firm ­ may be more likely to be able to operate independently, without political interference that could compromise quality. This is not a foregone conclusion, because there could be incentives to keep the client happy by not embarrassing the government. But a number of the early studies measuring new aspects of service delivery quality have used non- governmental agents to do the data collection, and Amin and Chaudhury (2008) argue that for transparency reasons, it is optimal to have facility surveys carried out by an independent survey firm or research organization. NGOs and other community based organizations may also be interested in collecting data on service delivery in order to play the role of an independent monitoring organization, encouraging transparency and accountability in the provision of public services. This is common in OECD countries, where outside organizations collect and publicize 24 information on government spending and the quality and availability of services.13 The World Bank has been working with the NGO Results 4 Development to pilot a program for building the capacity of community based organizations in developing countries to undertake surveys of service delivery, including PETS and absenteeism surveys to engage with governments on issues of accountability and quality of public health and education services. Another related question that needs to be addressed is whether the data collection for the indicators should be institutionalized as part of regular monitoring of performance, or should be carried out through occasional one-off surveys. To some extent, this overlaps with the by whom question: institutionalization may be more sustainable if the data collection is done by the government, at least if the process has government ownership. But clearly regular data-collecting efforts related to service delivery can also be carried out by NGOs, if they have the necessary funding. VIII. THE AUDIENCE FOR INDICATORS A related question is, who is the audience for the indicators? The multiple objectives for collecting indicators of governance and service delivery imply multiple audiences ­ policy makers, providers, politicians, NGOs and community leaders, beneficiaries, development partners and so on. Governments, first and foremost, need indicators to improve the quality of service delivery and outcomes. However, different levels of government and different stakeholders within the government need different information and different levels of disaggregation. The Minister of Education needs to look at a different dataset than the principal of a school for human resource planning. NGOs, community organizations, development partners and other stakeholders also need data for the reasons discussed in this paper. The implications are that indicators should be clearly defined, transparent, and understandable by people with different backgrounds and interests, as well as being easily available and accessible. IX. NEXT STEPS This paper is a contribution to the on-going effort of the HD Chief Economist`s Office (HDNCE) to identify HD governance and service delivery indicators that can be used by task teams and clients alike. In addition to developing the conceptual framework, HDNCE is working with task teams to pilot collection of governance and service delivery at the country level. The work will be taken forward in two directions: (1) defining indicators; and (2) collecting data. 13One example is the Center on Budget and Policy Priorities, which collects data on state and federal government spending and services in the US. 25 1. Defining Indicators Based on the long list of indicators included in this paper, we will develop a glossary of definitions. The aim is to have as many indicators as possible defined so that they can be incorporated into survey questionnaires and other data collection instruments. Where possible, existing definitions will be used based on international practices ­ e.g. drawing from OECD education indicators on teacher pay, and WHO indicators on pharmaceuticals. This work on definitions will also specify the type(s) of data sources that can be used to collect each indicator. In some cases, it will not be possible to have one agreed definition of an indicator due to country or regional variations. In these cases the glossary will note more than one option or suggested variations on the indicator. 2. Collecting Data In parallel, an effort is underway to collect data on governance and service delivery with the aim of building a cross-country dataset of HD indicators for the AGI database. The database will not be comprehensive in terms of indicators or country coverage, but will rather be a first step at identifying and compiling data from existing sources. Data will be more readily available on indicators of governance performance than on governance policies. As noted above, expert surveys may be the best source of information on policies across countries. Data collection efforts are currently underway that will help to populate the database of governance and service delivery indicators. In health and education, teams are piloting data collection with the support of seed funds from the Governance Partnership Facility (GPF) trust fund. In health, teams in the Philippines, India, Yemen, Turkey, and Zambia are fielding a range of instruments, including facilities surveys and scorecards to test collection of governance indicators. Similar indicators will also be used in the context of evaluations of Results-Based Financing (RBF) programs in health. The experience will be documented and synthesized for use by other teams. In education, a facility survey is being tested in India to look at the de facto performance of teacher policies.14 This will be expanded to other countries. On the side of governance policies, the Education Anchor is leading an extensive effort to collect policy data on human resources in education. This has involved application of a detailed questionnaire in five countries on all aspects of teacher policies, including recruitment, hiring, pay, retention, and evaluation. The instrument is being scaled down to a subset of indicators for application in a broader set of countries. 14Education facilities surveys will be conducted in additional countries. The countries currently under consideration are Indonesia, Mexico, Angola, Lesotho, and Nepal. 26 In addition to these efforts, there are other data sources that will also make major contributions to the knowledge base on governance and service delivery, including the PEFA surveys for budgeting data, and the Health in Africa data collection initiative of the Investment Climate Department. The data types and potential sources are summarized in the table below. As shown, there are gaps where data are not available, including information and critical inputs. The aim of this initiative is develop instruments to build the knowledge base in these areas over time. Table 1: Governance Policy and Performance, Data Sources Subarea of Service Governance Policy Governance Performance Delivery Human Resources Benchmarking of teacher Facilities surveys policies supported by the GPF Budgeting PEFA data PEFA data; public expenditure reviews; public expenditure tracking surveys Information PEFA data Critical Inputs WHO data on pharmaceuticals Provider Entry Investment climate Investment climate survey of Health in survey of Health in Africa Africa X. CONCLUSION Measuring governance and service delivery is an important and complex endeavor that can inform better results at the country level and better development projects. This paper aims to give orientation to task teams taking on this challenge. The framework and the potential indicators provide a starting point for identifying entry points and testing approaches. We hope to facilitate a learning process through on-going collaboration with task teams and country partners who are involved in piloting and collecting governance policy and performance data and integrating these indicators into country data systems. 27 XI. References Abadzi, H. (2006). The Economics of Instructional Time: How Efficiently are Funds Converted into Learning Activities? Washington, DC: World Bank. Amin, S. and N. Chaudhury (2008). An Introduction to Methodologies for Measuring Service Delivery in Education. Are You Being Served? New Tools for Measuring Service Delivery. S. Amin, J. Das and M. Goldstein. Washington, D.C., World Bank: 67-110. Amin, S., J. Das and M. Goldstein, Eds. (2008). Are You Being Served? New Tools for Measuring Service Delivery. Washington, D.C., World Bank. Chaudhury, N. and J. S. Hammer (2004). "Ghost Doctors: Absenteeism in Rural Bangladeshi Health Facilities." World Bank Economic Review 18(3): 423-41. Das, J., J. Hammer and K. Leonard (2008). "The Quality of Medical Advice in Low-Income Countries." Journal of Economic Perspectives 22 2: 93-114. Harding, A. and A. S. Preker, Eds. (2003). Private Participation in Health Services. Washington, DC, World Bank. Kremer, M., K. Muralidharan, N. Chaudhury, J. Hammer and F. H. Rogers (2005). "Teacher absence in India: A snapshot." Journal of the European Economic Association 3(2-3): 658- 67. Lewis, M. and G. Pettersson (2009). "Governance in Health Care Delivery: Raising Performance." Washington, DC: World Bank. Lindelow, M. and A. Wagstaff (2008). Assessment of Health Facility Performance: An Introduction to Data and Measurement Issues. Are You Being Served? New Tools for Measuring Service Delivery. S. Amin, J. Das and M. Goldstein. Washington, D.C., World Bank: 19-65. PEFA Secretariat (2005). Public Financial Management: Performance Measurement Framework. Washington, DC, World Bank. Public Expenditure and Financial Accountability Program. Private Sector Partnerships-One Project (2005). Performance Monitoring Indicators for the Private Sector Progam (PSP), with Explanatory Notes. Bethesda, MD, Abt Associates. Reinikka, R. and N. Smith (2004). Public expenditure tracking surveys in education. Paris, International Institute for Educational Planning. 28 Reinikka, R. and J. Svensson (2005). "Fighting Corruption to Improve Schooling: Evidence from a Newspaper Campaign in Uganda." Journal of the European Economic Association 3 2-3: 259-67. Savedoff, W. D. (2009). Governance in the Health Sector: Authority, Information and Motivation. Portland, Social Insight. School Ventures and Economist Intelligence Unit (2008). The African Private Schools Investment Index 2008. Alexandria, Virginia, School Ventures Inc. World Bank (2003). World Development Report 2004: Making services work for poor people. Washington, DC, Oxford University Press for the World Bank. --- (2007). Implementation Plan for Strengthening World Bank Group Engagement on Governance and Anticorruption. Washington, DC, World Bank Operations Policy and Country Services Network. --- (2008). "Doing Business 2009: Comparing Regulation in 181 Economies." 29