SOURCEBOOK SUMMARY | CHAPTER 9: ASSESSING THE PERFORMANCE OF SP DELIVERY SYSTEMS MAY 2022 CHAPTER 9 Assessing the Performance of Social Protection Delivery Systems This chapter provides an evaluative framework to assess the performance of the core building blocks of delivery systems for social protection programs, as well as to evaluate how these systems can promote inclusion and strengthen coordination. CONNECTING DELIVERY SYSTEMS TO PROGRAM OUTCOMES: A RESULTS CHAIN How Do Delivery Systems Contribute to Social Protection Program Results? Results chains typically focus on linking program activities to high-level outcomes. The focus of conventional theories of change tends to be on the causal chain linking activities of the program, such as cash transfers or job training, to high-level outcomes, such as the graduation of vulnerable populations from extreme poverty, or their enhanced resilience to shocks. In traditional theories of change, a set of underlying assumptions about delivery systems are usually spelled out including the accuracy of targeting and the ability of vulnerable populations to access the program’s benefits or services. Other assumptions might include the effectiveness of the referral system. However, theories of change are rarely explicit about how delivery systems contribute to effective and efficient social protection programs. In other words, those theories do not elaborate on what it takes for their assumptions to be fulfilled. Yet, as this Sourcebook has shown, delivery systems involve many processes, actors, and enabling factors to implement social protection programs. Without delivery systems, there is no implementation. Without implementation, there are no program activities. Without program activities, there are no outcomes. Performance Along the Delivery Chain Delivery systems are the backbone of social protection programs because they constitute the operating environment for implementing social protection benefits and services. That operating environment includes the core implementation phases and processes along the delivery chain the main actors (people and institutions), and enabling factors (communications, information systems, and technology). A well-performing delivery chain supports the dual objectives of effective and efficient delivery of benefits and services to the intended population. Effective delivery systems are by essence inclusive. They not only reach the intended population, but they do so in part by overcoming the challenge of including vulnerable groups and those who face specific access barriers. Delivery systems that function well also promote efficient program delivery: that is, clients and administrators can go through each phase of the delivery chain at a reasonable cost in terms of time and money. Thus, an efficient delivery system has necessarily overcome the challenges of coordination and fragmentation. Such a system exploits synergies within and between programs to minimize costs for administrators and integrates systems and delivery chains across programs to minimize costs for clients. Well-performing delivery systems ensure effectiveness and efficiency throughout the delivery chain, from outreach to routine oversight, and are supported by effective and efficient information systems, client interfaces, and institutions. Measuring Performance of Delivery Systems: Indicators FIGURE 9.3 Programs deliver services and benefits The performance narrative along the delivery GOAL effectively and efficiently and promote the inclusion of specific groups with access barriers. chain is laid out graphically in the upper tier. ASSESS ENROLL PROVIDE MANAGE RECURRING 2 3 4 5 6 7 CYCLE 8 9 OUTREACH ENROLLMENT PROVISION OF BENEFICIARIES Intended Population (IP), including Eligible applicants are onboarded BENEFITS & COMPLIANCE, Vulnerable Groups (VG) understands efficiently, witl minimal leakage to SERVICES UPDATING AND program & willing to apply ineligible population Enrolled GRIEVANCES INTAKE & REGISTRATION DECISION ON PACKAGE beneficiaries Information is kept up IP and VG are applying efficiently and their Benefits and service packages are receive appropriate to date, free of EFC, information is recorded accurately accurately services and responsive to clients ASSESS NEEDS & CONDITIONS benefits according evolving needs, & Applicants are accurately profiled and to service promote desired categorized standards behaviors ENABLING INFORMATION SYSTEMS are robust (with complete, accurate, unique information); secure (protect data privacy and ensures transaction safety); efficient (minimizes time and money involved in providing, curating, and accessing information) ; interoperable and dynamic FACTORS COMMUNICATIONS AND CLIENT INTERFACE provide appropriate information and support to citizens, especially those with access barriers, to guide them through the process in a user-centered way INSTITUTIONS are well-governed (with robust legal framework, clear roles and responsibilities, appropriate oversight); well-resourced (with skilled human resources and appropriate budgets and incentives); and well-coordinated 2 | SOURCEBOOK SUMMARY | CHAPTER 9: ASSESSING THE PERFORMANCE OF SP DELIVERY SYSTEMS MAY 2022 A performance indicator framework is a useful management tool to monitor the progress and contribution of delivery systems. The sourcebook chapter proposes several options for measuring performance along the delivery chain. The companion indicator framework presented in Annex 9A1 categorizes 100-plus indicators that capture various dimensions of performance. The compilation is not meant to suggest that a program should adopt the entire framework, but rather to offer a list of options for social protection programs to choose from and adapt to their specific circumstances. Indeed, parsimony is an important feature of a quality indicator framework. The choice of indicators depends on the system’s level of maturity as well as on the overarching objectives of the program and how the system is intended to contribute to the program. Therefore, there is no ideal set of indicators that could be replicated for all programs. Only those indicators that are measurable, monitorable, and above all, useful for making decisions should be integrated. This section highlights the logic behind the indicator framework. Key Dimensions of Performance: Effectiveness and Efficiency The Sourcebook focuses on two broader categories of performance: effectiveness and efficiency. There are many dimensions to the performance of delivery systems, but most fit under the broad objectives of effectiveness and efficiency. For example, inclusion is explicitly included as a key component of effectiveness. FIGURE 9.4 Highlights key performance indicators along the delivery chain that capture dimensions of effectiveness and efficiency. ASSESS ENROLL PROVIDE MANAGE Eligibility Determination Provision of Beneficiaries Assessment and of Benefits Notification Benefits Compliance, Exit Decisions, Intake & of Needs and Enrollment and Service and and/or Updating and Notifications and Outreach Registration Conditions Decisions Package Onboarding Services Grievances Case Outcomes RECURRING 1 2 3 4 5 6 7 CYCLE 8 9 % of population that is % of intended population Processing times: # % of beneficiaries with updated registered Data source: enrolled in the program and of days to process information Numerator: Registry data ; % of benefits going to specific benefit payments; Data source: Administrative info Denominator: census data groups (e.g., poorest quintile) or % of beneficiaries Data source: ASPIRE coverage receiving payment % of sampled (or cross-checked) % of intended population and coverage data, administrative for each cycle beneficiaries without information registered (or of vulnerable data from program; deronunator: according to quality errors Data source: Periodic audits or groups) administrator, census, or HH cross-check standards (e.g.,< 30 Data source: Numerator: survey data days) Registry data with indicators % of registered grievances resolved Data source: Payment of characteristics of .ntended Processing times: # of days system data Data source: 6RM data population; Denominator: from application to eligibility Administrative, Census, NH notification; # of applications % of unemployed % of individual beneficiaries in survey data processed according to quality clients who move each category with conditionalities standards (e.g., < 30 days) into employment monitoring information % of registered population Data source: Administrative data within specific time Data source: Administrative data with up•to-riate basic information (e.g.. < 2 years old) band % of service clients with Data source: % of service clients with IAPs that Data source: Numerator: Registry Individualized Action data or Periodic audit data; Information system are monitored according to quality Plans (IAPs) Data source: for PES or contracted standards Denominator: Registry data Administrative data service provider Data source: Administrative data 1 See the Sourcebook page 410 for more details on the performance indicator framework for social protection delivery systems 3 | SOURCEBOOK SUMMARY | CHAPTER 9: ASSESSING THE PERFORMANCE OF SP DELIVERY SYSTEMS MAY 2022 When deciding which indicators to integrate in a delivery system’s performance measurement framework, a few principles should be kept in mind. • First, performance indicators are only as good as their use. It can be costly to generate performance measurement data and to make sense of them through reporting. Thus, when deciding which performance indicators to track, consultation with the intended users should take place to ensure the indicators will be used. • Second, generating the most useful and valid indicators might be costlier than collecting routine indicators that are less useful but are automatically generated by the system. Thus, being selective about indicators also means considering the value of each indicator against its cost. • Third, to track performance over time, it is helpful to generate longitudinal data. Some core performance indicators should not change every year. Others can be adopted for shorter periods to track the short-term effects of an innovation in the delivery system or to verify whether performance is improving after a problem in the delivery chain has been fixed. • Fourth, the performance indicator framework should evolve with the delivery systems. More mature delivery systems will be able to generate and absorb more advanced data sources. Given these four considerations, programs should make careful choices about which indicators to adopt for performance management. Data sources There are three primary sources of data that can be used to measure the main indicators of the delivery systems, performance framework: (1) delivery systems administrative data; (2) national data sources (from periodic surveys); and (3) data that require a separate dedicated collection effort. Note that there can be some pitfalls in combining various sources of data to craft performance indicators. One of these pitfalls is that of time discrepancy: while program data are potentially collected routinely, household and census data are much more infrequent. The results framework in the Sourcebook (annex 9A) indicates which data sources are appropriate to use for each suggested indicator. 4 | SOURCEBOOK SUMMARY | CHAPTER 9: ASSESSING THE PERFORMANCE OF SP DELIVERY SYSTEMS MAY 2022 EVALUATING DELIVERY SYSTEMS: TYPES OF ASSESSMENTS Different types of assessments can be used to evaluate the performance of delivery systems. Such assessments range from more general, comprehensive approaches to more focused, specific ones. Some assessments are interrelated and provide inputs for others. For example, business process reviews are usually an input for information systems reviews because they allow the mapping of processes that need to be automated by the information system. Similarly, some assessments, especially the more focused ones, can be part of larger, more general assessments. For example, while business process reviews can be conducted as a stand-alone evaluation, they can also be part of a larger process evaluation. For each of the assessments, multiple sources of data and information can be used to answer specific assessment questions. The choice between the various types of reviews and the methods used to conduct them should always be driven by the key performance questions that the program needs answered. Table 9.2 lists the eight types of assessments and recaps the reviews covered in the main chapter. TABLE 9.2 Types of Assessments to Gauge the Performance of Social Protection Delivery Systems Type of assessment Intended use Client-focused Assess the performance of the delivery chain from the clients' perspective performance assessment Process evaluation Assess what works and what does not work in the implementation of the program and find options to improve implementation Business process review Assess the effectiveness of specific business processes that are used in the delivery chain Compliance audit Assess whether the rules, guidelines, and protocols are followed Efficiency analysis Assess the relative costs to the outcomes of various systems or processes Benchmarking Compare the performance of two or more delivery systems Information systems Assess whether information systems are appropriate and maintain data review integrity and use Institutional review Assess whether the organizational structure, management systems, and staff are adequate Source: Original table for this publication 5 | SOURCEBOOK SUMMARY | CHAPTER 9: ASSESSING THE PERFORMANCE OF SP DELIVERY SYSTEMS MAY 2022