MEASURING MARKET DEVELOPMENT A HANDBOOK FOR FUNDERS AND IMPLEMENTERS OF FINANCIAL INCLUSION PROGRAMS PAT R I C K SPAVE N A N D KA RI NA BROENS NI ELSEN This work is available under the Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO) https://creativecommons.org/licenses/by/3.0/igo/. By using the content of this publication, you agree to be bound by the terms of this license. For attribu- tion, translations, adaptations, and permissions, see the provisions and terms of use at https://www.adb.org/terms-use#openaccess. Suggested citation: Spaven, Patrick, and Karina Broens Nielsen. 2017. “Measuring Market Development: A Handbook for Funders and Implementers of Financial Inclusion Programs.” Washington, D.C.: CGAP. All queries on rights and licenses should be addressed to CGAP Publications, 1818 H Street, NW, MSN IS7-700, Washington, DC 20433 USA; e-mail: cgap@worldbank.org. Consultative Group to Assist the Poor/World Bank Group MEASURING MARKET DEVELOPMENT A HANDBOOK FOR FUNDERS AND IMPLEMENTERS OF FINANCIAL INCLUSION PROGRAMS PATRICK SPAVEN AND KARINA BROENS NIELSEN ACKNOWLEDGMENTS The authors of this Handbook are Patrick Spaven, independent monitoring and evaluation spe- cialist, and Karina Broens Nielsen, CGAP financial sector specialist. Significant contributions have been made by Alice Negre, financial inclusion specialist, and Krisana Pieper, monitoring and evaluation specialist. Eric Duflos (UNSGSA) and CGAP’s Barbara Scola, Matthew Soursourian, Antonique Koning, Michael Tarazi, and Camille Busette provided technical review and support. This Handbook was developed with input from many organizations and individuals. Special thanks for their constructive feedback and suggestions go to Jim Tanburn, Nabanita Sen, and Donna Loveridge, from the Donor Committee and Enterprise Development; Diane Johnson, inde- pendent financial inclusion specialist; Marcus Jenal (BEAM Exchange); Rob Hitchins and Alan Gibson (the Springfield Centre); Tim Ruffer (ITAD); CGAP members, including Paul Nelson (USAID), Abdoul Karim Coulibaly (Mastercard Foundation), Jonathan Ndaa Agwe (IFAD), and Andrew Fyfe and John Tucker (UNCDF). CGAP’s collaboration with Rich Williams, Robert Stone, and Sukhwinder Arora from Oxford Policy Management to develop Impact-Oriented Measure- ment Guidelines for Financial Sector Development Africa and DFID also helped to inform the development of this Handbook. CGAP piloted this Handbook’s recommendations and the self-assessment tool with the UNCDF- SHIFT (Shaping Inclusive Finance Transformations) Program. The authors thank Deanna Morris and Muaz Jalil for their insightful comments and suggestions. Throughout the drafting of this Handbook, selected questions, topics, and principles were dis- cussed in CGAP’s Measuring Market Development working group, several consultative events, numerous interviews, and CGAP-organized measurement learning event for funders. The authors extend sincere thanks to all the working group members for the constructive questions and fruit- ful dialogue that helped shape this Handbook. Finally, we thank Anna Nunan (CGAP) for editing and production of the Handbook. TAB L E OF CO N T ENTS Executive Summary v What You’ll Find in This Handbook viii Introduction 1 MODULE 1. The Systemic Approach and Measurement 3 MODULE 2. The Measurement System 9 MODULE 3. Theory of Change and Results Chains 17 MODULE 4. Measurement Questions 29 MODULE 5. Results Frameworks 37 MODULE 6. Indicators and Data Sources 49 MODULE 7. Enhanced Monitoring 63 MODULE 8. Reporting and Knowledge Sharing 69 MODULE 9. Evaluation 77 MODULE 10. Portfolio-Based Measurement 85 Glossary 93 Self-Assessment Tool for Financial Inclusion Measurement Systems 95 iv  Measuring Market Development: A Handbook for Funders and Implementers of Financial Inclusion Programs EXEC UT I V E SUM M A RY This Handbook guides funders and their implementing partners on how to effectively monitor and measure results of financial inclusion programs that apply a systemic approach. It is primarily intended for measurement specialists who design and manage measurement systems of financial inclusion programs, oversee a portfolio of programs, or advise programs on how to measure results. The Handbook includes a self-assessment tool—a practical instrument for benchmarking and identifying gaps and areas for improvement. FINANCIAL INCLUSION AND THE • Rules and norms governing these functions. SYSTEMIC APPROACH For the financial services market to be efficient and inclusive, different market actors need to Governments and development partners around perform a variety of functions: the core func- the world share a vision of financial inclusion tions of demand and supply and the support- where all individuals and businesses have ing functions that shape, inform, and enable access to, and the ability to use, a range of transactions between customers and provid- appropriate financial services that are provided ers. There must be rules and norms gov- responsibly and sustainably. Despite significant erning such functions. Funders need to under- funding of financial inclusion efforts, approxi- stand the different functions in the financial mately 2 billion adults remain excluded from services market system and address the barri- the formal financial system in 2016 (Global Fin- ers that exclude the poor. They can do this dex), and there is still substantial variation in the through interventions that facilitate inclusive diversity, quality, and use of financial services. and responsible financial services markets. For the financial services market to be effi- CGAP’s “A Market Systems Approach to Fi- cient and inclusive, different market actors nancial Inclusion: Guidelines for Funders” (Bur- need to perform a variety of functions: jorjee and Scola 2015) encourages funders to • Core functions of demand and supply. apply a systemic approach. Instead of provid- ing piecemeal solutions through direct assis- • Supporting functions that shape, inform, tance, funders should support programs that and enable transactions between customers and providers.   v vi  MEASURING MARKET DEVELOPMENT use a systemic approach to help market actors easily quantifiable, but more sustainable strengthen weak functions and develop other outcomes. functions that are needed. The interventions 3. Programs need to have a broad view in should be designed to change the mindsets monitoring a complex financial system. and practices of financial services providers Measurement tends to focus narrowly on (FSPs), clients, regulators, supervisors, and interventions and intended outcomes that other market actors. are defined at the beginning of a program. Two other aspects of the systemic approach In systemic financial inclusion programs, it is stand out. First, because the financial system is important to broaden the scope of mea- complex and elements within it are dynamic surement beyond the intended results to and unpredictable, programs need to empha- include unexpected or unplanned outcomes size experimentation and learning, and be reg- and factors in the external environment that ularly adapted based on experience. Second, can affect progress both positively and neg- progress to sustainable financial inclusion at atively. Monitoring based exclusively on scale can be slow and not necessarily linear, so results framework indicators is inadequate program managers need to take a long-term for systemic financial inclusion programs view toward systemic change. and can be misleading. 4. Measurement tends to assume that pre- IMPLICATIONS FOR MEASUREMENT dicted change is influenced by or attribut- able to program interventions. Conventional The principal implications for measurement in intervention-led measurement—the “out- systemic financial inclusion programs are as ward” perspective—asks whether and to follows: what extent the interventions have led to intended results. Given the complexity of 1. The measurement system needs to be financial market systems, intervention-led designed to detect and assess whether and measurement needs to be done in conjunc- how systemic change is happening. The sys- tion with change-led measurement—the tem needs to include theories of change “inward” perspective. Change-led measure- (ToCs), results chains, and results frame- ment asks what has changed and what has works and indicators. Some outcomes and contributed to the changes. Applying these indicators will measure progress toward sys- two perspectives gives a broader view of temic change. Some will relate to program evidence of change, allows for triangulation, partners, while others go beyond partners to and mitigates dependency on just one per- other actors that are affected. ToCs should spective. Although it is difficult to attri- depict how interventions and intermediate bute specific outcomes to funders’ interven- outcomes contribute to systemic change tions, this dual perspective allows funders and how systemic change leads to a more to develop a rigorous and credible narra- inclusive financial system and, ultimately, to tive about the impact of their programs development outcomes like improved eco- and projects. nomic well-being. 5. ToCs and results frameworks should be 2. Expected systemic change outcomes often viewed as hypotheses and not as blue- relate to changes in capacity, practices, prints—results are difficult to predict. ToCs institutional processes and structures, and and results frameworks need to be regularly relationships between actors. These types challenged, and funders and program units of outcomes are often highly context-spe- must be prepared and enabled to change cific and nuanced. For example, indicators them. In this paradigm of accountability, the will need to measure degrees of change, or work of program units is assessed against a the quality of new processes or relation- range of factors, and they are not held to ships. It may be possible to summarize these targets that may no longer be reasonable or outcomes in numbers through rating scales, appropriate. but it is difficult to standardize this across projects and programs. Ratings scales need 6. Monitoring in systemic programs should not to be complemented by contextualized nar- only ensure interventions stay on track but rative. Results frameworks that focus exclu- should also provide frequent feedback so sively on quantitative targets can lead that interventions can be adjusted as program staff to orient themselves “to chase needed. Therefore, monitoring needs to be the numbers” within target timeframes linked to regular, data-informed, evaluative rather than to explore long-term, often less reviews. These reviews are critical in ensur- EXECUTIVE SUMMARY   vii ing that the data emerging from monitoring A BROADER MODEL OF are applied to decisions about the interven- ACCOUNTABILITY tions going forward. The reviews should be owned internally, but can benefit from the Conventional accountability in development participation of partners and from third- programs tends to center on achieving pre- party facilitation. defined targets. Because accountability usu- ally passes through a chain of actors—from 7. This type of measurement needs to be project staff, through to programs manage- closely integrated with project and program ment, to funders, and often on to funders’ gov- management. Because more participants ernance stakeholders, it makes sense to look are involved than in conventional measure- for standardized indicators so that key results ment, support capacity is needed, for exam- information can be aggregated and indicators ple, to understand the nature and value of can be framed as readily measurable targets. evidence. Also, staff need to have incentives In systemic financial inclusion programs, the to be open-minded and inquisitive, and to scope for quantitative indicators, particularly take risks. When senior program unit man- those that can be shared with other projects agement and funders show that they value and programs, is limited. And results are also measurement results, staff are more likely to less predictable because programs, teams, and see the value as well. partners need to be able to experiment and 8. Funders need to recognize that ToCs in sys- adapt. Despite these challenges, implementa- temic financial inclusion programs have a tion of financial inclusion programs still need longer time horizon than those in many to be accountable. This calls for a broader other fields. Funders should not expect model of accountability that measures a range quick results at the higher levels of the ToC.1 of demonstrated behaviors that indicate, for Instead, they should give due attention to example, enhanced forms of monitoring, effec- the intervening steps. This is particularly tive risk management, and evidence-based important in setting milestones and targets adaptation. Funders and senior program man- that often frame the incentives for program agement should create an environment that teams and partners. allows these behaviors. Reports to stakehold- ers need to reflect this broad view of account- 9. To understand how change happens, we ability. Although the reporting may be more need to measure impact. Although the type complicated than a conventional accountabil- of evaluative monitoring described in this ity model, it also has the added value of pro- Handbook can give us valuable insights into viding greater scope for learning. program impact, there is usually a need at some point to apply externally led impact Executive Summary References and evaluation, especially for large-scale pro- Resources grams. It is important to focus these evalua- Burjorjee, Deena M., and Barbara Scola. 2015. tions where significant sustained impact is “A Market Systems Approach to Financial likely to be detected. If an evaluation takes Inclusion: Guidelines for Funders.” Washington, place during, or shortly after, the life of the D.C.: CGAP. https://www.cgap.org/sites/default/ files/Consensus-Guidelines-A-Market-Systems- program, this type of impact is likely to be Approach-to%20Financial-Inclusion-Sept- found below the level of development out- 2015_0.pdf comes—for example, at the level of the World Bank. 2015. “The Global Findex Database inclusive financial system. The impact on 2014, Measuring Financial Inclusion around the development outcomes needs to be evalu- World.” Washington, D.C.: World Bank. http:// ated strategically and over the long term, www.worldbank.org/en/programs/globalfindex preferably in collaboration with other Note funders and market actors. 1. The levels of the ToC are set out in Module 3.   viii  MEASURING MARKET DEVELOPMENT W HAT YO U ’ L L F I N D I N TH IS H AND BO O K MODULE 1 (The Systemic Approach and Impli- ToCs in the context of systemic financial inclu- cations for Measurement) introduces the sys- sion programs and puts forward proposals for temic approach to financial inclusion pro- using them effectively. It explains the impor- gramming and explains its rationale. It sets out tance of managing assumptions and risks in the key principles of the approach and their the financial inclusion domain, and presents a implications for measurement. The module model risk register that complements a results describes a broader concept of stakeholder framework. It presents an example of a com- accountability that is appropriate in the con- pleted program results framework and an indi- text of financial inclusion. cator profile. MODULE 2 (The Measurement System) de- Module 6 (Indicators and Data Sources) exam- scribes the scope and components of a mea- ines the roles and limitations of indicators and surement system for programs in financial targets in systemic financial inclusion pro- inclusion. It discusses the importance of a grams. It proposes several indicator focus areas conducive environment for measurement and at different levels of the financial inclusion ToC a broader conceptualization of accountability. and identifies the types of data sources that It explains how the measurement system is might be used. It analyzes the mix of data mobilized through a program measurement sourcing opportunities and the challenges that strategy and project measurement implemen- market systems for financial inclusion interven- tation plans. tions face and how to address them. MODULE 3 (Theories of Change and Results Module 7 (Enhanced Monitoring) examines Chains) examines the roles of the theory of why traditional monitoring modalities often fall change (ToC) and results chains in financial short of what is required for systemic financial inclusion programs and projects and explains inclusion interventions. It describes the con- the distinction between the two terms. It iden- cept of “enhanced monitoring” and points to tifies criteria for effective ToCs and results some of its practical applications. chains and offers a conceptual ToC module for financial inclusion programs that apply a sys- Module 8 (Reporting and Knowledge Shar- temic approach. It also provides examples of a ing) describes the principal roles of reporting program ToC and a project results chain. in systemic financial inclusion programs and how these roles can fit into a single reporting Module 4 (Measurement Questions) explains mechanism. It discusses the inter-related how measurement questions help to sharpen issues of frequency and scope or content of the relevance of both monitoring and evalua- reports and points to their limitations for tion throughout the program and project cycle knowledge sharing with a wider range of and why it is important to frame the questions stakeholders. It provides an example of a pro- at the beginning of these cycles. It then pro- gram annual report template from program vides guidance on identifying key measure- staff to the funder. ment questions. Module 9 (External Evaluation) examines the Module 5 (Results Frameworks) describes uses of external evaluation in systemic finan- results frameworks, how they link to ToCs and cial inclusion programs. It addresses the con- results chains, and how they are used in devel- cept of impact evaluation in the context of opment programs generally. The module systemic financial inclusion programs and assesses the strengths and weaknesses of assesses different evaluation methodologies. WHAT YOU’LL FIND IN THIS HANDBOOK   ix Module 10 (Portfolio-Based Measurement) at less frequent intervals. This module is partic- explains how portfolio-based measurement, ularly relevant to funders. using standardized indicators and/or scoring systems, can produce insights for funders to Annexes improve their interventions and to make deci- A. Glossary sions about their overall strategy. It presents Self-Assessment Tool—a practical instru- B.  two portfolio measurement tools: portfolio ment for benchmarking and identifying gaps dashboards (usually applied more than once a and areas for improvement year) and portfolio reviews, which take place At what stage of programming should one start applying the guidance from the handbook? Program Design and Planning Program Implementation Program & Portfolio Evaluation MODULE 2. The Measurement MODULE 2. The Measurement System MODULE 9. External Evaluation System MODULE 3. Theories of Change and Results MODULE 10. Portfolio Review MODULE 3. Theories of Change Chains and Results Chains MODULE 5. Results Frameworks MODULE 4. Measurement MODULE 6. Indicators and data sources Questions MODULE 7. Enhanced Monitoring MODULE 5. Results Framework MODULE 8. Reporting and Knowledge MODULE 6. Indicators and Sharing Data Sources SELF-ASSESSMENT TOOL The Handbook Is for . . . Measurement specialists in funder organizations that have investments in the financial •  sector. Organizations—funders or program units—that want to produce their own •  financial inclusion measurement guidance. The Handbook is not designed to be an operational manual, but it can provide inspiration and material for such tools. Program and project units that construct measurement systems, strategies, and plans. •  x  Measuring Market Development: A Handbook for Funders and Implementers of Financial Inclusion Programs INTR OD UC T I ON The Market Systems Approach to financial mental organizations, government agencies, inclusion is designed to help market actors nongovernmental organizations (NGOs), and strengthen weak functions and develop other consultancy companies. An important objec- functions that are needed.2 This Handbook tive of this Handbook is to help both funders guides funders and their implementing part- and program unit staff to align their measure- ners on how to monitor and measure results of ment systems and agree on roles and responsi- financial inclusion programs that follow a sys- bilities. It also stresses the responsibilities of temic approach. It was developed through both parties for effective measurement. collaboration with many organizations that This Handbook is particularly useful in devel- work in financial inclusion, including CGAP oping new, or conducting a comprehensive re- members and implementing organizations view of existing, measurement strategies and such as UNCDF, USAID, IFAD, the Mastercard systems at the organizational and program- Foundation, FSD Africa, the Donor Committee matic levels. It includes a Self-Assessment Tool for Enterprise Development (DCED), and the (SAT)—a practical instrument for benchmark- BEAM Exchange. It draws on guidance on ing and identifying gaps and areas for im- measurement for market development, and provement. applies it to financial inclusion. Some guidance This Handbook and the SAT are not alterna- presented here depart from mainstream mea- tives to the DCED Standard for Results Mea- surement practices because this Handbook aims to accommodate the challenges in apply- FIGURE I-1. Comparing the DCED Standard and This Handbook ing the systemic approach to a dynamic finan- cial services market. DCED Standard and Audit Handbook Guidance and SAT This Handbook is intended for measurement specialists who design and manage measure- Describes the desired state Describes the journey from of what a market systems- mainstream approaches to ment systems of financial inclusion programs, oriented measurement measurement, toward a oversee a portfolio of programs, or advise pro- framework should be market systems-oriented gram leads on how to measure results. The in- measurement framework Focuses on the broader tended audience includes funder staff or staff private sector Focuses on financial sector of units that implement financial inclusion pro- grams (program units). These units could be Follows an eight-part Flexible and can apply operated by funders, but they are more typical- framework sections as approrpriate ly found in organizations such as Financial Provides an audit service Provides a Self-Assessment Tool Sector Deepening Trusts (FSDs), intergovern-   1 2  MEASURING MARKET DEVELOPMENT surement (DCED 2016) and its audit services, KEY TAKEAWAYS which are widely used by market development programs in a variety of private-sector pro- • This Handbook is written for measure- grams. Rather, this Handbook complements ment specialists who design and manage the DCED Standard and other relevant guid- the measurement systems and strategies ance, such as the BEAM Exchange (BEAM of financial inclusion programs, oversee a Exchange 2015 and O’Sullivan 2016). Its focus portfolio of programs, or advise program on financial inclusion is a practical resource leaders on measurement. It aims to pro- for funders and program unit staff working in mote effective measurement, thereby this field. helping to improve program design and performance, demonstrate program ef- Introduction References and Resources fectiveness and impact, and maximize BEAM Exchange. 2015. “Monitoring Guidance.” learning. London: BEAM Exchange. https://beamexchange. org/guidance/monitoring-overview/ • This Handbook synthesizes relevant DCED (Donor Committee for Enterprise Develop- guidance from other fields, particularly ment). 2016. “DCED Monitoring and Measuring market development, and applies it to a Results in Private Sector Development.” financial inclusion context. Cambridge, United Kingdom: DCED. http:// www.enterprise-development.org/page/ • Users will need to tailor this Handbook’s measuring-and-reporting-results guidance to their own circumstances. O’Sullivan, Fionn. 2016. “Impact Evaluations for Market Systems Programmes.” London: The BEAM Exchange. https://beamexchange.org/ guidance/evaluation-guidance/ Springfield Centre, The. 2015. “The Operational Guide for Making Markets Work for the Poor (M4P) Approach,” 2nd edition. The Springfield Centre. http://www.enterprise-development.org/ wp-content/uploads/m4pguide2015.pdf Note 2. See Module 1 for more detail on the systemic approach. M O D ULE 1 THE SYST E M I C A P P R OAC H AND M EAS UREM EN T THIS MODULE . . . • Explains the rationale for applying a systemic approach to financial inclusion. • Sets out the key principles of the systemic approach and their implications for mea- surement, including a broader concept of accountability. CONSTRAINTS TO FINANCIAL INCLUSION What is financial inclusion? Despite significant funding of financial inclu- A state where both individuals and busi- sion efforts—an estimated US$34 billion was nesses have opportunities to access, and committed as of 2015—approximately 2 billion the ability to use, a range of appropriate adults remain excluded from the formal finan- financial services that are responsibly cial system (Soursourian and Dashi 2016 and and sustainably provided by formal finan- Findex 2016). And there is still substantial vari- cial institutions. ation in the diversity, quality, and use of finan- cial services. To advance financial inclusion, constraints to • Supporting functions that shape, inform, providing relevant financial services and pre- and enable transactions between custom- venting poor and low-income people from ac- ers and providers. cessing and using financial services need to be addressed. The dearth of providers that serve • Rules and norms governing both core and low-income segments is often a symptom of supporting functions. underlying constraints that may be rooted in Constraints to financial inclusion often stem different areas of the financial system. from knowledge gaps and other capacity An inclusive financial system requires that a weaknesses, misaligned incentives, or weak variety of functions be effectively performed relationships and coordination among a range by different market actors (see Figure M1-1): of market actors. Insufficient financial resources • Core functions of demand and supply (cus- are often not the most significant problem. tomers and providers).   3 4  MEASURING MARKET DEVELOPMENT FIGURE M1-1. The market system and main market functions What is systemic change in the con- text of inclusive financial markets? SUPPORTING FUNCTIONS A change in the underlying dynamics of a system (the financial system or an Coordination interconnected system) that leads to Capital markets changes in the system’s structure, scale, Infrastructure sustainability, and resilience and im- provements in the poor’s participation Skills & in financial services markets. Information capacity SUPPLY CORE DEMAND Standards Informal norms • The financial system encompasses a wide Regulations range of different actors that have interde- Codes of pendent roles and relationships. conduct • Financial markets are unpredictable and Supervision dynamic. RULES AND NORMS • Progress to sustainable financial inclusion at scale can be slow and not necessarily linear. These features lead to the five key principles of the systemic approach and its measurement: • Aim for systemic change APPLYING A SYSTEMIC APPROACH • Play a facilitative role • Experiment, learn, and adapt To promote financial inclusion, funders should • Adopt a long-term view first assess the different functions in the finan- • Take a different perspective on account- cial system and address constraints that pre- ability vent these functions from working effectively to include the poor. CGAP’s “A Market Systems KEY PRINCIPLE #1: Aim for systemic change Approach to Financial Inclusion: Guidelines for The systemic approach is about sustainably Funders” (Burjorjee and Scola 2015) encour- changing the underlying dynamics and struc- ages funders to use a systemic approach in ture of a financial system to enable it to be their financial inclusion programs. Instead of more inclusive. The measurement system being market actors that step in to fill a gap, needs to be designed to detect and assess they can apply interventions that use a sys- whether and how systemic change is happen- temic approach to strengthen weak functions ing. This calls for a distinct level in program and develop other required functions. The Theories of change (ToCs) and results frame- interventions are designed to change the mind works (and often in the project equivalents) sets and practices of financial services provid- that defines systemic change outcomes and ers (FSPs), clients, regulators, supervisors, and their indicators in the specific program and other market actors, thereby altering the under- project contexts. Some outcomes relate to lying system dynamics—a process called sys- program partners; others reach beyond pro- temic change. Systemic change is expected to gram partners to a wider range of actors. ToCs eventually lead to changes in system structure should depict how interventions and interme- and in financial inclusion itself. diate outcomes contribute to systemic change and how systemic change leads to a more inclusive financial system, and ultimately to Key principles and their implications development outcomes like improved eco- for measurement nomic well-being. Applying a systemic approach to financial A broad range of indicators is needed to inclusion has important implications for the measure systemic change. Expected outcomes way programs are designed and carried out often relate to changes in capacity, practices, and for how they are measured. Three key fea- institutional processes and structures, and re- tures of the financial system and its relation- lationships between financial system actors. ship with financial inclusion stand out: Meaningful quantitative indicators are often MODULE 1   5 difficult to discern. It may be impossible to standardize indicators across projects and pro- Programs and projects grams. In addition, results frameworks that fo- cus exclusively on quantitative targets can lead In the Handbook, “program” is a coherent set of proj- program staff to orient themselves “to chase ects with a common focus. A program usually has a lon- the numbers” within target timeframes rather ger life than any single project within it. A project is a than to explore long-term, often less easily tightly focused set of interventions with a common quantifiable, but more sustainable, outcomes. work plan. ToCs apply to programs, while results chains Principle #1 is closely related to Principle #5. apply to projects. KEY PRINCIPLE #2: Play a facilitative role Funders and program unit staff need to think of themselves as facilitators who help to financial services market has evolved and fo- change system dynamics and enable a broad cus on understanding how they contributed to range of market actors to perform market this change. functions more effectively, without having to provide missing functions themselves. Facilita- KEY PRINCIPLE #3: Experiment, learn, tors need to be able to step back to under- and adapt stand, monitor, and address the market system. The systemic approach allows for uncertainty Market systems are made up of the interac- through its flexible design and implementa- tions of multiple market actors that perform tion. Funders that use this approach are able different functions, operate in their own inter- to encourage experimentation and adapta- connected systems, and are influenced by tion. The program design should allow for wider contextual factors beyond the funders’ program staff to engage with a variety of control. Funders and program unit staff need market players, enter into and exit partner- to constantly monitor the overall financial ships as needed, adapt strategies based on system, relevant interconnected systems— new information, and use funding opportunis- such as services markets for capacity building tically to spur innovation. or data—and contextual factors that affect This approach has profound implications for these systems. measurement. Above all it implies that results Measurement tends to incorporate the as- are difficult to predict. ToCs, results chains, and sumption that predicted change is influenced results frameworks—the cornerstones of mea- by or attributable to program interventions. In surement—need to be viewed and managed systemic financial inclusion programs, it is im- differently from each other. Although these portant to broaden the scope of measurement frameworks should be based on the best avail- beyond the intended results to include unex- able evidence of how change happens in finan- pected or unplanned outcomes and factors in cial inclusion, inherent unpredictability dictates the external environment that can affect prog- that they still should be viewed as hypotheses ress both positively and negatively. and not blueprints. ToCs and results frame- Conventional intervention-led measure- works need to be regularly challenged, and ment—the “outward” perspective—asks wheth- funders and program units must be prepared er and to what extent the interventions have and enabled to change them as needed. This led to intended results. Given the complexity requires a new paradigm of accountability of financial market systems, intervention-led where program unit activities are assessed measurement needs to be done in conjunction against a range of factors and are not held to with change-led measurement—the “inward” static targets. perspective. Change-led measurement asks There are also implications for measurement what has changed and what has contributed processes and culture. Monitoring should not to the changes. be only about ensuring interventions stay on Applying these two perspectives allows for track. Monitoring should provide frequent feed- a broader view of evidence of change and tri- back so that interventions can be adapted angulation. It also mitigates dependency on when necessary. Therefore, monitoring needs just one perspective. Although it is difficult to to be linked to regular, data-informed, evalua- attribute specific outcomes to funders’ inter- tive reviews. These reviews are critical in ensur- ventions, this dual perspective allows funders ing that the data emerging from monitoring are to develop a rigorous and credible narrative applied to decisions that shape the interven- about the impact of their programs and proj- tions going forward. The reviews should be ects. When evaluating and analyzing results, owned internally and can be enriched by partic- funders should take a broad view on how the ipation of partners and by outside facilitation. 6  MEASURING MARKET DEVELOPMENT To take root, these approaches to measure- evaluated strategically and over the long term, ment need capacity building and incentives. preferably in collaboration with other funders Measurement of this type is integrated more and market actors. closely with project and program management than conventional measurement. This means KEY PRINCIPLE #5: Take a different there are more participants, and many of them perspective on accountability need capacity support—for example, to under- Conventional accountability in development stand the nature and value of evidence. There programs tends to focus on achieving pre- also needs to be incentives for staff to be defined targets. Accountability is a key feature open-minded and inquisitive, and to take risks. of the relationship not just between program Staff who see that senior unit management and funder, but also project and program on and funders value and use results will likely do the one hand, and funder and governing stake- the same. holders on the other. For this reason, there is a strong inclination to look for common indica- KEY PRINCIPLE #4: Adopt a long-term view tors so that key performance information can The paths from interventions to sustainable be aggregated from the project level along the financial inclusion at scale are long and unpre- chain of accountability. Where common indi- dictable. Sustained impact of financial inclusion cators are not possible or appropriate, there is on people’s lives—development outcomes— still the expectation that quantitative indica- takes even longer to achieve. tors, which can be converted into more readily Funders should not push for quick results at measurable targets, need to be used. the highest levels of ToCs. They should focus on In systemic financial inclusion programs, the intervening steps instead. How, when, and there is limited scope for quantitative indica- which milestones and targets are set can help tors, particularly for those that can be shared to define incentives. Funders should expect an with other projects and programs. Results are extended horizon for program interventions also less predictable long in advance, because and recognize that, while some progress can staff need time to experiment and adapt be achieved in the lifetime of a program, in the program. Despite this, financial inclusion some cases, programs will be contributing to a programs still need to incorporate account- long-term strategic vision in a country context. ability. A broader model of accountability may To understand how change happens, we be needed. This broader model requires pro- need to measure impact. Although the type of grams to be designed to demonstrate a range evaluative monitoring described in this Hand- of behaviors, such as enhanced forms of moni- book can provide valuable insights into pro- toring, effective risk management, and evi- gram impact, there is usually a need at some dence-based adaptation. It also obliges funders point to apply externally led impact evaluation and senior program management to create a especially regarding large-scale programs. It is conducive environment for, and to engage important to focus these evaluations where with, these behaviors. significant sustained impact is likely to be de- The scope and content of reporting to tected. If an evaluation takes place during, or stakeholders must adequately cover the broad shortly after, the life of program, this type of aspects of accountability. This approach has a impact is likely to be found below the level of more complex responsibility on reporting than development outcomes—for example, at the that for a conventional accountability model, level of the inclusive financial system.3 The im- but it also creates an opportunity to combine pact on development outcomes needs to be accountability with learning. MODULE 1   7 KEY TAKEAWAYS 1. Three key features of the financial system and its relationship with financial inclusion need to be kept in mind when shaping a systemic approach and how it is measured: • The financial system involves a wide range of different actors with interdependent roles and rela- tionships. • Financial markets are unpredictable and dynamic. • Progress to sustainable financial inclusion at scale can be slow and not necessarily linear. 2. The five key principles of the systemic approach in financial inclusion programs and their most important measurement implications are as follows: • Aim for systemic change. The measurement system needs to be able to detect and assess whether and how systemic change is happening. Results frameworks need a broad range of indicators, some of them qualitative. • Play a facilitative role. Enhance the scope of measurement beyond the intended results to include unexpected or unplanned outcomes and factors in the external environment that can affect progress both positively and negatively. Measure inward from change as well as outward from interventions. • Experiment, learn, and adapt. Monitoring needs to be linked to regular, data-informed, evaluative reviews. ToCs and results frameworks need to be regularly challenged, and funders and program unit staff must be prepared and enabled to change them. • Adopt a long-term view. Pressure to show results in a short amount of time at the highest levels of ToCs should be avoided. Instead, monitoring and impact evaluation should focus on the intervening steps in ToCs. • Take a different perspective on accountability. Accountability confined to the achievement of predefined targets is less appropriate in systemic programs. Use a broader model of accountabil- ity that requires programs to demonstrate a range of behaviors, such as enhanced forms of mon- itoring, effective risk management, and evidence-based adaptation. Module 1. References and Resources Sinha, Sunil, Johan Holmberg, and Mark Thomas. 2013 “What Works for Market Development: A Review of the Burjorjee, Deena M., and Barbara Scola. 2015 “A Market Evidence.” UTV Working Paper. Stockholm: Swedish Systems Approach to Financial Inclusion: Guidelines for International Development Cooperation Agency. http:// Funders” Washington, D.C.: CGAP. https://www.cgap. www.enterprise-development.org/wp-content/uploads/ org/sites/default/files/Consensus-Guidelines-A-Mar- what_works_for_market_development-1.pdf ket-Systems-Approach-to%20Financial-Inclusion-Sept- 2015_0.pdf Soursourian, Matthew, and Edlira Dashi. 2016. “Taking Stock: Recent Trends in International Funding.” Wash- Davies, Gareth. 2016. “Getting to Scale: Lessons in Reach- ington D.C.: CGAP. http://www.cgap.org/publications/ ing Scale in Private Sector Development Programmes.” taking-stock-recent-trends-international-funding London: Adam Smith International. https://beamex- change.org/resources/785/ Springfield Centre, The. 2015. “The Operational Guide for Making Markets Work for the Poor (M4P) Approach,” Fowler, Ben, and Elizabeth Dunn. 2014. “Evaluating 2nd edition. The Springfield Centre. http://www. Systems and Systemic Change for Inclusive Market enterprise-development.org/wp-content/uploads/ Development.” Washington, D.C.: USAID. http://pdf. m4pguide2015.pdf usaid.gov/pdf_docs/PBAAC412.pdf World Bank. 2015. “The Global Findex Database 2014, Jenal, Marcus, and Mollie Liesner. 2017. “Causality and Measuring Financial Inclusion Around the World.” Attribution in Market Systems Development.” London, Washington, D.C.: World Bank. http://www.worldbank. United Kingdom: The BEAM Exchange. https://beamex- org/en/programs/globalfindex change.org/resources/950/ Kessler, Adam. 2014. “Assessing Systemic Change. Imple- Note mentation Guidelines for the DCED Standard.” Cam- 3. The levels of the ToC are set out in Module 3. bridge, United Kingdom: DCED. http://www.enter- prise-development.org/wp-content/uploads/Systemic_ Change_DCED_Guide_August2014.pdf 8  Measuring Market Development: A Handbook for Funders and Implementers of Financial Inclusion Programs M O D ULE 2 THE ME ASURE M EN T SYST E M THIS MODULE . . . • Describes the scope and components of a measurement system for systemic programs in financial inclusion. • Discusses the importance of a conducive environment for measurement and a broader conceptualization of accountability. • Explains how the measurement system is mobilized through a program measurement strategy and project measurement plans. THE SCOPE OF A MEASUREMENT plans at the project level. These strategies and SYSTEM plans should also encompass complementary processes, such as measurement resourcing, An effective measurement instrument needs to role allocation, capacity building, and incentiv- be designed holistically and managed as a ization. cohesive system, rather than as a set of stand- Figure M2-1 illustrates the main components alone tools. The system includes several com- of a measurement system and demonstrates ponents: the sequence in which the features are typically developed and iterated through both short- • Frameworks define what is to be measured and long-term feedback loops. In the interest and how. They include ToCs and results of simplicity, Figure M2-1 includes only the chains, sets of measurement questions, longer-term feedback loops, which are depict- results frameworks, and risk registers. ed by broken lines. • Processes and tools are used to collect, Although every program should have all the process, assess, report, and use the infor- following components, the nature and scale of mation. these components should be tailored to the needs of a specific program. The measurement system needs to be mobi- lized through a measurement strategy at the Diagnostic process. In systemic approaches, program level and more detailed measurement diagnostics aim to go beyond symptoms to   9 10  MEASURING MARKET DEVELOPMENT FIGURE M2-1. Measurement System Components and Cycle Conducive environment Learning among Carry out program stakeholders and diagnostic wider audiences Articulate the program theory of change Consolidate, report, and share Conducive environment information and Learn, improve, and adapt Develop program measurement knowledge the projects and program strategy, including measurement questions, risk register, and results framework Conducive environment Analyze and review/ Articulate project results chains evaluate the evidence !"#$%&'()*)#('+"#,)#-* Develop project measurement Collect and manage data plans, including measurement on results and risks questions, risk register, and results frameworks Feedback loops identify the root causes of the problem—that Measurement strategies and plans. A mea- is, the constraints that prevent poor and low- surement strategy provides broad and long- income people from accessing and using fi- term guidance for measurement throughout a nancial services. A diagnostic is a continuous program or an organization. It principally cov- process that not only serves program design, ers high-level measurement matters, such as but also allows project management to regu- principles and standards, standardized indica- larly adapt interventions. From a measurement tors, and program-level evaluation. Project perspective, this first data-gathering effort can measurement plans complement the strategy. provide useful background information. For The plans provide more detailed information at more information, see CGAP (2015). the project level. They define what is to be measured, how, when, by whom, and at what Theory of change and results chains. A pro- cost. For more information, see the last section gram ToC defines pathways from interventions of this module. to different levels of expected outcomes; it is usually expressed in the form of a diagram. It Measurement questions. Measurement ques- should include an explanation of the hypothe- tions help to shed light on what stakeholders ses on which progress along the pathways is want and need to know about the program or based (the contribution hypotheses) and ad- project. They cover the whole spectrum from dress the assumptions about external factors economy and efficiency in the use of resources that support or at least do not hinder that to the program’s impact and relevance, ad- progress. A ToC is applied at a program level; a dressing both accountability and learning. results chain is the equivalent at a project level. From a learning perspective, the most import- A project results chain is narrower in scope ant questions tend to relate to the ToC and re- and more context-specific than a program sults chain contribution hypotheses and ToC, which should be broad enough to accom- assumptions. Program measurement ques- modate a range of projects. For more informa- tions tend not to change during the program. tion, see Module 3. Project implementation tends to reveal new measurement questions at the project level, MODULE 2   11 sometimes superseding some of the original Review. Regular internal review of the evidence set. For more information, see Module 4. emerging from monitoring and the assessment of the implications for adaptation (called Results frameworks. A results framework is a enhanced monitoring in this Handbook) are planning and measurement instrument that essential. Although they are internal processes, usually takes the form of a matrix. It includes a reviews can include external stakeholders and summary of intended interventions and out- can be facilitated by external staff, such as comes that is directly related to the program measurement consultants. In some cases—for ToC or project results chain and measure- example, at a project’s mid-point—reviews can ment-related information, such as indicators, take the form of internal formative evaluations. baselines, targets, and means of obtaining For more information, see Module 7. data. The most common type of results frame- work is the logical framework or logframe. Adaptation. The adaptation process is strongly Project results frameworks need to dovetail linked to evidence-informed reviews, and it with their counterparts at the program level— should be part of the measurement system. For for example, through shared outcomes and, in more information, see Module 7. some cases, standardized indicators. For more information, see Module 5. Reporting. Reporting to stakeholders—both internal and external—plays an important role Risk register. A program or project risk register in the measurement system. Good quality and is a framework of identified risks to interven- useful reporting flows from, rather than runs tions and outcomes, usually with ratings and independently of, review processes. Well- information about risk management. Risk reg- presented, outcome-rich reporting leads stake- isters, particularly at the program level, may holders to request more such reports, which in also cover risk categories such as reputation, turn create incentives to further improve the fraud, staff, and data security. For more infor- measurement system. For more information, mation, see Module 5. see Module 8. Data collection. Data are collected in a variety Evaluation. Internal review involves evaluative of ways, depending on whether they are pri- processes, but there is also a role for externally mary or secondary. Primary data are usually led evaluation at some point in the program or collected through surveys, observation, or self- project cycle. Internal and external evaluative monitoring. Collecting secondary data involves processes should complement each other. For access to what has already been recorded, usu- more information, see Module 9. ally for a different original purpose. For more Figure M2-1 includes a component for learn- information, see Module 6. ing among stakeholders and wider audiences, which is outside the normal scope of a measure- Management of data. Once collected, data ment system. However, in systemic programs, need to pass through several processes before knowledge generated from measurement can they are ready to be analyzed and used to be valuable to wider audiences. review, adapt, evaluate, report, and share know- ledge. These processes include the transfer of data from the data collection points, quality A CONDUCIVE ENVIRONMENT FOR assurance, storage, and retrieval. They involve MEASUREMENT IT applications—these do not need to be sophis- ticated or expensive. For more information, see Measuring market systems programs requires a the last section of this module. range of program staff to take initiative and feel a sense of ownership. Measurement needs to Analysis. Data rarely speak for themselves. be done in an environment that They need to be analyzed and presented in an • Confers legitimacy on measurement and accessible way before they can be useful for incentives to participate in it fully. review and adaptation. Analysis should involve competent project staff and should not be • Provides adequate resources. done exclusively by measurement specialists. • Provides capacity building and on-going For more information, see the last section of support. this module. 12  MEASURING MARKET DEVELOPMENT Legitimacy and incentives. Measurement sys- of systemic change that is likely to lead to sus- tems are intricately connected to the organiza- tainable upscaling. tion in which they operate, and they are For example, program units should be ac- influenced by its culture. Program-level mea- countable for the following: surement systems are also affected by external • Designing projects based on thorough diag- stakeholders, particularly funders. To increase nostics and with an appropriate level of its chances of being effective, measurement ambition. depends on several organizational factors, including a sense of ownership by program • Putting in place and operating a proportion- and project staff, strategic positioning of the ate and connected measurement and learn- measurement coordination unit, incentives to ing system. be open about results that are suboptimal, • Providing credible evidence for results that and explicit demand for and use of evidence cannot be expressed quantitatively. by leaders. • Managing risk effectively. Resources. Measurement systems require ade- • Identifying aspects of interventions that quate financial and human resources. A pro- have not worked well and making well- gram or large project would typically have a constructed cases for adaptation to inter- dedicated measurement specialist to manage ventions and their measurement frameworks. its measurement system. Five percent of pro- Funders and senior management should do gram budget is the minimum needed to fund the following: the program or project; 10 percent or more of program budget may be required for innova- • Participate constructively in project formu- tive programs and projects where evidence is lation. particularly important. In financial inclusion • Encourage, resource, and support the programs, measurement expertise combined development of measurement and learning with financial sector experience is hard to find, systems. and these measurement specialists may com- mand a high fee. • Give program units appropriate levels of autonomy in their interventions. Capacity building and support. Measurement • Legitimize and define clear protocols for capacity among program staff is likely to be an adaptation, both in measurement frame- ongoing challenge, especially where measure- works and interventions themselves, engage ment is the work of several partners. Adequate constructively with proposals for adapta- investments in capacity building and support tion, and learn from them. is needed. • Avoid disproportionate demands for re- porting. AN ALTERNATIVE MODEL OF • Have an open dialogue with program units ACCOUNTABILITY about the data implications of their account- ability to stakeholders such as executive Both funders and senior management in their boards, parliaments, or the general public. program implementing organizations need to This broader conceptualization of accountabil- create a conducive environment for measure- ity will take time to achieve. ment. The main objective is to create incen- tives for measurement practices that are transparent and that produce reliable data, MEASUREMENT SYSTEMS ARE while generating cycles of learning and adap- INTERCONNECTED tation. Accountability that focuses exclusively on quantitative targets can create the wrong Project and program measurement systems are incentives by tempting program staff to tailor usually so closely related that the project sys- their interventions “to chase the numbers” tem is said to be “nested” in the program sys- within target timeframes rather than explore tem. Funders and the features of program unit longer-term, often less easily quantifiable but measurement systems also need to be aligned, more sustainable, outcomes. Scale is essential for example, in choosing intended results and and needs to be measured quantitatively, but reporting achievements. at the right level in the results chain, and at the Many organizations struggle to create and right time. The quantitative measurement maintain coherent measurement systems be- needs to be complemented by measurement cause they can be complicated. Systems and MODULE 2   13 their dependencies need to be fully mapped level, planning takes the form of measurement so that they are understood by all stakeholders strategies that address measurement matters and that efforts can be made to streamline that transcend individual projects. These strat- them. For example, implementing organiza- egies chart a long-term course for monitoring tions often receive funding from more than and evaluation across organizations and large one donor. If a system is not streamlined, pro- programs, and they paint a broad picture of the gram units may find themselves working with program-level measurement system, including several different results frameworks, and re- high-level frameworks. They include strategic porting in different ways at different times. measurement questions and can include mea- Stakeholders should be amenable to harmo- surement policy if this is not set out elsewhere. nizing these frameworks and processes as This information is often contained in mea- much as possible.4 surement manuals. Funders are usually preoccupied with man- Measurement strategies should be comple- aging their own systems and rarely have in- mented by more detailed and regularly updat- sights into how program unit systems look and ed measurement plans at the project level. perform. However, the quality of data entering Measurement plans cover all the frameworks funder systems from the programs depends on and processes used for project monitoring and how effective the program measurement sys- evaluation. They provide details on what is to tems are. Funders need to provide incentives be measured, how, by whom, when, and at and support to program units to enable these what cost. They enable project managers to units to build and maintain systems. ensure that resources, capacity, and processes are in place to meet the project’s measurement needs and stakeholder requirements. The mea- MEASUREMENT STRATEGIES AND surement plan should include the project’s key PLANS frameworks: the project results chain, set of learning questions, results framework, and risk Organizations, programs, and projects need to register. Measurement strategies and plans plan to ensure that measurement systems are may overlap, but they should not contradict fit for purpose and effective throughout the each other. Table M2-1 provides a comparison life of the program or project. At the program of measurement strategies and plans. 14  MEASURING MARKET DEVELOPMENT TABLE M2-1. The Scope of Measurement Strategies and Plans ASPECTS STRATEGY PLAN Scope Entire program, including matters that are Project common to all projects Principles and standards Measurement throughout the program Covered by strategy Measurement stakeholders Program stakeholders Project stakeholders Management and governance Management and governance of measurement Roles and responsibilities for measurement at program level; oversight of project in the project, including among partners measurement—roles and responsibilities Human resources Guide to human resourcing of measurement Any measurement awareness-raising and throughout the program, including a competency capacity-building required for project staff framework (relates to principles and standards) and partners Frameworks included Program ToC Project results chain Strategic measurement questions Project measurement questions Program results framework, including project- Project results framework level standardized indicators Program risk register, including, but not limited to, Project risk register—usually related to results those related to ToC assumptions chain assumptions Reference sheets for indicators Reference sheets for indicators Monitoring •  Reference sheets for standardized indicators • Reference sheets for indicators • Guidance on monitoring does not depend on • Planning for baseline data collection project data (e.g., monitoring of program (cross-referring to reference sheets for management and communications) indicator baseline data) • Guidance on monitoring beyond indicators: processes for capturing wider information about intended outcomes, unplanned results (positive and negative), and factors in the external environment that may be influencing the project Data management Processes for data channelling from the projects, Processes for data monitoring, including and subsequent management of data, including quality assurance and preliminary analysis quality assurance and preliminary analysis Internal review, adaptation, • Processes for conducting an internal program • Processes for doing an internal review of and risk management review, including a standard set of review the project, including a standard set of questions review questions • Processes for adapting and continually • Processes for adapting and continually managing risk managing risk Reporting and knowledge Processes for reporting and sharing knowledge Processes for reporting and sharing sharing with program stakeholders knowledge with program management and other project stakeholders Evaluation Initial planning for evaluation, describing the Initial planning for project evaluation(s), type, scope, resourcing, and timing of describing the type, scope, resourcing, and evaluation(s) at program level and pointers timing for project evaluations MODULE 2   15 KEY TAKEAWAYS • A measurement system consists of frameworks (e.g., a ToC or results chain, a results framework, and a risk register), processes (e.g., for data collection and management, review and adaptation, and evaluation), and related tools (e.g., data collection instru- ments, reporting templates, data processing, and storage applications). • Measurement systems, particularly in systemic programs that demand regular and deep engagement by program staff and partners, need a conducive environment characterized, for example, by incentives to be open about results and capacity building and support. Funders and senior management need to foster a conducive environment. • Project measurement systems have multiple linkages to the system in the governing program. Program systems often have touchpoints with the measurement systems of donors and other stakeholders, such as governments. These linkages and touchpoints need to be mapped and managed. • Measurement systems need to be mobilized through high-level measurement strategies at the organizational or program level, and complementary, detailed, measurement plans at project level. Module 2. References and Resources FSD Africa and Oxford Policy Management. 2015. BEAM Exchange, The. 2015. “Monitoring Guidance.” “Towards Impact-Oriented Measurement (IOM) London: The BEAM Exchange. https://beamex- Systems for the FSD Network” Nairobi: FSD change.org/guidance/monitoring-overview/ Africa and Oxford Policy Management. https:// www.fsdafrica.org/knowledge-hub/documents/ Burjorjee, Deena M., and Barbara Scola. 2015. “A towards-impact-oriented-measurement-for- Market Systems Approach to Financial Inclusion: the-fsd-network/ Guidelines for Funders.” Washington D.C.: CGAP. https://www.cgap.org/sites/default/files/ Görgens, Marelize, and Jody Zall Kusek. 2009. Consensus-Guidelines-A-Market-Systems- “Making Monitoring and Evaluation Systems Approach-to%20Financial-Inclusion-Sept- Work: A Capacity Development Toolkit.” 2015_0.pdf Washington, D.C.: World Bank. https://elibrary. worldbank.org/doi/abs/10.1596/978-0-8213- DCED (Donor Committee for Enterprise Develop- 8186-1 ment). 2016. “DCED Monitoring and Measuring Results in Private Sector Development.” Cam- Note bridge, United Kingdom: DCED. http://www. 4. This was one of the principles articulated in the enterprise-development.org/page/measuring- Paris Declaration on Aid Effectiveness of 2005. and-reporting-results 16  Measuring Market Development: A Handbook for Funders and Implementers of Financial Inclusion Programs M O D ULE 3 THEORY O F C HA NGE A N D RES ULTS C HA I N S THIS MODULE . . . • Examines the roles of ToCs and results chains. • Offers a model conceptual ToC for financial inclusion programming. • Illustrates a program ToC and a project results chain. DEFINING THE THEORY OF CHANGE A program ToC is often developed from a AND RESULTS CHAIN broader, more conceptual ToC. This process can be managed incrementally as program de- For use in program measurement, the program sign progresses. A conceptual ToC is valuable ToC needs to be a simplified reflection of the at an early stage in program formulation and program strategy that traces the anticipated planning. It helps to frame the program within progression from the interventions through the wider context of how change is under- successive levels of change to the highest out- stood to happen in the development domain in come in focus—usually what this Handbook question. (A development domain is defined calls the “development outcome.” The ToC as a sector, subsector, or broad focus area, should be based on the best evidence avail- such as financial inclusion, for development in- able of what works and in what circumstances. terventions.) A more context-specific program This type of ToC is usually expressed as a dia- ToC is progressively developed, as the diag- gram, but it may also include narratives. These nostics, conducted by the funder and/or the elements should include the key assumptions program unit, clarify the specific focus of the about external factors on which this progres- program and the intended outcomes at each sion relies. It may also be accompanied by a level. It will, for instance, give more details on definition of the problem or opportunity that the components of the financial systems that the program is designed to address and by the program aims to influence and on the “contribution hypotheses,” which explain the types of intervention with which it intends to underlying logic behind the progression from do so. At the start of program development, one level of the ToC to the next. Contribution funders and program units should agree on the hypotheses are valuable when the program is conceptual ToC they will work from. At a later being reviewed or evaluated.   17 18  MEASURING MARKET DEVELOPMENT stage, after diagnostics have reached an ad- ToC levels vanced stage, both should participate in de- The conceptual ToC in Figure M3-1a has five signing the program ToC. levels, listed in order of highest to lowest level The term results chain is sometimes used in- (for a vertical ToC): terchangeably with ToC. This Handbook uses results chain to refer to a ToC at project rather • Development outcome than program level. • Inclusive financial system A project results chain is rooted in the pro- • Systemic change gram ToC, but because it refers to a specific • Intermediate outcomes project, it is narrower in scope and more con- • Interventions textualized. The results chain defines the path The levels are different in terms of timeframe from the interventions to the intended out- (the higher the level, the longer the timeframe) comes of a project, with more granularity— and sphere of influence (the higher the level, more steps—than the program ToC. Because it generally the weaker and less direct the pro- is project-specific, it can be more precise gram influence). These two distinctions have about the types of partners, the interventions, clear implications for measurement; measuring and their anticipated scale. Results chains are outcomes will require different approaches, usually designed by the program unit, after a depending on the level. thorough diagnostic of the environment. Proj- ect interventions and outcomes should be Development outcome easily matched with those in the ToC, although In this Handbook development outcome refers the results chain may not reach the higher lev- to the highest level of the ToC. Impact is often els of outcome in the program ToC because of found at this level, but this Handbook avoids the project’s narrower scope and generally its use in the context of ToCs and results chains shorter timeframe. for two reasons. First, impact in measurement Program ToCs and project results chains are is taken to mean the net effect of interven- primarily used to set the agendas for measure- tions.5 However, the higher levels of a ToC or ment—both monitoring and evaluation. But results chain do not signify the net effect of they are also valuable tools for communicating the lower levels.6 On the contrary, the higher with program partners and wider audiences. the level, the greater the contribution of exter- nal factors. Second, as pointed out in Module 1, the impact of the intervention in this domain A CONCEPTUAL TOC should be measured at more than one level in the ToC or results chain. Most importantly, it A conceptual ToC for the systemic approach to needs to be measured at the inclusive financial financial inclusion, shown in Figure M3-1a, system level. would typically be used at the beginning of the The development outcome usually captures development of a program strategy—for orien- change in the well-being of poor people. In tation—and as the touchstone for the later the conceptual ToC, the chosen development development of a program ToC. The ToC in outcome is “improved economic well-being,” Figure M3-1a does not prescribe the scope though some programs may aim for other pro- that programs of this nature should have. poor outcomes, including one or more relevant Some programs—such as single-country facil- Sustainable Development Goals. Although it is itation programs—may aim to contribute to common to specify only one outcome at this developing all major parts of the financial level, more than one could be included. market system, and therefore may have ToCs Depending on the status of the financial that cover the same ground as this concep- system at the beginning of the program, large- tual model. Their building blocks, however, scale and sustained development outcomes would be more specific because a program may not happen for many years after programs ToC needs to be evaluable—to translate read- begin. Evidence of change on a small scale ily into a results framework, the central instru- should be detectable early on, but major pro- ment for measurement in programs and gramming decisions should not be based on projects (see Module 5). this early evidence. MODULE 3   19 FIGURE M3-1a. A conceptual theory of change for financial inclusion Development Improved economic well-being (or any relevant SDG) of target group(s) outcome Inclusive financial 1 system A wide range of inclusive financial services supplied at scale Substantial, 2 sustained use of right quality Comprehensive and sustainable financial services 3 supporting functions for expanded by poor people Well inclusive financial services functioning financial Adequate levels of financial capability system of substantial proportion of small enterprises and poor people A comprehensive set of rules and norms conducive to the expansion of inclusive financial services OUTCOMES 4 Systemic change Changes in Noncompeting system actors respond to the innovations (respond) underlying dynamics of Nonpartner competing institutions copy/adapt innovations (expand) the focus market(s) Partners institutionalize the innovations (adapt) 5 Intermediate outcome Partners launch new/improved products/ Nonpartners appreciate services/regulation/etc. (adopt) the relevance to them of partner innovations Initial changes in partners’ practices Changes in awareness, knowledge, LEGEND attitude, and capability in partners The numbers 5 refer to contribu- tion hypotheses and assumptions listed in Figure Interventions Interventions M3-1b 20  MEASURING MARKET DEVELOPMENT FIGURE M3-1b. Contribution Hypotheses and Assumptions HYPOTHESIS #1: A fully functioning financial market system that is supported and sustained by 1 financial inclusion has a substantial impact on livelihoods, vulnerability, and other positive development outcomes. Assumption: Poor people have access to other basic services (affordable energy, housing, education, etc.). HYPOTHESIS #2: A fully functioning market system promotes financial inclusion, enables and 2 incentivizes access to relevant, affordable financial services for underserved people, who have the capability to take up and benefit from these services. Assumption: Enabling social, economic, and political context (no economic crisis, no conflict, etc.). HYPOTHESIS #3: Financial inclusion leads to sustained demand for relevant, affordable financial 3 services that reinforce the incentives of other market actors and other components of the financial market system to provide these services. Assumption: Enabling social, economic, and political context (no economic crisis, no conflict, etc.). HYPOTHESIS #4: Systemic change reaches a tipping point where it becomes self-reinforcing and 4 permanently changes the incentives, inter-relationships, and other dynamics among and between market actors. This leads to a fully functioning system. Assumptions: Time is allowed for market development; enabling social, economic, and political context (no economic crisis, no conflict, etc.). HYPOTHESIS #5: The changes generated by the program are visible and credible to other market 5 actors who begin to invest in similar practices or respond with complementary ones; partner organizations are incentivized to continue to invest in innovation. Assumptions: Nonpartner organizations have the incentives, capability, and resources to copy/ adapt or respond; innovations supported by the program are successful. Inclusive financial system actors, and the dynamics of their engagement This level encompasses two interdependent with markets that are likely to contribute to the elements: changes in the financial system (well- development of an inclusive financial system.7 functioning financial system) and changes in For instance, outcomes will include compelling financial inclusion (substantial, sustained use of a substantial number of nonpartner FSPs to high-quality financial services by underserved provide sustained inclusive financial services— people). Financial inclusion flows from and crowding-in—a crucial feature of an inclusive reinforces development of the financial system: financial system.7 Similarly, while the outcome weaknesses in the system prevent advances in at the financial-system level might be a com- financial inclusion, and without substantial, prehensive set of rules and norms conducive to sustained advances in financial inclusion, the expansion of inclusive financial services, at advances in the financial system are likely to the systemic change level, a satisfactory out- unravel (e.g., market actors would stop provid- come might be the early stages of the transla- ing supporting functions because of lack of tion of policy reform into improved regulation. demand, and regulators would lose focus on Significant change at this level should be the bottom-of-the-pyramid segment). detectable by the end of a 4–6-year program. Depending on the condition of the financial system at the beginning of the program, sub- Intermediate outcomes stantial systemwide advancement may be be- Intermediate outcomes primarily capture the yond the scope and timeframe of a 4–6-year changes directly supported by the program. program. However, significant changes in parts These outcomes mainly happen with direct of the system should be evident before the partner institutions (e.g., FSPs, support- end of a program. function providers, and government agen- cies) or key individuals within them. They Systemic change start with initial, but encouraging, changes This level represents tangible signs of changes that can be cognitive (awareness and knowl- in the orientation and practices of a range of edge), affective (attitudes), and behavioral MODULE 3   21 (practices). These types of change should be Contribution hypotheses and assumptions detectable early in the program. They indicate that partners are moving toward adopting the A ToC should contain or be accompanied by innovation that is expected to be completed explicit and reasonable assumptions that iden- by the end of the program. tify the main external factors of progression Intermediate outcomes also capture the ini- through the ToC. It is helpful to also formulate tial steps toward scaling up the outcomes of contribution hypotheses that explain the logic the program, by attracting both competing behind the progression from one level to the and noncompeting nonpartner market actors next. Figure M3-1b on page 20 provides some to contribute to the innovation in their respec- generic assumptions and hypotheses to illus- tive ways. For example, a competing financial trate how they might accompany the concep- services provider might introduce a similar in- tual ToC. The numbers correspond to those novative financial service, and a government inside the green arrows in the figure. agency might respond to the innovation by re- moving restrictive regulation. Interventions de- signed to attract these nonpartners need to be THE AAER FRAMEWORK explicitly made in the ToC, especially in a sys- temic approach to financial inclusion. Pro- The conceptual ToC focuses on systemic grams that demonstrate notable outcomes change. One of the best-known models for con- often attract other actors, but experience has ceptualizing the process of systemic change is shown that this demonstration effect rarely the Springfield Centre’s Adopt, Adapt, Expand, happens on its own and that program leaders Respond (AAER) framework (see Figure M3-2) need to be more intentional about it. (The Springfield Centre 2015). This Handbook interprets the framework in the context of Interventions financial inclusion programs. This level encompasses the support given to The change process in financial inclusion primary program partners, such as FSPs, sup- begins with Adopt in the lower, left box of the port function providers, and government model. This reflects program-supported inno- agencies, together with the action needed to vation, which, despite the support, requires a promote scaling up by organizations not considerable degree of buy-in from partners. regarded as primary program partners. The Systemic change encompasses the Adapt, most frequent interventions include direct Expand, and Respond phases. Adapt signifies funding, technical assistance, and capacity (i) a high degree of autonomous strengthening building, and less direct action such as advo- and (ii) adaptation by the partners of the initial cacy, the production and dissemination of innovation of their modes of operating in gen- knowledge products, and the facilitation of eral. Both aspects would be underpinned by networking. partner investment of resources. FIGURE M3-2. The AAER Framework ADAPT RESPOND Partners institutionalize the innovation, Noncompeting actors adjust their allocate resources independently of the practices in response to the program, and begin to expand and innovation. diversify their range of services, regulatory framework, etc. ADOPT EXPAND Innovation is introduced by partners Push the boundaries of the and plans to institutionalize it innovation. are developed. Similar or competing actors copy or add diversity to the innovation. Adapted from The Springfield Centre 22  MEASURING MARKET DEVELOPMENT The right side of the model represents • Granularity and context at the intermediate broader systemic change. Expand refers to in- outcome level. vestment by competing market actors (e.g., • At the intervention level, the program ToC nonpartner FSPs or providers of supporting should refer to the mechanisms—such as functions). Expand involves some degree of direct funding, technical assistance—likely copying the behavior and models of the pro- to be used but mostly without tying them to gram partners. specific outcomes, to ensure that individual Respond represents the actions of noncom- projects have the built-in flexibility neces- peting actors—actors who are not competing sary for their specific contexts. with program partners in the market—who ad- just their practices, services, or regulations in The program ToC should be the following: response to the innovation brought about by • Grounded the program. They respond because they per- • Progressive ceive opportunities to pursue their objectives • Plausible or mandate. For example, services providers • Transparent might perceive new, complementary market • Dynamic opportunities or government agencies might • Simple identify a need to adjust regulation. • Evaluable These components of the AAER framework are embedded in the conceptual ToC. The Grounded. The ToC should spring from and Adopt phase, where the innovation is progres- address a central and clearly defined problem sively introduced by program partners, is an or opportunity. It should be based on a rigor- intermediate outcome. The Adapt phase, in- ous diagnostic process that leverages evidence volving the institutionalization of the innova- from research and evaluation and different tion by partners, is at the systemic change stakeholder perspectives for its construction. level, although some programs might consider it to be an intermediate outcome if they intend Progressive. The different levels of the ToC to continue to support partners beyond the should represent a progression. The levels launch of the initial innovation. Expand and Re- should be thought of as clear building blocks spond are integral parts of systemic change in that identify the intended outcomes and the a ToC. They provide evidence of change in the proposed steps needed to achieve these out- underlying dynamics of the market system, be- comes. A higher level should be more than the yond the partners. sum of the results at the lower level; it should represent a change that builds on the lower- level results. From the conceptual ToC to a program ToC Plausible. The transition from one level to the A conceptual ToC (see Figure M3-1a on page next (the progression narrative) must be plau- 19) is used to broadly frame a program, before sible and must reflect the challenges in the a more thorough diagnostic is performed. With program environment and the scale of the this diagnostic, the funder and/or program intervention. Two key aspects of plausibility unit—in varying combinations—can design the are relevance (are the interventions or out- program and construct a more specific ToC comes at one level likely to contribute to the that can help to frame projects within the pro- outcomes at the next level?) and sufficiency gram and form the platform for program-level (are these contributions likely to be powerful measurement. Although the program TOC enough to lead to the outcomes at the next should have the same levels as the conceptual level, given the accompanying assumptions?). ToC, the program ToC should be more specific at each level. It should include, for example, the Transparent. The ToC is a theory—it is a hypoth- following: esis, not a blueprint. The underlying assump- tions and hypotheses should be established • Aspects of a well-functioning financial sys- and periodically tested. Assumptions that are tem that will be targeted by the program deemed to be unreliable should be clearly and more details on the systemic changes identified, and may require specific activities, necessary to improve the way the market such as advocacy, knowledge dissemination, functions.8 increased market coordination, and in some cases, additional projects, to strengthen them. MODULE 3   23 Dynamic. The ToC should be dynamic. It should planned, apart from “softer” interventions, such be developed at a level high enough to set as awareness-raising and networking. broad long-term parameters, but it should be The program is expected to lead to at least flexible enough to allow for adaptation during one project in each of the three countries. Each implementation. The upper levels of the ToC, project will have a coherent set of interven- from development outcome to systemic tions. A results chain for a possible project is change, should be defined with the expecta- illustrated in Figure M3-4. tion that they are unlikely to change during the The ToC in Figure M3-3 is an illustration; it program. The lower levels should be broad should not be used as is. Funders or program enough to allow for the envisaged range of units need to adapt this example to address projects but also be open to adaptation in light their specific program contexts and diagnostic of experience. Market systems are unpredict- needs. able and inevitably need to be flexible to ensure interventions are relevant. Caveats about ToCs Simple. ToCs are often criticized for their sim- Because they are linear and deterministic (the plicity, especially in the complex and unpre- sense that if X is achieved, then Y is bound to dictable context of market development, but follow), ToCs can be challenged by the com- this simplicity is what makes them useful. They plexity and unpredictability of financial inclu- should enable stakeholders, both internal and sion programs. Even nonlinear visual tools, external to the program, to readily understand such as systems maps with complex feedback the nature, scope, and objectives of the pro- loops, are simplifications of programs and their gram and how the interventions are intended environments. The key to ensuring a ToC con- to contribute to them. tinues to work for a program is to make maxi- mum use of its internal flexibility and at the Evaluable. The building blocks of the program same time be prepared to adapt it if the pro- ToC need to be clear and evaluable. First, it gram does not unfold as predicted. should be obvious to stakeholders what the dif- ferent levels and building blocks mean. Second, the outcomes should be clear enough to be interrogated through measurement. This means TIPS: DESIGNING A TOC that translation of the program ToC to its results framework should be straightforward. • Funders and program units should Figure M3-3 on page 24 illustrates a pro- collaborate so that they both own and gram ToC—excluding the contribution hypoth- use the program ToC, including its eses and assumptions, which would be similar iterations. to those in the conceptual ToC. The program • The program ToC should be based on envisaged focuses on promoting pro-poor a thorough understanding of the mar- client-centered financial services with financial ket dynamics that is gained through services providers with two critical support obtaining and triangulating evidence functions—capacity building and information from research and evaluation and dif- services for client centricity—as the primary ferent stakeholder perspectives. targets. In this example, the program design builds on diagnostics that identified a lack of • ToCs are not designed once and for client centricity among financial services pro- all. Their components, including the viders as an obstacle to pro-poor financial assumptions and contribution hypoth- market development in three countries of in- eses, should be reviewed and tested terest to the funder. An assessment indicates periodically as the program advances. that government and financial services indus- They should be adapted, along with try associations are likely to be receptive—and the interventions themselves, if evi- have the capacity—to adapt the regulatory dence suggests that the hypotheses frameworks and member voluntary codes. Sig- are not viable. nificant investment of effort in that area is not 24  MEASURING MARKET DEVELOPMENT FIGURE M3-3. An illustration of a program theory of change Development Improved economic well-being of poor people outcome Inclusive financial A more inclusive financial system system A wide range of financial services relevant to Substantial, poor people sustained use of right quality Comprehensive, relevant information available financial services on clients’ financial services needs and use by poor people A broad range of capacity-building services available supporting client-centered innovation A regulatory environment that enables client- centered financial services innovation Systemic change Regulators and financial services industry associations revise regulation and codes to encourage client-centered innovation Nonpartner FSPs adjust their business models—or enter the market—and embed practices to be more client centered OUTCOMES Partner FSPs adapt and broaden their service o er for poor people Partner capacity building and information services providers adapt and broaden their 5 service o er relevant to client centricity Nonpartner information service and capacity-building providers adjust their business models—or enter the market—to develop services relevant to client centricity Intermediate outcome Policy makers and New/improv- Partner capa- Partner FSPs Nonpartner financial services ed, a ordable, city-building embed client- organizations industry associa- client-related providers centered appreciate the tions commit to information launch new/ practices and relevance to revising regulation services are improved launch new/ them of invest- and voluntary launched by services sup- improved ing in, support- codes to enable partners porting client- financial ser- ing or enabling client-centered centered vices for poor client-centered innovation in innovation people approaches financial services Initial changes in partners’ practices Changes in awareness, knowledge, attitude, and capability in partners and policy makers    Interventions Flexible Direct Technical Awareness-raising, creation, and dissemina- deployment financing assistance and tion of knowledge products, facilitation of of: capacity building networking and cooperation Note: FSPs = financial services providers MODULE 3   25 FIGURE M3-4. An example of a results chain for a client centricity project Range of courses on market Wide range of client-centered Substantial sustained use research and client centricity financial services available of financial services by available poor people Partner capacity builder Demand for and Nonpartner capacity Nonpartner FSPs Partner FSPs create launches adapted market take-up of new builders develop introduce client- a wider range of research and client courses grows new courses centric approaches client-centered centricity and develops to service financial services new courses development for poor people Partner capacity builder Nonpartner capacity Nonpartner FSPs Partner FSPs launch client- pilots market research building providers perceive appreciate the centered services and client centricity business opportunities in importance of client courses o ering similar courses centricity in service development Partner FSPs develop client- centered services for poor people Technical assistance to Awareness raising and Capacity building and partner capacity builder to networking on client centricity facilitation provided to develop market research and targeting nonpartner FSPs partner FSP key sta in Partners recruit new client centricity courses and capacity builders market research personnel and make changes in processes to embrace client centricity LEGEND Suitable capacity builder Technical assistance on commits to partnership developing a client-centric Inclusive financial business model system Systemic change Awareness raising on client Suitable FSPs Coaching on client centricity Intermediate centricity among potential partner commit to to strategic managers in outcomes FSPs and capacity builder(s) partnership partner FSPs Interventions Note: FSPs = financial services providers RESULTS CHAINS from the overall program strategy, a project’s results chain should be rooted in the overall Projects need their own ToCs. This Handbook program ToC. This means that the results chain uses the term “results chains” for ToCs at the outcomes should be identifiable in the pro- project level because there are nuances about gram ToC; however, they are usually narrower their form and use that set them apart from in scope and context-specific. the type of program ToC illustrated by Figure A results chain should be tightly focused on M3-3. The program ToC provides a coherent the field of influence of the project, therefore, framework for the various projects that fall it might not include the development outcome, under the program. Because projects flow or even the inclusive financial system level. It 26  MEASURING MARKET DEVELOPMENT may not include the full range of systemic should be in the framework, and therefore change outcomes. have indicators, and which ones can be left to Interventions in the results chain are more less formal monitoring.9 detailed: They often refer to specific partners, The program unit is likely to take the lead on and they are linked to specific outcomes. It can developing the project results chain. The re- be helpful for results chains to be more granu- sults chain is constructed once there is a thor- lar than the ToC, for example, identifying more ough understanding of the current state of the steps at both the intervention and intermedi- financial system in the specific markets to be ate outcome levels. These two levels often in- covered by the project. Despite the high level teract in the results chain, which reflects the of specificity in the results chain at any one fact that interventions and intermediate out- time, it must maintain its flexibility. The results comes can be iterative and build on each other. chain needs to be reviewed and adjusted in A project results chain needs to be translat- line with adaptations in the project itself. ed into a results framework or similar measure- Figure M3-4 presents an example of a re- ment instrument. In a granular results chain, sults chain for a project stemming from the decisions need to be made about which steps program illustrated in Figure M3-3. KEY TAKEAWAYS 1. A conceptual ToC is a valuable instrument at the early stage of program design and for wider communication. It needs to be complemented by a more specific program ToC and a more focused results chain that align with the program ToC for projects. 2. Program ToCs and project results chains have foundational roles in measurement. They lead directly to the construction of results frameworks and are the platforms for enhanced monitoring and for evaluation. 3. ToCs and results chains need to satisfy several criteria, particularly: • The progression narrative needs to be based on the best available evidence, and there needs to be clear and reasonable assumptions and hypotheses behind this narrative. • The building blocks of ToCs and results chains need to be evaluable. • ToCs and results chains need to be understood and treated as hypotheses and not blueprints. They should be reviewed and adapted during program implementation. MODULE 3   27 Module 3. References and Resources Vogel, Isabel. 2012. “Review of the Use of ‘Theory of Fiala, Nathan, and Markus Pilgrim. 2013. “Interven- Change’ in International Development.” London: tion Models and Performance Indicators of the DfID. https://www.gov.uk/government/news/ ILO Small Enterprises Unit.” Geneva: ILO. http:// dfid-research-review-of-the-use-of-theory-of- www.enterprise-development.org/wp-content/ change-in-international-development uploads/ILOModels.pdf Kessler, Adam, and Nabanita Sen. 2013. “Guidelines Notes to the DCED Standard for Results Measurement: 5. In this Handbook, net effect is defined as the Articulating the Results Chain.” Cambridge, positive and negative, primary and secondary, United Kingdom: DCED. http://www.enter- medium- to long-term, intended or unintended prise-development.org/wp-content/uploads/ effects of interventions. 1_Implementation_Guidelines_Results_Chains_ 6. Higher assumes a vertical ToC. ToCs can also be Apr_2015.pdf presented horizontally. Nippard, Daniel, Rob Hitchins, and David Elliott. 7. Markets at this level are not limited to the 2015. “Adopt-Adapt-Expand-Respond: A financial market. To sustainably change the way Framework for Managing and Measuring financial systems work, financial inclusion Systemic Change Processes.” Briefing Paper. programs might have to focus also on changing Durham: The Springfield Centre for Business in how interconnected markets work (e.g., those for Development. http://www.springfieldcentre.com/ capacity-building or data services). wp-content/uploads/2014/06/2014-03-Adopt- 8. Program ToCs could include all aspects of the Adapt-Expand-Respond-Briefing-Paper1.pdf market system, with some form of differentia- Springfield Centre, The. 2015. “The Operational tion—such as color-coding—to indicate which Guide for Making Markets Work for the Poor are specifically targeted by the program. This (M4P) Approach,” 2nd edition. The Springfield can result in a complex ToC. The alternative is to Centre. http://www.enterprise-development.org/ address the nontargeted aspects of the system wp-content/uploads/m4pguide2015.pdf through explicit assumptions. Stein, Danielle, and Craig Valters. 2012. “Under- 9. The latter would constitute an example of standing Theory of Change in International enhanced monitoring, which is covered in Development. A Review of Existing Knowledge.” Module 7. London: London School of Economics and Asia Foundation. http://www.theoryofchange.org/ wp-content/uploads/toco_library/pdf/ UNDERSTANDINGTHEORYOFChangeStein ValtersPN.pdf 28  Measuring Market Development: A Handbook for Funders and Implementers of Financial Inclusion Programs M O D ULE 4 M EAS UREM EN T Q U E ST IO NS THIS MODULE . . . • Explains how measurement questions help to sharpen the relevance of both monitoring and evaluation throughout program and project cycles. • Provides guidance on identifying key measurement questions. WHY FRAME MEASUREMENT use. Data that are critical for the evaluation— QUESTIONS? both at baseline and during implementation— may not have been collected. Cycles of review Data collection is time-consuming and expen- and adaptation throughout the program will sive. It is easy to be overwhelmed with data. be weaker. Data collection and analysis need to focus on Measurement questions should be identi- what is important to know. Too often, mea- fied at the start of a program or project. This surement is shaped by what can easily be mea- is particularly important in complex environ- sured rather than what stakeholders need to ments, such as financial market systems, know. Far better an approximate answer to the where the landscape for monitoring is less right question than an exact answer to the well-defined and needs the compass that wrong question. In fact, measurement that measurement questions can provide. Identify- lacks focus not only wastes time and money ing measurement questions early on does the but can mislead program stakeholders. It is following: important to be clear about what questions • Guides monitoring. ToCs tell us what inter- monitoring and evaluation need to answer. ventions and outcomes to include in the results frameworks, but they do not say much about monitoring. Measurement ques- WHY DEFINE QUESTIONS AT THE tions point to where monitoring is particu- BEGINNING OF THE CYCLE? larly important and provides clues as to the nature and number of indicators required. Measurement questions are customarily identi- fied when launching an evaluation or impact • Drives the regular use of data for learning assessment. But this is too late for their optimal and decision-making. Monitoring is not just   29 30  MEASURING MARKET DEVELOPMENT about collecting data. Enhanced monitoring defined in the OECD-DAC framework (OECD should probe and test the hypotheses on 1991).10 Appendix A for this module provides which the program is founded. It should ask examples of generic questions and indicates why and why not questions and identify whether they are more likely to be answered what the data and analysis mean for the through monitoring or evaluation, or both. program going forward or for future pro- These generic questions are organized along grams. Posing these questions at the begin- the different levels of the ToC as presented in ning of the cycle helps to ensure that Module 3. program stakeholders regularly apply this At the intervention level, they include ques- type of thinking to the data that are col- tions on the type of resources provided and lected and use them effectively. the timing, scope, and relevance of the activi- ties. At the various outcome levels, they cover • Supports evaluation by ensuring that rele- achievement of expected outcomes and vant baseline data are collected and that changes other than those that were planned. monitoring continues to collect the types of Questions also cover the overall relevance and results information that eventually will be coherence of the program and project: Was needed. the program the right thing to do? Could it • Shapes research agendas. Not all questions have been done better, or by other parties? Is of interest to stakeholders can be ade- this type of program still needed? quately answered through measurement Program measurement questions are broad within a program or project. Defining these and high level. Measurement questions for questions early in a program or project can projects are more detailed and focused. Pro- help funders and other stakeholders decide gram measurement questions are usually de- whether and where to invest in complemen- termined before projects are formulated, and tary research, such as impact evaluation, to to some extent they shape the measurement more fully answer the questions. questions for those projects. At the same time, some program questions may come from those • Promotes consensus around accountability defined at project level, particularly where and learning priorities. Framing measure- more than one project independently identifies ment questions can help funders, program the same or similar measurement question. units, and other stakeholders see eye-to-eye Questions address specific program or on measurement. Funders tend to ask ques- project priorities. For example, it may be im- tions about both the impact of programs portant to track or evaluate cross-cutting di- and the value for money. Program units will mensions of your program or project. Some be more interested in how effective their stakeholders may be particularly interested in interventions are in the near term so that gender equality and women’s empowerment. they can adjust their focus and strategies if Measurement specialists and other stakehold- necessary. Other stakeholders may be inter- ers for mature programs may be more inter- ested in the effectiveness of the program or ested in longer-term outcomes and impact, project to make decisions about their own while those of start-up projects will want to interventions. A collaborative approach to know if the intervention approach is working. defining measurement questions and apply- The set of measurement questions defined for ing them promotes clarity and balance in each program or project should reflect these the measurement system, and realistic ex- specific interests. pectations of what should be measured and The program ToC or project results chain when. Having constructive dialogue around will be the main reference frame for specific measurement questions at the beginning of measurement questions. Questions can be the program cycle will help to ensure that asked about any level, and about relationships measurement is adequately resourced. between the levels. However, some areas of the ToC may be more important than others in a specific context and time. The following are IDENTIFYING THE QUESTIONS some examples: Measurement questions are usually a mix of • It is important to agree on the highest level generic and ToC-specific questions. The at which to assess impact. For example: generic questions are likely to be relevant for What has been the impact of the program every program or project, particularly those on development of the market system or related to effectiveness and efficiency as economic well-being? MODULE 4   31 • Some linkages may need more interroga- interventions. How have these changes come tion than others. For example: Did the regu- about? What are the main factors that have latory improvements achieved through the contributed to them? This perspective pro- program lead to expanded provision of vides an “inward” view from the changes in to- appropriate financial services or increased ward the interventions. Inward approaches take-up of those services? should be based on a comprehensive under- standing of the forces at work in the program • The relative effectiveness of different types environment. of intervention may be of interest.11 For Figures M4-1a and M4-1b show how the example: Is stimulating the growth of a mar- combined outward and inward approach to ket for training providers a more effective measurement questions for an intervention use of program resources than direct train- aiming at improving rules and norms is used. ing interventions? Is financial education The example shows that inward questions are provided by FSPs more sustainable than restricted to the middle zone of the ToC. At the financial education provided by NGOs? development outcome level, in the context of whole or large sections of populations, there Outward and inward questions are so many potential factors at work that in In measurement, interventions tend to be at most cases it would be unrealistic for a typical the center of the program environment. The program measurement strategy to apply in- need for accountability reinforces this. Most ward questions such as “What led to improve- indicators and targets will be about what the ments in economic well-being?” This requires interventions deliver and what effects they evaluative research on a different scale. On the have had. other hand, at the lowest levels of the ToC, However, this tendency carries risks. It can within the “program envelope,” contribution is lead to “intervention bias”—a tendency to more transparent and more easily mapped, so overemphasize the influence of the interven- inward approaches are less relevant. tion and underemphasize or ignore the contri- bution of other factors. It also reduces the chances that stakeholders will understand the APPLYING MEASUREMENT QUESTIONS dynamics of the market system: what is influ- encing what, and therefore, where should the Measurement questions are meant to steer program lend its support to be most effective measurement planning. They are a key first at any time. This is particularly the case in fi- step in planning and shaping monitoring and nancial systems because of their complex and evaluation because they clarify the questions changing dynamics. they should help to answer. Together with Measurement efforts in financial inclusion the results framework, measurement questions need to ask questions about change from two determine the measurement strategy, which perspectives. What were the effects of the in- encompasses all aspects of the measurement terventions? This perspective provides an system. “outward” view the intervention. The other It is important to periodically review mea- perspective is that of changes identified in surement questions and how to answer them, areas of interest to the program or project and to adjust the measurement plan where stakeholders, without initial reference to the necessary. 32  MEASURING MARKET DEVELOPMENT FIGURE M4-1a. Outward and inward measurement questions based on the conceptual ToC Development Improved economic well-being (or any relevant SDG) of target group(s) outcome Inclusive financial MQ1.9 system A wide range of inclusive financial services supplied at scale Substantial, MQ MQ sustained use MQ1.8 2.1 2.2 of right quality Comprehensive and sustainable financial services supporting functions for expanded by poor people Well- inclusive financial services functioning financial Adequate levels of financial capability system of substantial proportion of small enterprises and poor people A comprehensive set of rules and norms conducive to the expansion of inclusive financial services OUTCOMES MQ1.7 Systemic change Changes in Noncompeting system actors respond to the innovations (respond) underlying dynamics Nonpartner competing institutions copy/adapt innovations (expand) of the focus market(s) Partners institutionalize the innovations (adapt) MQ 2.3 MQ1.6 Intermediate outcome Partners launch new/improved products/ Nonpartners appreciate services/regulation/etc. (adopt) the relevance to them of partner innovations MQ1.4 Initial changes in partners’ practices MQ 5 MQ 1. MQ1.3 1.5 Changes in awareness, knowledge, attitude, and capability in partners MQ1.2 Interventions Interventions MQ1.1 MODULE 4   33 FIGURE M4-1b. Outward and inward measurement questions based on the conceptual ToC OUTWARD from interventions to financial inclusion  INWARD from financial inclusion to (and in some cases to development outcomes) systemic change MQ 1.1 Have the interventions been delivered efficiently, MQ 2.1. To what extent has there been substantial, to the intended beneficiaries, and of sufficient scale sustained use of right-quality financial services and quality? by poor people; what have been the drivers and obstacles? MQ 1.2 To what extent and in what ways have the partners’ awareness, attitudes, knowledge, and capacity been MQ 2.2 To what extent and in what forms has affected by the program interventions? there been development of a well-functioning financial system and what have been the drivers MQ 1.3 To what extent and in what ways have the partners’ and obstacles? practices been affected by the program interventions? MQ 2.3 To what extent and in what forms has MQ 1.4 Have the changes in the partners’ practices sup- there been systemic change and what have ported by the program enabled the partners to produce been the drivers and obstacles? new/improved products/services/regulation/etc. of the right quality? MQ 1.5 To what extent have program interventions and/or exposure to partner innovations promoted an appreciation among non-partners of the relevance for them of the innovations? MQ 1.6 To what extent and how have the intermediate outcomes of the program—including unplanned outcomes— led to partners adapting their innovations and nonpartners expanding, and responding to, them? MQ 1.7 To what extent and how has the early systemic change progressed and matured in the form of a well- functioning financial system? MQ 1.8 To what extent and how has the development of the financial system promoted financial inclusion (substantial, sustained use of right quality financial services by poor people)? MQ 1.9 To what extent and how has financial inclusion promoted development outcomes (such as improved economic well-being)? KEY TAKEAWAYS • Measurement questions, which are identified at the beginning of the program or project cycle, set the scope and focus of both monitoring and evaluation, to ensure measure- ment captures useful information for both learning and accountability. • Measurement in systemic interventions needs to avoid intervention-centricity, by ask- ing questions about change from two perspectives: starting from the intervention (out- ward questions—what are the effects of the intervention?) and starting from the identified change (inward questions—how has this change come about?). • Measurement questions usually include generic questions, especially those related to OECD-DAC criteria, and complement these with specific questions relating to the pro- gram ToC or project results chain. • As with other aspects of measurement, it is important to periodically review and revise measurement questions. 34  MEASURING MARKET DEVELOPMENT Module 4. References and Resources Notes IFAD (International Fund for Agricultural Develop- 10. The broad OECD-DAC evaluation criteria— ment). 2002. “Deciding What to Monitor and relevance, effectiveness, efficiency, impact, Evaluate. Managing for Impact in Rural Develop- and sustainability—are likely to be appropriate ment: A Guide for Project M&E.” Rome: IFAD. for program-level evaluation, but impact and https://www.ifad.org/documents/10180/ sustainability may not be relevant to all project b916757f-3eff-45a6-80ca-3f3f53cd0e8f evaluations or to monitoring, which tend to OECD (Organization for Economic Co-operation have shorter time horizons. and Development). 1991. “The DAC Principles 1 1. Comparisons between different types or areas for the Evaluation of Development Assistance.” of intervention may not yield firm conclusions, Paris: OECD. http://www.oecd.org/dac/ but they can uncover issues that need further evaluation/daccriteriaforevaluatingdevelopment exploration. assistance.htm Preskill, Hallie, and Nathalie Jones. 2009. “A Practical Guide for Engaging Stakeholders in Developing Evaluation Questions” Princeton, N.J.: Robert Wood Johnson Foundation. http:// www.rwjf.org/en/library/research/2009/12/a- practical-guide-for-engaging-stakeholders- in-developing-evalua.html MODULE 4 | APPENDIX   35 APPENDIX M4-A. Generic measurement questions FOCUS OF FOCUS OF QUESTIONS MEASUREMENT EXAMPLES OF MEASUREMENT QUESTIONS Interventions Project • Are these the right types and quantity of financial, human, and other resources? monitoring • Are the delivery partnerships functioning well?a • To what extent are the interventions being carried out as planned in terms of timing and specification? If not, are there good reasons for deviation? What can be learned? • Are the intervention processes likely to generate ownership among the target institutions and groups? • To what extent are the right customer segments and organizations being reached and on a sufficient scale? • What is the quality of the interventions in terms e.g., of their relevance and usability for their intended target institutions and groups? • Are the interventions being delivered at the right cost? • If interventions are below expectations, why? • What needs to be done to improve or adapt the interventions going forward? Intermediate Project and • To what extent is expected progress being made with planned outcomes (i.e., those defined outcomes program by the results framework)? To what extent have the planned outcomes been achieved? monitoring and • What significant changes have taken place in the program environment—other than the evaluation planned outcomes—to which the intervention may have contributed? • What would have happened if there had been no interventions? (counterfactual question) • To what extent and in what ways are program partners progressing beyond the program outcomes in contributing to market systems development? • How sustainable do the positive outcomes (planned and unplanned) appear to be? • How and why did these outcomes happen? Why did expected progress or outcomes not happen? • In what ways did the intervention contribute to what happened? In what ways did other factors contribute? • Are the assumptions about external factors affecting the interventions proving to be relia- ble? Have the assumptions stood up in practice? What significant changes have happened in the program environment that may have affected the program or may in the future? • What should the projects do—or have done—differently to be more effective? Systemic Program and • What is the evidence of systemic change? change project – To what extent and how have partners institutionalized the innovations? monitoring and – To what extent and how have nonpartner competing actors copied or adapted evaluation the innovations? –  To what extent and how have nonpartner noncompeting actors responded to the innovations? • To what extent has the program or project interventions contributed to the changes? What other factors have influenced and constrained systemic change? • What should the projects do—or have done—differently to be more effective? Inclusive Program • What changes happened at the financial-systems level—including the status of financial financial monitoring inclusion—beyond the intermediate and systemic change outcomes? systems and • What was the contribution of the lower-level outcomes and the interventions? What were evaluation the contributions of other factors? • What does the status of financial systems outcomes tell us about where and how we should target interventions going forward? Development Program • What development outcomes in the program environment might have been outcomes evaluationb influenced by financial inclusion? • What were the contributions of financial inclusion to those outcomes? • What do the findings about the relationship between financial inclusion and development outcomes tell us about where and how we should target interventions going forward? Relevance and Project and • Was the program or project the right thing to do? Did it have any significant negative effects? coherence program • Were the parties involved the right ones to have undertaken it? Could it have been evaluation done better by other parties? • Is this type of program or project still needed? If so, which parties are best placed to take it forward? a. A delivery partner is an organization or person that contributes to the interventions with, e.g., funding, expertise, logistics, or actual implementation. b. It may not be possible to answer questions at this level through every program evaluation. 36  Measuring Market Development: A Handbook for Funders and Implementers of Financial Inclusion Programs M O D ULE 5 RES ULTS F RA M EWO R KS THIS MODULE . . . • Describes results frameworks and how they link to ToCs and results chains. • Assesses the strengths and weaknesses of results frameworks. • Puts forward proposals for the effective use of results frameworks. • Explains the importance of managing assumptions and risks, and presents a model risk register that complements a results framework. • Presents an example of a completed program results framework and an indicator profile. WHAT ARE RESULTS FRAMEWORKS? function because they present a more intuitive way to identify the key contributions and re- A results framework summarizes key measure- sults of a program or project and to align and ment information. The most common form of a link them logically. results framework is the logical framework— Results frameworks link the definition of or logframe—which has been widely used in results with their measurement strategies by development planning and measurement for identifying indicators and means of obtaining over 40 years. The terminology, precise for- data. Results frameworks at the program level mat, and scope of results frameworks vary to are used mainly for accountability to funders some extent, but they have principal purposes and for overseeing measurement at the pro- and core components in common. They are gram level. Project results frameworks are used at both program and project levels.12 mainly used for accountability to program Results frameworks were originally intro- management and for managing measurement duced to bring systematic and logical thinking in the project. The results framework is not into formulating and planning programs. Al- in itself a comprehensive measurement strat- though results frameworks still do this, ToCs egy or plan; it is a core component of these and results chains have largely taken over this instruments.   37 38  MEASURING MARKET DEVELOPMENT RESULTS FRAMEWORK COMPONENTS With projects using systemic approaches, it is important to monitor beyond the results A results framework is usually constructed as a framework indicators. A column can be includ- matrix and invariably includes the following ed in the project-level results framework for two components: pointers to where monitoring beyond the indi- cators might focus. This would not be relevant 1. A summary of planned interventions and at the program level, where this type of moni- their intended outcomes directly relating to toring does not take place. the ToC or results chain. The left-hand col- The results framework should reflect the umn of the matrix is often referred to as the program ToC or project results chain. Figure “narrative summary.” For many years, logical M5-1 shows the recommended relationships frameworks used standard terminology and among ToCs, results chains, and their respec- number of levels for the narrative summary: tive results frameworks. As noted in Module 3, a single “goal,” a single “purpose,” and a lim- project results chains may include multiple ited number of outputs and activities. This is steps, particularly for interventions and inter- no longer the case; users tailor the narrative mediate outcomes. The user may need to se- summary to their needs. Complex, long- lect fewer key steps when translating the chain term programs and projects in financial into the framework. inclusion, in particular, need more levels and the flexibility to have as many results at each Example of a program results framework level as require measurement. Appendix M5-A contains an illustration of a 2. Measurement variables, such as indicators, program results framework relating to the pro- baselines, projections (often known as mile- gram ToC in Module 3. Like the ToC, it is not stones and targets), and data sources or put forward as a model, but simply as an illus- “means of verification.” Most results frame- tration of what a program-level results frame- works have similar approaches to these work could look like. Points to note about the measurement variables. A few basic rules example include the following: should be followed. • In this program, the highest level to be mon- • Indicator development is often an itera- itored systematically to assess contribution tive process and requires time. is the inclusive financial system. No indica- tors are included at the development out- • Measurement variables should be aligned come level. Data on changes in development with the intervention or outcome to outcomes will be reviewed in the planned which they relate. end-term evaluation and principally to inform • Baselines should be captured as early as recommendations for future programming. possible, once indicators are agreed on. • Program-influenced change in an inclusive • Milestone projections do not have to be financial system is not expected to be signif- expressed in the same terms as in the icant until late in the program, so projections end-line target. For example, a mid-term are not included until the final year when projection for an indicator like “number they will be assessed by the end-term eval- of new financial services” could be a uation. Similarly, projections for systemic stage in product development rather change are not included for the first four than several new services (which is likely years of the program. Any evidence of sys- to be zero). temic change in earlier years will be cap- tured through monitoring and will be • Projections should not be firmly defined reported (see Module 8). until enough is known to make an informed prediction. • No intermediate outcome projections are made until the end of Year 3. This is because Most results frameworks also include a column the intermediate outcomes in the program for assumptions or risks relating to interven- results framework example are focused on tions and outcomes, taking their lead from the those expected to be achieved at or near ToC or results chain. This can be used to moni- the end of the projects, the first of which tor assumptions, which is an essential aspect will be approaching completion at this of enhanced monitoring (see Module 7). A point. This does not mean that evidence of more practical tool for comprehensively man- intermediate outcomes will not be captured aging risks is a risk register. Risk registers are and reported in earlier years. discussed in this module, and an example is included in Appendix M5-C. MODULE 5   39 FIGURE M5-1. Relationship between ToCs, results chains, and results frameworks Conceptual ToC Program ToC Development outcome Development outcome Inclusive financial system Inclusive financial system Project #1 results chain Project #2 Financial Financial results chain systems inclusion Inclusive financial system Systemic change Systemic change Systemic change Intermediate outcomes New services/ regulations Intermediate outcomes + ... + ... appreciate change Non partners Initial changes in practices Changes in aware- ness, attitude, knowledge, capability Interventions Intermediate Interventions Interventions outcome Assumptions Assumptions Data sources Data sources Projection 4 Projection 3 Projection 2 Projection 1 Milestone 2 Program Project Milestone 1 Indicators Indicators indicators (end line) summary summary Baselines Baselines Narrative Narrative Beyond Results Results Framework Framework Inclusive Systemic financial change system Systemic Systemic change change Systemic Interventions change • At the intervention level, because there is to be elaborated to ensure that the data needs little or no scope for meaningful common of indicators are unambiguous and that the indicators across projects, performance is data are collected reliably and on time. This is assessed using a rating scale at each level usually done in one of two ways. One approach with several set criteria. These assessments is to produce an expanded version of the are made during annual reviews and con- results framework, to provide the necessary tribute to a progress “dashboard.” guidance for meeting the data requirements of the indicators. This often ends up being a cum- • In the example, only one indicator is given bersome document. An alternative that pro- for each result for the sake of brevity. Out- motes ownership by the persons who are comes often have two or more indicators. monitoring the indicator is to create a separate • No actual milestone data or projections are microplan or “profile” for each indicator. These given in the example. should be developed as part of the measure- ment strategy (for a program) or plan (for a Indicator profiles project) (see Module 2). A results framework usually contains only a The expanded results framework or the in- summary of the information necessary for dicator profile should include the following: effective measurement. This information needs 40  MEASURING MARKET DEVELOPMENT • Additional information about the indicators tures of how the results should be measured. to ensure they are clear and unambiguous. Bringing this information together and align- ing it can facilitate dialogue and understanding • Sufficient detail about data source(s) and in the late stages of planning, and go on to the instruments (e.g., surveys) and processes serve as a reference for measurement during for collecting the data. and after an intervention. The fact that results • Responsibilities and timelines for collecting, frameworks with similar formats are so widely analyzing, and reporting the data. used also means that stakeholders can readily • Risks to data collection and any mitigating interpret what they see across different proj- steps that need to be taken. ects, programs, and organizations. • Costs, particularly if surveys are involved. Understanding the limitations and Appendix M5-B provides an example of an mitigating the risks indicator profile that complements the results framework for a client-centered innovation Be realistic about the intervention’s field of program. influence. ToCs in financial inclusion present hypotheses about how change happens up to and including the development outcome level. STRENGTHS AND WEAKNESSES Because results frameworks are primarily mea- OF RESULTS FRAMEWORKS surement instruments, it can be misleading to include indicators at levels beyond what is Using and managing results frameworks re- likely to be the measurable influence of the quires appreciation of their benefits, an under- interventions. standing of their limitations, and the mitigation of the risks associated with them. Results Adopt a mixed basket of indicators to measure frameworks have been criticized for their lin- progress beyond financial services access and ear, intervention-centric, and restricted view of use. Results frameworks in financial inclusion outcomes. Results frameworks can also turn programs and projects typically emphasize into binding and inflexible contracts that “snapshots” of inclusion indicators that are restrict measurement and create obstacles to linked to access and use. Results frameworks adaptation. These risks are a particular con- should complement these types of indicators cern for measurement in systemic programs with others that capture evidence of systemic because of their complex, dynamic, and unpre- change and financial system development. dictable environments. (See Module 6 for more on indicators and data sources.) This will also help to make it less Appreciating the benefits tempting to take shortcuts with interventions to maximize inclusion results (e.g., number of Results frameworks define, in one instrument, new accounts) that may look promising at first, the expected results of a program or project, but turn out to be unsustainable. their hypothesized relationships, and key fea- Why are results frameworks criticized? There are two main concerns about results frameworks: 1. Like results chains and most ToCs, results frameworks are, by definition, interven- tion-centric, linear, and deterministic. They present a simplified view of a project or program environment, which implies that the intervention is the principal contributor to the changes and that cause and effect are unidirectional. It confines external factors to the assumptions column. These issues are more acute in the results framework because they are often used as a type of contract between funder and program manager. 2. Indicators often present only a partial view of the results in the framework, particularly if they are exclusively quantitative. This can overshadow less concrete, but nevertheless important, aspects of the outcomes in question. However, these concerns need not outweigh their benefits as long as the concerns are recognized and managed. MODULE 5   41 Monitor results beyond indicators. Indicators This is not to say that program units should tend to become the principal, often the sole, change the content of results frameworks at focus of attention for monitoring. This may will. There should be clear processes and crite- limit the scope of monitoring and reduce the ria for change. Funders should fully consider potential for learning and adapting within the the rationales for changes, not least because program. Monitoring and evaluation needs to of the mutual understanding and learning that be open to aspects of intended results that are this can bring. not captured by indicators and unplanned out- comes beyond the results framework. See Module 7 for information on how this can work RISK MANAGEMENT in practice. Risks affect programs and projects from start Monitor and manage assumptions. Although to finish. The success of interventions and the assumptions are usually included in results realization of intended outcomes depend on frameworks, they are often not diligently moni- internal and external factors, which inevitably tored and managed. Completing the assump- carry risks. Although risks are encountered tions column of a results framework is often mostly at the project level, program manage- viewed as a compliance exercise. It is important ment needs to oversee how the projects iden- to monitor aspects of the program environ- tify and manage risks. ment that are not in the intervention’s direct Risks to interventions include (i) internal field of influence, but that could affect the risks (e.g., to the proper functioning of the proj- intervention and its outcomes. Although it is ect, such as funding or availability of compe- not possible to monitor everything in the pro- tent personnel) and (ii) exogenous risks from gram environment, an important place to start outside actors and factors (such as obstacles is the set of identified risks. The list of assump- created by governments, cultural barriers, tions or risks should be regularly reviewed and threats to security, or macroeconomic volatility updated as necessary. that can delay or frustrate interventions). Risks to outcomes are mostly exogenous— Do periodic reality checks and adjust results and more so at higher levels in the results frameworks, as needed. The narrative sum- chain. All outcomes depend on external actors mary of a results framework—like the ToC or and factors. The launch of a new product by a results chain—needs to be thought of as a partner financial services providers or a new sequence of hypotheses that are designed to regulation by a partner government agency, be tested through measurement processes. for example, depends on a high degree of au- This has three main implications: tonomous action by the partner and support • Measurement processes need to test if from the program and project. Outcomes also these hypotheses hold in practice. depend on a favorable operating environment. The higher up the results chain, the greater the • The same processes need to consider expla- dependency on external actors and factors to nations for outcomes beyond the effects of achieve those outcomes. the interventions (see modules 4 and 8). All programs and projects need to manage • The stakeholders for the results framework risks systematically. Most results frameworks need to be open to changes that reflect the include a column for assumptions or risks re- “discovered reality” of the program and its lating to interventions and outcomes. However, environment. a more practical vehicle for risk management is a risk register. A risk register comprises risks that need to be managed. These risks are iden- A broader model of accountability tified by assessing both the likelihood of the Results, indicators, and targets must be kept risk materializing and the impact on the proj- realistic and up-to-date. If they are not, and ect or program if it does. The major risks to they remain in the contract between funder interventions should be identified as early as and program unit or between program units possible when the project is being formulated. and project teams, they can impede adapta- Risks that are assessed to be significant but tion and innovation. In the broader model of manageable should be included in the risk reg- accountability, funders give program units ister. Risks identified as significant and unman- appropriate levels of autonomy to design and ageable—killer risks—should prompt some adapt their interventions as required, while the degree of project redesign or a decision to units put in place comprehensive measure- change the nature of the program or project ment and learning systems to account for their altogether to avoid the risk. For example, it may overarching objectives. 42  MEASURING MARKET DEVELOPMENT be decided to not run a project in a country client-centered innovation program envisioned where it is unlikely to make sufficient gains be- in the results framework in Appendix M5-A. cause of security or political concerns. Lower- All significant assumptions and risks at ev- level risks should be included in the register if ery level—activity, output, or outcome—need they may require action, for example, creating to be monitored. This is part of enhanced mon- a contingency plan to mitigate the risks before itoring, which is described in Module 7. Howev- they have a chance to materialize. An example er, there are limits to the risks to outcomes that of a risk register is included in Appendix M5-C. can be practically managed through mitiga- The example is populated with some interven- tion. Risk registers may not extend above the tion and outcome risks that might arise in the intermediate outcome level. KEY TAKEAWAYS 1. Results frameworks have an important role in presenting core measurement informa- tion for both programs and projects in a convenient and easy-to-understand format. They are mostly used for accountability and managing measurement. 2. It is important to understand the limitations of results frameworks, and to actively manage the risks associated with them. For example • Be realistic about the intervention’s influence when agreeing on the highest level of outcome that attracts indicators in the results framework. • Track a basket of indicators—both qualitative and quantitative—to measure progress in market systems development. • Extend measurement beyond indicators in results frameworks, which may be only the “tip of the iceberg.” • Do not make the results frameworks the only element in accountability. • Regularly review and, when necessary, revise results frameworks to avoid misaligned incentives and wasting resources used to collect irrelevant data. 3. A results framework is an integral part of any measurement strategy or plan, but it needs to be complemented by more detailed tools for managing indicators, such as the “indicator profile” presented in Appendix M5-B. 4. Risks to planned interventions and outcomes are inherent in systemic financial inclusion programs and projects. Actively monitor and manage key risks (using a risk register). Module 5. References and Resources Wanitphon, Phitcha, and Alexandra Miehlbradt. Bakewell, Oliver, and Anne Garbutt. 2005. “The Use 2015. “Developing a Program Specific Monitoring and Abuse of the Logical Framework Approach.” and Results Measurement Manual.” Cambridge, Stockholm: SIDA. http://www.mande.co.uk/docs/ United Kingdom: DCED. http://www.enter- LFA%20Review%20final.doc prise-development.org/wp-content/uploads/ RMCase7_MRM_Manualb.pdf Glaister, Leslie, and Nathalie Holvoet. 2013. “Feeling for the Intangible: A Framework for Donor’s World Bank. 2012, “Designing a Results Framework Monitoring and Evaluation of Capacity Develop- for Achieving Results: A How-To Guide.” Wash- ment Interventions.” Antwerp: Institute of ington, D.C.: World Bank. http://siteresources. Development Policy and Management, Universi- worldbank.org/EXTEVACAPDEV/Resources/ ty of Antwerp. https://www.uantwerpen.be/ designing_results_framework.pdf images/uantwerpen/container2143/files/ Publications/WP/2013/05-Leslie-Holvoet.pdf Note Itad (Information, Training and Development). 2012. 12. In this Handbook, the term “program” is used “GEMS Results Measurement Handbook.” to designate a coherent set of projects that Brighton, United Kingdom: Itad. http://itad.com/ together help to influence common high-level reports/gems-results-measurement-handbook/ outcomes. A project is a tightly focused set of Jensen, Greta. 2013. “The Logical Framework interventions with a common work plan. A Approach.” London: Bond. https://www.bond. program usually has a longer life than any org.uk/data/files/resources/49/The-logical- single project within it. framework-approach-How-To-guide-December- 2013.pdf MODULE 5 | APPEN DIX   43 APPENDIX M5-A. Example of a partially completed results framework for a client-centered innovation program in financial inclusion CRITICAL ASSUMPTIONS BASELINES AND PROJECTIONS (behind achievement of the Projection Projection Projection Projection NARRATIVE SUMMARY outcome at the same level) INDICATORS Baseline Year 3 Year 4 Year 5 Year 6 DATA SOURCES FOR INDICATORS Level: Development outcome Improved economic The development outcome level will not be systematically monitored in this program well-being of poor people Level: Inclusive financial system Targeted populations have An inclusive financial # available financial services in Program surveys of FSPs’ service system adequate access to relevant each country, in each category range services (i.e., savings, credit, insurance, and payments) that meet criteria Targeted populations have for relevance sufficient information about # suppliers of relevant information Program survey of information services and financial capability services available in each country service suppliers Political commitment to create Assessment of services by FSPs an enabling environment is % of client-centric FSPs in each Program surveys of FSPs maintained country whose information needs are met by affordable information services # suppliers of relevant central Program survey of capacity- bank services in each country building services Assessment of services by FSPs % of FSPs in each country who Program surveys of FSPs perceive (i) statutory and (ii) vol- untary regulation to be enabling client centricity in service development # poor people who report Where available, financial inclusion frequency and perceived value, insights surveys supplemented by using different types of financial program-initiated financial diaries services by target group samples Level: Systemic change Regulators and financial Political will is maintained % regulators and relevant industry Program surveys of regulators services industry associa- associations undertaking relevant and industry associations tions revise regulation Regulators and industry associa- regulation and code revision and codes to encourage tions have the capacity to revise client-centered regulation and codes innovation APPENDIX M5-A. continued 44  MEASURING MARKET DEVELOPMENT CRITICAL ASSUMPTIONS BASELINES AND PROJECTIONS (behind achievement of the Projection Projection Projection Projection Projection NARRATIVE SUMMARY outcome at the same level) INDICATORS Baseline Year 3 Year 4 Year 5 Year 6 DATA SOURCES FOR INDICATORS Level: Systemic change, continued Nonpartner FSPs adjust Nonpartner FSPs have the # and % of targeted nonpartner Triangulation of informal surveys, their business models, or incentives, capability, and resources FSPs adjusting their business networking information, media enter the market, and to copy or adapt models, or entering the market, references embed practices to be and embedding practices to be more client centered more client centered Partner FSPs adapt and Services launched with program % partner FSPs adapting and/or Program follow-up surveys of broaden their services support are successful in the broadening their services offer after partners offer for poor people markets disengagement from direct pro- gram support Partner capacity building Services launched with program % of partner (i) capacity building Program follow-up surveys of partners and information services support are successful in the and (ii) information services provi- providers adapt and markets ders adapting and/or broadening broaden their services their services offer after disengage- offer relevant to client ment from direct program support centricity Nonpartner information Nonpartner providers have the # and % of targeted nonpartner Triangulation of informal surveys, services and capacity- incentives, capability, and resources information services and capacity- networking information, media building providers adjust to develop the services building providers adjusting their references their business models, or business models or entering the enter the market, to market to provide services relevant develop services relevant to client centricity to client centricity Level: Intermediate outcomes Policy makers and Interest at the political level is Evidence of commitment to Collection of qualitative data financial services maintained revising regulation in each through engagement with political industry associations country processes by projects commit to revising re- gulation and voluntary codes to enable client- centered innovation in financial services New and/or improved, Political regimes continue to # of new and/or improved, Collection of quantitative and affordable, client-related permit data gathering relevant, information services qualitative data by projects through information services are launched and assessed as rele- engagement with partner capacity launched by partners vant and affordable by FSPs  building and information services providers. Assessment by a panel of FSPs (partners and nonpartners). APPENDIX M5-A. continued CRITICAL ASSUMPTIONS BASELINES AND PROJECTIONS (behind achievement of the Projection Projection Projection Projection Projection NARRATIVE SUMMARY outcome at the same level) INDICATORS Baseline Year 3 Year 4 Year 5 Year 6 DATA SOURCES FOR INDICATORS Level: Intermediate outcomes, continued Partner capacity build- Commitment sustained by # of new and/or improved, ing providers launch capacity-building providers relevant, capacity-building new and/or improved, throughout the services develop- services launched and assessed affordable services sup- ment process despite lack of as relevant and affordable by porting client-centered FSP demand FSPs innovation Partner FSPs embed Buy-in by FSP management and # new and/or improved services Collection of quantitative and client-centered practices governors launched by partner FSPs that qualitative data by projects through and launch new and/or Commitment sustained by FSPs have been developed using engagement with partner FSPs improved financial throughout the service develop- client-centric approaches and services for poor ment process despite lack of aimed at poor people people early-stage market data Nonpartner organiza- Receptiveness of nonpartner % of nonpartner organizations Project surveys of nonpartner tions appreciate the organizations to change (in each category and in each organizations relevance to them of market) engaged by the program investing in, supporting, Visible and credible partner or directly by FSPs that indicate or enabling client- progress (through their survey responses) centered approaches that they appreciate the relevance of client-centered approaches Level: Interventions Yr 1 Yr 2 Yr 3 Yr 4 Yr 5 Yr 6 Program successfully % of projects on track or better Annual program reviews of implemented in achieving results projects MODULE 5 | APPENDIX   45 46  MEASURING MARKET DEVELOPMENT APPENDIX M5-B. Example of an indicator profile for a client-centered innovation program INDICATOR % of partner (i) capacity building and (ii) information services providers adapting and/or broadening TITLE their services offers after disengagement from direct program support Outcome or intervention to Systemic change outcome: Partner capacity building and information services providers adapt and which it relates broaden their services offers relevant to client centricity Indicator owner (person Program measurement team leader reporting on this indicator) Indicator architecture, A partner is an organization that received support from a project worth at least US$100,000. including definition of terms To be included in the count, a partner would have invested over US$50,000 (equivalent) in services development to reconfigure one or more of the services created during the project and/or to add a new service to its range. Detailed description of the Semi-structured interviews will be conducted in person with the partner CEOs or heads of services means of collecting data for development. They will be asked to show or at least describe their current business plans. the indicator Schedule for collecting Baseline data will be collected by project teams on a rolling basis within two months of disengaging data for the indicator, with partners. including baseline  Progress data will be collected in Quarter 4 of program Year 5 for partners who have disengaged and in Year 6 during the summative evaluation scheduled for quarters 3 and 4. Schedule for reporting End Year 5; in line with summative evaluation reporting the indicator Person(s) collecting data In Year 5 data will be collected by projects (project managers will be accountable for ensuring this is done). In Year 6, the program measurement team leader needs to ensure this is done during the evaluation. Anticipated costs of each Baseline data costs are subsumed in project management. Year 5 survey costs estimated to be round of data collection  US$2,000 per project (included in project budgets). Year 6 survey costs estimated to be US$9,000 (included in evaluation budget). Baseline (copy and paste To be determined. (Baseline data to be collected within two months of disengaging with partners.) from results framework) with explanation if it is zero or N/A Projections (copy and paste To be determined in Year 4 (depends on progress in the projects) from results framework) Issues (e.g., risks to reliable Partners may be reluctant to reveal their business plans in detail. They should be assured about data collection and strategies strict confidentiality. If this is not sufficient, a judgment will be made as to whether the information for mitigation or contingency) available is sufficient to include or exclude them from the count. MODULE 5 | APPENDIX   47 APPENDIX M5-C. Program risk register partial example RISK RISK RISK INITIAL PLANNED RESIDUAL CATEGORY FOCUS DESCRIPTION RISK RATING MITIGATION RISK RATING CONTINGENCY Interventions Projects Insufficient Likely: M Decision to launch a Likely: L/M Decision to cease perform in line number of Impact: H project will be based Impact: H support and redeploy with expectations committed on an extensive funding to another partners found diagnostic of partner market to be made after to develop commitment. Part- 18 months momentum in ners will be required the projects to invest in services development before receiving extensive support. Projects do not Likely: H M&E capacity Likely: M More intense oversight provide ade- Impact: M support from pro- Impact: M of those projects that quate reporting gram in project still give rise to concern of their progress inception period Outcomes Partner FSPs Commitment is Likely: M Plan for early soft Likely: L/M Stop funding support to embed client- not sustained by Impact: M launch of a service Impact: M the partner(s) in centered practices some FSPs to demonstrate question and look to and launch new throughout the potential move funding to more and/or improved service develop- promising partners financial services ment process for poor people despite lack of market data in the early stages Regulators and Regulators and Likely: M Network regulators Likely: M Consider co-funding financial services industry associa- Impact: H and/or financial Impact: H technical support in industry associa- tions do not have services industry phase 2 tions revise regula- the capacity to associations with tion and codes to revise regulation counterparts in at encourage client- and codes least one other centered innovation country who have successfully revised regulation and/or codes Resourcing— financial, human, material Fiduciary Reputation Data security Security and health of personnel Note: The cells above have been left blank deliberately because this is only a partially completed register for illustration only. 48  Measuring Market Development: A Handbook for Funders and Implementers of Financial Inclusion Programs M O D ULE 6 IND ICATO RS A N D DATA S O U R C E S THIS MODULE . . . • Examines the roles and limitations of indicators and targets (projections). • Proposes focus areas, where indicators are needed at different levels of a conceptual ToC. • Categorizes types of data sources. • Analyzes data sourcing opportunities and challenges in systemic financial inclusion projects. THE ROLE OF INDICATORS Certain features of systemic approaches have implications for indicators and their use in Indicators make interventions and expected measurement systems: outcomes measurable. If indicator measure- • Because systemic financial inclusion pro- ment is needed more than once over time, or grams have long ToCs, indicators need to be across interventions, and consistent data col- identified at all measurable levels so that lection is feasible, indicators also enable com- progress can be effectively monitored. parisons. They also can be used as platforms for projections.13 Indicators and projections • Market systems development, particularly help funders and program managers monitor at lower levels in the ToC, implies changes in planned performance; they serve as the prin- the awareness of key actors and the prac- cipal focus for evaluative processes. tices and shape of institutions. These changes Because indicators tend to be tangible. need to be expressed qualitatively, through They often drive measurement systems, there- descriptors. by overshadowing or crowding out less mech- • Indicators need to be regularly reviewed to anistic means of capturing progress and ensure they are still relevant and have kept achievement, which are also needed in com- pace with adaptations in the project or pro- plex intervention environments. These less gram. Projections should be thought of as mechanistic alternatives are described in mod- hypotheses and should be reviewed regu- ules 7 and 9. larly and revised, if necessary.   49 50  MEASURING MARKET DEVELOPMENT INDICATORS IN PRACTICE Indicators need to represent, as closely as With these qualifications in mind, tables possible, the results that the interventions are M6-1 through M6-4 propose focus areas for in- expected to contribute to. At lower levels in dicators, using the outcomes in the Module 3 project results chain, they need to be context- conceptual ToC and typical data sources. specific. At higher levels, there is more scope These are illustrations and are not intended to for standardizing indicators across projects in be an exhaustive list. a program or even across different fund- er programs, as long as there is sufficient con- vergence around what needs to be measured. 1. Development outcomes There are several possible development outcomes, depending on the focus of the program. Improved economic well-being is usually a central objective of financial inclusion programs, although other aspects of well-being, such as empowerment, health, and education, can also be explicit either as a direct or indirect result of financial inclusion. Economic well-being can take the form of increased income, increased personal consumption (generally or relating to, e.g., educa- tion or nutrition) and reduced vulnerability to shocks and variation in income. If programs and projects include development outcomes in their ToCs and results chains, specific aspects of well-being—and their target groups—should be incorporated. TABLE M6-1. Development outcomes—Indicator focus areas and data sources INDICATOR FOCUS AREAS (illustrative) TYPICAL DATA SOURCES Development outcome: Improved economic well-being • Levels of household and/or enterprise asset ownership • National and international surveys and statistics • Patterns of income-generation trials • Program-led sample surveys of clients • Patterns of employment income • Client longitudinal surveys, diaries, etc., for • Consumption patterns controlled trials • Degree of consumption smoothing in response to a negative shock or to variable earnings • Degree of protection of savings from external pressure and temptation • Capacity for long-term financial planning and risk management 2. Inclusive financial system The inclusive financial system level contains outcomes in terms of both financial system develop- ment and financial inclusion. Measuring progress in financial inclusion requires both quantitative and qualitative evidence of the use of financial services by underserved people. Table M6-2 pro- poses areas of focus for indicators and typical data sources. Measuring financial system development requires an explicit description of what is expected to change. As discussed in Module 3, the program ToC should build on the conceptual ToC and provide more details on the targeted components of the financial system. Changes can be ex- pected at various levels: • Supply (e.g., a significant increase in the scale of reach and diversity of financial services provided). • Demand (e.g., customers are more able to engage constructively with the services on offer). • Supporting functions (e.g., increased quality, quantity, or sustainability of certain supporting functions such as market coordination, payment infrastructure, information, or capacity build- ing services). • Rules and norms (e.g., improved regulatory environment). MODULE 6   51 TABLE M6-2. Inclusive financial system—Indicator focus areas and data sources INDICATOR FOCUS AREAS (illustrative) TYPICAL DATA SOURCES Financial inclusion (substantial, sustained use of right-quality financial services by poor people) • Number and % of users, with various breakdowns: • Third-party supply- and demand-side surveys (e.g., – Clients segments (types of people or households, enterprises) Finscope, Findex, Financial Inclusion Insight Survey) – Types of financial services (savings, credit, insurance, payments, etc.) and national statistics – Duration • Program-led sample surveys of FSPs, clients, and nonclients – Regularity (e.g., more frequent use of savings accounts and digital payments than insurance and credit) • Attitudes of users and nonusers Financial system development (well-functioning financial system) Supply side • Number of services providers, in specified categories (e.g., type, scale, • Third-party supply- and demand-side surveys (Finscope, length of time in market) or specified types of services Findex, Financial Inclusion Insight Survey, etc.) and • Proximity of specified types of services to potential customers national statistics • Extent of choice available to specified categories of customers • Program-led sample surveys of FSPs, clients, and nonclients • Costs of specified types of services • Responses of providers to market opportunities and setbacks • Degree of customer centricity of providers’ business models • Degree of interoperability of points of service • Number and/or % of specified types of services meeting given quality criteria • Client satisfaction with specified types of services Demand side • Customer understanding of how to access and use specified types of services • Customer awareness of the range of services • Loan default rates among specified categories of customers Supporting functions • Infrastructure • Access to and use of independent points of sale by targeted customer segments; number and type of FSPs using independent infrastructure— and cost of use; volumes of payments transactions due to interoperability; degree of competition in provision of infrastructure • Sustainability of the infrastructure, user (both FSP and customer), satisfaction with infrastructure • Other supporting functions’ (information, capacity building, market coordination, product development, capital markets, etc.) relevance and comprehensiveness, access and use by FSPs or clients, user (FSP and customer) satisfaction, affordability (where relevant) Rules and norms • Comprehensiveness and relevance of policy, regulation, and • Public record documents relating to policy, regulation, and normative frameworks normative frameworks • Responsiveness of guardians of policy, regulation, and normative • Program-led surveys of rule-setters frameworks to perceived new needs • Rule-setters’ own monitoring • Quality of supervision frameworks • Third-party or program-led surveys of stakeholders (e.g., • Extent of enforcement FSPs, micro and small enterprises) • Extent of buy-in among services providers regarding voluntary industry • Third-party demand-side surveys codes • Extent to which FSPs and other market actors face regulatory barriers to entry or have regulatory incentives to enter the market • Perceptions of policy, regulation, and normative frameworks among stakeholders affected • Inclusiveness of policy-making and regulatory processes (e.g., are stakeholders consulted?) • Extent and depth of cooperation and collaboration among market actors 52  MEASURING MARKET DEVELOPMENT 3. Systemic change The systemic-change level captures changes in the dynamics of the market sys- tems in focus, particularly the behavior of the actors beyond the direct influence of the program interventions. (See Table M6-3.) TABLE M6-3. Systemic change—Indicator focus areas and data sources INDICATOR FOCUS AREAS (illustrative) TYPICAL DATA SOURCES Partners institutionalize the innovations that were fostered by the program (adapt) • Extent and scope of partners’ continuous • Structured and semi-structured inter- improvement of policies and practices views with principal partner interlocutors • Partners' use of customer feedback and other data • Partner qualitative self-monitoring in decision-making (e.g., with journals) • Partners’ use of diagnostics and other approaches • Partner meeting reports, MOUs, budgets, to scaling up and developing new services and other records that capture • Partners’ governance stakeholders’ approaches to decisions and changes innovation and risk • Structured organizational assessment • Partners’ engagement with risk identification, • Program staff or third-party observation monitoring, and management • Partners’ responses to shocks • Partners’ investment in human capacity building • Partners’ networking for knowledge and ideas Nonpartner competing actors copy or adapt partner innovations (expand) Extent to which nonpartners . . . • Develop or adopt similar practices • Third-party industry-level surveys • Acquire technology similar to that of partners’ • Program-led interviews with nonpartner • Develop similar services institutions • Engage more actively with target segments or with actions in relation to target segments of population Nonpartner noncompeting system actors respond to partner innovations (respond) • Extent to which noncompeting market actors • Third-party industry-level surveys adjust or develop new services and/or regulations, • Program-led interviews with etc., in response to innovation by partners nonpartner institutions and expansion by nonpartners 4. Intermediate outcomes Intermediate outcomes mostly relate to the partner institutions, such as FSPs, infrastructure providers, industry bodies, and government agencies. Inasmuch as there have been interventions to stimulate crowding-in and scaling-up of the program innovation, intermediate outcomes should extend to those actors as well. • Among partners, this level of the ToC captures (i) initial changes triggered by the interventions (changes in mindset and capability, commitments to change direction); (ii) initial changes in practices, processes, and structures that build on the new mindset or capability; and (iii) the adoption of these practices by partners, illustrated by upgrading an existing product, service, or regulation or developing a new one. Changes in awareness, attitude, knowledge, and capability can be measured directly through questionnaires and testing, but a more practical and reliable approach is to look for evidence of these changes in actual behaviors. Table M6-4 suggests focus areas for indicators to track these changes. • Intermediate outcomes also capture initial outcomes beyond partners, that is, the initial steps to the crowding-in and/or scaling up phase. At this level, out- comes are expected to show that nonpartners appreciate the relevance of the change at the partner level to their own strategy and/or operations. MODULE 6   53 TABLE M6-4. Intermediate outcomes—Indicator focus areas and data sources INDICATOR FOCUS AREAS (for illustration only) TYPICAL DATA SOURCES Changes in awareness, knowledge, attitude, and capability in partners • Partner staff awareness, knowledge, and attitudes • Structured and semi-structured interviews with principal • Confidence expressed by partner staff in their own and their organiza- partner interlocutors tion’s capability • KAP surveys of wider groups of partner staff • Application of new knowledge and skills • Partner governance and management meeting reports, • Extent to which discussion in partners’ governance forums reflects capturing of dialogue interest in engaging with greater financial inclusion • Program staff observation • Extent, scope, and nature of partner engagement in dialogue with other • Public media output actors about financial inclusion • Public declarations of intent by partners Initial changes in partners’ practices • Create budgets for developing services for underserved segments • Structured and semi-structured interviews with principal • Adjust strategies and/or structures to focus on developing services for partner interlocutors underserved segments • KAP surveys of wider groups of partner staff • Obtain and analyze data on underserved segments • Partner meeting reports, MOUs, budgets, and other records • Engage with internal capacity assessments related to developing new that capture decisions and changes services • Partner qualitative self-monitoring (e.g., with journals) • Procure external services aimed at advances in financial inclusion • Program staff or third-party observation • Public media output Partners launch new, improved products, services, regulation, etc. (adopt) Supply: New or improved inclusive financial services provided by partner • Structured and semi-structured interviews with principal institutions partner interlocutors • Number, type, and specific features (e.g., cost, transparency, flexibility) • Partner qualitative self-monitoring (e.g., with journals) of partners’ services targeting underserved segments • Partner meeting reports, MOUs, budgets, and other records • Accessibility of partners’ services by underserved segments that capture decisions and changes • Efforts made by partners to develop customer awareness and under- • Partner business data (for private-sector partners) standing of new services • Public record of changes in policy, regulation, and normative • Take-up of new partners’ services by targeted segments frameworks (for program working on rules and norms) • Client satisfaction with new partner services • Client, user, stakeholder surveys Supporting functions: New or improved services provided by the partner institutions • Types and specific features (e.g., scope, robustness, ease of use, customer affordability, transparency) • Customer reach • Affordability • Use by targeted clients and user satisfaction Rules and norms: New or improved rule or norm or process by partner institutions • Types and specific features (e.g., scope, transparency, relevance to underserved segments, and the providers of services targeting them) of new or improved partner-generated policy, regulation, and normative frameworks • Capacity of actors to apply, supervise, or enforce the policy, Nonpartner institutions appreciate the relevance to them of partner innovations • Engagement by nonpartner institutions with partners’ business models • Third-party, industry-level surveys for underserved segments • Program-led interviews with nonpartner institutions • Competing nonpartner institutions identify the case for copying or adapting the innovations • Noncompeting market actors identify the case for modifying their own practices in response to the innovations Note: KAP = Knowledge, attitude, and practice; MOUs = memoranda of understanding 54  MEASURING MARKET DEVELOPMENT DATA SOURCES is guided by the measurement plan,14 includ- ing the measurement questions that apply to Sourcing data for measuring financial inclusion monitoring (see Module 3). Data for systemic programs presents few conceptual but many financial inclusion programs may be needed practical difficulties. Data are needed in three on the following: stages: before the program or project begins, • Indicators in the program or project results during the program or project, and after it has framework, at the intervention, intermediate, been completed. and systemic change levels. Initially, baseline data may be needed, particularly for out- Before the interventions begin. Data are need- comes, and this should be collected as early ed before interventions begin, both to diag- as possible in the intervention, if not before. nose market barriers and opportunities and to Baseline data are the main references for establish baselines for future measurement of setting targets and for assessing change. change. Data might be needed on any of the following: • Aspects of planned results not adequately captured by indicators. These data are often • Actual and potential client perceptions of qualitative. financial services: access, use, affordability, risks, cultural fit, capacity, and information • Unplanned or unexpected outcomes, both deficits. positive and negative. • Current status and potential for advancing • Reasons why planned interventions and financial services provision: number and outcomes have or have not been achieved, types of providers and of services, extent including the contributions of actors out- and nature of outreach and use of services, side the interventions.15 types of customers, services provider knowl- • Pre-identified assumptions and risks and edge of (and attitudes toward) the low- other relevant dynamics beyond the field of income market segment, types of services, direct influence. providers’ strategies, and openness to inno- vation, capacity, and information deficits. At the end of or after the program or project. • Current status and potential for developing Data are needed at the end of, or after, the pro- supporting infrastructure and services: num- gram or project, for the summative and impact ber and types of providers and of services, evaluation, which is usually conducted exter- extent and nature of use, types of services, nally (see Module 8). At this stage, it is often providers’ strategies and openness to inno- necessary to collect additional data to fill gaps vation, capacity and information deficits, in the monitoring data, to complement it and availability of technology. through triangulation and to assess contribu- tion. Data collection is normally directed by a • Current structure and application of policy, detailed evaluation framework, led by the regulation, and other rules and norms. Atti- broad measurement questions set at the begin- tudes and capacity and information deficits ning of the program cycle, and further speci- among policy makers, regulators, and other fied through judgment criteria, indicators, and rule or norm shapers and guardians. other means of guiding data collection. Data at Data collection at this stage should be influ- this stage might be needed on the following: enced by a measurement strategy, including • Results framework indicators at systemic the program ToC, if one exists. (See Module 2 change outcome level—if not already or for a description of measurement strategies.) adequately collected—and at the financial Diagnostics at such an early stage (program or system and, where relevant, development project formulation) are not strictly measure- outcome levels. ment activities; however, data collected at this point can be used for measurement baselines if • Perspectives on planned results not cap- data are relevant and up-to-date—and comple- tured by indicators, for triangulation and mented by indicator-specific baseline data, deeper understanding. These data will be once the results framework has been created. mainly qualitative. • Unplanned or unexpected outcomes, both During the program or project. Data are positive and negative. needed during the program or project to • How and why outcomes have or have not monitor intervention performance and critical been achieved, including the role of actors aspects of the environment. Data collection outside the intervention. MODULE 6   55 Criteria for data collection. Even if there are quantitative as well as primary and secondary) appropriate data sources, choosing the right can be used at different levels of the ToC. data to collect and on the right scale can be challenging. Organizations can end up drown- ing in data and/or being faced with incomplete DEMAND- AND SUPPLY-SIDE DATA or poor-quality data when it is too late to SOURCES improve data quality. Successful data sourcing and manage- When focusing on the intermediate outcome, ment is often guided by use cases—that is, systemic change, and financial system levels of the specific potential use of the data. Rele- the ToC, it is important to distinguish between vance, intention, and incentive to use are im- supply- and demand-side data collected by portant criteria for data collection. However, third parties. Demand-side data originate from it is also important to be aware of other actual or potential users of financial services: strengths and limitations of data sources. individuals, households, enterprises. Supply- The following is a suggested checklist of cri- side data originate from services providers. teria for data sources. Both vary in several ways that are important for financial inclusion program use, including Practicability frequency of collection and how representa- The data source exists and is accessible. tive they are. Access is affordable; the cost of access is These differences need to be known before proportionate to expected value. deciding to choose them as data sources for Reliability  indicators. Comparing the attributes of differ- Trustworthy—data are sufficiently complete, ent data sources to understand their strengths representative, and free from unintentional and limitations can help guide you to the bias or deliberate manipulation. sources and indicators that are most relevant. C  onsistent—data in the same categories can be In recent years, data sources for financial in- collected every time they are needed. clusion have become richer and more compli- Precision cated to navigate. There are numerous data  The data sets are sufficiently granular (e.g., sources on the demand side. Although having data are collected by target groups and ample data to measure progress is an asset, locations of interest). each of these sources treats data in different Timeliness ways, which can lead to confusing or mislead- Data are available when needed and are ing results. Depending on what data source sufficiently up to date. you use, you can arrive at significantly different Ethics conclusions about financial inclusion. See Data are collected ethically and sustainably,  Table M6-5 for the best-known demand-side ensuring that human or institutional sources data sources; more details are provided in will be prepared to provide data in the future. Appendix M6-A. On the supply side, sources vary in scope and in survey attributes as shown in Table M6-6. Data sources for each level of the ToC The higher the level of the ToC, the more pro- grams are likely to rely on secondary or shared data, although this should be triangulated C OLLABORATE TO SOURCE DATA TIPS:  where possible with program-generated data. At the lower levels of the ToC, interventions Where there are no relevant data, it may be possible to need to generate their own context-specific collaborate with other funders, program units, and primary data. Data triangulation should be country stakeholders. This is not only important for pursued when possible, especially when pos- funders’ and program units’ measurement needs, but a ing how and why questions. For market sys- well-functioning financial system also relies on collect- tems programs and projects, assessing ing, processing, and interpreting data. Funders can changes in knowledge, attitude, capacity, and facilitate market development by collaborating with behavior (intermediate outcomes) is essential each other to develop sustainable data sources at the to understand what, how, and why the under- industry or country level. Collaboration with other lying dynamics and incentives are changing. funders can help to develop local capacity, ownership, Hence, primary—often qualitative—data are and a robust financial market data infrastructure. essential. Tables M6-1 through M6-4 illustrate how multiple data sources (qualitative and 56  MEASURING MARKET DEVELOPMENT TABLE M6-5. Demand-side data sources for financial inclusion QUANTITATIVE INDIVIDUAL/ SUB- DEMAND-SIDE DATA OR LAUNCH AND HOUSEHOLD/ NUMBER OF SAMPLE NATIONAL EXAMPLE INDICATORS OF DATA SOURCES DESCRIPTION QUALITATIVE FREQUENCY FIRM COUNTRIES SIZE DATA “ACCESS” AND OTHERS Global Findex Global trends and Qual 2012; Individual 148 1,000+ No • % holding an account cross-country 3 years at a formal FI comparison • % obtaining a loan from an FI within the past year FinScope Individuals’ Quant + 2002; Individual; 17 1,000– Regional • % operating an account financial mgmt; Qual 4 years some 21,000 with an FI perceptions household • # of adults holding an regarding financial conclusions account with a SACCO services; formal and informal FinAccess/ Individuals’ Quant + 2006 in Individual; Kenya, 8,250– Regional • % formally included Access to financial mgmt; Qual Kenya; 3–4 household Nigeria, 20,850 • % owning a mobile Financial perceptions years; others phone Services regarding financial varies by • % access by region, Surveys services; formal country gender, education and informal Financial Individual Quant + 2013–2015; Individual 10 + add. 3,000– Regional • % with a bank account Inclusion perception and Qual yearly or countries 45,000 • Average distance to Insight behavior regarding twice/year likely to closest banking facility Surveys formal digital come soon • % with a mobile financial services money acct Financial Trends in Quant + 2012–2015; Household 3 3,000– Regional • % of HHs owning or Inclusion households’ Qual yearly + (panel) (Tanzania, 5,000 with access to a Tracker financial behavior; quarterly Pakistan, mobile phone Surveys trends in poverty Uganda) • % of HHs with active SIM levels of mobile • % of mobile money money users accts/region WB Financial Assessment of Qual 2008; Individual 14 1,000– Urban/ • % who correctly answer Capability and people’s under- varies 2,000 rural financial knowledge Consumer standing of questions Protection financial • % who budget Surveys concepts • % with formal credit IFC Enterprise Estimation of the Quant + 2010; as Firm 177 Varies, No • % of MSMEs with Finance Gap number of MSMEs Qual data (panel) 1,000 access to credit Database in the world and the becomes data • % constrained by degree of financial available for select access to credit inclusion countries Living Household Quant 1985; Household 38 800– Regional • Average monthly Standards welfare and (limited) varies (panel) 36,000 expenditure by type Measurement behavior Qual and region Study % of HHs operating •  nonfarm enterprises Note: FI = financial institution; SACCO = Savings and credit cooperative organization MODULE 6   57 TABLE M6-6. Supply-side data source to measure financial inclusion—examples and features SUPPLY-SIDE INFORMATION NUMBER OF ANALYTICAL GEOSPATIAL EXAMPLE INDICATORS OF DATA SOURCES SOURCE FREQUENCY COUNTRIES TOOLS SUBNATIONAL “ACCESS” AND OTHERS IMF Financial Central banks in Yearly, 2004– 189 No No • Bank branches per Access Surveys 189 countries 2012 jurisdictions 1000km2 • ATMs per 100,000 adults MIX Maps MFIs; increasingly other Quarterly; 15 Tableau Yes • # of financial institutions, financial services depends on visualization; by type or region providers and regulators country and bench- • Locations per 100,000 providers marking adults • Locations per 1000km2 fspmaps.com Georeferencing 1 round; 7 Analytical Yes • # of poor people living providers in-country sustainability geospatial within 5km of a financial plans under online tool access point (urban/ discussion rural divide) • Locations of financial access points GSMA Mobile Mobile money providers; Since 2011, 208 Benchmark- No • # of registered MM users Money Adoption telcos yearly operators in ing • # of active MM agents Survey 83 countries • Value and volume of transactions Microinsurance Landscape studies by Launched in 102 Benchmark- No • # and % of people Centre Landscape the Munich Re 2016; yearly ing; online covered by different Studies Foundation, IADB, visualization types of insurance MFW4A, and GIZ WB Remittance Surveys; certified Launched 2016; 32 sending; Benchmark- No • Average cost of sending Prices Worldwide national and regional 2011 onwards 89 receiving ing or receiving remittances databases available online from specific countries • Cost of sending and receiving by institution type World Bank’s Central banks 2 years 139 Benchmark- No • Volume of transactions Global Payment (2008, 2010) ing Survey • # of countries with consumer protection legislation Mftransparency.org Microloan providers Launched 2006 26 Benchmark- No • # of transparent yearly and as ing and products data become online • % of products with one available graphing or more fees (2008–2013) tool • APR by institution type Note: MM = mobile money 58  MEASURING MARKET DEVELOPMENT KEY TAKEAWAYS • Indicators can be used to evaluate outcomes and other subjects of interest. If mea- sured more than once over time, or across interventions, indicators enable compari- sons. • Because they are tangible, indicators may be used to drive measurement systems. Although indicators may seem to offer a practical solution to the measurement needs of hard-pressed stakeholders, they can overshadow or crowd out less concrete means of capturing progress and achievement that may be more appropriate in complex intervention environments. • Indicators need to represent the results that the intervention is expected to bring about or contribute to. At the systemic change level and below, they need to be inter- vention-specific. Qualitative indicators may be more appropriate than quantitative. • There is some scope for standardizing indicators at higher levels in the ToC. • Data are needed throughout the project or program lifecycle. This requirement can create technical, logistical, and budgetary challenges. It is important to focus on use cases—on relevance and a clear intention to use—when considering data collection and to apply a range of other criteria for data fitness. • Different levels of the ToC or results chain—and therefore different stages in the pro- gram or project cycle, respectively—require different data sources and instruments. Qualitative data are more relevant at the lower levels of the ToC, while quantitative data feature more at higher levels where there is a greater focus on clients. • Collaboration is needed to ensure consistent, good quality, data sourcing at the national and international levels for financial inclusion. Module 6. References and Resources Demand-side Data Sources Church, Cheyanne, and Mark M. Rogers. 2006. • FinAccess. http://fsdkenya.org/publication/ “Designing for Results: Integrating Monitoring finaccess2016/ and Evaluation in Conflict Transformation • Financial Inclusion Insight Survey. http://www. Programs.” Washington D.C.: Search for intermedia.org/data-home/ Common Ground. http://www.dmeforpeace.org/ • Financial Inclusion Tracker Survey. http://www. sites/default/files/SFCG_Designing%20for%20 audiencescapes.org/fits Results_Ch4.pdf • FinScope. http://www.finscope.co.za/pages/ Kessler, Adam, and Jim Tanburn. 2014. “Why default.aspx Evaluations Fail: The Importance of Good • IFC Enterprise Development Database. http:// Monitoring.” Cambridge, United Kingdom: financegap.smefinanceforum.org/ DCED. http://www.enterprise-development.org/ wp-content/uploads/Why_Evaluations_Fail_ • Living Standards Measurement Survey. http:// Aug2014.pdf iresearch.worldbank.org/lsms/lsmssurveyFinder. htm Sen, Nabanita, and Adam Kessler. 2015. “Guidelines to the DCED Standard for Results Measurement: • World Bank Financial Capability and Consumer Defining Indicators of Change.” Cambridge, Protection Survey. http://responsiblefinance. United Kingdom: DCED. http://www.enter- worldbank.org/surveys/users-of-financial- prise-development.org/wp-content/uploads/ services 2_Implementation_Guidelines_Indicators_ • World Bank, Global Findex. 2015. “The Global May_2015.pdf Findex Database 2014, Measuring Financial Inclusion Around the World.” http://www. worldbank.org/en/programs/globalfindex • World Bank. Financial Capability and Consumer Protection Survey. http://responsiblefinance. worldbank.org/surveys/users-of-financial- services MODULE 6   59 Supply-side Data Sources • World Bank. Global Survey on Consumer • Bank for International Settlements (BIS) Protection and Financial Literacy. http:// Payment Systems Statistics. http://www.bis.org/ responsiblefinance.worldbank.org/surveys/ list/cpss/tid_57/index.htm providers-of-financial-services • Finclusion Lab. MIX. http://finclusionlab.org/ • World Bank. Remittance Prices Worldwide. • Fspmaps.com. http://fspmaps.com/ http://remittanceprices.worldbank.org/ • GSMA Mobile Money Adoption Survey. http:// www.gsma.com/mobilefordevelopment/ Notes programmes/mobile-money-for-the-unbanked 13. An indicator should be neutral and framed as, • International Monetary Fund. Financial Inclusion e.g., “the number of FSPs who launch new Access Survey. http://fas.imf.org/ inclusive products,” rather than “x new firms • Mftransparency.org. http://www.mftransparency. launch new products” or “an increase of y% in org/microfinance-pricing/ the number of FSPs who launch new inclusive products.” • Microinsurance Centre Landscape studies. http:// www.microinsurancecentre.org/landscape- 14. Measurement strategy is linked to the ToC; studies.html measurement plans are linked to the results chains and are more detailed. See Module 2. • World Bank. Global Payment Survey. http:// www.worldbank.org/en/topic/paymentsystems 15. There will be limits to how rigorously this can be remittances/brief/gpss done outside the scope of external evaluations. 60  MEASURING MARKET DEVELOPMENT APPENDIX M6-A. Selected demand and supply data sources There is no single best source for data on financial inclusion. To find out what data you need, start by asking yourself what you want to know. You also need to ensure that the data are reliable and valid. DEMAND-SIDE DATA SOURCES 3. FinAccess and Access to Financial Services sur- veys are similar to FinScope surveys, but they 1. Global Findex is the only global demand-side are not conducted by FinMark Trust. To a large data source that allows for global and regional extent, FinAccess in Kenya and Access to Finan- cross-country analysis. It includes data from 148 cial Services in Nigeria (and the like) follow the countries and collects information on 506 indi- same principles as FinScope, but they are not cators from at least 1,000 individuals over 15 conducted by nor owned by FinMark Trust. As years old within each country. The sample is with FinScope, these surveys are designed nationally representative and randomly selected. through industry consultation, which means Since the survey is a module added to the Gal- they have the potential to meet many needs lup World Poll, it combines information about and answer many questions. They have some of sociodemographic conditions and access to or the same drawbacks as FinScope: they are use of financial services. Global Findex is mainly not designed for cross-country comparison. used to analyze global trends and to make Because these surveys are commissioned and cross-country comparisons using key financial carried out by various entities, data quality may inclusion indicators, such as the number of be inconsistent. In addition, across both Fin- adults with access to formal bank accounts. A Scope and FinAccess surveys, there is no stan- drawback is that the data are not subnationally dard definition of “financially included,” so the representative, which means that they are less meaning behind this term varies according to useful for in-country policy makers because the definitions used by local stakeholders. For there is not sufficient granularity. Also, the defi- instance, some countries might include pay- nition of formal financial services used in this ment cards or mobile wallets that are not linked source is based on people’s perception of to an account, while others may not. whether their provider is a formal financial insti- 4. Financial Inclusion Tracker Surveys (FITS) are a tution, which is not necessarily aligned with the nationally representative panel survey design- regulatory and supervisory framework of a ed to collect trend data about households’ country. The sample is randomized at the indi- financial behavior over time. The Bill & Melinda vidual level, which allows users to aggregate the Gates Foundation’s Financial Services for the data by individual characteristics, such as Poor team in partnership with InterMedia income and gender, but this also makes the data designed these surveys to run over a three-year incompatible with household-level surveys. period in three countries. The sample size is 2. The FinScope Survey is a globally recognized 3,000 households each in Uganda and Tanzania demand-side data source that allows for finan- and 5,000 households in Pakistan. The survey cial inclusion indicators to be measured at a sub- will measure the same households throughout national level. It was launched in 2002 and is the entire period. Data from the survey repre- trademarked and owned by FinMark Trust. Fin- sent collective behavior and use patterns for all Scope is a nationally representative survey that members of a particular household. The data explains how individuals manage their financial are used to estimate trends in poverty levels of lives. It also provides insight into attitudes and mobile money users. This focus on households, perceptions about financial products and ser- while a useful perspective, can also be a short- vices. Sample sizes vary widely across countries; coming because it is not as helpful for analysis surveys have included responses from 1,000 to at the individual level. Furthermore, the survey 21,000 individuals. The unit of sampling is at the has been carried out in only three countries. individual level, but the survey does enable some 5. Financial Inclusion Insight (FII) Surveys were conclusions at the household level. Eighteen launched by the Bill & Melinda Gates Founda- countries have conducted or are in the process tion in partnership with InterMedia. FII data of conducting FinScope surveys. Often industry first became available in 2013. Unlike FITS, FII players will pay some of the cost of the survey surveys are not panel surveys because they do and help tailor the questionnaire to meet multi- not track the same household over time. They ple stakeholders’ needs. FinScope data are not focus more on measuring individual percep- comparable across countries on all indicators. tion and behavior, making them comparable to FinScope, FinAccess, and the like. However, their intentional focus on mobile money and digital financial services sets these surveys apart. The surveys are typically conducted MODULE 6 | APPENDIX   61 annually, with commitment from the Gates 3. Word Bank’s Global Payments Systems Survey Foundation to fund the surveys in eight coun- (GPSS). GPSS is a comprehensive survey that is tries (Kenya, Tanzania, Uganda, Nigeria, India, conducted in 139 countries. It provides informa- Pakistan, Bangladesh, and Indonesia) over tion on the status of national payments and three years. CGAP has also engaged InterMe- securities settlement systems worldwide. It cov- dia to conduct FII surveys in Rwanda and ers all aspects of national payments systems— Ghana (released in 2015) and Myanmar and from the legal and regulatory environment, to Côte d’Ivoire (data collected 2016–2017). Fur- infrastructure, technological and business thermore, UNCDF’s Mobile Money for the Poor model innovations, international remittances, program has hired InterMedia to conduct FII and oversight framework—and combines quan- surveys in Benin and Senegal. The sample size titative and qualitative measures of payments is typically high to allow for subnational repre- system development. GPSS is expected to sentation (even more so than, e.g., FinScope). guide reform efforts in the payments system The surveys include welfare measures based arena both nationally and globally. The 2008, on the Grameen Progress Out of Poverty index, 2010, and 2012 surveys provide a snapshot of which is unique to FII surveys. Because FII sur- the payments and securities settlement sys- veys mainly focus on insights into digital finan- tems in both advanced and emerging econo- cial services, they do not capture many mies. Data from the fourth and latest iteration of indicators around access to and use of nondig- the survey were expected in late 2016. ital financial services. 4. MIX’s Finclusion-Lab. The MIX Market is the pre- mier source of public information on microfi- SUPPLY-SIDE DATA SOURCES nance institutions (MFIs) and their financial and social performance. MIX offers a suite of popu- 1. The IMF Financial Access Survey (FAS) is the lar analysis reports at the global, regional, and most comprehensive global supply-side data on country levels, including global analyses of key financial inclusion. In addition to providing pol- issues for the sector. MIX has expanded its icy makers and researchers with annual geo- focus through geospatial mapping efforts to graphic and demographic data on access to aggregate and visualize data for FSPs, includ- basic consumer financial services worldwide, ing and beyond MFIs. To date (2016), 23 coun- FAS is one of the main data sources for the G20 tries have subnational data visualized through Basic Set of Financial Inclusion Indicators the portal. MIX’s move to visualize geospatial endorsed by the G20 leaders at the Los Cabos subnational supply-side data through publicly Summit in June 2012 as well as the more com- available geospatial maps has enriched the sup- prehensive G20 Financial Inclusion Indicators. ply-side data landscape. This will be a challenge The FAS database contains annual data for 189 because frequent data collection can be expen- jurisdictions, including all G20 economies, cov- sive and/or ad hoc, depending on when data ering a 10-year period (2004–2014). Countries become available. manage their data and metadata. FAS has the 5. Fspmaps.com is a website funded by the Bill & same functionality as Findex’s cross-country Melinda Gates Foundation in partnership with comparison advantage for demand-side data, Spatial Development International. Fspmaps. but for supply-side data. The IMF FAS is not com provides analytical tools to answer several subnationally representative, and the data financial access questions. As with MIX, the depend on countries’ ability to capture data website leverages geospatial information for from FSPs. Moreover, FAS includes data on pru- financial inclusion tracking and analysis. It hosts dentially regulated FSPs only. comprehensive geospatially referenced finan- 2. GSMA Mobile Money Adoption Survey. In 2011, cial access point data and high-resolution pop- Mobile Money for the Unbanked (MMU) initiated ulation data, including poverty densities and a global adoption survey to give managers of other demographic attributes. Its analytical mobile money deployments better insights into tools can be used to obtain detailed information the performance of their services relative to about where people—including poor people— each other. Subsequently, GSMA created its live in relation to financial services access points. Global Annual Mobile Money Adoption Survey— Underserved areas can be better detected this its annual results are published as part of the way. Another analytical tool allows you to drop MMU Annual State of the Industry report. In the a pin on a map and calculate population served 2015 survey, 107 mobile money providers from with mobile coverage but without adequate 67 countries participated in the survey. While financial access. The website also allows users the database itself is not public, MMU publishes to import private datasets. Fspmaps.com cur- an analysis of the aggregated results. The sur- rently hosts data for Tanzania, Uganda, Nigeria, vey offers a snapshot of the mobile money Kenya, Bangladesh, and India. For most coun- industry every year and provides benchmark tries, the data are snapshots in time and have data to mobile money services providers. not been collected over time. 62  Measuring Market Development: A Handbook for Funders and Implementers of Financial Inclusion Programs M O D ULE 7 ENHAN C ED M ON I TO R I NG THIS MODULE . . . • Examines why traditional monitoring falls short of what is required for systemic financial inclusion programs. • Describes the concept of “enhanced monitoring” and points to some of its practical applications. This module will be useful to staff of program mum implementation strategies and the pre- units; funders will also benefit from enhanced cise nature and timing of intermediate out- monitoring to promote a supportive environ- comes. For example, certain FSPs or other in- ment. Enhanced monitoring has implications stitutional partners chosen at the outset of a for resourcing and capacity building in mea- project may turn out to be unprepared or not surement, and for how accountability is inter- committed to stay the course in developing preted. new financial services, support functions, or rules. There are several reasons for this, in- cluding change of leadership or policies. In THE NEED FOR ENHANCED these circumstances, staff need to adapt im- MONITORING plementation strategies and expected inter- mediate outcomes to the new and changing Traditionally, monitoring is used to track per- circumstances. formances and to capture data for evaluations. The monitoring process is used to map the Monitoring tends to be thought of as a narrow, terrain and support the interventions’ naviga- technical process that is conducted in the mar- tion through it. The main challenge lies in gins of program and project management. This the unpredictability of results. Traditional model is being challenged in many develop- monitoring is inadequate and can yield mis- ment fields, but practice has been slow to leading information because it is usually con- adapt to the change in thinking. fined to predefined indicators and relies on In systemic financial inclusion programs structured surveys conducted at infrequent and projects, it is difficult to predict the opti- intervals for data on outcomes. The monitor-   63 64  MEASURING MARKET DEVELOPMENT ing process needs to have a broad scope, to • Regular evaluative reviews should be con- be conducted more frequently, and to be inte- ducted. Reviews should focus on outcomes grated with evaluative internal review and ad- that are based on evidence. They should aptation processes: include partners, where possible and appropriate. • Outcomes should be regularly monitored, at least as far as the systemic change level in • Agreed processes and criteria need to be in the ToC or results chains, through indicators, place so that the project or program (and its and more broadly, it should capture the nar- measurement frameworks) can be adapted rative behind the indicators. based on the reviews. • Monitoring should be designed to be broad enough to capture unplanned or unex- ENHANCED MONITORING IN DETAIL pected results and aspects of the environ- ment that are affecting the program or Three characteristics of systemic financial inclu- project. These environmental concerns may sion programs and projects call for enhance- have been identified as assumptions or risks ments to monitoring and adjustments to its or may be unexpected. relationship with external evaluation. These • Monitoring should be designed to have a characteristics and their implications for moni- broad perspective about what is driving toring are set out in tables M7-1 through M7-3. results. It should not be intervention-centric; it should include both an inward and out- ward perspective (see Module 4). TABLE M7-1. Implications for monitoring—Complex environment INTERVENTIONS OPERATE IN A COMPLEX ENVIRONMENT IMPLICATIONS FOR MONITORING The complexity and unpredictability of financial • Results monitoring needs to be flexible. Predefined outcomes, indicators, and markets mean that targets are important, but they need to be reviewed and adjusted or replaced if • It is difficult to predefine results, indicators, and they turn out to be inappropriate. This is easier if they are not locked into an targets with confidence inflexible results framework. • Assumptions and risks relating to the external • Results monitoring needs to take a broad approach that extends beyond predefined environment play a major role in intervention indicators. Results-focused mission reports and logs or journals kept by partners management and project staff are the most common tools for this type of monitoring. See Appendix M7-A for an example of a monitoring log. • The intervention impacts its external environment, but the environment also affects the intervention and its outcomes. Aspects of the external environment need to be monitored. This is chiefly done through identifying and monitoring assumptions behind, or risks to, the progression narrative of the ToC or results chain. However, it is not always possible to identify important external factors in advance, so the intervention’s active set of assumptions and risks needs to be regularly reviewed and adjusted. TABLE M7-2. Implications for monitoring—Change cannot always be quantified CHANGE CANNOT ALWAYS BE QUANTIFIED IMPLICATIONS FOR MONITORING Change at lower levels in the ToC and results 1. The monitoring of intermediate outcomes and systemic change exclusively through chains tends to be about attitude, behavior, quantitative indicators is likely to be inadequate and sometimes misleading. Useful organizational processes, and structure. These data at these levels are often qualitative and highly context-specific. They often take changes can be nuanced and often refer to very the form of narrative descriptions, especially where there are few units of change small numbers of people and organizations. (e.g., organizations). The two types of indicators need to be appropriately balanced. Qualitative data pose challenges to the aggregation of results across programs or 2.  portfolios and comparative assessments between them. Unlike standardized quantitative indicators, there is no way to automatically aggregate and compare the results of interventions if they are expressed qualitatively. However, in some cases, these types of results can be categorized to enable comparisons through scoring systems (see Module 9). MODULE 7   65 TABLE M7-3. Implications for monitoring—Dynamic environment INTERVENTIONS OPERATE IN A DYNAMIC ENVIRONMENT IMPLICATIONS FOR MONITORING The intervention environment is dynamic, 1. Data need to be collected regularly and in a timely fashion. This is easier to do if the particularly where technology is involved, and data are collected by the partners themselves or project staff on the ground, rather our understanding of it needs to keep pace. than by remotely located staff during field visits. Relying on an infrequent outcome Interventions need to enable rapid response to survey is not enough. Data need to be regularly reviewed, and their implications for remain relevant.  the intervention assessed. Reviews should be conducted with both internal and external key stakeholders, where appropriate. Reviews should make use of data previously collected through monitoring, but can 2.  also be used to generate data themselves in real time by asking measurement questions during the review (e.g., about significant changes that participants have observed or experienced). Answers to these questions should be triangulated with data already recorded or should be followed up for corroboration. Do not assume that changes identified can be attributed to the interventions. 3.  Program and project staff should try to identify plausible drivers of the change in question and examine the evidence to help determine which drivers may have been the most influential. This “inward” perspective in monitoring and internal review can realistically be done robustly only up to a certain level of the results chain—in some cases, no higher than the systemic change outcome level. The demands on data collection and analysis at higher levels may be too great. Processes for making necessary adaptations to both the interventions and the 4.  measurement plans should be systematized. For example, this could be a mandatory item in review agendas. HOW TO MAKE IT WORK Beyond processes, staff capacity is crucial. The organization should enable staff to iden- Enhanced monitoring needs legitimacy, time, tify evidence of change in a complex interven- resources, guidance, and support. Funders tion. The measurement plan should be used and senior program management need to to point to the types of data to look for and create an enabling environment for enhanced explain how they should be recorded. At the monitoring. same time, staff should be encouraged and The proposed enhancements to monitor- empowered to create and own a narrative ing can be demanding. The following are that captures their experiences, observations, some tips on tailoring processes and tools to and partner feedback. fit the program or project and to manage Enhanced monitoring, enabled by commit- measurement capacity of a unit’s staff. ment from management and effective pro- Two overarching instruments promote ef- cesses, should lead to improved implementa- fective enhanced monitoring: measurement tion and results. It is likely to happen only if questions and a project measurement plan. funders allow for flexibility in results frame- As Module 4 demonstrates, clear measure- works and program implementation. Chang- ment questions at the beginning of the inter- ing course, after due process, needs to be an vention cycle will set the agenda for both acceptable option and encouraged. monitoring and evaluation. For example, ques- tions about whether assumptions in the ToC Transitioning from traditional to and results chains have turned out to be reli- enhanced monitoring able, and whether they were the right ones, are relevant for both monitoring and ex-post Transitioning from traditional to enhanced evaluation. To develop a project measurement monitoring can be a big change for program plan, the details of what is to be monitored, unit or organization staff. With enhanced mon- how, when, by whom, and at what cost need itoring, more people—often most project team to be worked out. The plan provides a realistic members, partners, and in some cases, their context and helps to systemize the approach clients—are involved in collecting and report- (see Module 2). However, it is also important ing relevant data. Implementing enhanced not to “over-engineer” the data collection monitoring requires substantial change to pro- tools and processes. If the processes are cum- cesses, appropriate expertise in design and bersome and rigid, they will be seen as a coordination, and more financial resources bureaucratic chore. than that of traditional monitoring. Staff may 66  MEASURING MARKET DEVELOPMENT find it difficult to move directly from traditional vant data have been collected and that a indicator-based monitoring to the new para- degree of analysis and interpretation has digm of enhanced monitoring. This section taken place. Identifying measurement ques- addresses managing this transition. tions at the beginning of the program or project can help. • External evaluators can add value to mea- INTEGRATING MONITORING WITH surement planning at the start of a program EVALUATION or project cycle. They can help to identify measurement questions, construct the In an enhanced model, monitoring takes on a results framework, identify areas for moni- new relationship with evaluation. Enhanced toring beyond indicators, build capacity, monitoring is not a substitute for evaluation. It and steer baseline research. In some cases, cannot achieve the scope or rigor of a well- external evaluators are retained to work as designed and well-conducted evaluation. There real-time or developmental evaluators. is usually a place for evaluation in a project and invariably in a program. • Evaluations should inform future monitor- Monitoring and evaluation can be integrat- ing in the same or similar programs by ed in three principal ways: highlighting what needs to be monitored and how. • Where evaluations are planned—whether internally or externally led—monitoring can Evaluation and its relationship with monitoring support evaluations by ensuring that rele- is explored further in Module 9. T RANSITIONING TO ENHANCED MONITORING TIPS:  Enhancing monitoring does not have to be implemented in one fell swoop. Instead of taking one big step, teams can move forward incrementally. • The first step can be to improve reviews, so that they can be conducted frequently; include all necessary stakeholders; are supplied with adequate, well-presented data; and lead to actionable conclusions. • Another step is to foster a sense of ownership of data collection. Ensuring that the staff who collect data are involved in interpreting the significance of the data provides an incentive for them to perform this function more effectively. • Encouraging staff to broaden their field of vision in looking for evidence of change may be difficult at first, but the effort is rewarded with a richer engagement with data. For the project team, this may mean having closer and more regular contact with stakeholders in the field (partners, clients, and other market actors). • Piloting enhanced monitoring in one or two projects and demonstrating how it can lead to improved implementation can drive buy-in, because staff will be able to see the positive effects of their efforts. MODULE 7   67 KEY TAKEAWAYS • It is widely recognized that “traditional” monitoring, which is based exclusively on results framework indicators, is inadequate and can be inappropriate for programs that apply a systemic approach to financial inclusion and other sectors. • Indicator monitoring needs to be complemented by a wider range of data about planned and unplanned results. • The external environment will affect the intervention and its outcomes both positively and negatively, predictably and unpredictably. These external factors need to be moni- tored mainly through the lens of assumptions and risks. • Systemic financial inclusion programs need to be adaptable. Data collected through monitoring need to be regularly reviewed, and the conclusions of the reviews should lead, through due process, to adaptations to both the program or project itself and its results framework. During the review process, it is important to be clear about what is driving the identified changes: the extent to which the changes were driven by the inter- vention and other players and factors. • Monitoring and external evaluation have different roles, but there is a middle ground where monitoring staff and evaluation staff need to collaborate. Measurement strategies (Module 2) need to identify how the roles are different and what aspects call for collab- oration. • Monitoring enhancements amount to substantial organizational change. Staff involved in the enhanced monitoring should have incentives and support. Module 7. References and Resources Dunn, Elizabeth, et al. 2014. “Monitoring Facilitation Ripley, Matthew, and Daniel Nippard. 2014. “Making Activities: Report to USAID Uganda.” Washing- Sense of Messiness: Monitoring and Measuring ton, D.C.: USAID. http://pdf.usaid.gov/pdf_docs/ Change in Market Systems: A Practitioner’s PBAAC328.pdf Perspective.” Durham, United Kingdom: Hummelbrunner, Richard, and Heather Britt. 2014. Samarth-NMDP and The Springfield Centre. “Synchronizing Monitoring with the Pace of http://www.enterprise-development.org/wp- Change in Complexity.” Washington, D.C.: content/uploads/Samarth-Bangkok-5Mar14.pdf USAID. file:///C:/Users/patri/Downloads/ Samarth-NMDP. 2015. “Results Measurement Pace%20of%20Change%20in%20Complexi- System: User Manual.” Lalitpur, Nepal: Samarth- ty%202014-09-25%20FINAL.pdf NMDP. http://samarth-nepal.com/resource/ Perrin, Burt. 2012. “Linking Monitoring and Evalua- results-measurement-system-user-manual tion to Impact Evaluation.” New York: Rockefel- USAID (United States Agency for International ler Foundation. https://www.interaction.org/ Development). 2016. “Program Cycle Guidance. sites/default/files/Linking%20Monitoring%20 Complexity-Aware Monitoring.” Washington, and%20Evaluation%20to%20Impact%20 D.C.: USAID. https://usaidlearninglab.org/library/ Evaluation.pdf complexity-aware-monitoring-discus- sion-note-brief 68  MEASURING MARKET DEVELOPMENT APPENDIX M7-A. Example of a project monitoring log The log is designed to capture observations—first hand or from reliable secondary sources—about: 1.  Project interventions. Changes in the project environment (outcomes) that appear to be influenced by project interventions, 2.  ether planned or unplanned. wh Other (contextual) factors in the project environment that have had, or are likely to have, a bearing 3.  on the project and its outcomes. Ref # Short account of what has been observed (provide hyperlink How, when, and Relevance of the observation: or other reference to any sources with fuller information). where was this • If it relates to an intervention or outcome Explain what happened, to or with whom, and when. If it information (and, where relevant, indicator) in the results is an outcome, explain why you believe it was influenced captured? framework, specify which. by a project intervention and which one. Note also if any • If it is about a contextual factor, indicate action is needed to corroborate the link with an intervention. which pre-identified assumption or risk  (if any) it relates to. (If the observation implies a heightened risk factor, it should be followed up with risk management.) 1 Example: We received feedback from Paul FF that the Phone call from Intervention: “Coaching on client centricity to coaching he received so far is not practical. He finds it Paul on 25.04.16 strategic managers in partner FSPs.” to be too theoretical. He is not asking for the coach to be replaced but for us to discuss alternative approaches Indicator: Satisfaction rating. with her. A recording of the call can be accessed at. . . . (Paul was aware that the call was being recorded.) 2 Example: We learned that one of our FSP partners, PPP We were Intermediate outcome: Partners recruit new Bank, has signed a contract with TTT Insight to conduct informed by TTT staff and make changes in processes to qualitative research among savings groups in three counties Insight in a embrace client centricity. in western Kenya. This is its first venture into this type of meeting at its research and is likely to have been triggered by the office on participation of its marketing director in our Nairobi 04.05.16. workshop on client research in January 2016. (We will need to follow up with the manager to confirm this.) 3 Example: We have become aware that Miriam LL, the HHH Bank Intermediate outcome (assumption): board member of HHH Bank most committed to promoting website seen Commitment sustained by FSP management client centricity in the bank, has left. This could represent a 11.05.16 and governance throughout the services risk to board-level commitment to the continued process. development process. M O D ULE 8 REPORTING AND KNOWLEDGE SHARING THIS MODULE . . . • Describes the principal roles of reporting in systemic financial inclusion programs and how reporting and knowledge sharing can be captured in one report. • Discusses the inter-related issues of frequency and scope and/or content of reporting. • Provides an example of a program annual report template. REPORTING ROLES needed. Reporting needs to be carefully craft- ed around key stakeholders’ agreed vision of Reporting is a critical element to achieving accountability. This might include, for example, both accountability and learning. It also tends the obligation to demonstrate through report- to focus on results expressed through indica- ing the effective management of risk and of tors and targets. Reporting usually follows a adaptation and the credibility of qualitative chain that begins with projects and moves results. The key to successful reporting is en- through to programs, funders, and finally to gaging with stakeholders to determine their funders’ governance stakeholders. To stream- needs and expectations for reporting. line the performance information passing Reporting of systemic approaches to finan- along this chain, funders often want programs cial inclusion programming needs to be strong- to use common indicators so that key informa- ly knowledge-oriented. Measurement in these tion can be aggregated. programs is designed to inform funders and With financial inclusion, as with other inter- program units about whether the program is ventions in complex systems, there is limited on track and whether the track is the right scope for common indicators, particularly at one—and when and how to change course if it the lower levels of the ToC and results chains. appears not to be. With systemic approaches, This makes it difficult to aggregate information funders and other stakeholders have a strong gathered along the chain, and alternative ap- interest in this broader program perspective, proaches to accountability reporting may be which can be gained through reporting.   69 70  MEASURING MARKET DEVELOPMENT Integrating reporting into internal review interest to broad audiences, such as manag- ers and other staff of institutions that have Module 2, Figure M2-1, demonstrates how similar or complementary objectives, beyond reporting and knowledge sharing should flow a program’s immediate stakeholders. Formal directly from data analysis, review, and evalua- reports often are not the most engaging tion. Reporting consumes a lot of time and media for wider audiences. Audiences may be other resources. Reporting that is integrated more receptive to relevant narratives in other with review and evaluation is both more effi- formats, such as blogs, live broadcasting, cient and more effective than when it is a webinars, or face-to-face workshops. detached process designed solely for account- ability. Review and evaluation can transform data into knowledge, which can be harnessed Frequency of reporting in reporting. Knowledge-oriented reporting can lead to Frequency of reporting depends primarily on learning for all program stakeholders up and how useful the reports are to recipients—the down the reporting chain. Meaningful report- level and the nature of the interventions. Most ing to direct stakeholders can stimulate inter- systemic financial inclusion programs span a est in the program, promote more constructive long period, and substantive change at the engagement, and trigger valuable feedback intermediate level does not usually happen that can used to improve and adapt the pro- quickly or often. Many funders do not have the gram. Furthermore, well-presented, outcome- time to make sense of detailed quarterly or rich reporting can generate demand for more even semi-annual reports. An exception would such information, thus creating incentives to be if the program is operating in a high-risk further improve the measurement system. environment, such as a fragile or conflict- affected state, in which case the funder would have an interest and a duty to be kept informed Broader information and knowledge about the risk landscape. sharing Project staff usually report to program unit Figure M2-1 demonstrates that review and staff more frequently than program staff re- evaluation can produce knowledge that is of port to funders because the capacity for en- FIGURE M2-1 (repeated from Module 2). Measurement System Components and Cycle Conducive environment Learning among Carry out program stakeholders and diagnostic wider audiences Articulate the program ToC Consolidate, report, and share information and Develop program measurement Conducive environment Learn, improve, and adapt knowledge the projects and program strategy, including measurement questions, risk register, and results framework Conducive environment Analyze and review/ Articulate project results chains evaluate the evidence !"#$%&'()*)#('+"#,)#-* Develop project measurement Collect and manage data plans, including measurement on results and risks questions, risk register, and results frameworks Feedback loops MODULE 8   71 gagement and granularity of interest between Reporting against projections takes place the former is usually greater than that of the only when projections are scheduled. In sys- latter. Under average risk conditions, detailed temic financial inclusion programs where signif- project reports should be generated for pro- icant change takes time, projections are usually gram stakeholders once a year; less detailed not scheduled every year. Between projections, reports may be submitted mid-year. Apart progress is still reported against the interven- from the value to the recipients, semi-annual tions and outcomes in the results framework. reporting should create incentives for project These reports are guided by, but are not neces- staff to conduct in-depth reviews, but only if sarily limited in scope to, the indicators. the reviews are integrated with reporting—that At the program level, indicators are often is, timed to add value to the reports. broad and lack context. Information from the Reporting from program staff to funders projects in program reports should illustrate could follow the same timetable as reporting and expand on the basic indicator data. The of projects to program unit staff. Preferably, results framework is the principal perspective the timing of the reporting will allow for proj- on intended results, but reporting, especially ect reporting to inform program reporting. for systemic programs, is much broader. Re- However, funders that do not have the capac- porting should be complemented by a broad ity to peruse two reports a year should not narrative that provides a full picture of what ask for them. Funders may find it more useful has happened in the program domain—both to engage with program staff and project intended and unintended changes—and what staff between annual reports through face-to- have been the main drivers of and inhibitors of face or virtual meetings that involve a less for- these changes—whether in the program’s mal dialogue about what is working, what sphere of influence or outside of it. Although isn’t, and what lessons can be drawn. reporting does not require the full rigor of an evaluation, it should include credible evidence of the program’s contribution to change. In THE SCOPE AND CONTENT OF constructing this narrative, it is important to REPORTS refer to the assumptions about the external en- vironment—whether they have proved to be Module 4 shows how the different information reliable, or whether risks have materialized and and knowledge needs and interests of a pro- how the program responded. gram’s stakeholders can be expressed through Reporting is not just about the past; it measurement questions. These questions not should also look to the future. It should include only help to shape the measurement con- information about lessons learned that have a ducted during a program, they also indicate bearing on the current program and concrete the scope and focus of reporting to these proposals for adaptation, including revisions to stakeholders. measurement and risk frameworks, based on The core of a project or program report in- these lessons. variably focuses on the results framework. This It is difficult—even risky—to prescribe a par- applies irrespective of the frequency of the re- ticular format for reporting. The form and con- port, although reports that are more frequent tent of reports should be designed around than annual usually focus mainly or exclusively stakeholder needs and interests. Some contexts on interventions rather than outcomes. The in- demand more structure than others. Neverthe- terventions and outcomes in the results frame- less, an example of a template for a program- work, nuanced and calibrated by indicators to-funder annual report is given in Appendix and projections, are what the project or pro- M8-A, with these reservations in mind. gram is intended to achieve. 72  MEASURING MARKET DEVELOPMENT KEY TAKEAWAYS 1. Reporting is fundamental to accountability, and it can also address stakeholders’ learn- ing needs and appetites. It is essential to identify what these needs are and to design reports to meet these needs. Measurement questions can be important guide posts for this process. 2. Reporting should be integrated into review cycles. 3. Significant changes do not happen quickly or frequently in systemic approach pro- grams, and the length of reporting cycles should take this into account. 4. The interventions and outcomes in the results framework, which are nuanced and cali- brated by indicators and projections, should be the focus of the reports. In addition, there should be reporting beyond the results framework, including reporting on unplanned outcomes and relevant external developments. 5. Reports should present a plausible narrative of how the program has contributed to change. 6. Reports should describe what lessons were learned and how they will be applied going forward. 7. Information and knowledge brought together for reporting can serve wider audiences, although they usually need to be reformulated and disseminated in more appropriate formats. Module 8. References and Resources Forum Syd. 2014. “Manual on Reporting.” New Philanthropy Capital, et al. 2012. “Principles Stockholm: Forum Syd. http://www.forumsyd. of Good Impact Reporting.” New Philanthropy org/PageFiles/5098/Manual%20on%20 Capital. http://www.thinknpc.org/publications/ Reporting.pdf?epslanguage=sv the-principles-of-good-impact-reporting-2/ Jones, Harry, and Enrique Mendizabal. 2010. Schacter, Mark. 2010. “The Art of the Performance “Strengthening Learning from Research and Dashboard.” Ottawa: Mark Schacter Consulting. Evaluation: Going with the Grain.” London: ODI. http://docs.wixstatic.com/ugd/dadb01_ https://www.odi.org/sites/odi.org.uk/files/ 67024a8b0ac5243ee0d4e8f668ca1e4a.pdf odi-assets/publications-opinion-files/6327.pdf MODULE 8 | APPENDIX   73 APPENDIX M8-A. Example of a program-to-funder annual report template This example assumes a coherent program similar to the example in modules 3 and 5, with an overall results framework that focuses on outcomes. In this example, it is assumed that the number of projects in the program is small enough—say under five—to enable digestible sum- mary reporting on each project. For programs with many projects, the project narratives may need to be integrated into a large and more granular summary section, although the project ratings could be retained. Even with just a few projects, some funders may prefer more granularity than is proposed in this template, for example, differentiated narrative reporting on each program outcome in the results framework or separate reporting on certain cross-cutting themes, such as gender equity or sustainability. Risks and/or assumptions from a ToC perceptive are covered in this template. However, com- prehensive risk management reports, including adjustments to the risk register, are normally made more frequently. Reporting should be accompanied by guidance on terminology, rating criteria for projects, and so forth. The template should specify the word-count range for each subsection. A. PROGRAM NARRATIVE Program staff may prefer to complete this section after completing sections B and C. Most significant areas of progress and challenge • With particular reference to the program ToC—including the defined assumptions and/or risks and contribution hypotheses—summarize progress and challenges with the program over the past year. • Identify the principal factors in program implementation that have helped and hindered progress. • Describe the most significant developments in the program’s external environment and the effects these have had on the program, both positive and negative. Program-facing lessons, implications, and adaptations Describe the main lessons learned in the past 12 months about internal and external factors, their implications for the program, and the principal adaptations to the program (i) that have already been made in the period and (ii) are proposed to be made in the next 12 months. Adaptations should include those to program design, resourcing, and management and to the ToC, risk register, learning questions, results framework, and other aspects of measurement. Wider insights Describe any lessons or other insights that you have drawn from your experience with the program and its environment in the past year that may have wider application. Suggest which stakeholders, groups, or institutions might be interested and how they might be engaged. 74  MEASURING MARKET DEVELOPMENT B. PROJECT PERFORMANCE This section is for reporting on project performance. Project 1. [title] Summary narrative of the project and its progress in the past year. RATING Provide an overall rating for the project as whole. Choose from one of the following categories: Progress significantly above expectations Progress with all the project’s interventions can be shown to be satisfactory and, for at least one, is above expectations. The lessons from this have been captured for wider dissemination. Progress broadly in line with expectations Progress with all the project’s interventions can be shown to be satisfactory. Progress moderately below expectations Progress with at least one of the project’s interventions cannot be shown to be satisfactory. Steps are being taken by the project to address this. Progress substantially below expectations The project is seriously underperforming. This has been escalated to the program management, which is addressing the issue with project management. A further report will be issued before the mid-point of the reporting year. Rating Rating narrative Summarize the reasons for the rating. Project 2, etc. [title]—Repeat for each project C. PROGRAM OUTCOME PROJECTIONS This section is for reporting on outcome projections at all levels in the program results frame- work. Reporting should be completed only where indicator projections were scheduled in the reporting year. Intermediate Outcome 1—[title] 1. Indicator data against projections Indicator 1 xxxx Baseline Interim projection Interim projection End-line projection (date) (date) (date) Planned Actual Indicator 2 xxxx Baseline Interim projection Interim projection End-line projection etc. (date) (date) (date) Planned Actual 2. Narrative For each indicator whose projections were scheduled for this year, provide a short com- mentary on “actual” data, explaining the difference, if any, between the projected and actual result. Repeat for each Intermediate Outcome MODULE 8 | APPENDIX   75 Systemic Change Outcome 1—[title] 1. Indicator data against projections Indicator 1 xxxx Baseline Interim projection Interim projection End-line projection (date) (date) (date) Planned Actual Indicator 2 xxxx Baseline Interim projection Interim projection End-line projection etc. (date) (date) (date) Planned Actual 2. Narrative For each indicator whose projections were scheduled for this year, provide a short com- mentary on “actual” data, explaining the difference, if any, between the projected and actual result. Repeat for each Systemic Change Outcome Inclusive Financial System Outcome 1—[title] 1. Indicator data against projections Indicator 1 xxxx Baseline Interim projection Interim projection End-line projection (date) (date) (date) Planned Actual Indicator 2 xxxx Baseline Interim projection Interim projection End-line projection etc. (date) (date) (date) Planned Actual 2. Narrative For each indicator whose projections were scheduled for this year, provide a short com- mentary on “actual” data, explaining the difference, if any, between the projected and actual result. Repeat for each Inclusive Financial System Outcome D. UNPLANNED OUTCOMES Describe any unplanned outcomes, positive or negative, for which you have evidence. Refer to specific projects where relevant. Cite the nature of the evidence. Describe the contributions of the program to these outcomes. M O D ULE 9 EVALUAT I O N THIS MODULE . . . • Examines the uses of external evaluation in systemic financial inclusion programs. • Explores the concept of impact evaluation in the context of these programs. • Proposes the most appropriate evaluation designs and methodologies. TIMING AND PURPOSES OF EXTERNAL ence did the program or project make? Do EVALUATION the changes appear to be sustainable? What are the implications for replicating Generally, the conventional timing and pur- this type of program or project? poses of evaluations are as follows: This basic perspective provides a broad over- • Mid-term evaluations are usually conducted view of the types of evaluation frequently to determine what can be improved and found in a wide variety of settings. However, adapted—How can we do things better? evaluating systemic financial inclusion pro- What do we need to do differently? grams requires a certain amount of reposi- tioning. • End-term evaluations are usually conducted External evaluation is expensive—monetari- for accountability purposes and to take ly, and perhaps more importantly, regarding advantage of strategic learning—Did the the time and effort spent by everyone involved, program or project achieve what it was including partners. To maximize efficiency, intended to? Was it value for money? What funders and program unit staff need to clearly lessons should we learn and share with oth- understand what accountability, attribution, ers for future interventions? and impact mean in their programs. They also • Ex-post evaluations are mainly conducted need to reconsider the evaluation and moni- to capture strategic learning—What differ- toring relationship.   77 78  MEASURING MARKET DEVELOPMENT Uses of evaluation range of measurement questions about multi- ple interventions. Examples are as follows: For formative learning and improvement Because financial inclusion programs usually • Is it possible for funder interventions to have take place in uncertain and dynamic environ- a significant impact on the development of ments, the need for learning and adaptation, market systems that promote financial inclu- both at operational and strategic levels, is more sion? If so, how? Under what circumstances? pronounced than in relatively well-understood • Does financial inclusion have a significant development interventions, such as some in impact on poverty reduction and welfare? If the health sector. Internal enhanced monitor- so, with which types of financial services ing and self-evaluation in financial inclusion and channels? In what circumstances? How? programs and projects can and should address areas that are traditionally addressed by exter- • Should the funder continue with a given line nal evaluation, particularly formative evalua- of intervention or invest its resources in tion. Internal staff are likely to understand the other ways? complexities of the program environment bet- These types of evaluations are major undertak- ter than external evaluators, and learning and ings that need to be planned strategically and its application are more likely to flow from an allocated considerable amounts of time and evaluative process in which staff are deeply other resources. involved. Furthermore, because of its cost in time and money, only one formative external For accountability evaluation is likely to be conducted in the life of Conventional use of evaluation for account- a program or project. Because enhanced mon- ability calls for a high degree of detachment— itoring takes place more frequently, it provides which can be provided more credibly by shorter feedback loops. outsiders. It often focuses on validating the External evaluators are still able to contrib- achievement of predefined results. ute to enhanced monitoring through a collabo- However, in systemic programs, predefined rative model of measurement in which internal results are only a part of a wider mosaic of reviewers and external evaluators work togeth- change that needs to be pieced together by an er. Although most evaluators will not have the evaluation. Attribution can be a contested con- same depth of knowledge of the intervention cept. Accountability in systemic programs and and its environment that internal staff have, projects needs to have a broad perspective they can offer an outsiders’ perspective by ask- (see Module 2). It should complement the re- ing questions and identifying patterns in the quirement to meet specific targets. responses. External evaluators can contribute External evaluation plays an important role to learning and improvement through internal in a redefined accountability model. It can help reviews and monitoring/evaluation hybrids to put together a broader set of evidence and such as developmental and real-time evalua- bring an outside perspective to judgments tion. Appendix M9-A explains the different about it. It contributes to learning in ways that types of evaluations and their relevance for enhanced monitoring cannot easily do. How- funders using a systemic approach. ever, funders should not rely exclusively on Regarding ToCs, which were presented in evaluations. They should engage with pro- Module 3, collaboration between internal mon- grams enough to know whether other key itoring and external evaluators is most relevant management processes are in place. They where the focus is on the intermediate out- should encourage and support enhanced mon- come level. Intermediate outcomes are often itoring that will contribute to the required qualitative and are more open to interpretation body of evidence. than results at other levels. Their assessment benefits from a collaborative approach. At this level, formative learning is a particularly im- APPROACHES TO IMPACT EVALUATION portant use of evaluation and can be effective- IN SYSTEMIC FINANCIAL INCLUSION ly leveraged through a collaborative model. PROGRAMS For strategic learning “Impact” is among the most commonly used A more pronounced and independent role for words in measurement, and yet it is interpreted external evaluators is often appropriate for in several different ways. OECD-DAC defines strategic learning at program, portfolio, or impact in a broad, inclusive manner as “posi- organization levels, to help answer a broad tive and negative, primary and secondary MODULE 9   79 long-term effects produced by a development The complex and long ToCs in systemic pro- intervention, directly or indirectly, intended grams often make it impractical to measure or unintended.” This definition distinguishes effects at the development outcome level impact from the narrower concept of effec- through evaluations that take place at the end tiveness in measurement by also referencing of a typical 4–5-year program. These measure- unintended and negative effects. Effectiveness ments could also be misleading. On the other is normally confined to outcomes that are pos- hand, the aim of systemic programs is to pro- itive and intended. Although the definition mote large-scale, sustainable change at the de- does not prescribe a specific design for impact velopment outcome level, through development evaluation, development stakeholders tend to of the financial system and greater financial in- use the term exclusively in reference to evalua- clusion. The implications for impact evaluation tions that use a control group or counterfac- of systemic programs are threefold: tual and randomized designs. • The most suitable design is likely to be The principal factors for approaches to im- theory-based evaluation (TBE), which is pact evaluation in systemic financial inclusion generally compared against experimental programs include the breadth and complexity impact evaluations.16 However, it is possible of the financial system, and the time it takes and often desirable to combine TBE with for change at this level to have a large-scale, quasi-experimental approaches, or even ran- sustainable impact on financial inclusion and, domized control trials (RCTs), to test specific in turn, on development outcomes. Stake- links at lower levels in the results chain. holders of programs that promote change in more than one part of the financial system at • TBE may focus more on impact at the inclu- the same time are likely to struggle to control sive financial system level, than on develop- for factors external to the interventions. Ran- ment outcomes. domization is not an option, and even nonran- • Researching the impact of financial inclu- domized quasi-experimental designs may be sion on poverty and well-being is best difficult to apply because of the lack of an ap- approached strategically through a combi- propriate control group or because statistical nation of initiatives, including multiple exper- approaches are hampered by the small num- imental or quasi-experimental evaluations ber of units (e.g., partner institutions) at the and large-scale longitudinal approaches. intermediate outcome level. Combined, these initiatives would be beyond It is important to establish how change hap- the scope of many programs because of the pens in a systemic approach. Although experi- significant resources involved. This points to mental designs may include data on how the need for funders and other actors to change happens (e.g., by incorporating client collaborate strategically to advance knowl- diaries) this is not usually a program’s primary edge in this area. purpose. Using experimental designs for specific links in the results chain Experimental methods are not the most appropriate evaluation tool to measure change at the market-system level. Experimental and quasi-experimental methods require a credible counterfactual. When the unit of evaluation is the entire market system, it is impossible to create a counterfactual because there is only one market system. Nevertheless, evaluators can use these methods to test specific links in results chains. They can tell us whether individual, relatively discrete, interventions contributed in the way we expected them to. Experimental methods are particularly helpful during pilot stages to confirm a hypothesis before scaling up. For example, if a project objective is to stimulate demand for a specific product (e.g., microinsurance), then evaluators may choose to use experimental methods to compare the effectiveness of different forms of outreach to potential customers. This evaluation would not reveal whether the program had a systemic impact on financial inclusion, but it could clarify whether specific interventions were leading to their intended intermediate outcomes. Because RCTs can be expensive, their utility should be carefully assessed. 80  MEASURING MARKET DEVELOPMENT KEY TAKEAWAYS • External evaluation is expensive and needs to be justified by robust use cases. At the proj- ect level, particularly for formative evaluation, there may be a case for using external evaluators in collaboration with internal staff. • TBE is the most appropriate methodology for systemic programs and projects in financial inclusion. Rigor in TBE depends on several approaches, including deeply probing the ToC and capturing data about other interventions and contextual factors to explore plausible alternative contributions to the identified changes. At the heart of TBE is the understand- ing that evaluators are looking for contribution. • Theory-based impact evaluation at the project and even program level should focus on the financial system level and financial inclusion. Measuring the impact of financial inclu- sion on development outcomes, such as increased economic well-being, needs to be done strategically through a combination of methodologies and preferably through collabora- tion between funders and other stakeholders. • Experimental and quasi-experimental evaluation designs are unlikely to be relevant to most areas of systemic financial inclusion programing. However, they can contribute to strategic research for testing the link between financial inclusion and development outcomes. Module 9. References and Resources Better Evaluation. n.d. “Contribution Analysis.” O’Sullivan, Fionn. 2016. “Impact Evaluations for Retrieved 17 October 2015. http://bettere Market Systems Programmes.” London: The valuation.org/plan/approach/contribution_ BEAM Exchange. https://beamexchange.org/ analysis guidance/evaluation-guidance/ Carter, Becky. 2012. “Theory-Based Evaluation Puri, Jyotsna. 2014. “Quasi-Experimental Methods Approach. Governance and Social Development for Impact Evaluations.” New Delhi: Asian Resource Centre.” London: DfID. http://r4d.dfid. Development Bank and International Initiative for gov.uk/Output/192576/ Impact Evaluation. http://www.betterevaluation. Collier, David. 2011. “Understanding Process org/en/resources/guide/quasi-experimental_ Tracing.” Political Science & Politics, 44, methods_for_impact_evaluations No. 4: 823–30. http://www.ukcds.org.uk/sites/ Ruffer, Tim, and Elise Wach. 2013. “Review of M4P default/files/uploads/Understanding-Pro- Evaluation Methods and Approaches.” London: cess-Tracing.pdf DfID. http://www.itad.com/reports/review-of- Dozois, Elizabeth, et al. 2010. “A Practitioner’s m4p-evaluation-methods-and-approaches/ Guide to Developmental Evaluation.” Scriven, Michael. 1991. “Prose and Cons about Montreal: J.W. McConnell Family Foundation Goal-Free Evaluation.” American Journal of and International Institute for Child Rights and Evaluation, 12: 55. https://thefutureofbusinessed- Development. http://vibrantcanada.ca/files/ ucation.wikispaces.com/file/view/Goal+Free+ development_evaluation_201_en.pdff Evaluation+Scriven.pdf Harrison, Tom. 2015. “RCT and Market Systems: Stern, Elliot, et al. 2012. “Broadening the Range of An Opportunity to Gain Insights.” Blog post, Designs and Methods for Impact Evaluations.” 22 January. http://beamexchange.org/en/ London: DfID. https://www.gov.uk/dfid-research- community/blogs/blog-entry/2015/1/22/ outputs/dfid-working-paper-38-broadening-the- tomharrison/ range-of-designs-and-methods-for-impact- Mayne, J. 2008. “Contribution Analysis: An evaluations Approach to Exploring Cause and Effect.” ILAC Brief 16. Rome: Institutional Learning and Note Change Initiative. http://www.betterevaluation. 16. See, e.g., O’Sullivan (2016). org/en/plan/approach/contribution_analysis and https://beamexchange.org/uploads/ filer_public/f5/55/f5553494-f9cf-4f2f- 90ee-105877dddd71/contribution-analysis.pdf MODULE 9 | APPENDIX   81 APPENDIX M9-A. Evaluation designs and methodologies The lexicon of evaluation methodology is Experimental and quasi-experimental de- dense and can be confusing. At the risk of signs in isolation tell us only whether an inter- over-simplification, it is possible to see a con- vention has had an effect, not how. They are vergence in recent years on two principal often described as “black box” studies—leaving types of evaluation, particularly where the the box containing the effect mechanism un- focus is impact: experimental and quasi- opened. In complex environments, it is im- experimental evaluation on the one hand, and portant to know how change happened or nonexperimental TBE on the other. In devel- why it did not. Without this knowledge, repli- opment, generally, these two evaluation types cation and scale-up are unlikely, because con- have the same goal. In many areas of market text and process are important. Even if these systems development, TBE is considered to evaluation designs met the controlled envi- be the most appropriate approach to external ronment criterion, they would need to be evaluation (Ruffer and Wach 2011). accompanied by qualitative assessment of process and a consideration of other influenc- ing factors, which will always be unique to the Experimental and quasi-experimental intervention’s immediate environment. designs As such, experimental evaluation is unlikely Experimental and quasi-experimental evalua- to be relevant to most areas of systemic finan- tion seek to identify the effects of the interven- cial inclusion programming. It can, however, tion by isolating the interventions from other contribute to strategic research for testing factors. It uses focused baseline and end-line the link between financial inclusion and devel- definitions of the status of what is being evalu- opment outcomes. ated, and it uses data in the same way data are used in a counterfactual or control group. In Theory-based evaluation some designs—particularly RCTs—the control group at baseline needs to be a close (ideally In sharp contrast, TBE is well-suited for sys- identical) analogue of the “treatment” group temic programs. At its simplest, TBE follows (i.e., the intended beneficiaries of the interven- the path of the ToC or results chain from the tion). In RCTs, the control group is drawn from interventions through the various levels of the same population as the group that receives expected outcomes, gathering evidence as to the treatment. In other designs, differences are whether the changes at each level have, as permitted between the treatment and control hypothesized, been influenced by the inter- groups, but these differences need to be vention and whether the supporting assump- clearly identified. tions have held up. TBEs will be able to test These conditions are difficult to achieve in the ToC or results chain. Some forms of TBEs systemic programs and projects because these are able to estimate the strength of the inter- programs and projects have long ToCs and vention’s influence and that of other contrib- results chains (respectively), their environ- uting players and factors. ments are complex and dynamic, and their TBE depends on a ToC or results chain. In boundaries are blurred. Depending on the adaptive programs like those typically found state of development of the financial system, it in financial market development, evaluators may take years before the effects of the inter- should ensure that the ToC or results chain is ventions on customers and their enterprises up to date. TBEs can be used not only to as- can be detected. Systemic programs do not sess whether expected results have occurred, provide the controlled environment that they also look at how they occurred or why experimental, or even quasi-experimental, they did not. designs require. It is often difficult to find a Appropriate rigor needs to be applied in true counterfactual in the conventional sense TBEs through several actions: of the word. Furthermore, the necessarily • Having a robust ToC or results chain that adaptive, opportunistic nature of systemic has been tested and adapted, where neces- financial inclusion interventions often dimin- sary, during the intervention. ishes the role of original baselines because the focus of the intervention shifts in response • Defining measurement questions at the to obstacles or new opportunities. beginning of the program or project to enable monitoring to maximize its contribu- tion to the evaluation. 82  MEASURING MARKET DEVELOPMENT • Identifying baselines for key anticipated Strict causality, beyond the most direct of in- changes. termediate outcomes, is difficult if not impos- sible to prove. • Capturing and assessing data about antici- pated and unanticipated outcomes. Process tracing is similar to contribution analy- • Capturing data from a sufficient number of sis in that it analyses whether results are con- different sources, both quantitatively and sistent with the program theory and whether qualitatively, to enable “triangulation” and alternative explanations can be ruled out. It use of credible strategic informants. does this by applying a highly structured methodology that involves the systematic • Establishing plausible connections between examination of “diagnostic evidence selected the interventions and the identified changes. and analysed in the light of research ques- • Capturing data about other interventions tions and hypotheses posed by the investiga- and contextual factors (e.g., economic and tor” (Collier 2011). political) to explore other possible contribu- tions to the identified changes. Realistic evaluation also takes a similar approach to contribution analysis, but it places even more Contribution analysis, process tracing, and emphasis on context. The following evaluation realistic evaluation are three prominent TBE questions would be asked in combination: methodologies. What works? For whom? In what circum- stances? In what respects? How? A ToC in real- Contribution analysis, like other forms of TBE, istic evaluation is highly context-specific and is does not set out to prove causality but rather usually seen to have weak external validity. to provide “evidence and a line of reasoning Replication would require the ToC to be heavily from which a plausible conclusion can be recontextualized and adjusted. drawn that, within some level of confidence, the program has made an important contribu- Inward–outward perspectives in TBE tion to the documented results” (Better Evalu- ation n.d.). Conventional effectiveness-focused evaluation Appropriate evaluation questions may in- largely follows the ToC from the intervention clude the following: upwards to the highest level of evaluation interest (e.g., advances in financial inclusion). • Was the intervention and its pre-identified This outward approach is also the perspective supporting factors and assumptions suffi- of conventional approaches to monitoring. cient to produce the intended result? However, explicit in TBE, and particularly in • What role did the intervention have in this? contribution analysis, is that an intervention- How did its contribution compare to those of centric perspective needs to be set aside at other factors, both pre- and post-identified? some point to examine the possibility of other explanations for the changes discovered. In • How and why did the intervention make—or this inward approach, measurement is not led not make—a difference? by pre-set outcomes, it looks for changes that Implicit in this approach is that the evaluator have taken place in the program environment should look beyond the intervention for contri- and works backwards to identify the plausible butions to the changes discovered. Before contributions to the changes, including the in- concluding that the intervention played a cen- tervention in question. tral role, the evaluator needs to look at other Evaluation that includes inward perspectives plausible explanations for the changes. allows for greater triangulation and reduces Also implicit is that evaluators are looking the intervention centricity that can distort the for plausible attribution, or in some cases sole findings and conclusions of effectiveness- contribution, rather than strict causality. Most oriented evaluation. These approaches cap- financial inclusion interventions attempt to in- ture information on outcomes in the project fluence a system that has many interdepen- or program terrain, but without reference at dent parts because numerous nonprogram first to the intervention itself. The inward per- factors contribute to change. Change often spective can be applied in evaluation and in happens episodically, after periods of inac- enhanced monitoring. tion. This makes it difficult to isolate the There are advantages and disadvantages in change the program has caused. Evaluation both inward and outward approaches to mea- expectations need to be set so that plausible surement. A combination of these approaches attribution or contribution can be discovered. is most likely to address the inherent draw- MODULE 9 | APPENDIX   83 backs of applying either approach on its own, which directions hold promise and which although this implies requiring more resourc- ought to be abandoned; and suggests what es. Figure M4-1 and Table M4-1 in Module 4 il- new experiments should be tried” (Dozois et lustrate measurement questions that could be al. 2010). applied in this way. Although there is a range of methodologi- cal approaches to real-time and developmen- tal evaluation, there are some commonalities: Developmental methodologies • They take place during implementation. In this handbook, the term “developmental methodologies” refers to a range of approaches • They take place iteratively rather than as a that challenge the conventional separation of one-off, and they are integrated into the monitoring from evaluation. Developmental program cycle. methodologies can work well with the collabo- • The emphasis is on immediate lesson- rative model described earlier in this module. learning more than on impact assessment The key principle underlying these develop- or accountability. mental methodologies is that they are intended • They aim to produce context-specific to have an immediate impact on programming. understandings that inform ongoing inno- The evaluator’s primary role is to bring evalua- vation rather than generalizable findings tive thinking and the use of data into program across time and space. and project management on a regular, or even continuous, basis. The most common exam- With these methodologies, evaluation ques- ples of these approaches are real-time and tions and intended results should be allowed developmental evaluation. to evolve as evaluation progresses. They Real-time evaluation originated in the hu- remain theory-based. manitarian sector; the Office of the United Nations High Commissioner for Refugee insti- Goal-free methodologies tutionalized it in 2000. Emergency relief is arguably the most complex, dynamic, pro- The term “goal-free evaluation” applies to a gramming environment. Real-time evaluation range of methodologies in which the evaluator in these situations emphasizes the use of con- conducts the evaluation without particular stant feedback loops to improve processes. knowledge of, or reference to, predetermined Real-time evaluation has been applied more results. The emphasis is on “finding out what recently in nonhumanitarian contexts. the program is actually doing without being Like its real-time equivalent, developmen- cued as to what it is trying to do. If the program tal evaluation is designed for complex, dy- is achieving its stated goals and objectives, namic environments, which makes it suitable then these achievements should show up; if not, for much of financial inclusion programming, it is argued, they are irrelevant” (Scriven 1991). particularly for measurement at the lower lev- “Outcome harvesting” is a recent variation els of project results chains. It fits neatly into of the goal-free approach. Evaluators or proj- a collaborative model because it comple- ect stakeholders collect evidence of what has ments enhanced monitoring. It requires mea- changed, and work backwards to determine surement itself to be flexible so that the focus whether and how the project or intervention and mechanisms for measurement can contributed to the change. Goal-free ap- change as intended outcomes evolve or shift. proaches in financial inclusion should comple- “Developmental evaluation facilitates as- ment outward perspectives in both monitoring sessments of where things are and reveals and evaluation; it should not be a stand-alone how things are unfolding; helps to discern evaluation methodology. M O D ULE 10 PORTFO L I O - BASED ME AS U R E ME N T THIS MODULE . . . • Explains how portfolio-based measurement, using standardized indicators and/ or scoring systems, can produce insights for funders to improve their interven- tions and make decisions about their overall strategy. • Presents two portfolio measurement tools: portfolio dashboards, which usually are applied at least once a year, and portfolio reviews, which take place less frequently. For funders and other organizations that THE IMPORTANCE OF A PORTFOLIO implement multiple programs, taking a portfo- APPROACH TO MEASUREMENT lio approach to measurement provides insights and helps report results to constituencies Unlike individual program evaluations from across the whole portfolio or across specific which lessons may be difficult to extrapolate subsets of programs.17 Comparing performance beyond the program, portfolio-based monitor- across systemic programs is challenging. Con- ing and review compares results across all or text is important, and progress is often mea- substantial parts of a portfolio. Portfolio sured in terms of changed practices, processes, reviews are used to identify reasons for varia- and structures—making considerable use of tions in results. These evaluative reviews can qualitative indicators that do not lend them- produce insights that are broadly valid and selves to direct comparison or aggregation. that help funders make evidence-based deci- Nevertheless, at higher levels in the ToC, there sions. Portfolio reviews also fulfill an overarch- is scope for direct comparability through stan- ing accountability function. They can be used dardized indicators. In addition, well-designed to assess whether funders are achieving their and consistently applied methods of indirect objectives, whether they are achieving results comparison can be used. efficiently, and to the extent possible, whether these results are relevant in their sectoral and market contexts.   85 86  MEASURING MARKET DEVELOPMENT TOOLS FOR PORTFOLIO-BASED Dashboards typically include the following: EVALUATIONS AND MONITORING • Program categories (e.g., by region, type of program, type of partner) Portfolio reviews and dashboards are two complementary tools funders and program • Disbursements units can use to aggregate results at the port- • Values of standardized indicators folio level. Portfolio reviews of financial inclusion are • Performance against projections (see Mod- thorough assessments of completed or well- ule 6) advanced programs that are done by indepen- • Timeline tracking dent evaluators and are repeated every three • Overall performance ratings (based. for ex- to five years. They can be used to assess ample, on a red/amber/green “traffic light” whether a funder is delivering on its financial rating scale) inclusion strategy. Portfolio reviews typically assess the com- • Risk ratings position and the performance of the portfolio. • Short summaries of progress and of issues They include five phases: preparation, portfolio that arise snapshot, performance scoring, stakeholder consultation (including field visits as much as Dashboards are usually created by program or possible), and overall analysis and reporting project staff. They should not require much, if (Figure M10-1).18 A portfolio review can take up any, additional research beyond existing moni- to a year from initiation to the submission of toring. Dashboards should incorporate as the final report to complete. many visual elements as possible, for example, Portfolio dashboards are designed to regu- color-coding to highlight high or low perform- larly inform boards and constituencies about ers and important changes made since the overall portfolio progress, and to help senior previous reporting period. management identify programs that need at- Because dashboards rely on quantification, tention and to react appropriately. They can scoring through categorization, and monitor- indicate whether current programs are “on ing against predicted timelines, it may be diffi- track.” Deeper scrutiny can be triggered if the cult to illustrate standardized indicators at the dashboard indicates that a program appears intermediate outcome level because compari- to be diverging from expectations. Some sons of program progress need to be put in dashboards incorporate a facility to automat- context. Change often happens episodically, ically “drill down” to greater program—or even so judging progress against predicted time- project-specific—data. lines can be misleading. Dashboards should be Because dashboards are meant to be read viewed as an entry point for deeper analysis. and understood quickly and easily, they are usually graphic presentations of data. Dash- boards that illustrate program aspects that are STANDARDIZATION subject to change within a short timeframe are often revised more than once a year. Standardization is a controversial concept in market systems development because the sys- temic approach requires flexibility and context- FIGURE M10-1. Portfolio review components specificity throughout the program. However, the ability to standardize and aggregate per- 1. Preparation Assess the composition of the formance information is necessary for both portfolio: Is the funder investing dashboards and reviews. Standardization and 2. Portfolio snapshot in the right programs? aggregation are required for a holistic per- 3. Perfomance scoring spective of the portfolio, and to enable com- + parisons. They do not need to undermine 4. Stakeholders Assess program performance flexibility in implementation. consultation/ throughout the portfolio: Have The focus should be on the middle levels of field visits these programs performed? the ToC (as defined in Module 3)—later inter- Have they generated systemic mediate outcomes, systemic change, and the change? Has the funder and/or implementing agency been inclusive financial system—to enable standard- 5. Overall analysis and reporting e cient in implementing ization in portfolio-based measurement. Devel- programs? opment outcomes require different evaluation instruments, as discussed in Module 9. MODULE 10   87 Experience with portfolio reviews Portfolio reviews to help revisit strategic priori- Portfolio reviews to evaluate strategy implemen- ties—Agence Française de Développement (AFD) tation—African Development Bank (AfDB) A six-month portfolio review of AFD’s microfinance In 2013, AfDB leaders decided to evaluate its micro- operations was conducted by independent consul- finance policy, strategy, and operations from 2000 tants in 2010–2011. The review was intended to pro- to 2012. There was no automated system in place vide, for the first time, an external and critical that allowed for the extraction and consolidation of viewpoint on the performance of AFD’s microfi- information about all financial inclusion projects. nance projects. The review involved the use of a pre- The evaluation included a thorough portfolio review. determined methodology to score a representative In this review, microfinance projects were identified, sample of active projects and was based on docu- performance data were collected, and the projects ments and field visits in a selected number of coun- were assessed through standardized indicators. The tries. The operational diagnostic made it possible to review identified several discrepancies between the identify several strategic priorities, based both on strategy and the operations. For example, whereas market needs and AFD’s strength and added value. the strategy specified private-sector department The findings contributed to developing the defini- leadership for financial inclusion operations, most of tion of AFD’s financial inclusion strategy. the projects were still channeled through govern- ments. AfDB used the findings to update its strategy. At these middle levels, the tension between are made for changes in program manage- standardization and flexibility can be recon- ment and strategy. ciled through “smart standardization” that ac- Whereas the snapshot phase focuses on objec- counts for context and does not require tive and quantifiable information, the scoring projections to be set in stone from the onset of and analysis phases include contextual and the program. Smart standardization can be other qualitative information and subjective achieved through a mix of aggregation of stan- assessment. dardized quantitative indicators (A in Figure M10-2) and subjective scoring, which reflect Design. In the design phase, the evaluator (or context (B). the program managers in the case of portfolio Although there is no standard methodology dashboards) and the funder (senior manage- for results aggregation, funders can use the ment) work together to adapt the evaluation following guidance and generic set of scoring framework to the funder’s information needs factors to develop a methodology that meets and determine the scope of the review. A cen- their specific needs and context and that ac- tral element of this phase is agreement on the counts for data, time, and budget constraints. evaluation questions to be answered. The over- The following is a four-phase approach to ag- arching question a portfolio review is designed gregate performance information while ac- to answer is whether the funder delivers on its counting for context: financial inclusion strategy (i.e., is the funder 1. Design—objectives are set and tools are investing in the right programs and projects developed. and is it achieving its objectives?). Funders may have other questions that portfolio-based 2. Snapshot—portfolio composition and per- measurement can be used to answer (e.g., formance data are collected, and where about specific links in a portfolio-wide ToC). possible, aggregated using standardized Like all measurement activities, the design and indicators. use of portfolio reviews and dashboards should 3. Scoring—program performance is rated be guided by measurement questions from the using standardized scoring criteria, taking funder’s measurement strategy (see Module 4). into account other monitoring data and context, and then aggregated. Snapshot. The evaluators collect information on programs and consolidate this information. 4. Analysis—performance is compared, pat- The objective is to get a good understanding terns are identified, and recommendations of the composition of the funder’s portfolio 88  MEASURING MARKET DEVELOPMENT FIGURE M10-2. Smart standardization for performance aggregation Specific performance indicators Program M&E How to aggregate information across programs? Standardized performance indicators ➔ Using standardized indicators, and compiling data (Box A) ➔ Creating a scoring system, and compiling scores (Box B) Specific performance indicators Program M&E Standardized performance indicators A. Snapshot: aggregated Scoring B. Aggregated performance data system performance scoring Specific performance indicators Information on contexts Program M&E Standardized performance indicators Note: M&E = monitoring and evaluation and its performance. What are the characteris- Committee evaluation criteria: tics of the programs? What were their initial • Relevance and coherence. Are programs objectives in terms of intermediate and market that are aligned with a portfolio-wide ToC system outcomes? Who were the funder’s designed to promote financial system devel- partners? How have funds been used? Have opment? Are they relevant to country needs program objectives been achieved? and coherent with country long-term strate- Table M10-1 summarizes the types of data gies? needed for the snapshot. Consolidating these data enables a thorough analysis of the com- • Effectiveness. Have objectives been achiev- position of the portfolio and helps gather key ed? or How likely are they to be achieved? information for program scoring. Data includ- • Efficiency. Are program interventions com- ed in the snapshot should be transparent and mensurate with allocated resources? objective. Funders should make sure their own and their partners’ data management systems • Impact. What is the nature and scale of can automatically generate what is needed change at the relevant levels of the ToC to (program documents should be structured in a which the programs have contributed? way that guarantees the necessary information is collected). • Sustainability. Are programs’ positive out- comes being sustained or are likely to be Scoring. In this phase, evaluators (i.e., program sustained? managers in the case of portfolio dashboards) Criteria that assess the extent to which a pro- score the performance of programs and con- gram met a certain objective, rather than solidate scores across the portfolio. Using granting scores on a binary (i.e., yes or no) scoring criteria makes it possible to include basis, should be used in the scoring grid. Eval- results that are not measurable through quan- uators should use their judgment in scoring titative indicators. Using a well-defined scoring and incorporate contextual factors. methodology reduces potential bias in the The scoring process should follow clear assessment process. guidelines to allow for flexibility and to ensure Evaluators score all programs, based on the the comparison is relevant.19 Guidelines may data collected during the snapshot phase and address the following: on information from key informant interviews and field visits. The scoring grid example in • A scoring scale that explains the meaning of Appendix M10-A can be used as a starting each score should be designed (see Figure point for funders to adapt and develop their M10-3). own grids. This example is based on an adap- • Qualified evaluators should be selected and tation of OECD’s Development Assistance trained. Evaluators are necessarily a key ele- MODULE 10   89 TABLE M10-1. Data for the portfolio snapshot DESCRIPTION Performance against Program Program unit Program partners results frameworks Information about the Information on who is imple- Information on the partners (e.g., For private-sector entities, there program. The measurement menting the program. Examples: regulators, FSPs, sector coordina- should also be information on out- system should collect this funder staff, consultancy firm, tion facilities, membership organi- reach (size and type), size (number information from programs local implementer, government zations, universities, statistics of employees, etc.), and financial in a standardized way. unit. bureau). Examples: type of situation. Information on the levels of Examples: region or country, institution, years of operation, achievement of the program’s key dates, funding amount, range of services. For private- expected outcomes, normally at the funding instruments, type sector entities, there should also intermediate, systemic change, and of program. be information on outreach (size inclusive financial systems levels.a and type), size (number of Standardized indicators, if available, employees, etc.), and financial are used so that performance can be situation. aggregated. Data normally come from partners’ own monitoring systems. ntermediate outcomes focus on changes in attitudes, behaviors, institutional processes, etc., that are difficult to capture with standardized indicators and, a. I therefore, are unlikely to be included in the snapshot phase. Data on outcomes at this level should be introduced selectively during the scoring phase. FIGURE M10-3. Sample scoring scale FIGURE M10-4. Portfolio review—Evaluations profiles 6 Exceptionally successful. The operational responsibility for a portfolio review should be with an independent team of 5 Very good. Worth extending or replicating. evaluators. Independence is important not only 4 Satisfactory/all right. Good for this specific situation, but not to achieve accountability, but also to gain an worth replicating/extending. external view on the relevance and performance 3 Weak but minimally acceptable. The evaluator wouldn’t of programs. support such a project, but wouldn’t veto it or stop it. The evaluation team needs to combine evaluation 2 Unacceptable. Unlikely to produce a decent development skills with expertise in financial inclusion, a good return on the funds invested. The evaluator would veto such knowledge of the market system development a project. approach, and, last but far from least, a good knowledge of the countries where the funder 1 Exceptionally poor. Probably does serious damage, or operates. represents extreme bad practice. ment of the “smart standardization” pro- previous phases to answer the measurement cess. Accounting for context relies on their questions defined for the portfolio review. knowledge of the context and their analyti- They combine findings and analyze the com- cal skills (see Figure M10-4). piled data for trends and correlations to extract lessons on what works, what does not, and • The scoring process should be transparent. why. It is important at this stage to triangulate Evaluators should list available evidence and findings with country-level market data and to justify their scores. In the case of large port- verify them with key stakeholders. The evalua- folios, to save time, the evaluators might tors should develop recommendations for the focus on justifying the lowest and highest funder based on these findings. scores. To test whether the scoring system is The findings and recommendations should used consistently across different evalua- be consolidated in the final report. It should be tors, evaluators could score a sample of a coherent standalone product that a reader projects and check whether the scores and can understand without having to refer to in- justifications are consistent. For quality termediary reports and other documents pro- assurance, to the extent possible, all proj- duced during the review process. It should be ects should be scored independently by two concise and draw the reader’s attention to the evaluators before a final score is assigned. main findings the evaluators consider import- Analysis and action. At this stage, sufficient ant and realistic for the funder to act on. The information should be available for analysis. reports are usually shared with key stakehold- Evaluators need to extract findings that will ers. By contrast, portfolio dashboards are usu- help the funder achieve better results with its ally shared internally. portfolio. The evaluators use data collected in 90  MEASURING MARKET DEVELOPMENT KEY TAKEAWAYS • Portfolio-based monitoring and evaluation, based on aggregated information on pro- grams and their performance, are useful for both learning and accountability purposes. • Portfolio reviews and dashboards are two examples of results measurements at the portfolio level. Dashboards aim to detect whether or not existing programs are going in the right direction. Reviews aim to assess whether the programs are delivering on funder strategies, how, or why not. • Like all measurement activities, the design and use of portfolio reviews and dashboards should be guided by measurement questions from the funder’s measurement strategy. • Some degree of data standardization of program composition and performance indica- tors is necessary to enable aggregation at the portfolio level. But there are limits to the practicability and validity of standardized indicators in systemic financial inclusion pro- grams, particularly at the intermediate outcome level. Portfolio reviews need to com- bine standardized indicators with scoring, which, while systematic and based on clear criteria, allows evaluators to use their judgment based on contextual factors and non- standardized performance data. Module 10. References and Resources Biscaye, Pierre, et al. 2015. “Evaluating Donor-Level OECD (Organization for Economic Co-operation Results Measurement Systems.” Seattle: Bill & and Development). 1991. “The DAC Principles Melinda Gates Foundation. https://evans.uw. for the Evaluation of Development Assistance.” edu/sites/default/files/EPAR_UW_300_ Paris: OECD. http://www.oecd.org/dac/ Donor%20Results%20Measurement%20 evaluation/daccriteriaforevaluatingdevelopment Systems_08.21.15_0.pdf assistance.htm Burjorjee, Deena M., and Barbara Scola. 2015. “A Smart Campaign. “Smart Campaign Client Market Systems Approach to Financial Inclusion: Protection Principles.” Washington, D.C.: Smart Guidelines for Funders.” Washington D.C.: Campaign. http://www.smartcampaign.org/ CGAP. https://www.cgap.org/sites/default/files/ about/smart-microfinance-and-the-client- Consensus-Guidelines-A-Market-Systems- protection-principles Approach-to%20Financial-Inclusion-Sept-2015_ UNPRI (United Nations-Supported Principles for 0.pdf Responsible Investment). n.d. “Principles for Development Gateway. 2016. “What Systems and Responsible Investment.” London: UNPRI. Processes Do Donors Use for Measuring Aid https://www.unpri.org/about Effectiveness?” Blog post, 25 March. http://www. developmentgateway.org/blog/what-systems- Notes and-processes-do-donors-use-measuring- This module builds on Scola and Nègre (2012). 17. aid-effectiveness See Scola and Nègre (2012) for detailed 18. Negre, Alice, and Barbara Scola. 2012. “Portfolio guidance for each of these phases. Reviews: Resource Guide for Funders.” Washington D.C.: CGAP. http://www.cgap.org/ Scola and Nègre (2012) provide additional 19. publications/portfolio-reviews-resource-guide- guidance on how to define a scoring grid and funders processes. MODULE 10 | APPENDIX   91 APPENDIX M10-A. Scoring grid example RELEVANCE AND COHERENCE Scoring Criteria Do programs follow an appropriate ToC? R1  Extent to which the program, based on a portfolio-wide ToC, meets a set of effective- ness criteria (see Module 3). Are program objectives consistent with the R2 Extent to which the program results framework is aligned with objectives in the funder’s overall strategy or ToC? funder’s strategy and ToC. R3 Extent to which the program complements the funder’s other programs, especially in the same country. Are the programs in a given country aligned R4 Extent to which the objectives of programs in a given country are aligned with or with national strategies to promote financial complementary to the objectives in national strategies for financial inclusion where inclusion? they exist. R5 Extent to which the program design intentionally addresses diagnosed market system development needs. Are the programs in a given country aligned R6 Extent to which the program is based on clear and documented analysis of the with the efforts of other funders and non- funder’s value-added, taking into account the funder’s comparative advantage and government agencies? other agencies’ contributions. Are programs following internationally R7 Extent to which the program is in line with internationally recognized best practices recognized best practices? (see Burjorjee and Scola [2015]); Principles for Investors in Inclusive Finance; Principles for Responsible Investment 20 Are program designs relevant to meet their R8 Degree of consistency between program objectives and partners’ needs, strategy, intended objectives?  and capacity to implement the program and extent to which consultation ensured effective ownership by recipients. R9 Extent to which programs included a clear crowding-in strategy beyond the program partners. R10 Extent to which instruments, funding amounts, and planned timeframe were consistent with the program objectives and provided the necessary flexibility. EFFECTIVENESS Scoring Factors Are measurement systems fit to track the ES1 Extent to which outcomes were clearly specified, at all levels of the ToC, with relevant effectiveness of the programs? indicators, baseline values, projections, and measurement sources. ES2 Extent to which the program has established a measurement system that allows for  enhanced monitoring of unintended, positive, and negative outcomes, as well as intended outcomes. ES3 Extent to which performance data are sufficient in all relevant categories, quality assured, and available for comparison across programs. Are planned outcomes up to and including ES4 Extent to which intermediate outcomes have been or are being achieved. the systemic change level achieved or likely ES5 Extent to which systemic change outcomes have been or are being achieved. to be achieved? ES6 In case of loans or refinancing, extent to which the recipient has repaid or is repaying the installments on time. EFFICIENCY Scoring Factors Is the funder enabling partners and third- EY1 Extent to which the program and its projects were designed, negotiated, and finalized party implementers to deliver on their within reasonable time and/or agreed timeframes. commitments in a timely and efficient EY2 Extent to which financing was provided in line with activities or other financing manner? needed or extent to which disbursement took place according to plan. EY3 Extent to which the funder has an accountability regime that matches criteria for effectiveness in the context of market systems development. Are contracting systems designed to EY3 Extent to which the funder has used transparent, competitive procurement processes ensure efficiency?  for contracting resources to implement the program (e.g., consultants, services providers, etc.). EY4 Extent to which the funder’s contracting (e.g., performance-based contracts) and accountability mechanisms incentivize high performance. Have program interventions been delivered EY5 The extent to which planned interventions have been delivered according to planned according to plan? factors: timing, scale, specification, quality. Have program intervention results been EY6 Extent to which resources used (e.g., money, staff, and materials) are commensurate with the resources used? commensurate with the interventions. 92  MEASURING MARKET DEVELOPMENT APPENDIX M10-A. Scoring grid example, continued RELEVANCE AND COHERENCE Scoring Criteria IMPACT Scoring Factors What are the impacts of the programs on I1 Extent to which the program has had a positive impact on partners’ provision of inclusive financial system development and services, regulation, etc. (where relevant) on development outcomes? I2 Extent to which the program has had a positive impact on the range and the scale of inclusive financial services. I3 Extent to which the program has had a positive impact on the financial capability of individuals, households, and enterprises. I4 Extent to which the program has had a positive impact on market infrastructure and supporting functions. I5 Extent to which the program has had a positive impact on rules and norms. I6 Extent to which the program has had a positive impact on the uptake and use of financial services by poor and low-income people. I7 Extent to which the program has contributed to positive development outcomes. I8 Extent to which the program has contributed to negative displacement effects and other negative outcomes, socially, economically, or environmentally. SUSTAINABILITY Scoring Factors Are there clear exit strategies for the S1 Extent to which exit options were assessed and defined in program design and funders’ support? subsequent work plans, and carried out. Are partners able to sustain their operations S2 Extent to which partners have increased their financial sustainability over the funding without continued funding? period, and have the financial capacity to address recurrent costs. S3 Extent to which partners have enhanced capacity to innovate and withstand shocks beyond the boundaries of program interventions. Has market sustainability and resilience been S4 Extent to which the program has influenced systemic change and sustainable change strengthened by the programs? in the inclusive financial system. Note  ttps://www.unpri.org/about & http://www.smartcampaign.org/about/smart-microfinance-and-the-client- 20. h protection-principles G LOSSA RY Adaptive management Purposeful experimentation to answer questions that are critical to the achievement of results; includes pivots in strategy and tactics based on feedback loops coming from the results of detailed, planned experiments or pilots. Attribution The degree to which an outcome can be shown to be caused by one or more interventions. Catalyzing Increasing the rate of a reaction through the participation of an additional substance: a catalyst. With a catalyst, reactions occur faster and uses less energy. The catalyst itself does not become part of the change. Contribution The demonstration of a plausible link between an outcome and one or more interventions, but without fully isolating the effect of the intervention(s) from other factors. Contribution hypothesis An explanation of the logic behind the progression from one level of a theory of change or results chain to the next. Development domain A sector, subsector, or broad focus area, such as financial inclusion, for development interventions. Facilitation Working with system actors to catalyze the desired systemic change process. Financial inclusion A state in which all individuals, households, and businesses have the choice to access, and the ability to use, a range of appropriate financial services, responsibly and sustainably provided by institutions enabled to offer such services. Financial system An umbrella term that includes a market system for different financial services (e.g., payments, savings, credit, insurance, leasing, Sharia-compliant financial products, etc.). Funder Public or private organizations that support financial inclusion to achieve a developmental mission or mandate, including bilateral and multilateral development agencies, private foundations, and development finance institutions. Funders typically fund the programs; they may also act as an implementer. Funders are not considered market actors because they only temporarily intervene in market systems. Impact Positive and negative, primary and secondary, medium- to long-term, intended or unintended effects produced by interventions. Innovation New practice or approach adopted by a market player. May include new goods, services, supporting functions, rules, and norms produced as a result of the new practice or approach. Intervention One or more activities, carried out by an implementing organization, that involve interac- tions with and including products, services, or funding delivered to, targeted groups or institutions. Market system The interaction of multiple market actors performing multiple market functions, including a core function (such as the demand and supply of financial services), supporting functions, and rules and norms. Broadly describes the complex and dynamic interactions between all market actors, including those from the private and public sectors.   93 94  MEASURING MARKET DEVELOPMENT Measurement plan Complements the strategy as it provides more detailed information at the project level: defining what is to be measured, how, when, by whom, and at what cost. Measurement strategy Provides broad and long-term guidance for measurement at the level of an organization or large program. Principally covers matters relating to measurement throughout the organization or program, such as principles and standards, standardized indicators, and program-level evaluation. Measurement system Consists of frameworks (e.g., a ToC or results chain, a results framework, and a risk register), processes (e.g., for data collection and management, review and adaptation, and evaluation), and their related tools (e.g., data collection instruments, reporting templates, data processing, and storage applications). Outcome A change in an organization or persons brought about, at least partly, by an intervention. Portfolio A set of programs in a given development domain or country or region. Program A coherent set of projects with a common, but often broad, focus. Usually has a longer life than any single project within it. Program unit Units implementing financial inclusion programs; may be operated by funders themselves, (or implementer) but more typically they are found in Financial Sector Deepening Trusts, intergovernmental organizations, international or national nongovernmental organizations, educational or research organizations, consultancy companies, or government agencies. Commonly referred to as implementer or implementing partner. Progression narrative The description (explicit or implicit) of the transition from one level of a theory of change or results chain to the next. Project A tightly focused set of interventions with a common work plan, managed by a single implementing agency. Results chain A sequence of changes, usually in the form of a diagram, that are expected to be influenced by specific interventions in a project. The expected changes—outcomes—along with the interventions and the assumptions behind the changes, are more context-specific and narrowly focused than in a program theory of change. (See Module 3.) Results framework A planning and results measurement instrument, usually in matrix form. Includes a summary of a program or project’s intended interventions and outcomes, together with measurement-related information, such as indicators, means of obtaining data for these indicators, baselines, and targets for the indicators. May also contain assumptions and risks. (See Module 5.) Rules and norms Shape incentives for market actors and determine who can participate in a financial system and under what conditions. Formal rules include laws and regulations issued by the legislator and public authorities (e.g., banking regulation, licensing criteria for financial institutions, or know-your-customer procedures). Also includes rules issued by industry bodies (e.g., industry standards or codes of conduct). Informal rules are the product of local culture and generally accepted practices. Supporting functions A range of functions that fall outside of the core of a market system, but that significantly affect the strength or weakness of that market. Support, shape, inform, and enable transactions between demand and supply actors. Systemic approach An approach to development that aims to catalyze systemic change. It aims to (or market development promote inclusivity, scale, sustainability, and resilience in the financial system and to have approach)  positive development impacts. Key features include the role of facilitation, adaptive management, and monitoring of a long-term view. Also referred to as a market develop- ment approach, a market systems approach, a market facilitation approach, or making markets work for the poor. Systemic change A change in the underlying dynamics of a financial market that leads to significant change in terms of scale, sustainability, and resilience of the financial system. Systemic change occurs if market players beyond those interacting with program interventions adopt new practices that improve poor people’s participation in the financial services market. Theory of change A set of pathways from interventions to different levels of expected outcomes, usually presented in the form of a diagram. Progress along the pathways is based on contribution hypotheses and is governed by assumptions, the most important of which are made explicit. In this Handbook, the term is used in connection with programs rather than projects (see results chain). (See Module 3.) SELF -ASSESSM EN T TO O L FOR FI N A N C I A L I NC LU S I O N M EAS UREM EN T SYST E MS The Self-Assessment Tool (SAT) is designed to funders, or government bodies). However, help funders and program managers improve funders could also benefit from collaborating the measurement strategies, systems, and with program units in these assessments. Fur- practices used in financial inclusion portfolios, thermore, some of the questions will be rele- programs, and projects, particularly those vant to funders’ own “upstream” measurement using systemic approaches. systems, including portfolio-based monitoring and evaluation. It is important that the SAT process is sanctioned by, and preferably has The composition of the tool the participation of, senior management. SAT is based on a set of building blocks. For Changes triggered by the process may have each building block, SAT proposes a series of resourcing and other strategic implications. assessment questions (first column of the As a whole, SAT proposes a comprehensive tool's grid). The aspect of measurement cov- set of measurement practices. It is unlikely that ered by a question is then scored according to all of the practices will be highly relevant to ev- a standard methodology that is set out in the ery user. Furthermore, some will be relevant but second column. Depending on the score, brief not a high priority for action (e.g., because of action steps or other comments are entered in the stage in the program cycle). For example, the third and final column, to be followed up the development of ToCs and results frame- later with more detailed action planning. works are most relevant at the beginning of a SAT is closely aligned to CGAP's Measuring cycle; detailed evaluation planning will be more Market Development Handbook. Although relevant once the program is well under way. some terms are defined in footnotes in SAT, SAT enables the collaborative identification more comprehensive explanations of concepts of priorities for improvement and the trigger- and terms are given in the Handbook. ing of action. SAT could be conducted with or A few questions in the tool are in bold to without external stakeholders, and with or indicate that they are critical for effective mea- without external expert facilitation. surement. However, it is important to view the blocks as interdependent and every question Scoring as potentially relevant. Questions that are closely related are sepa- Assessors should consider two aspects of each rated by broken lines. of the questions. First they should consider the extent to which the question is relevant to their portfo- Audience lio, program, or project, given the scope of the A majority of questions in SAT are oriented assessment under way. The SAT scale for rele- around implementation and will be more rele- vance is A = highly relevant; B = moderately vant to program units, whether they are relevant; C = not relevant. funders themselves or external implementers If a question is considered “not relevant” in (e.g., facilitators, consulting firms, grantees of the context of the review, there is no need to   95 96  MEASURING MARKET DEVELOPMENT go further. If the question is considered to be lection, data management, review, and evalua- fully or moderately relevant in the assessment tion). They need to be seen as a system, context, assessors should consider the degree interdependent, and connected. SAT maps the of fitness of measurement in the portfolio, pro- key components of the system with available gram, or project in relation to the question. resources, measurement status and location in Fitness has two dimensions. The first is the the organization, capacity building for mea- extent to which the portfolio, program, or proj- surement (including at the partner level), and ect is equipped with any necessary frame- linkages with reporting and the way evidence works, processes, or tools in relation to a produced by the system influences decision- specific assessment question. The second is making. human capacity: Are there sufficient staff with For more information, see Module 2. the necessary knowledge, skills, and mind set in relation to the assessment question? The 3. The environment for measurement SAT scale for fitness is 3 = fully fit (no action The measurement system should be set in a needed on frameworks, processes, tools, or ca- conducive environment that enables and pacity); 2 = moderately fit (some action need- incentivizes staff to participate fully in mea- ed); 1 = unfit (considerable work needed). surement activities. This is particularly import- A score for a question combines the two ant for systems in financial inclusion programs elements (e.g., a score of A2 would mean highly that adopt systemic approaches—where mea- relevant and moderately fit, and therefore some surement requires innovation and initiative by action is needed). Where action is needed, the a range of program staff. SAT covers support next steps and any other comments should be received by management, incentives for staff recorded in the last column of the tool. The to participate, collaboration between funders scale of action and the urgency may depend on and program units, and a broader notion of whether the relevant score is A or B. Action may accountability. mean putting in place or improving frameworks, processes, or tools; it may also mean support- For more information, see Module 1 and Module 2. ing human capacity development. 4. Measurement strategies and plans Organizations, portfolios, and large programs SAT building blocks need measurement strategies that address SAT comprises 13 building blocks that are measurement matters that transcend individ- widely considered to be essential for an effec- ual projects. These may include measurement tive measurement system. policy, if that policy is not set out elsewhere. Measurement strategies should be comple- 1. Key concepts mented by detailed and regularly updated There is no clear consensus on some concepts, project measurement plans. The boundaries such as financial system, systemic change, and between the two instruments will vary. SAT financial inclusion, which are nevertheless covers how measurement strategy, plans, and important for funders to clarify to be able to questions were developed and their content. effectively measure progress in market system development for financial inclusion. Funders For more information, see Module 2. need to have a plausible model of the system and concepts of systemic change and financial 5. Theories of change (ToCs) and results chains inclusion that are clear and that stakeholders ToCs and results chains play foundational roles have bought into. To be able to compare prog- in measurement and link with program and ress across interventions, the model and defi- project planning for financial inclusion. They nition of the concepts should be applied pave the way for results frameworks and are consistently in ToCs and results frameworks. the platform for enhanced monitoring and SAT lists questions around key concepts to evaluation, particularly theory-based. ToCs check whether these are properly defined and and results chains can be viewed as similar communicated to relevant program or project instruments: ToCs are broader in scope and stakeholders. less context-specific than results chains and are, therefore, found in programs rather than For more information on these key concepts projects. SAT covers the quality and relevance see Module 1 and Module 3. of ToCs and results chains and the way they are used (hypothesis rather than blueprints) 2. The measurement system and updated. Measurement needs frameworks (e.g., ToCs, sets of measurement questions, and results For more information, see Module 3. frameworks) and processes (e.g., for data col- SELF-ASSESSMENT TOOL FOR FINANCIAL INCLUSION MEASUREMENT SYSTEMS   97 6. Measurement questions by the monitoring of a wider range of results Measurement questions clarify what we want to data: qualitative aspects, unplanned and un- learn about our programs, projects, and portfo- expected outcomes, assumptions and risks, lio of programs. They help to define the scope and other aspects of the external environ- and direction of monitoring and evaluation. ment that may affect the intervention. SAT Framing measurement questions at the begin- covers enhanced monitoring’s content and ning of program and project cycles is important. processes, including feedback into interven- For more information, see Module 4. tion design and management. For more information, see Module 4 and 7. Results frameworks: Structure Module 7. Results frameworks are the most practical ways to summarize the chain of expected 11. Reporting and knowledge sharing results and the means of measuring them. The Reporting to stakeholders plays a big part in frameworks need to reflect the related ToC or measurement. It is fundamental to account- results chains. This means allowing for more ability, which is one of the most important than one level of outcome and a reasonable functions of measurement, and it can feed number of results at each level. The most com- stakeholders’ learning needs and appetites. mon version of a results framework is the logi- Combining these two functions need not be a cal framework or logframe. This building block challenge, but other outreach channels may covers the structure and terminology of the need to be used to reach broader audiences. results framework. For more information, see Module 8. For more information, see Module 5. 12. External evaluation 8. Results frameworks: Indicators and targets External evaluation is a core component of the This building block looks inside the results measurement system because it adds value in framework at indicators and targets. It assesses several areas, including an unbiased standpoint how indicators and targets are selected and to calibrate results data, assess contribution, reviewed, how relevant they are, and whether validate performance for accountability, and the framework links to adequate data sources. provide expertise and experience to turn find- For more information, see Module 5 and ings into recommendations for change. Evalu- Module 6. ation is expensive in terms of time and other resources and needs to be led by robust use 9. Data sourcing cases. Evaluation should be closely integrated Sourcing data for the measurement of financial with monitoring for the lower levels of the ToC inclusion programs presents few conceptual because external evaluators bring a detached but many practical difficulties. Data are needed perspective to bear on the program staff’s in three stages: before the program or project deep knowledge of the intervention environ- begins, during the program or project, and ment. Collaboration in, and reciprocal sharing after it has been completed. SAT assesses data of knowledge from, evaluation is particularly quality, including baseline data, and the pro- important in the relatively unmapped territory cesses for collecting them. of market system development programs for financial inclusion. SAT covers the policy gov- For more information, see Module 5 and erning external evaluation, the type of meth- Module 6. odologies used, integration with monitoring, and collaboration with other stakeholders. 10. Monitoring, review, and adaptation Market systems, especially immature financial For more information, see Module 4, Module 7, systems, are dynamic and unpredictable. and Module 9. Because of this project monitoring takes on additional responsibilities compared to tradi- Portfolio reviews (only for funders) 13.  tional approaches. Monitoring based exclu- Portfolio-based monitoring and evaluation, sively on results framework indicators is based on aggregation or systematic scoring of inadequate for market system development information on programs and their perfor- and similar program approaches, and the mance, can be useful to funders for both learn- results can be misleading. Enhanced monitor- ing and accountability. SAT encourages funders ing that goes beyond tracking predefined to check whether there is a regular, systematic, indicators and straddles the boundaries and “smart” monitoring of interventions at the between monitoring and both formative eval- portfolio level, and whether findings are applied uation and risk management should be con- in decision making. sidered. Indicators need to be complemented For more information, see Module 10. 98  MEASURING MARKET DEVELOPMENT Building block 1. KEY CONCEPTS Assessment questions Scoring Next steps and/or comments Do we have a clearly defined model of a financial 1.1   market system that we use consistently across our program(s) and, where relevant, share with key stakeholders? To what extent is the model clearly identified in 1.2  the program ToC?a Do we have a definition of systemic change that 1.3  we use consistently across our program(s) and, where relevant, share with key stakeholders? Is the definition evaluable (e.g., is it possible to 1.4  represent it in a TOC or results chain and translate it into indicators)? Do we have a definition of financial inclusion that 1.5  we use consistently across our program(s) and, where relevant, share with stakeholders? Is the definition evaluable (e.g., does it easily 1.6  translate into indicators)? “Theory of Change” covers constructs that may also be called results chains, logic models, or impact pathways. Results frameworks—such as logframes—also a.  contain indicators and other measurement information. For more information, refer to modules 3 and 5. Building block 2. MEASUREMENT SYSTEMS Assessment questions Scoring Next steps and/or comments 2.1 Is there a clear picture of what the measurement system is composed of and how it is connected? Is this picture shared with stakeholders who 2.2  need to know? 2.3 Is the system adequately resourced and managed?a Is system management appropriately positioned 2.4  in the organizational structure to give it the necessary credibility, authority, and access? 2.5 Is the system proportionate in terms of depth and cost in money and human resources?b 2.6 Are the data collected adequately managed within the system (e.g., quality assured, accessible where and when needed, effectively analyzed and presented)?c Given that sustained changes at the financial system 2.7   level may not be detectable until after an inter- vention is over, are there institutionalized processes for continuing to monitor beyond the life of specific interventions?d 2.8 Is evidence produced by both monitoring and evaluation systematically reviewed at a strategic level and within projects? Are findings and con- clusions purposefully applied to new decision- making?e 2.9 Is there a systematic approach to identifying and monitoring risks in both the delivery and outcomes of interventions?f If risks are identified, is appropriate action taken 2.10  (e.g., adjust the intervention to mitigate or avoid the risk)? SELF-ASSESSMENT TOOL FOR FINANCIAL INCLUSION MEASUREMENT SYSTEMS   99 Building block 2. MEASUREMENT SYSTEMS, continued Assessment questions Scoring Next steps and/or comments 2.11 Does internal and external reporting flow from review processes, rather than being conducted as an isolated exercise without learning benefits?g Is reporting to different stakeholder groups 2.12  sufficiently streamlined to avoid unnecessary duplication of effort? 2.13 Is capacity building for measurement sufficiently resourced and institutionalized?h Does this include capacity building for stakeholders 2.14  (e.g., project partners) who are engaged in measurement with us? 2.15 Is it backed by accessible, user-friendly tools, templates, and written guidance? 2.16 Are there ready sources of human support, including peer-to-peer (e.g., through a community of practice)? All measurement systems need resources and need to be managed. In small projects, this can sometimes be done without specialist expertise and a.  dedicated measurement. But this is unlikely to be the case with larger, more complicated interventions. The measurement system needs to meet defined needs and no more. Overly elaborate measurement requirements are unsustainable and can damage b.  the whole system. The quality of data—both in its raw state and in the way it is analyzed and presented—affects its reliability and use. Data management is an important c.  part of any system. One of the benefits of having a program-wide measurement strategy is that data collection can continue beyond the life of individual projects to support d.  impact evaluation. The most consistently highlighted weakness in measurement systems is in the application of evidence to new decision-making. Yet this is probably e.  the most important reason for measurement. f. Identifying and monitoring significant risks are essential for effective program delivery and outcomes. They are also valuable processes for learning about what is feasible and what promotes progress in interventions, especially in the relatively unmapped territory of financial inclusion. Reporting is one of the least loved activities in the intervention or institutional cycle. It is often seen solely as a compliance activity. Constructing g.  reports as a consequence of a collaborative review process can inject learning into report-writing and improve the quality of the reports. It is often possible to avoid multiple report formats and timing through negotiation with report stakeholders. Measurement does not come naturally to many of the staff involved. It requires investment in regular capacity-building and other support mechanisms. h.  This is particularly true of partners who may not have any background in measurement. A community of practice can help both specialist measurement staff and program staff without specialist support from feeling isolated. Building block 3. ENVIRONMENT FOR MEASUREMENT Assessment questions Scoring Next steps and/or comments 3.1 Is measurement embedded in program management and not seen as a technical function done by measurement specialists? a 3.2 Does senior management visibly promote measurement? b 3.3 Are adequate incentives applied to staff to collect data systematically and to take time to review it? c 3.4 Are staff encouraged to identify weaknesses in implementation and results? 3.5 Are staff encouraged to take reasoned risks, even if the chances of failure might be considered high? 3.6 Is there a collaborative approach to measurement between funders and program units? If not, are steps being taken to improve them? d continued 100  MEASURING MARKET DEVELOPMENT Building block 3. ENVIRONMENT FOR MEASUREMENT Assessment questions Scoring Next steps and/or comments 3.7 Are the following program factors considered in interactions about accountability between program units and funders? e a.  Designing well-formulated projects based on thorough diagnostics and with an appropriate level of ambition b. Putting in place and operating a proportionate full-cycle measurement and learning system c. Systematically identifying, monitoring, and managing risks d.  Having processes that ensure the quality of activities and outputs and delivery at a reasonable cost e. Being a proportionate contribution to valued outcomes f. Delivering robust cases for results frameworks Measurement does not function properly as a technical add-on, particularly in market systems development programs. It should be part of mainstream a.  intervention management. As with many other activities in organizations, if measurement is not seen to be valued and used by senior management, it will be given a low priority by b.  other staff. Examples: regularly demanding good-quality, timely data on both implementation and outcomes and being seen to use it in operational and strategic decisions; or supporting the production of these data by adequately resourcing and engaging with it. Many aspects of measurement, especially enhanced monitoring, benefit from a culture of enquiry and openness. This, and the need for time to be devoted to c.  it outside the normal routines of intervention management, requires a re-setting of incentives for staff throughout the program or organization. Funders and program units have a mutual interest in ensuring that interventions are designed and implemented effectively and that lessons are learned d.  and carried forward. Measurement plays a big part in this. It makes sense to collaborate in ensuring measurement meets these needs and serves accountability. Program unit accountability in environments like those of most market systems development programs for financial inclusion, should not be equated solely e.  with delivering predefined, quantitative targets. This narrow concept of accountability can even be damaging and can create inappropriate incentives and risk aversion. Accountability conceived more holistically, is likely to be more relevant and effective in developing and maintaining conducive environments and promoting good outcomes. If this approach to accountability is lacking, it should be possible for program units at least to open a dialogue about it with funders. Building block 4. MEASUREMENT STRATEGIES AND PLANS Assessment questions Scoring Next steps and/or comments 4.1 Is measurement operationalized through robust and practical measurement strategies and plans? 4.2 Do measurement strategies and plans include the following? a a. An introduction that includes a description of key measurement issues and the measurement questions that should be addressed b. Resourcing and management details c. Relevant ToC(s) d. Results framework(s), including baselines and targets e. A risk register and clear processes and responsibilities for risk monitoring and management f. Detailed guidance on the monitoring for each indicator g. Guidance on monitoring beyond indicators (unplanned results, etc.) h. Guidance on baseline data collection i. Guidance on processes for data management j. Guidance on internal review processes, including key measurement questions k. Guidance on internal and external results reporting l. Guidance on external evaluationb and use of evaluators m. Details of further capacity building and support for measurement in the intervention Some items, such as a, b, and c, are more appropriate for a measurement strategy; others, such as f, as applied to project-specific indicators will be found in plans. a.  There is always a degree of overlap between the coverage measurement strategies and plans. b. This is only preliminary guidance for evaluation. Every evaluation will have a detailed evaluation plan. SELF-ASSESSMENT TOOL FOR FINANCIAL INCLUSION MEASUREMENT SYSTEMS   101 Building block 5. THEORY OF CHANGE AND RESULTS CHAINS Assessment questions Scoring Next steps and/or comments Do we have a ToC for the overall financial inclusion 5.1  program and, where relevant, broad themes within financial inclusion, such as savings? 5.2 If there is a strategy that covers financial inclusion? Is the ToC aligned with it? Is the ToC broad and inclusive enough to encom- 5.3  pass the full range of interventions we anticipate delivering or supporting under our strategy? 5.4 Do individual projects have their own results chains? Do they clearly relate to the program-wide ToC? 5.5  If not, is there an agreed case for divergence (e.g., to pilot an innovative approach)?a Are the ToCs and results chains based on adequate 5.6  evidence? Are they developed with the participa- tion of key stakeholders?b 5.7 Are the progression narratives of the ToCs and results chains plausible?c 5.8 Are they based on explicit contribution hypotheses? d 5.9 Are the assumptions (including the contributions of other stakeholders) behind the progression narratives explicit in the models or supporting documentation? e 5.10 Have the assumptions been identified through a proper process of analysis involving relevant stakeholders? 5.11 Are the key assumptions—or their counterparts, risks—carried forward into an actionable risk register? 5.12 Are the building blocks of the ToCs and results chains evaluable to the extent that they can readily be converted into results frameworks and easily translated into indicators? 5.13 Are the ToCs and results chains readily understood by program staff and key stakeholders? f 5.14 Where detail is left out to prevent the ToCs and results chains from becoming too complicated, is it included in an accessible accompanying document? 5.15 Are the ToCs and results chains presented and understood as hypotheses and not as blueprints?g 5.16 Are they reviewed at appropriate intervals or after critical events and, if necessary, adjusted to reflect changes in the program strategy, external players, and program circumstances? a. Projects stem from program strategies so their results chains should normally align to some extent, unless a divergence has been agreed, e.g., to test out a new area of intervention. b. The process of developing ToCs and results chains is important partly for their credibility and the buy-in of stakeholders; and also because their robustness depends on the quality of inputs to the process: evidence from research and evaluation where available, and stakeholder perspectives about what is plausible and feasible. c. The progression narratives (description of the transition from one level and its different elements to the next; it usually includes a set of “If….then” statements) and their underlying hypotheses—including assumptions about factors beyond the intervention—are what bind the blocks of the ToCs and results chains together. They need to be grounded through a proper process of analysis. d.  A contribution hypothesis is an explanation of the logic behind progression from each output to its related outcome, and outcomes at one level to the next. e.  Assumptions and/or risks are important in market system development programs and need to be monitored and managed effectively. f. ToCs and results chains are valuable as communication vehicles both internally and externally. They should not be too complicated. At the same time they should attract credibility through not appearing simplistic or over-deterministic. Balancing these two considerations may be challenging but is always worthwhile. g. ToCs are designed to be reliable, but in complex and unpredictable territory like market systems development for financial inclusion, they are bound to be hypothetical and will need to be tested and adapted if found wanting. 102  MEASURING MARKET DEVELOPMENT Building block 6. MEASUREMENT QUESTIONS Assessment questions Scoring Next steps and/or comments Are comprehensive sets of measurement 6.1  questions that need to be addressed by monitor- ing and different types of evaluation developed at the beginning of program and project cycles?a 6.2 Whenever measurement questions are developed, do they take the OECD-DAC evaluation criteria into account? b 6.3 Do they relate closely to the ToC and its explicit or implicit contribution hypotheses and assumptions? 6.4 Are measurement questions developed jointly between funders and implementers and, where relevant, with the participation of other stakeholders? c Measurement questions set the overall agenda for measurement of a project or program. It is important to frame them at the beginning of the program or a.  project cycle to maximize the potential of monitoring and support evaluation by ensuring that critical data are collected and analysis is undertaken at optimal times. The OECD-DAC criteria provide the broad evaluation agenda. The ToC contextualizes it enough for measurement questions to gain traction, but without b.  narrowing their focus as much as the results framework. OECD DAC: http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm Developing the questions jointly helps to promote coherence and align expectations on accountability and learning priorities. c.  Building block 7. RESULTS FRAMEWORKS: STRUCTURE Assessment questions Scoring Next steps and/or comments 7.1 Do results frameworks exist for interventions at both project and program level? 7.2 Are the results aligned with the program ToC and project results chains? 7.3 Are the frameworks proportionate in their size and complexity to those aspects of the project or program? 7.4 Is the structure (the number of levels, the permitted number of outcomes at each level) of the frameworks practical and easy to work with? 7.5 Is the terminology of the different levels of the framework (e.g., outputs, intermediate outcomes) used consistently across the program? a 7.6 Is the rationale for the levels and their terminology understood and appreciated by stakeholders? There is no universal language for results framework terminology and this often generates debate and uncertainty. To reduce the uncertainty in the circles in which a.  it moves, every organization or program needs to base its own terminology on a clear rationale, explain it to stakeholders, and use it consistently. SELF-ASSESSMENT TOOL FOR FINANCIAL INCLUSION MEASUREMENT SYSTEMS   103 Building block 8. RESULTS FRAMEWORKS: INDICATORS AND TARGETS Assessment questions Scoring Next steps and/or comments 8.1 Where there are preidentified measurement questions that point to important aspects of the expected results, are they reflected in the results framework indicators?a 8.2 Have the indicators been developed by staff who are involved in and have a good understanding of the intervention?b 8.3 Are the indicators relevant to the related results? For example, is there at least one indicator for each distinct element of a result? 8.4 Are qualitative indicators used to complement—or instead of—quantitative indicators where the latter are not able to tell the story adequately?c 8.5 Are steps taken to exclude indicators of marginal relevance?d 8.6 Is there appropriate alignment between project and program indicators?e 8.7 Where relevant, is there appropriate alignment (results and indicators) with frameworks of other stakeholders (e.g., between programs and funder portfolios)? 8.8 Are the indicators precise enough to be understood in the same way by different stakeholders? 8.9 Are the indicators linked to accessible, reliable, and affordable data sources and instruments?f 8.10 Where there are targets, have they been establish- ed through an evidential process? g 8.11 Are current targets and milestones appropriate and feasible, reflecting what is likely to have been achieved at that point in the intervention? h 8.12 Are the contents of results frameworks reviewed at appropriate intervals and adjusted, where necessary, so that projects and programs do not to get locked into inappropriate or out-of-date results frameworks? Measurement questions can suggest nuances for indicators because of themes such as gender, which are of particular interest and importance, but which may a.  not be explicit in the ToC outputs or outcomes themselves. They can also point to where indicators need to be particularly intensive because of a high degree of interest in the result. b. Indicators are central to most measurement systems. In complex environments like market system development, they need to be based on solid knowledge and understanding of where change is likely to happen. Participation is also important for buy-in. c. Results cannot always be adequately captured through quantification. Changes in a single policy, institutional structure or process, or the behavior of a group, for example, are often better described in words. Indicators require much effort and other resources. It is important that they carefully target areas where they are most needed. d.  It is important to avoid imposing standardized indicators where they are not sufficiently relevant to the intervention because this can lead to inappropriate e.  incentives and distortion in implementation Although measurement plans will provide greater detail on data sources and instruments like surveys, it is important to establish that these sources exist and f.  that they can be accessed through data collection instruments, reliably, and affordably, before the indicator is “signed off”. For example, have baseline data and information on the trend leading to the baseline been collected? Have there been consultations with stakeholders who have g.  a good understanding of what is feasible, and of how reliable the assumptions behind the targets are? Targets create expectations. They need to be realistic. In market system development programs, milestones on the road to end-line results, such as increased access to services, may need to be about changes in h.  behavior and institutional structure. 104  MEASURING MARKET DEVELOPMENT Building block 9. DATA SOURCING Assessment questions Scoring Next steps and/or comments 9.1 Are baseline data collected for indicators and other areas where change is expected to be measured? a Are the baseline data collected early enough in 9.2  the intervention cycle to be valid? Are steps taken to avoid collecting baseline data 9.3  where there is not a clear use case?b Where data—whether for baselines, ongoing 9.4  monitoring, or evaluation—are collected from samples of a population, is it ensured that they are sufficiently representative of the population for the purpose at hand?c Where secondary sources are relied on, are they 9.5  sufficiently up-to-date and granular for the purpose at hand?d If they are not, are steps taken to influence the 9.6  timing and granularity of these sources, or to fill in the gaps with primary data collection?e Where existing data are not available and sourcing 9.7  data ourselves would be unaffordable or disproportionate, is collaboration with other stakeholders (e.g., funders, government agencies) considered to generate the data, either one-time or regularly? a.  Baseline data are essential if we are to measure change, e.g., in the status of stakeholders or institutions that already exist at the start of the intervention. b. To avoid data overload, it is important to be clear what changes we are likely to want to measure before planning baseline data collection. c.  Data sampling is governed by well-established techniques. Standards need to be observed if the data are to be relied on and credible. d. At the higher levels in financial inclusion ToCs, secondary sources are often the most practical option for measurement. e. Secondary sources may not give us the data when and in the form we need. It is important to research this at the beginning of the intervention cycle and consider investing in new data sources or extensions to existing ones. Collaboration may be the most feasible way of doing this. Building block 10. MONITORING, REVIEWING AND ADAPTING Assessment questions Scoring Next steps and/or comments 10.1 Is there systematic monitoring of indicators? Is there systematic monitoring of qualitative 10.2  aspects of planned results (outputs and outcomes) not adequately captured by the indicators in the results frameworks? Is there systematic monitoring of outcomes that 10.3  are unplanned and unexpected, both positive and negative? Is there systematic monitoring of identified 10.4  assumptions or risks and other aspects of the external environment that may matter for the interventions?a Is there regular and systematic review both of the 10.5  implementation and results of interventions? Do relevant stakeholders participate in the review 10.6  processes? Do relevant measurement questions drive the 10.7  reviews? Is relevant available evidence assembled and 10.8  presented in these reviews in usable formats? continued SELF-ASSESSMENT TOOL FOR FINANCIAL INCLUSION MEASUREMENT SYSTEMS   105 Building block 10. MONITORING, REVIEWING AND ADAPTING Assessment questions Scoring Next steps and/or comments Are there clear criteria and a robust process, 10.9  acceptable to key stakeholders, for adapting interventions based on review findings?b a. The external environment will affect the intervention and its outcomes both positively and negatively, predictably and unpredictably. These external factors need to be monitored mainly through the lens of identified assumptions and risks. During the review process it is important to keep an open mind about what is driving results that may appear at first sight to have been primarily influenced by the intervention. Market system development interventions need to keep pace with the dynamics of their environment. Data collected through monitoring needs to be regularly b.  reviewed, and the conclusions of the reviews should lead, through due process, to adaptations to the intervention itself and its results framework. Building block 11. REPORTING AND KNOWLEDGE SHARING Assessment questions Scoring Next steps and/or comments Is reporting designed to serve both accountability 11.1  and stakeholders’ learning needs and interests? Have measurement questions been taken into 11.2  account in determining the scope and content of reports? Alternatively, have key stakeholders been consulted recently on their information and know- ledge needs and interests that could be served by the reports? Do progress reports focus on the results framework 11.3  as well as provide ample space for reporting beyond planned results, including unplanned outcomes and relevant developments in the external environment? Do reports present a plausible narrative of how the 11.4  program has contributed to change? Do reports describe what lessons have been learned 11.5  and how they are to be applied going forward? Is the frequency and scope of reporting required 11.6  from projects to programs and from programs to funders proportionate and in line with how that information will be used? Is reporting integrated into cycles of review, so 11.7  that reporting flows in a timely way from evidence- informed reviews, and is not prepared in isolation? Is knowledge stemming from review and evaluation 11.8  shared with other stakeholders and broader audiences through appropriate dissemination channels? Building block 12. EXTERNAL EVALUATION Assessment questions Scoring Next steps and/or comments 12.1 Is there an up-to-date and relevant policy or strategy governing external evaluation? a 12.2 Are there criteria and a robust process for ensuring that there is a clear use case for every evaluation? 12.3 Are those criteria and that process followed consistently? Are evaluations based on relevant ToCs and 12.4 measurement questions? b 12.5 Where use is made of experimental methodologies, is it appropriate?c continued 106  MEASURING MARKET DEVELOPMENT Building block 12. EXTERNAL EVALUATION Assessment questions Scoring Next steps and/or comments 12.6 Do theory-based evaluations:d a. Capture and assess data about both anticipated and unanticipated outcomes? b. Capture data from a sufficient number of sources, both quantitative and qualitative, to enable “triangulation”? c. Use credible strategic informants? d. Establish plausible connections between the interventions and the changes identified? e. Capture data about other interventions and contextual factors, to explore other possible contributions to the changes identified? 12.7 Is evaluation closely integrated with monitoring? Have external evaluation staff been used, when 12.8  relevant, to facilitate internal reviews and/or conduct developmental or real-time evaluation? e Is there collaboration with other stakeholders in 12.9  evaluating progress in market system development? Are findings and conclusions from our evaluations 12.10  being shared with stakeholders, including key national actors? This is best included in the type of measurement strategy described Building Block II. a.  If ToCs and/or measurement questions do not exist, they need to be created in the evaluation inception period. If they exist, they should be revisited and refined, b.  if necessary. Experimental methodologies have limited roles in systemic programs. They are more likely to be used in testing links between financial inclusion and poverty c.  reduction. TBE is the most appropriate methodology for market system development programs for financial inclusion. Rigor depends on several; approaches, including d.  combining top-down with bottom-up perspectives. The terms “developmental” and “real-time” evaluation are used to describe evaluation work that takes place during project and program implementation, is e.  conducted iteratively, and where the emphasis is on immediate lesson-learning and its application, rather than on impact assessment or accountability. Building block 13. PORTFOLIO REVIEWS (funders only) Assessment questions Scoring Next steps and/or comments Is there regular and systematic monitoring of the 13.1  portfolio of financial inclusion interventions at the portfolio level, using an instrument such as a portfolio dashboard, to see if the interventions are going in the right direction? Is there periodic (e.g., once every 2–4 years) 13.2  evaluation of the portfolio against a strategy for financial inclusion? Is there a system that generates sufficient relevant 13.3  data to enable comparisons of interventions across the portfolio?a Are these comparisons applied “smartly,” bringing 13.4  context and complexity into play?b Are insights from the reviews applied in new 13.5  strategic decision making? Portfolio reviews are more effective if their needs are anticipated at the beginning of intervention cycles through categorization in intervention plans and in a.  the use of standardized expression of results and indicators, where this is appropriate. (See Building Block 8.) Comparisons across market systems development interventions can be difficult and even invidious. It is important to treat standardized results as the entry b.  points for further analysis and interpretation. Consultative Group to Assist the Poor 1818 H Street, N.W., MSN IS-700 Washington, DC 20433 USA Phone: +1 202 473 9594 cgap@worldbank.org www.cgap.org www.microfinancegateway.org