Series: Are We Ready for Strategic Social Accountability? GPSA Note 5 ADAPTIVE LEARNING This is the fifth note in a six-part series discussing whether the social accountability field is already primed with the knowledge and capabilities to design, implement, fund, and learn from strategic interventions. This note takes stock of the way entries from the first two rounds of GPSA applications use learning, monitoring and evalu- ation to provide immediate feedback to improve the effectiveness of their social ac- countability interventions. It presents results from systematic analysis of more than 600 applications submitted to the Global Partnership for Social Accountability. August 2014 I. THE GLOBAL PARTNERSHIP FOR SOCIAL ACCOUNTABILITY AND LEARNING The Global Partnership for Social Accountability Our approach to drawing lessons from GPSA (GPSA) is a global multi-stakeholder coalition Applicants: of donors and development actors. It provides strategic and sustained funding and knowledge •Selection of a sample of 40 of 644 GPSA ap- plications support to civil society organizations (CSOs) •Scored each for their strategic political ap- working together with governments to solve proach critical governance and development prob- •Extra attention to the best and worst lems. The GPSA has requested and awarded applications applications for funding in 2013 and 2014. All •4 Components of social accountability strategies coded as present, partial or absent applicants were asked to explain how they would contribute to the GPSA’s fourth pillar, Keep in mind that our analysis did not fully reas- focused on Learning for Improved Results. sess the country context to determine whether In addition, applicants were expected to an application responded precisely to local cir- use learning in their applications and their cumstances. Rather, we assessed to what extent the applicant discussed, analyzed and showed projects. evidence of responsiveness to their own assess- ment of the local context. This note analyzes a sample of 40 out of the 644 applications focusing on how they propose to Want to learn more about the GPSA’s selection learn, but more importantly on how they use process and our methodology? Check out GPSA- past learning to explain and justify what they Note 1. have written into their projects. This Note Series has been developed by Florencia Guerzovich (Consultant to the GPSA Knowledge Component, florcig@gmail.com) and Maria Poli (GPSA Team Member, mpoli@worldbank.org) with support from Jonathan Philips ( jonathanphillips@fas.harvard.edu). To access the complete Notes series, go to www.thegpsa.org /1 We understand learning as the extent to which each GPSA applicant “gains and uses Adaptive Learning: Some Helpful Points knowledge, from both its own work and Check out Rakesh Rajani & Twaweza’s that of others, to influence its policy, strat- take: monitoring and learning is not egy, plans and actions.” about having an answer about what works, but organizations should have mechanisms to: We want to share data and reflect: how are we •Look into what works and what doing in the strategic use of learning to adapt doesn’t and what we think it can work our social accountability work as it evolves? without pretending we are sure, •Look and measure what is going on What should we individually and collectively be during the implementation of our idea, doing to improve our individual and collective •Use the feedback to interact with it use of adaptive learning? and adapt our ideas and approach, •Change what we are doing accordingly. II. PINPOINTING ADAPTIVE LEARNING IN GPSA APPLICATIONS There is much talk in the development and social In this scenario, it can be hard to find indivi- accountability fields about the importance of in- dual examples to illustrate concretely and brief- corporating systematic, real-time learning, as well ly what we mean by adaptive learning. But it is as monitoring and evaluation into the operation not impossible. We think that the experience of of social accountability interventions, often re- Samarthan’s adaptive learning approach to drive ferred to as adaptive learning (e.g. see here and improvement in the transparency and effective- here). ness of government spending on an employ- ment guarantee program in rural India is useful At the same time, for too long research, evalu- here. Initially, the CSO decided to use a set of ation, and learning have been seperated from tactics including developing a social watch group strategies and operations. So much so that many and organized wage earners in workers’ unions. of our colleagues think that the context for work Structured review meetings with the field team, in social accountability and development of- regular feedback from government officials and ten fails to encourage adaptive learning and threats to the field staff, as well as lack of results, the application of this knowledge into our prompted Samarthan to adapt its approach. decision-making (for different assessments and The organization engaged 1000 villages in diffe- courses of action see e.g. here, here, here, here, rent regions, downloaded information from the here, here, here). Note: there is no conclusive website, identified two educated youths in each evidence that these “new” principles for learn- village, and presented a formal report to senior ing in the development world are more (or officials. They did all of this while tracking and less) conducive to wider governance reforms documenting their choices. If you want to check than business as usual; these principles are a bet what happened during and after strategic adap- towards a better future. tation see this case study. /2 It is even harder for funder-grantee rela- country programs by the World Bank tionships to institutionalize learning and in- among other resources. We used the questions tegrate it systematically throughout the life in Table 1 to verify whether GPSA applicants cycle of their joint projects, starting with the used and planned to use learning as a basis to application. Still, the GPSA’s theory of change justify and improve on their proposed social ac- and results framework move in this direction, countability projects. Our list of questions is not taking a cue from a range of consultations and exhaustive. As a caveat, we considered current this 2013 evaluation on knowledge-based debates as well as the material we were wor- king with. Table 1: Key Questions to identify adaptive learning in GPSA applications 1 Does the application provide a clear rationale to justify how past successes and failures inform strategies and tactics, including risk management? 2 Does the project approach learning as an exercise to course-correct and improve its work rather than as a tool to market success/best practices for others to take up? 3 If the project aims to produce lessons for others to take up, (does it presume that the world can learn from their experience or) does it take specific measures / propose concrete MEL approaches that will explain under which conditions their experience will be useful for others? 4 What is the projects approach to scale? Does the project spell out a consistent approach to learning for scaling? Does this approach lend itself to taking into account contextual and politico-institutional factors are likely to influence the implementation of these lessons? 5 Does the project explain how the M, E, L system will be used to help ensure its sustainability (i.e. the continuation of the project beyond the duration of GPSA funding)? 6 Does the project identify the timing and nature of opportunities for learning and iteration of approaches? 7 Does the project fully specify the trade-offs between alternative strategic interventions and provide clear justification for the chosen approach? 8 Does the application draw on and customizes systematic evidence from social accountability initiatives in other countries or sectors? III. GPSA’s LEARNING BY DOING ABOUT LEARNING The GPSA has a mandate to work on learning Round 2 because the questions were simpli- and asks all its grantees to contribute, too. In fied. Instead, the questions were integrated into this spirit, the application template for both of the justification of the proposed social accoun- its calls for proposals requested applicants to tability approach, by asking the applicant to re- develop a stand-alone “Knowledge & Learning fer to their prior experiences, what worked and (K&L) component”. This component should link did not work, and why the proposed approach and use the proposed interventions as oppor- should work this time around. tunities for learning about strategic pathways. Overall, responses did not meet the standard Round 1 included three additional questions on set in the knowledge and learning questions. lessons learned and alternatives considered. To some extent, the fact that questions about These three questions weren’t included in adaptive learning were less clearly flagged in /3 Round 2 seems to have elicited fewer construc- opportunities for learning and iteration along tive responses in this case. Still, in both rounds the lines of the Samarthan case presented the applicants had the opportunity to lay out above. This approach encourages the devel- a knowledge and learning component, and to opment of a culture of on-going, free and full connect the proposed intervention to their past communication about what does and does not and ongoing work. Taking a detailed look, we work. This model of problem-driven iterative explored how applicants engaged with diffe- adaptation is documented. (here and here) rent dimensions of learning: • Building on existing tacit and explic- it knowledge to define the strategy and Tips (& recognition) for making adapting insights to the problem and con- the most of failures? text: We checked whether applicants provided http://blogs.wsj.com/at- a clear rationale to justify how past successes work/2013/03/29/the-manag- and failures inform strategies and tactics, and er-who-kept-a-six-year-diary- 12 did. Only two proposals fully specified the of-her-mistakes/ trade-offs between alternative strategic inter- http://www.theguardian.com/ ventions and provided clear justification for the global-development/pover- chosen approach given the problem they face ty-matters/2011/jan/17/ in their own context. The applicants’ record ngos-failure-mistakes-learn- regarding assessment of systematic evidence encourage?CMP=twt_gu from documented social accountability inter- http://this.org/maga- ventions fared worst. However; the merits of zine/2011/12/01/admitting- building on and adapting existing knowledge failure/ in the social accountability field are clear. One proposal learnt that “National CSOs have at- tempted to implement social accountabil- ity programmes using local accountability • Learning as a process to increase sustain- networks, much of the work being done by ability: Thirty out of forty applications failed to external experts, with little participation of link their monitoring, evaluation, and learning citizens and local government authorities. approach to the prospects for the sustainability Unfortunately, this has left no sustainable of the project. The opportunities from linking networks of empowered local social audi- learning to sustainability are promising. One tors. Bringing together and empowering project worked to align its monitoring and eval- local networks to carry out social audits is uation indicators with the statistical system of sustainable.” The lesson learned by the CSO the national statistical office so that the project was that it is more sustainable to bring together could become a valuable source of compara- and empower local networks to carry out social tive data and attract continued support from audits, rather than bringing in external experts. a wide range of stakeholders interested in the Accordingly, the proposal sought to train local data, increasing the likelihood of sustainability. people as social auditors to empower the local community. • Learning as a tool to influence others: Twenty proposals approached learning as a • Learning as an exercise to immediately tool to communicate success and/or identify refine social accountability interventions: best practices for others to replicate elsewhere while ten proposals approached learning as an (despite widespread agreement in the field that exercise to course-correct and improve their there are no magic bullets). Only one of the work, only three identified specific times and proposals that aims to produce lessons for /4 others explains how they will bind their lessons Beyond this, however, this hypothetical exam- to specific contextual conditions so that they ple captures a broader trend in terms of con- are useful for others to learn from and adapt. ceptualizing scale and its relationship to learn- ing. For us, it is striking that a similar logic is • Learning as a path to scale (or grow im- applied in so many countries and sectors. Even pact): Only three of the projects have a clear if partial decentralization were ubiquitous, the approach to scaling up in the case of success. specific decisions made in each level can and All too often, the proposal went like this: “We often do vary by sector and country (on the will implement a pilot project in a range of lo- political economy of decentralization see here cal settings we have identified carefully and and here). We need a more sophisticated un- where we will work with local stakeholders to derstanding of their politico-institutional con- ensure adoption and implementation. Work in text and a more nuanced approach to learning. these areas of primary focus will help us iden- tify best practices that could be replicated else- On a related issue: strategic social accounta- where in the country. However, we realize that bility seeks to find the practices that make sense many of the key decisions about the process we technically and politically for a particular setting care about are made at the national level – i.e. (whether national or sub-national) which can be not where we are working most of the time subject to a limited scale-up rather than a best in this project. Hence, we will employ advoca- practice that works everywhere. And yet, too cy and awareness raising activities for national often projects that embrace a problem-driven decision-makers taking advantage of the na- approach and work with what they find in the tional networks the applicant already belongs context for designing and implementing their to. These networks will facilitate sharing of best core, pilot interventions turn to a best-solution practices and lessons learned to the wider na- driven approach when it comes to growing their tional level audience and through the media for impact. It is exceptional to find entries that qual- making a strong case for wider adoption of the ify the replication exercise to particular types of model. The final phase of the knowledge and sub-national context or take into account the learning component of the project will focus links between national and sub-national politics on advocacy at the national level for country to implement reforms (see e.g. here and here). wide adoption of the model developed by the This probably reflects a major gap in our under- project. This process of wider dissemination and standing about achieving scale in development: advocacy will contribute significantly to enhanc- for what types of problems and under which ing the knowledge base on local government conditions can we replicate external solutions dynamics, practices and intervention needs.” while avoiding problem identification on-the ground and iterative development of solutions Note that this approach does make an explicit (here)? attempt to integrate national and subnation- al work, a step forward in terms of learning if In short, even if the idea of learning starts compared to many cases from the past that out as politically aware for some geograph- tended to concentrate social accountability ical areas in a country, there is a tendency work in capital cities and at the national level. to turn it into a technocratic effort that es- This overall idea is consistent with our knowl- chews political analysis and learning when edge and strategic social accountability in so considering scaling-up. far as it seeks to integrate work across levels of government taking into account where public policy decisions are made (see GPSA Note 3). /5 We, as a field, probably need to give a lot more mental organizations] around the world con- thought to this apparent tension in our ap- firms the existence of a universal NGO language proach to social accountability. There are some that owes nothing to the specific problems or examples where this approach seems to have cultural traits of individual countries”. The chal- worked (see e.g. here). lenge for the GPSA and others goes beyond Still, the apparent universality of this ten- ensuring that funders pick up on this and signal sion could be suggesting what others found they want to overcome this culture. Undifferen- many years ago: “similarity among the state- tiated changes (or calls for changes) in scaling- ments of goals issued by NGOs [non-govern- up are unlikely to take us very far. IV. PUTTING FINDINGS INTO CONTEXT A final set of thoughts about the bigger-picture. In fact one of us explored with 100+ colleagues We need to put these sometimes less-than- working on social accountability what they were encouraging findings in context. We want to (and were not) doing regarding learning and reiterate that the problem and the solution are concluded that funder-grantee-expert relation- unlikely to rely solely on civil society grantees. ships are one key source of the problem and Many CSOs indicate in their GPSA applications the solution (see here). that they have been working on the problem/ The root causes of the struggle are likely to be issue for a number of years, receiving support in an ecosystem (see here) that is larger than from donors, yet they are unable to summarize the GPSA (To learn about an initiative trying what they have learned that leads them to their to work on funder-grantee-CSO incentives on own proposal. Our analysis suggests that this is learning see here, here, and here). This state- regularly found across a majority of GPSA ap- ment, of course, is not meant to ignore what the plicants and projects that have been supported GPSA can do to lead, in particular along with by donors in the past. its own grantees. But the road ahead probably Even if the GPSA provides funds to its 22 gran- calls for action on incentives, institutions, and tees for large projects, it is a small piece of the processes available for learning in collaboration funding pie in terms of social accountability with others. The GPSA’s network of Global Part- funding. If we researched how many funders ners, which has 180+ members including other and civil society organizations are learning in grant-making organizations, might be able to practice, few would probably fare well. help out particularly when it comes to the GPSA itself. /6