Engaging Citizens for Better Development Results AN INDEPENDENT EVALUATION © 2018 International Bank for This work is a product of the staff of The RIGHTS AND PERMISSIONS Reconstruction and Development / World Bank with external contributions. The material in this work is subject The World Bank The findings, interpretations, and to copyright. Because The World 1818 H Street NW conclusions expressed in this work Bank encourages dissemination Washington, DC 20433 do not necessarily reflect the views of of its knowledge, this work may be Telephone: 202-473-1000 The World Bank, its Board of Executive reproduced, in whole or in part, for Internet: www.worldbank.org Directors, or the governments they noncommercial purposes as long as full represent. attribution to this work is given. Attribution—Please cite the report as World Bank. 2018. Engaging Citizens The World Bank does not guarantee Any queries on rights and licenses, for Better Development. Independent the accuracy of the data included in including subsidiary rights, should be Evaluation Group, Washington, DC: this work. The boundaries, colors, addressed to World Bank. denominations, and other information World Bank Publications, The World shown on any map in this work do not Bank Group, 1818 H Street NW, imply any judgment on the part of The Washington, DC 20433, USA; fax: World Bank concerning the legal status 202-522-2625; e-mail: pubrights@ of any territory or the endorsement or worldbank.org. acceptance of such boundaries. Nirmal Hurry, Claiming Identity II (detail shot), 2012, fibers, terracotta and porcelain. Artist from Mauritius. From the World Bank’s Art Collection Nirmal Hurry received his MA in fine arts from the Jamia Millia Islamia University New Delhi as well as a Dilplome Superieur d’ Art Plastique from Ecole Nationale Superieur des Beaux Arts, Paris. In this piece, black, beige, white, and red hands are grasping fibers. The hands are cut off at the wrists. With this lack of connection to anything larger than itself, each hand gains attention for its individual color, the way the fingers fold, and its contribution to a larger wave-like composition. The variety yet repetition of colors, expressive gestures, and placement of the hands bobbing in and out of the fibers hint at the merging of difference, and sameness. The artist states that the complex multi-cultural Mauritian society is a source of inspiration, and that this piece is “an exploration of the ambiguities of the social issues of fragmentation within the society.” Engaging Citizens for Better Development Results AN INDEPENDENT EVALUATION Careful observation and analysis of program data and the many issues impacting program efficacy reveal what works as well as what could work better. The knowledge gleaned is valuable to all who strive to ensure that World Bank goals are met and surpassed. contents abbreviations  v acknowledgments  vi overview  vii management response  xviii management action record  xxvi report to the board from the committee on development effectiveness  xxxi 1. Introduction  1 The Landscape for Citizen Engagement and Scope of the Evaluation  2 Evaluating the Bank Group Support to Citizen Engagement  5 2. Meeting the Target  11 the Results and Performance of 13 Embedding Citizen Engagement in Investment Operations  17 Bringing Citizens to the Table in Policy and Strategy Discussions  Monitoring the Effort  World Bank Group 2015 18 Implementing the Mandate: Organization, Resources, and Incentives  21 AN INDEPENDENT EVALUATION 3. Enhancing the Quality of Citizen Engagement  26 Aiming for Results  29 Continuous Engagement and Closing the Feedback Loop  30 Ensuring Inclusion  32 Building Capacity to Engage  35 4. Engaging for Results  40 Behavior and Relationship Change  42 Operational Change  45 Institutional Change  46 Supporting a Higher Level of Corporate Ambition  49 Careful observation and analysis of program data and the many issues 5. Conclusions and Recommendations  impacting program 54 efficacy reveals what works as well as what References58 could work better. The knowledge gleaned is Boxes valuable to all who strive to ensure that World  our Lines of Inquiry Guiding the Evaluation  Box 1.1 | F 6 and Bank goals are met surpassed.  bjective of the Strategic Framework  Box 1.2 | O 7 Independent Evaluation Group | World Bank Group iii  takeholder Engagement in International Finance Corporation Performance Standards  Box 2.1 | S 16  he Challenge of Inclusion in Lao People’s Democratic Republic SUFORD Project  Box 3.1 | T 34 Concrete Operational Changes in a Conditional Cash Transfer Program  Box 4.1 |  47 Building on the Strength of the Strategic Framework  Box 4.2 |  50 Synergies and Differences between the New Environmental and Social Framework and the Box 4.3 |  Citizen Engagement Agenda  53 Figures  oad Map of the Evaluation Report  Figure 1.1 | R 8  itizen Engagement Mechanisms Used  Figure 2.1 | C 15  ommunity Engagement Mechanisms in International Finance Corporation Planned Investments  17 Figure 2.2 | C Discrepancy between What Is Planned and What Is Tracked (FY11–16)  Figure 2.3 |  20  ain Constraints to the Implementation of the Agenda, According to Staff  Figure 2.4 | M 23 Staff and Management Perception of Resource Adequacy for Type of Engagement  Figure 4.1 |  52 Table Table 1.1. Main Entry Points for Citizen Engagement  4 Table 4.1. Path to Results: Typology of Proximate and Intermediary Outcomes  42 Table 4.2. Observed Results of Citizen Engagement Mechanisms in Project Case Studies  44 Appendixes Appendix A. Methodological Approach  64 Portfolio Review of Citizen Engagement Activities in Investment Project Financing  83 Appendix B.   itizen Engagement in World Bank’s Development Policy Lending  Appendix C. C 103 Analysis of Citizen Engagement Activities in International Finance Corporation Investment Projects 119 Appendix D.   itizen Engagement in the Bank Group’s Country Engagement Model  Appendix E. C 135 Appendix F. Staff Survey  144 Appendix G. Partnership Review of the Global Program for Social Accountability Program  154 iv Engaging Citizens for Better Development Results | Contents abbreviations CDD community-driven development CPF Country Partnership Framework CSO civil society organization CSR corporate social responsibility DPF development policy financing E&S Environmental and Social ESF Environmental and Social Framework FY fiscal year GP Global Practice GPSA Global Partnership for Social Accountability GRM grievance redress mechanism ICR Implementation Completion and Results Report IEG Independent Evaluation Group IFC International Finance Corporation IPAC Participatory Anti-Corruption Initiative IPF investment project financing MIGA Multilateral Investment Guarantee Agency NGO nongovernmental organization OGP Open Government Partnership OP Operational Policy PS Performance Standard PSFM participatory sustainable forest management RC Reportes Comunitarios SCD Systematic Country Diagnostic USAID United States Agency for International Development All dollar amounts are U.S. dollars unless otherwise indicated. Independent Evaluation Group | World Bank Group v acknowledgments This evaluation was prepared by an Independent Evaluation Group (IEG) team led by Elena Bardasi under the overall direction of Caroline Heider, director-general, Evaluation, and with the guidance and supervision of Emanuela Di Gropello, manager, Corporate and Human Development, and Auguste Tano Kouame, director, Human Development and Economic Management. The core team comprised Estelle Raimondo, who cowrote the report and oversaw the overall methodology design and implementation, including the production of case studies; Eduardo Fernandez Maldonado, who managed the investment project financing portfolio review; and Gisela Garcia, who conducted the Country Partnership Framework analysis. The team also comprised Sahra Abdi Nur; Anahit Anna Aghumian; Anna Amato; Viengxong Chitavong; Shana Cohen; Francesco Cuomo; Anis Dani; Fredrik Galtung; Anteneh Girma; John Heath; Lauren Kelly; Sama Khan; Gurkan Kuntasal; Javier Lanza Ferrera; Chris Nelson; Kimberly Parekh; Aryamala Prasad; Mari Noelle Roquiz; Pallavi Sengupta; Kathryn Steingraber; Giovanni Tanzillo; Daniel Walker; Maurya West Meiers; Qijin Wu; and Jingwen Zheng. Rasmus Heltberg was co-task team leader of the evaluation until April 2017 and continued to provide advice to the team. Marie Gaarder provided guidance and supervision as manager of Corporate and Human Development until September 2017. Administrative support was provided by Yezena Yimer, Carla Fabiola Coles, and Lalrinpari Sailo. The editor was Amanda O’Brien. The report was peer reviewed by Cyprian Fisiy (former World Bank Director); Jonathan Fox (Professor, School of International Service, American University); and Ritva Reinikka (former World Bank director). The team is grateful to the many staff who engaged with us and gave generously of their time, especially the counterparts in the Citizen Engagement Secretariat, and to all those who accepted interviews, responded to our surveys, and participated in focus group discussions. Thanks are also due to the country offices of the Dominican Republic, Ethiopia, the Kyrgyz Republic, the Lao People’s Democratic Republic, Mali, Morocco, Pakistan, and West Bank and Gaza for the support provided during our country visits. vi Engaging Citizens for Better Development Results | Abbreviations highlights 1 The World Bank Group has progressively integrated citizen engagement in lending instruments, country strategies preparation, and policy formulation. The 2014 strategic framework Overview made this agenda a top priority for the institution. The concept of engagement evolved from an initial focus on risk management to a more proactive interaction with citizens. 2 The corporate commitment to mainstream citizen engagement generated awareness and buy-in among senior management and staff; this change in mind-set is a major achievement of the strategic framework. The number of World Bank projects with citizen engagement mechanisms and indicators has increased substantially over the past few years. 3 The flipside of the corporate emphasis on mostly quantitative targets has been the limited application of quality standards in the design, implementation, and monitoring of citizen engagement. Projects increasingly resort to multiple mechanisms and pay closer attention to inclusion; yet, in most cases, engagement remains “thin” and capacity building for engagement is limited. Crucially, closing the feedback loop with citizens is not properly mainstreamed or monitored. vii 4 The share of projects that include beneficiary feedback indicators in their results frameworks has increased markedly since the introduction of the corporate commitment. However, indicators rarely track results, many mechanisms are not measured in results frameworks, and reporting is still insufficient. 5 The International Finance Corporation leveraged Performance Standard 1 to encourage its clients to engage more systematically with stakeholders throughout their business cycle. In general, reporting provides little information on implementation and results. 6 The corporate arrangements in place to oversee and coordinate the commitment work well, but limited dedicated staff time is available to operationalize the approach beyond ensuring compliance with the corporate target. 7 Achieving “thick” citizen engagement to scale is an ambitious agenda requiring a more selective and strategic approach that responds to local context and client demand. A concerted effort will be needed to enhance capacity building and learning, mobilize resources, and achieve better synergies with other relevant agendas. viiiviii Section Engaging Citizens for Better Development Results | Overview Citizen Engagement in the World Bank Group C I T I Z E N E N G AG E M E N T H A S A LO N G H I S TO RY in the World Bank Group. Engagement with “stakeholders,” “communities,” “beneficiaries,” or “citizens” has been an integral part of the Bank Group’s operational activities since the 1980s, with the introduction of environmental and social safeguards as part of the design of projects affecting indigenous peoples. Collaboration with beneficiaries in operational work followed in the 1990s, and consultations with stakeholders in policy and strategy formulation at the macrolevel started in the mid-1990s. The concept of “engagement” evolved from reputational risk management to more proactive interaction in operations and macropolicies and strategies. The evolution of citizen engagement activities in the Bank Group culminated in a corporate commitment in 2013 and the adoption in 2014 of a strategy outlined in the Strategic Framework for Mainstreaming Citizen Engagement in World Bank Group Operations. In 2013, the Bank Group committed to integrate, by fiscal year (FY)18, beneficiary feedback into 100 percent of investment projects where beneficiaries can be clearly identified. The 2014 strategic framework aimed to provide a more coherent approach to incorporating citizen engagement across Bank Group operations. The framework builds on lessons from Bank Group–financed operations across regions and sectors and underscores the importance of country context, government ownership, and clear objectives for citizen engagement. Several recent developments have confirmed the Bank Group’s commitment to a more holistic approach to citizen engagement. These include the new country engagement model introduced in 2014, the World Development Report 2017: Governance and the Law, and the recently reformed Environmental and Social Framework (ESF), which introduces an Environmental and Social Standard Definition of Citizen Engagement on stakeholder engagement and Citizen engagement denotes multiple interactions information disclosure. The ESF between citizens and governments, or the private reform follows the 2012 Performance sector, within the scope of World Bank Group’s Standards on Environmental interventions. It is a two-way relationship that and Social Sustainability update, implies the existence of a tangible response which expanded the magnitude to citizens’ feedback. As described in official of stakeholder engagement in Bank Group statements, the objective of International Finance Corporation (IFC) mainstreaming citizen engagement in operations operations. is to give citizens a stake in decision making to This corporate evaluation aims improve development outcomes. to inform the Board of Executive Independent Evaluation Group | World Bank Group ix Directors and the management of the Bank Group institutions on the effectiveness of mainstreaming citizen engagement to support development processes and outcomes. It allows for an assessment of the early implementation of this important corporate agenda and intends to provide a timely contribution to any review and update of the corporate citizen engagement goals, targets, and approaches on completion of the 2018 corporate citizen engagement commitment. The overarching question of this Independent Evaluation Group (IEG) evaluation is, “How effectively has the Bank Group mainstreamed citizen engagement at the project, country, and corporate levels, and what is the evidence on how this process contributes to the achievement of development outcomes?” The evaluation is structured around the conceptual framework found in the 2014 Strategic Framework for Mainstreaming Citizen Engagement in World Bank Group Operations. It is informed by the Bank Group rationale stated therein for involving stakeholders and citizens: “Supporting client engagement with citizens where such engagement can improve development outcomes is a key component of the World Bank Group’s strengthened focus on results” (5). The evaluation adopted the five principles for mainstreaming citizen engagement in Bank Group–supported operations proposed by the strategic framework: focusing on results, engaging throughout the operational cycle, seeking to strengthen country systems, applying context specificity, and using a gradual rollout approach, as well as the additional cross-cutting principle of ensuring inclusion. The evaluation adopted a multilevel, mixed methods, and case-based design. To gauge the extent of mainstreaming across Bank Group instruments and assess trends over time, the evaluation team conducted six portfolio reviews. To deepen the findings, the evaluation team conducted 8 case studies of country-level engagement and 40 case studies of engagement mechanisms embedded in 19 projects across 11 countries, corroborated by evidence from literature reviews. To assess the corporate environment, the evaluation team conducted a survey and interviewed staff and management and participated in an online consultation with civil society organizations (CSOs). Embedding Citizen Engagement in World Bank and IFC Projects With the introduction of the corporate commitment, the institution succeeded in raising awareness, especially internally, of the importance of citizen engagement in the Bank Group’s work. Staff and management agree that citizen engagement is a responsibility of the Bank Group and a useful strategy to strengthen accountability in service delivery, mitigate risks, and anticipate problems. This change in mind-set, whereby the value of engaging citizens is now encouraged across the Bank Group’s work, is a major achievement. Progress on the commitment to achieve 100 percent mainstreaming of citizen engagement in investment projects where beneficiaries can be clearly identified is tracked using three process indicators: the percentage of investment projects with a “citizen-oriented design” (having at least one citizen engagement mechanism among the project activities); the percentage of investment projects with at least one beneficiary feedback indicator in their results framework; and the percentage of projects that report on a beneficiary feedback indicator (the latter is still in the process of being rolled out). x Engaging Citizens for Better Development Results | Overview The corporate committment helped increase the percentage of projects with citizen engagement mechanisms, especially those that are not safeguards related. Based on a portfolio of 299 randomly selected World Bank investment projects, the percentage of projects with non-safeguards-related citizen engagement mechanisms has increased from 67 percent to 76 percent between FY11–13 and FY14–16, likely owing to the increased buy-in of the mandate. More than 90 percent of investment projects approved during FY14–16 plan to use at least one citizen engagement mechanism—that is, have a citizen-oriented design. Although more engagement mechanisms are included in project design, their type has not changed much from before the introduction of the corporate mandate. Consultations and grievance redress mechanisms (GRMs) remain the most common mechanisms. Moreover, mechanisms implying a light degree of engagement (“informing” and “consulting”) are much more frequent than more intense forms of engagement (“collaborating” and “empowering”). Crucially, IEG found that the definition of citizen-oriented design and the corporate-level indicators used to track progress do not systematically consider whether the feedback loop was closed. IFC and the Multilateral Investment Guarantee Agency (MIGA) have systems for approaching stakeholder and community engagement that differ from the World Bank’s. These institutions require clients to adhere to a set of eight Performance Standards (PSs), which constitute an integrated sustainability and risk management framework, revised in 2012 by IFC and 2013 by MIGA. This evaluation analyzed a portfolio of IFC investment projects from FY07–11 (137 projects) and FY15–17 (30 projects). It excluded MIGA because of the highly intermediated relationship between MIGA and final beneficiaries. Since the 2012 update of the PSs, IFC has expanded the magnitude and improved the depth of its stakeholder engagement, in accordance with PS 1 (Assessment and Management of Environmental and Social Risks and Impacts), which requires stakeholder engagement to occur in all investment projects, with special attention to riskier ones. Three positive patterns emerge: First, IFC clients more frequently plan to engage strategically with communities, via a range of mechanisms. Second, the mix of activities changed to include more proactive activities rather than reactive and risk-mitigation ones, such as GRMs, throughout the project cycle. Third, since the 2012 update, IFC has more systematically promoted forms of stakeholder engagement directly related to clients’ business and the project activities, and clients have progressively moved away from corporate social responsibility as the only form of engagement. Despite this progress, IFC clients do not fully comply with PS 1. Several projects are still without GRMs or a system for disclosing project information—both required by PS 1, particularly for projects that may have an adverse impact on local communities. Stakeholder engagement plans are absent in the majority of IFC projects approved between FY15 and FY17. Bringing Citizens to the Table in Policy and Strategy Discussion The Bank Group has started to leverage the new country engagement model adopted in 2014 to consult with a wider range of stakeholders during the preparation of the Systematic Country Independent Evaluation Group | World Bank Group xi Diagnostics and Country Partnership Frameworks (CPFs). This evaluation reviewed all 46 CPFs approved between FY15 and FY17 and found that almost all of them refer to having consulted with civil society and, increasingly, with the private sector. Development policy financing (DPF) is not systematically used to support the broader citizen engagement agenda. Based on IEG’s review of a sample approved between FY11 and FY16, almost all DPF included some reporting on consultations with stakeholders during the preparation phase, as required by operational policy. However, most consultations did not focus specifically on the reforms at stake but were conducted in the framework of the discussion of a country’s national development plan or poverty reduction strategy. Moreover, very scant information exists on the consequences of the consultations undertaken. Monitoring and Reporting The share of projects that include a beneficiary feedback indicator in their results framework has increased markedly over the past two years. Before the corporate commitment, 42 percent of projects had at least one such indicator; after the introduction of the commitment, this increased to 63 percent in FY14–16, with a steady annual increase up to 95 percent in FY16. This increase hides three significant challenges. First, the level of reporting during implementation is low. Only 57 percent of the projects approved in FY14–17 with engagement indicators in their results frameworks have reported on at least one of the citizen engagement indicators included in their results frameworks. Second, a major discrepancy exists between mechanisms included in project design and what the organization reports on. Consultations and GRMs are the most common mechanisms but the least measured ones. Third, very little is known about the contribution of citizen engagement to project outcomes based on Bank Group monitoring, as most indicators are process or output oriented, and reporting in self-evaluation is scarce. IFC reporting is driven by compliance with PSs and focused more on processes and outputs rather than outcomes achieved. IFC does not systematically enforce high reporting standards; IEG found that a fourth of projects provide too little information to even understand how the client engaged with the community. Reporting is slightly more complete in high-risk projects, although also generally insufficient in this case. Enhancing Citizen Engagement Quality This evaluation assessed the extent to which the Bank Group’s citizen engagement activities incorporate four quality principles: aiming for results, closing the feedback loop, ensuring inclusion, and building citizens’ and governments’ capacity to engage. Aligning closely with the empirical literature on participation and social accountability, the strategic framework identified high-level principles of engagement that should underpin the design and implementation of engagement activities in a context-sensitive way. The case studies undertaken for this evaluation corroborate the xii Engaging Citizens for Better Development Results | Overview literature and shed light on the characteristics of citizen engagement that are more likely to contribute to improved development outcomes. Aiming for results. Since the introduction of the strategic framework, more World Bank citizen engagement activities combine multiple tools, an approach likely to improve the quality of engagement. Case studies conducted for this evaluation confirm the literature finding that “thick” approaches—those combining multiple tools to enable collective action and public sector responsiveness—are more promising than “thin” approaches—those that are not matched with vertical integration of independent monitoring and oversight or do not include support to increase a government’s capacity to respond. In the World Bank’s portfolio, the share of projects with thick engagement has increased from 27 percent in FY11–13 to 38 percent in FY14–16 and even more markedly in FY17. Closing the feedback loop. World Bank projects are more likely to plan at design how to act on feedback collected during citizen engagement activities than to inform citizens about the actions taken following the consultations. The empirical literature, echoed by this evaluation’s case studies, provides robust evidence that giving a tangible response to citizen feedback is fundamental to producing results, sustaining participation, and improving trust. In investment lending, there is some confusion on how “closing the feedback loop” should be measured. When this is interpreted as “acting on the feedback of the citizens,” as per Bank Group’s guidance notes, IEG found that about 70 percent of the mechanisms included in a sample of FY16 projects met this criterion. The compliance rate was drastically lower (only 4 percent) when closing the feedback loop is interpreted as “informing those engaged how the information they provided has been used,” as per the definition included in the strategic framework. Ensuring inclusion. The World Bank’s portfolio is showing encouraging progress in the level of attention dedicated to inclusion at project design, a critical feature to prevent elite capture, but there is still room for improvement. The share of project designs describing how the participation of women and vulnerable groups would be ensured for at least one engagement mechanism increased from 40 percent in the FY11–13 sample to 58 percent in the FY14–16 sample. Yet, only a minority of projects report at least one beneficiary feedback indicator disaggregated by gender or other social group. The early implementation of the new country engagement model foreshadows a more inclusive process that reaches a broader set of stakeholders than in the past. CPF teams now consult much more frequently with local governments and local nongovernmental organizations. In some cases, partnering with local CSOs helped improve the outreach and quality of consultations. Building citizens’ and governments’ capacity to engage. Citizen and state actors’ lack of capacity and willingness to engage represent major constraints to successful engagement per the literature and Bank Group staff. Creating capacity requires a long-term investment that, although not feasible in all projects, is essential to improving collective action and producing lasting change beyond the life of the project. Yet, less than a quarter of projects (23 percent) reviewed had at least one engagement mechanism that included capacity enhancement activities in their design. Independent Evaluation Group | World Bank Group xiii The Global Partnership for Social Accountability (GPSA) is one of the few channels that the World Bank has for building capacity, but its potential is still underexploited. Grants provided to CSOs have been used to build the capacity of a wide network of CSOs and local community groups in, for example, social monitoring or local budgeting processes. Many grantees report constructive engagement with local government offices in charge of overseeing delivery of public services. However, despite positive exceptions, better synergies between GPSA grants and World Bank projects taking place in the same country would have helped the World Bank extend its engagement with CSOs in broader governance reforms, while improving the prospects of institutionalizing GPSA- supported social accountability mechanisms and sustaining their results. GPSA’s global level efforts in gathering and sharing practical knowledge on social accountability is impressive but could be better used by the World Bank Group. Engaging for Results Engaging citizens in development operations can have a tangible impact on the quality of services and on development outcomes. Evidence from this evaluation’s case studies agrees with the literature that if the conditions of high-quality design and implementation are met, and activities are well-embedded in the local context, impact on development outcomes is more likely (and vice versa). Because a causal link between mechanisms used to engage citizens and the development outcomes is hard to establish, this evaluation focused on intermediate changes at the level of behaviors and relationships; operations; and institutions. Behavior and relationship change. The case studies conducted for this evaluation found that service providers are more responsive when citizen engagement mechanisms succeed in providing an interface between citizens and service providers, so that they can jointly identify problems and find solutions. Increased sensitization of citizens and service providers can reduce conflicts and increase ownership of the project. The improvement of trust in public services can be achieved, according to the literature, through rigorous participatory methodologies embedded in durable institutional systems that can bring resolution to citizens’ feedback and complaints. IEG observed this in several case studies. Operational change. The Bank Group is not using citizens’ feedback sufficiently to adapt its strategies, portfolio composition, and project design and implementation in response to citizens’ demands. IEG found that stakeholder consultations often confirmed preidentified priority areas of attention and only sporadically triggered a change in CPF priorities, based on analysis of the documents and interviews with CPF task leaders. At the project level, citizen feedback can strengthen operations by informing project design or triggering course correction, including in how services are delivered, but this was not found to be the norm. Institutional change. Institutional change is important to sustain citizen engagement, and several entry points exist to trigger this change. Evidence from the structured reviews and case studies showed that the Bank Group can facilitate institutional changes by (i) convening multiple voices in xiv Engaging Citizens for Better Development Results | Overview policy dialogue in the formulation of CPFs; (ii) triggering reforms that create an enabling institutional environment for citizen-state collaboration through DPF prior actions; and (iii) using investment projects to spark institutional changes that can reinforce country systems through scale-up or replication. The Bank Group used these entry points to varying extents. IEG found several examples where the Bank Group promoted citizen engagement through its country strategies; it used its convening power to bring citizens to the policy discussion table outside of the CPF consultations less frequently. A good example of the latter is the Dominican Participatory Anti-Corruption Initiative, which facilitated a long-term dialogue between government and civil society. As for DPF, only 10 percent of all prior actions in DPF since the early 1990s have broadly supported citizens’ capacity to act by improving access to law and justice; introducing conflict prevention mechanisms; or enhancing state capacity to respond to citizens’ demand through decentralization or judicial or civil service reforms. Finally, the evaluation identified a few good examples of projects that strengthened country systems for engagement and sectorwide operations that embedded citizen engagement practices. Internal Arrangements for Moving the Citizen Engagement Agenda Forward A tension remains between meeting the mostly quantitative corporate targets and ensuring the quality of engagement. The commitment to increase beneficiary feedback in 100 percent of projects has created momentum toward better engaging citizens, but the emphasis on the corporate indicators and their targets risk generating a “check-the-box” attitude, to the detriment of quality. The structure to support operational teams and monitor progress in implementing the mandate is working, but there is still margin for improvement. The Citizen Engagement Secretariat has implemented the monitoring of the corporate requirement with good results. The secretariat worked on operationalizing definitions to screen activities and projects. The regional citizen engagement focal points have been effective in providing expertise and support to operational teams. The sectoral focal points have not been as effective. In general, Global Practices have been less active than the Regions in providing guidance and gathering knowledge of what works. This evaluation identified areas for the institution to improve to fulfill the ambition of the strategic framework and move the agenda to the next level. These areas include improving monitoring, creating opportunities for thick engagement, investing in generating evidence, strengthening staff capacity, and exploiting internal synergies. Improving monitoring. Overall, the institution is not sufficiently monitoring key elements identified in the strategic framework as essential to ensure successful integration of citizen engagement in projects. The quality of engagement is poorly tracked; documenting whether and how the feedback loop was closed (including by identifying the right indicators in results frameworks) requires more attention. Not only does this generate inaccurate reporting of progress, it also provides little incentive to teams to “raise the bar” and deprives the institution of systematic information from which to learn. Independent Evaluation Group | World Bank Group xv The Quality Index adopted in Europe and Central Asia is a promising tool to improve the tracking of key quality characteristics. Creating opportunities for thick engagement. The institution has positively responded to the corporate mandate by embracing citizen engagement as a regular practice in Bank Group’s work. Although it is crucial to discourage a perfunctory adoption of the strategic framework, it is also important to recognize that regular and continuous thick engagement that uses multiple tools and is embedded in country systems cannot realistically be achieved in all projects. Creating strategic opportunities for this type of engagement is essential, but these are best identified at the country or regional level, as the goals and implementation modalities of thick engagement are highly contextual and influenced by client demand and local capacity. Generating evidence. For a results-driven agenda, little investment has been made in generating robust evidence of what works, where, and why. There is room to coordinate fragmented research initiatives and to promote systematic investment for transformational approaches, including application of new tools, such as information and communication technologies. Building capacity. Key tasks such as building internal and external capacity are not systematically provided, and this may pose risks to implementation and sustainability. Limited capacity and insufficient training are among the most relevant internal and external constraints to the agenda. Creating a critical mass of experts to regularly support Bank Group teams and clients is crucial and emerges as a priority. Improving synergies with other relevant agendas, primarily the ESF. Important connections can be established between the citizen engagement and other agendas of the Bank Group, such as social inclusion, gender, and especially the new ESF. The recent ESF reform, approved in August 2016 and to become operational over the next two years, moves safeguards from a predominantly risk management perspective to a more proactive engagement with stakeholders and provides the closest and strongest connection with the citizen engagement agenda. The ability of the Bank Group to bridge the two agendas can raise the level of ambition in terms of depth and quality of citizen engagement activities and meet expectations of many in the World Bank. Recommendations The Bank Group has succeeded in making citizen engagement a top priority and creating awareness and buy-in. However, aspects related to quality are typically not given sufficient attention at the design and monitoring stages. To sustain and deepen the effectiveness of the Bank Group’s citizen engagement efforts, the evaluation proposes the following: Recommendation 1. As it defines future corporate priorities for citizen engagement, the World Bank should reflect in those priorities the need to achieve greater depth and quality of the citizen engagement activities it supports. This will entail building on the progress achieved under xvi Engaging Citizens for Better Development Results | Overview the strategic framework to promote a more strategic approach that incorporates deeper citizen engagement where opportunities arise. These future corporate priorities will need to be clearly communicated to staff and stakeholders. Recommendation 2. The World Bank should encourage and support efforts of its regional, country and Global Practices teams to establish, where appropriate, thick citizen engagement that is regular and continuous, uses multiple tools, and is embedded in country systems. This could be achieved by more systematically using existing channels of dialogue and stakeholder engagement (such as Systematic Country Diagnostics, CPFs, and DPF) and applying tools (such as road maps and indexes) to plan, monitor, and assess results achieved at the various levels (Region, country management unit, Global Practice). This would entail better synergies between World Bank operations and other programs that support citizen engagement, including GPSA-financed activities; mobilizing adequate expertise to support World Bank teams; investing in training for staff and stakeholders; and improving knowledge management. Recommendation 3. The World Bank should strengthen the monitoring of its citizen engagement activities by systematically adopting results framework indicators that are results oriented. This will entail tightening the alignment between the citizen engagement mechanisms used in projects and the indicators that measure their quality and results, with special emphasis on indicators that show how feedback loops were closed and how diverse groups of stakeholders were included. Recommendation 4. The World Bank should seize the opportunity of the implementation of the ESF to leverage citizen engagement mechanisms—beyond consultations and GRMs—to reach the objectives of managing social risks, strengthening country systems, and promoting social inclusion. This will entail better drawing on the existing expertise on citizen engagement and making citizen engagement more prominent in the ESF trainings, both for staff and for project implementation units. Recommendation 5. IFC should ensure that its clients’ stakeholder engagement activities required by PS 1 in projects with affected communities are carried out during appraisal and supervision of the projects and systematically documented. This will entail mobilization of adequate expertise to systematically support clients and reporting that is comprehensive and results focused. Independent Evaluation Group | World Bank Group xvii management response World Bank Management Response The management of the World Bank welcomes the Independent Evaluation Group (IEG) evaluation, Engaging Citizens for Better Development Results. The evaluation outlines an overall positive review of the rollout of the first phase of the Strategic Framework for Mainstreaming Citizen Engagement in the World Bank Group.1 The evaluation provides helpful lessons to inform the future direction of citizen engagement at the World Bank. General Comments Management appreciates the report’s many positive findings about the World Bank’s increasingly proactive engagement with citizens. As the report states, the World Bank has progressively integrated citizen engagement in lending instruments, country strategy preparation, and policy formulation. Citizen engagement has evolved from initially focusing on risk management to a more proactive interaction with citizens as a way of improving development outcomes and promoting sustainability. Management welcomes the report’s findings that the strategic framework has played a key role in fostering institutional uptake and that a “change in mind-set is a major achievement of the strategic framework.” As the strategic framework highlights, citizen engagement can help governments achieve sustainable development results. Successful mainstreaming of citizen engagement facilitates this. Mainstreaming citizen engagement is a process in which the strategic framework was an important milestone. The implementation of the strategic framework started in mid-FY15, with full implementation in FY16. The focus of the initial implementation has been on the corporate commitment to beneficiary feedback, introduced by the president in 2013. Following the first phase of implementing the corporate commitment, a large number of investment project financing (IPF) operations now have citizen engagement mechanisms at the design stage. The second phase of the corporate commitment, now being rolled out, focuses on citizen engagement during IPF implementation. The strategic framework outlines entry points for deepening government engagement with citizens across the World Bank’s product portfolio, and management is committed to making the comprehensive offer of these entry points more attractive to governments. There are entry points to strengthen government’s engagement with citizens across the diagnostic (Systematic Country Diagnostic), strategy (Country Partnership Framework [CPF]), and portfolio levels (trust funds, Program-for-Results, IPF, development policy financing, policy dialogue, and Advisory Service and Analytics). There is room for a broader and more systematic approach in offering governments entry points at all three of these levels across the World Bank portfolio. The country engagement model can be a starting point for this approach, particularly when countries identify citizen engagement as a priority that can be supported with a variety of instruments across the engagement cycle. Monitoring and Reporting on Beneficiary Feedback The report finds that the share of IPFs that include beneficiary feedback indicators in their results frameworks has increased markedly over the past two years, a major achievement of the strategic framework. Before the corporate commitment, 42 percent of IPFs had at least one xviii Engaging Citizens for Better Development Results | Management Response beneficiary feedback indicator. This share increased to 63 percent in FY14–16, with steady annual increases to 95 percent in FY16. The World Bank is now rolling out the next phase of monitoring the corporate commitment to beneficiary feedback, capturing citizen engagement during the implementation of IPFs. This next phase is particularly important in terms of monitoring the two-way interaction of providing a response to citizen feedback (“closing the feedback loop”). The current guidance is consistent with the strategic framework, and during the rollout of this second phase, management will provide support to task teams, disseminating guidance to them and strengthening the guidance as necessary. In coordination with the Citizen Engagement Secretariat, the Regional and Global Practice citizen engagement focal points will continue to assist operational teams to design citizen engagement mechanisms and corresponding beneficiary feedback indicators. The World Bank works continuously to improve the results focus of its indicators. Management appreciates the report’s emphasis on strengthening the results orientation of the citizen engagement indicators, closing the feedback loop, and including diverse groups of stakeholders. In the West Bank and Gaza Municipal Development Project, for instance, it has mainstreamed citizen engagement in a context of fragility, conflict, and violence, and developed a results chain from citizen engagement to outcomes. As a result of this outcome-focused approach, the client is better able to quantify the relationship between citizen engagement mechanisms and performance toward the objectives. In measuring results, it is often important to pay specific attention to inclusion of disadvantaged groups (for example, women, the poorest), and the report finds that operations are increasingly doing so. Progress is noted in the level of attention to inclusion throughout the World Bank’s portfolio.2 In a total of 24 service delivery IPF and Program-for-Results operations in International Development Association countries approved during FY18, eight included gender-inclusive beneficiary feedback indicators in their results frameworks at design. For example, Ethiopia’s Enhanced Shared Prosperity through Equitable Services Project includes a disbursement-linked indicator that requires the participation of women in prebudget discussions at the district level so that citizens’ voices are reflected in the budget cycle.3 While management recognizes that there is room for improvement in advancing inclusive citizen engagement—for example, greater participation of women and other traditionally marginalized groups—in beneficiary feedback indicators, it is encouraging that more operations are addressing inclusive citizen engagement according to their project appraisal documents. Quality and Selectivity in Approaching Citizen Engagement The World Bank is continuing to work on improving the quality of citizen engagement across entry points—for example, by assisting countries in strengthening their own systems and capacity to engage with citizens and by engaging more comprehensively where country conditions allow. As the IEG evaluation notes, this approach will require selectivity and trade-offs. The Social Development Unit in the Europe and Central Asia Region has developed the Citizen Engagement Quality Index, which is used to assess the quality of citizen engagement at project design. It looks at a number of parameters—depth, opportunity, frequency, and scope—and will be expanded to inclusion and Independent Evaluation Group | World Bank Group xix responsiveness. The index was applied to all 200-plus active projects in the Region. The results of this indexing largely show that quality has improved: frequency of engagement has improved, the number of channels for engagement has increased, the restrictions on the scope of engagement or feedback are being removed, and most significantly, there are more direct interfaces between citizens and the state. Management recognizes the importance of building citizens’ and governments’ capacity to engage as a key element in enhancing quality, but this requires time and commitment. The goal is to strengthen government policies and systems that enable greater engagement with citizens. Capacity building for citizen engagement is a long-term endeavor, but it offers countries the opportunity to improve development outcomes and strengthen stability. Capacity building can be supported across the full range of World Bank instruments, although, as the report recognizes, major capacity building aspects are not feasible in all projects. The World Bank is committed to support capacity building focused on citizen engagement tools and mechanisms that are customized to fit the project context—for example, peer-to-peer exchanges, trainings, clinics for project implementation units, in-country workshops, and production of learning materials in different languages. Management agrees that comprehensive (what the report calls “thick”) citizen engagement— which is regular and continuous, combines multiple tools to enable collective action, and is embedded in country systems—supports sustainability beyond the project cycle and agrees that it needs to be selective. Since the introduction of the strategic framework, more World Bank operations with citizen engagement activities combine multiple tools, and—as outlined in the strategic framework and confirmed in the IEG evaluation—they are more likely to improve the quality of engagement. According to the report, the share of IPFs reviewed by IEG in the World Bank portfolio with “thick” engagement has increased from 27 percent in FY11–13 to 38 percent in FY14–16, and to 56 percent in FY16. Considering that full implementation started only in FY16, we can expect this positive trend to continue. Comprehensive citizen engagement is resource intensive. Client demand and context specificity are key factors for all citizen engagement, especially when attempting to deepen engagement, so the need for selectivity is clear. Management appreciates the finding that the country engagement model is a good basis for a more inclusive citizen engagement process. The model provides opportunities for wide citizen participation in defining challenges and priorities for World Bank engagement through the Systematic Country Diagnostic and CPF processes. Identification of citizen engagement as a key constraint or development priority is a strong entry point for the World Bank to offer a comprehensive package of support across instruments and use the CPF monitoring process to track progress. For example, Guinea, Guinea Bissau, and Mali adopted citizen engagement as one of the pillars of their CPF, providing a good basis for comprehensive engagement. Management would like to emphasize that governments are responsible for engaging with citizens when designing a development policy financing project. The responsibility to initiate a participatory process and design its scope rests with the government, but the World Bank can facilitate, support, and advise on the engagement. xx Engaging Citizens for Better Development Results | Management Response Link to Environmental and Social Framework It is important to delineate the differences between the requirements of the new Environmental and Social Framework (ESF) in IPFs and broader citizen engagement. Stakeholder engagement as addressed in the ESF is a specific aspect of the broader citizen engagement. The ESF uses two main avenues of stakeholder engagement (consultations and grievance redress), while the strategic framework has a broader menu with seven approaches along a continuum of citizen involvement in decision making (consultations, grievance redress, reporting on citizens’ input, collaboration, empowerment, and capacity building). The ESF obligates the borrower, while citizen engagement is developed through a dialogue with the borrower. In the context of the ESF, stakeholder engagement is a continuing and iterative process by which the borrower facilitates a dialogue on the project’s environmental and social (E&S) risks and impacts with both the people affected by its decisions and activities and others who have an interest in the implementation and outcomes of the project. Stakeholder engagement provides opportunities for borrowers to inform their decision-making on E&S risk management matters for a project; where appropriate, to course-correct from the experience, knowledge, and concerns of the affected and interested stakeholders; and to manage stakeholders’ expectations by clarifying the extent of the borrower’s responsibilities and resources. The main avenues for conducting stakeholder engagement as described in the ESF are meaningful consultations (facilitated by information disclosure) and grievance redress. Through the ESF, the World Bank will help borrowers build capacity to develop their own systematic approach to integrating stakeholder feedback into development projects. The Environmental and Social Standard 10 on stakeholder engagement requires borrowers to „„Establish  a systematic approach that will help borrowers identify stakeholders and build and maintain a constructive relationship with them, particularly project-affected parties; „„ Assess the level of stakeholder interest in and support for the project and enable stakeholders’ views to be taken into account in project design and E&S performance; „„ Promote and provide means for effective and inclusive engagement with project- affected parties throughout the project life-cycle on issues that could affect them; „„Ensure  that appropriate project information on E&S risks and impacts is disclosed to stakeholders in a timely, understandable, accessible, and appropriate manner and format; and „„ Provide project-affected parties with accessible and inclusive means to raise issues and grievances, and allow borrowers to respond to and manage such grievances. Response to Recommendations The World Bank broadly agrees with all the recommendations. Recommendation 1. Management agrees with the focus on achieving depth and quality of citizen engagement by comprehensively offering entry points for governments to engage with citizens and continuing to make it a corporate priority to do so. Management reaffirms its commitment to Independent Evaluation Group | World Bank Group xxi the strategic framework, which lays out the menu of entry points across the World Bank’s product portfolio. The depth of citizen engagement at the operation and portfolio level is context dependent. Selectivity and client demand are key in taking citizen engagement forward to achieve greater depth of engagement where appropriate. Through the country engagement process, the World Bank can provide support across a range of instruments when countries identify citizen engagement as a priority. Recommendation 2. Management agrees with the focus on maintaining and deepening the corporate arrangement for support across Regions, countries, and Global Practices to foster comprehensive engagement where appropriate. As the report states, the World Bank has a well-functioning corporate system to support the corporate commitment to beneficiary feedback. Systematic support for mainstreaming citizen engagement can be strengthened through the country engagement cycle and across instruments. One important aspect of that support is assistance in identifying entry points where citizen engagement has the potential to improve outcomes and in developing the relevant results chains from engagement to expected development outcomes. As an example of a more systematic approach, country citizen engagement road maps have been developed and included in 14 CPFs in the Europe and Central Asia Region. These road maps serve as action plans to articulate country-level objectives for citizen engagement, outline areas of focus, define responsibilities, and set out concrete actions. The road maps also allow Country Management Units to set goals and standards for the implementation of requirements. Management also recognizes the importance of continuous knowledge and learning on citizen engagement and provision of timely expert support to task teams and clients to foster deeper, thick, and quality citizen engagement at both design and implementation. Recommendation 3. Management agrees with the need to ensure that indicators are results oriented and that they reflect how the feedback loop was closed and how diverse stakeholders are included. Citizen engagement and its measurement are not without cost. Measuring citizen engagement outcomes is challenging because of the difficulty of isolating the contribution of citizen engagement and establishing the direction of causality, all in a particular context. In rolling out the second phase of the strategic framework, management will confirm the consistency of guidance to staff, facilitate its communication to operational teams, and adjust it as appropriate as new insights are gained. As projects that include citizen engagement in their design begin to mature, the focus will be on compiling best practices on indicators at the implementation stage. Efforts to ensure the results focus of indicators and to design and use indicators that measure the closing of the feedback loop and reflect how stakeholders are included need to be balanced against the danger of an overly complex set of indicators for which it may be difficult in practice to compile and analyze data on their contribution to results. Because of the contextual nature of results chains, it will be difficult to aggregate the development outcomes of citizen engagement across projects and countries. Recommendation 4. Management agrees on the need to cultivate synergies between the new ESF and the broader citizen engagement agenda, while clearly recognizing their differences. The ESF strengthens the World Bank’s commitment to mainstreaming stakeholder engagement across its IPF operations through the dedicated Standard on Stakeholder Engagement and Information xxii Engaging Citizens for Better Development Results | Management Response Disclosure. The World Bank is developing a deep-dive learning course on stakeholder engagement to build staff capacity in this area. This will be useful for supporting broader citizen engagement, as a larger set of staff gain expertise in stakeholder engagement. IFC Management Response International Finance Corporation (IFC) management would like to thank IEG for its evaluation of the World Bank Group’s approach to citizen engagement. The evaluation benefited from extensive consultations with staff on projects and led to a detailed, well-written, and thoughtful report. The report identifies areas where the concept of community or stakeholder engagement has evolved in IFC, and it also identifies areas where further improvements are recommended. The report pulled together an extensive set of data points and information, analyzing a portfolio of IFC projects, 137 from FY07–11, and 30 from 2015–17. The report points out areas where IFC can clearly take action and IFC management will take the findings of the evaluation into consideration moving forward. The report also pointed out the variation in definitions between policy, guidance notes and other support materials, which we will seek to align. Environmental and social sustainability is critical to the success of private sector business and sustainable development. Part of that success includes enabling the participation of stakeholders and communities in the planning, design, construction, and operational phases of their projects. As pointed out in the report, IFC’s approach to community engagement is based on a risk management framework. The report recognised that since the 2012 update of the Performance Standards (PSs), IFC has expanded the magnitude and improved the depth of its stakeholder engagement. PS 1 (Assessment and Management of Environmental and Social Risks and Impacts), requires stakeholder mapping to be undertaken in all investment projects, followed by stakeholder engagement tailored to the level of risk, and an external communication mechanism. IEG noted that three positive patterns have emerged. First, IFC clients more frequently plan to engage strategically with communities, and through a range of mechanisms. Second, the mix of activities changed to include more proactive activities rather than purely reactive and risk-mitigation ones, such as solely grievance redress mechanisms, throughout the project cycle. Third, after the 2012 update, IFC has more systematically promoted forms of stakeholder engagement directly related to clients’ business and the project activities. This has resulted in clients progressively moving away from corporate social responsibility as a form of engagement to two-way discussions with communities around issues of concern related directly to project impacts and mitigation measures. IFC strives to ensure that staff and clients better understand the requirements and various steps in the stakeholder engagement process, including the depth that is needed based on the nature, frequency, and level of effort scaled to the projects risks and impacts. All clients should have an external communications procedure which should include a process to receive, screen and address communications from the public, document responses and adjust the E&S management program. Not all clients currently have these communications procedures, which if systematically implemented, Independent Evaluation Group | World Bank Group xxiii could address the IEG’s comments regarding a baseline level of community engagement for all projects. In many cases, the requirement for an external communications procedure may be met by implementation of the client’s grievance mechanism. All clients should identify stakeholders through a stakeholder mapping exercise. In business activities where there are Affected Communities (as defined in the PS 1), stakeholder mapping should be followed by a stakeholder engagement planning process and a Stakeholder Engagement Plan. The PS define affected communities as, “any people or communities located in the projects near geographical proximity, particularly those contiguous to the existing or proposed project facilities who are subject to actual or potential direct project related risks and/or adverse impacts on their physical environment, health or livelihoods”. In the case of corporate investments or projects where the exact location of the project is unknown but is reasonably expected to have adverse impacts, a Stakeholder Engagement Framework should be developed by the client. The framework should inform approaches to information disclosure, consultation (a two- way dialogue ongoing throughout the project cycle), the grievance mechanism and ongoing report to communities on the client’s action plans. In 2014, IFC did an internal review of stakeholder engagement in our portfolio projects. Several findings from that review are consistent with some within this evaluation. At that time, reference material was produced including a Tip Sheet on PS 1 Stakeholder Engagement Requirements and on Preparing a Stakeholder Engagement Plan, updates to other existing reference material and updates to the Environmental and Social Review Procedure. There was training for specialists to understand all the required steps of the stakeholder engagement process, and how to assess the appropriate level of stakeholder identification, analysis or mapping, and engagement initiated by a client. Also, Compliance Advisor/Ombudsman Advisory Service produced a grievance mechanism tool kit and provided training to specialists in association with the toolkit’s launch. In addition, over the past several years, IFC has enhanced its supervision of this issue and specialists have given more feedback to clients on how to improve stakeholder engagement and grievance mechanism procedures in relation to their operations. IFC also reviewed and recommended enhancements with respect to the reporting received from the clients in Annual Monitoring Reports (AMRs). However, it is recognized that further staff training is needed to ensure greater consistency in our specialists’ evaluation of clients’ stakeholder engagement processes. As is to be expected through application of a risk-based framework, it is notable that stakeholder engagement in E&S category A projects receives much more scrutiny than in category B projects. Also, E&S category A projects often benefit from the assignment of a dedicated social development specialist, as well as an environmental specialist, to the processing team, whereas this is less usual in less complex category B projects. In addition, complex (category A) sectors such as extractives and infrastructure typically have better stakeholder engagement than found in forestry and agribusiness or other direct investment sectors. There are a number of reasons for this disparity including the lower risk profile and consequently reduced client E&S management staffing and resources in certain sectors. The Environmental, Social and Governance Department has introduced a training xxiv Engaging Citizens for Better Development Results | Management Response core curriculum to align all specialists with best practices and key areas of compliance. In addition, much time is spent at Environmental, Social and Governance Department training events on lessons learned, emerging issues and areas where practice could be improved, one such being more detail around stakeholder engagement and greater focus on implementation of company grievance mechanisms. Additionally, several lessons have emerged from Compliance Advisor/Ombudsman cases over the past few years, which are disseminated to specialists, with recommendations on improvements, some of which have also included issues around stakeholder engagement and grievance mechanisms. Regarding recommendation 5 on improving and more systematically documenting and reporting comprehensive results-focused information on Community/Stakeholder Engagement activities carried out by clients in IFC projects, IFC is in agreement as we seek to improve current practice and systems. 1 https://openknowledge.worldbank.org/handle/10986/21113. 2 Under IDA18’s Governance and Institutions the WorldBank made a policy commitment to inclusive citizen  engagement: “support projects in at least 10 IDA countries in the development and implementation of user feedback and/or enhanced [grievance redress mechanisms] for service delivery that ensure participation by women in these processes.” http://documents.worldbank.org/curated/en/348661486654455091/pdf/112728-correct-file-PUBLIC- Rpt- from-EDs-Additions-to-IDA-Resources-2-9-17-For-Disclosure.pdf. 3  Disbursement-linked indicator 7 “65% of Woredas [districts] have conducted pre-budget discussions, with at least 30% women participating in such discussion” (target for 2019–2020). Independent Evaluation Group | World Bank Group xxv management action record Defining and Tracking Quality IEG FINDINGS AND CONCLUSIONS The World Bank Group has rightfully made the quality of citizen engagement a cornerstone of its mainstreaming agenda. However, aspects related to quality are not reflected in the corporate commitment and are not given sufficient attention at the design and monitoring stages. The evaluation found that none of the quality criteria highlighted in the strategic framework were embedded in how citizen-oriented design is defined or tracked at the corporate level. The emphasis on the corporate indicators and their targets risk generating a “check-the-box” attitude, to the detriment of quality. The evaluation recognizes that mainstreaming citizen engagement across the Bank Group’s operations to achieve better development outcomes is an ambitious commitment. It requires a gradual approach, as stressed in the strategic framework, and it needs to be tailored to context. At the same time, as improving the quality of citizen engagement activities is critical to influence development results, incentives to teams need to support quality of engagement. IEG RECOMMENDATIONS Recommendation 1. As it defines future corporate priorities for citizen engagement, the World Bank should reflect in those priorities the need to achieve greater depth and quality of the citizen engagement activities it supports. This will entail building on the progress achieved under the strategic framework to promote a more strategic approach that incorporates deeper citizen engagement where opportunities arise. These future corporate priorities will need to be clearly communicated to staff and stakeholders. ACCEPTANCE BY MANAGEMENT Agree. MANAGEMENT RESPONSE Management agrees with the focus on achieving depth and quality of citizen engagement by comprehensively offering entry points for governments to engage with citizens, and continuing to make it a corporate priority to do so. Management reaffirms its commitment to the strategic framework, which lays out the menu of entry points across the World Bank’s product portfolio. The depth of citizen engagement at the operation and portfolio level is context-dependent. Selectivity and client demand are key in taking citizen engagement forward to achieve greater depth of engagement where appropriate. Through the country engagement process the World Bank can provide support across a range of instruments when countries identify citizen engagement as a priority. xxvi Engaging Citizens for Better Development Results | Management Action Record Promoting “Thick” Engagement IEG FINDINGS AND CONCLUSIONS “Thick” engagement that is regular and continuous, uses multiple tools, and is embedded in country systems cannot realistically be mainstreamed in all projects. Moreover, promoting thick engagement and building capacity at the country level depend on the country context, the commitment of the client government to citizen engagement, and the World Bank’s ability to seize the opportunity to influence the space and the quality of engagement. The evaluation found that this type of engagement requires intensifying training, ensuring that adequate expertise is mobilized, improving synergies with other programs (such as the GPSA), and strengthening knowledge management. These activities are critical to strengthen support to Regions and country management units, which are uniquely positioned to take into account country-specific factors that affect the scope and time frame for mainstreaming and to apply the instruments, tools, and approaches at their disposal to facilitate citizens’ participation and monitor results. IEG RECOMMENDATIONS Recommendation 2. The World Bank should encourage and support efforts of its regional, country and Global Practices teams to establish, where appropriate, “thick” citizen engagement that is regular and continuous, uses multiple tools, and is embedded in country systems. This could be achieved by more systematically using existing channels of dialogue and stakeholder engagement (such as Systematic Country Diagnostics, Country Partnership Frameworks, development policy financing) and applying tools (such as road maps and indexes) to plan, monitor, and assess results achieved at the various levels (Region, country management unit, Global Practice). This would entail better synergies between World Bank operations and other programs that support citizen engagement, including GPSA-financed activities; mobilizing adequate expertise to support World Bank teams; investing in training for staff and stakeholders; and improving knowledge management. ACCEPTANCE BY MANAGEMENT Agree. MANAGEMENT RESPONSE Management agrees with the focus on maintaining and deepening the corporate arrangement for support across Regions, countries, and Global Practices to foster comprehensive engagement where appropriate. As the report states, the World Bank has a well- functioning corporate system to support the corporate commitment to beneficiary feedback. Systematic support for mainstreaming citizen engagement can be strengthened through the country engagement cycle and across instruments. One important aspect of that support is assistance in identifying entry points where citizen engagement has the potential to improve outcomes and in developing the relevant results chains from engagement to expected development outcomes. As an example of a more systematic approach, country citizen engagement road maps have been developed and included in 14 Country Partnership Frameworks in the Europe and Central Asia Region. These road maps serve as action plans to articulate country-level objectives for citizen engagement, outline areas of focus, define responsibilities, and set out concrete actions. The road maps also allow Country Management Units to set goals and standards for the implementation of requirements. Management also recognizes the importance of continuous knowledge and learning on citizen engagement and provision of timely expert support to task teams and clients to foster deeper, thick, and quality citizen engagement at both design and implementation. Independent Evaluation Group | World Bank Group xxvii Adopting Results-Oriented Indicators IEG FINDINGS AND CONCLUSIONS The corporate commitment to mainstream citizen engagement and associated targets have generated awareness and buy-in among senior management and staff. An increasing number of World Bank projects include citizen engagement mechanisms in their design, and indicators in their results frameworks. However, monitoring of citizen engagement remains inadequate beyond tracking progress of corporate targets. Indicators rarely track results; many mechanisms are not measured in results frameworks. Inclusion of women and marginalized groups is not regularly tracked. Crucially, closing the feedback loop with citizens is neither mainstreamed or tracked. IEG RECOMMENDATIONS Recommendation 3. The World Bank should strengthen the monitoring of its citizen engagement activities by systematically adopting results framework indicators that are results oriented. This will entail tightening the alignment between the citizen engagement mechanisms used in projects and the indicators that measure their quality and results, with special emphasis on indicators that show how feedback loops were closed and how diverse groups of stakeholders were included. ACCEPTANCE BY MANAGEMENT Agree. MANAGEMENT RESPONSE Management agrees with the need to ensure that indicators are results- oriented and that they reflect how the feedback loop was closed and how diverse stakeholders are included. Citizen engagement and its measurement are not without cost. Measuring citizen engagement outcomes is challenging because of the difficulty of isolating the contribution of citizen engagement and establishing the direction of causality, all in a particular context. In rolling out the second phase of the strategic framework, management will confirm the consistency of guidance to staff, facilitate its communication to operational teams, and adjust it as appropriate as new insights are gained. As projects that include citizen engagement in their design begin to mature, the focus will be on compiling best practices on indicators at the implementation stage. Efforts to ensure the results focus of indicators and to design and use indicators that measure the closing of the feedback loop and reflect how stakeholders are included need to be balanced against the danger of an overly complex set of indicators for which it may be difficult in practice to compile and analyze data on their contribution to results. Because of the contextual nature of results chains, it will be difficult to aggregate the development outcomes of citizen engagement across projects and countries. xxviii Engaging Citizens for Better Development Results | Management Action Record Leveraging the Environmental and Social Framework to Increase Citizen Engagement IEG FINDINGS AND CONCLUSIONS There are important connections to be established between the citizen engagement and other agendas of the World Bank Group, such as social inclusion, gender, and especially the new Environmental and Social Framework. The recent Environmental and Social Framework reform, approved in 2017 and to become operational over the next two years, moves safeguards from a predominantly risk management perspective to a more proactive engagement with stakeholders. The ESF agenda is more endowed with staff and resources than the citizen engagement agenda, it is in the process of being strengthened through new hiring and a massive capacity building effort, and has clear enforcement mechanisms. Linking the two agendas could support more substantial work through pooling of resources, capacity, and experience. IEG RECOMMENDATIONS Recommendation 4. The World Bank should seize the opportunity of the implementation of the ESF to leverage citizen engagement mechanisms—beyond consultations and grievance redress mechanisms—to reach the objectives of managing social risks, strengthening country systems, and promoting social inclusion. This will entail better drawing on the existing expertise on citizen engagement and making citizen engagement more prominent in the ESF trainings, both for staff and for project implementation units. ACCEPTANCE BY MANAGEMENT Agree. MANAGEMENT RESPONSE Management agrees on the need to cultivate synergies between the new ESF and the broader citizen engagement agenda, while clearly recognizing their differences. The ESF strengthens the World Bank’s commitment to mainstreaming stakeholder engagement across its IPF operations through the dedicated Standard on Stakeholder Engagement and Information Disclosure. The World Bank is developing a deep-dive learning course on stakeholder engagement to build staff capacity in this area. This will be useful for supporting broader citizen engagement, as a larger set of staff gain expertise in stakeholder engagement. Independent Evaluation Group | World Bank Group xxix Improving IFC Reporting IEG FINDINGS AND CONCLUSIONS International Finance Corporation (IFC) reporting provides especially poor information on implementation and results. Reporting by IFC clients of stakeholder engagement activities is partial, and largely driven by compliance rather than focused on outcomes achieved. A fourth of all IFC clients (including high-risk ones) provided too little information to understand how the client engaged with the community, despite IFC considered them in compliance. IEG RECOMMENDATIONS Recommendation 5. IFC should ensure that its clients’ stakeholder engagement activities required by Performance Standard 1 in projects with affected communities are carried out during appraisal and supervision of the projects and systematically documented. This will entail mobilization of adequate expertise to systematically support clients and reporting that is comprehensive and results focused. ACCEPTANCE BY MANAGEMENT Agree. MANAGEMENT RESPONSE IFC is in agreement with this recommendation and we seek to improve current practice and systems going forward. Management will improve and more systematically document and report comprehensive results focused information on Community or Stakeholder Engagement activities carried out by clients in IFC projects. xxx Engaging Citizens for Better Development Results | Management Action Record report to the board from the committee on development effectiveness The Committee on Development Effectiveness met to consider Engaging Citizens for Better Development Results and the draft management response. The committee welcomed the evaluation and acknowledged that it was well-timed for the World Bank Group to incorporate lessons learned from early implementation and make course corrections. Members were pleased to see findings by the Independent Evaluation Group (IEG) showing increased corporate commitment and awareness among staff. They congratulated management for the progress made. While acknowledging the complexity of doing meaningful citizen engagement, they also highlighted room for improvement and making the comprehensive offering of World Bank products attractive for governments to engage with their citizens. Members also requested more information on planned learning and dissemination activities for the evaluation. Several members reinforced that they expected citizen engagement to be increasingly “thick,” as the IEG report put it, with strong links to better development outcomes and impacts. They recognized the evaluation’s finding that the Strategic Framework had led to a change in mind-set, and members underscored that they expected the Bank Group to use citizen engagement as more than a risk management tool or a “tick in the box” exercise. Members asked how management was going to support staff in this regard. Members highlighted the importance on working on client demand, commitment, and capacity for citizen engagement. They encouraged management to include citizen engagement in Country Partnership Frameworks and diagnostic tools, to develop communication strategies, to strengthen borrower capacity and to embed citizen engagement more deeply in the development of country systems. Members asked management to explain how they planned to enhance the quality of citizen engagement across operations, and citizen engagement indicators at the corporate and operational level. Some Members suggested replicating the Citizen Engagement Quality Index from the Europe and Central Asia Region to other regions. Other members required management to explain why results-oriented indicators were insufficiently adopted in results frameworks to monitor citizen engagement activities and expressed concerns it might be a systemic issue, as discussed in other evaluations. They also asked management to elaborate plans on how to close the feedback loop and how to ensure an inclusive approach for marginalized or vulnerable groups. Members also required the International Finance Corporation to encourage client’s proactive commitment to early resolution of issues through citizen engagement and asked for specific plans to implement IEG recommendations. Independent Evaluation Group | World Bank Group xxxi xxxii Engaging Citizens for Better Development Results | Management Action Record 1 I N R E C E N T Y E A R S , the world has seen formidable manifestations of citizens’ engagement. By taking to the streets to condemn corruption scandals, by rallying on social media to address growing inequalities, or by participating in global Introduction consultations to develop the Sustainable Development Goals (SDGs), ordinary citizens are increasingly eager and able to make their voices heard. At the same time, after several decades of progress, the space for citizens’ voices is shrinking globally as governments raise legal barriers to constrain actions by civil society organizations (CSOs) and to muzzle the media (World Bank Group 2017; Lührmann et al. 2017). In this context, the World Bank Group’s commitment to citizen engagement can catalyze change. This is even more important because achieving the SDGs and the twin goals rests on the active involvement of citizens and local governments. Citizen engagement is the term adopted by the Bank Group to denote the two-way interaction between citizens and governments or the private sector within the scope of Bank Group interventions. As described in official statements, it is an approach that gives citizens a stake in decision making with the goal of improving development outcomes. It is a relationship that implies the existence of a tangible response to citizens’ feedback (or “closing the feedback loop”). Creating this relationship may require shifting or rebalancing power between governments and governed or private companies and their customers. Often, it requires opening or expanding spaces for citizens’ voices. Yet, the World Bank is primarily the governments’ partner, and the International Finance Corporation (IFC) is the companies’ partner. Mainstreaming citizen engagement must be assessed in light of this inherent tension. The history of stakeholder engagement at the Bank Group is documented as far back as the 1980s. Engagement has happened with different stakeholders and at multiple levels, 1 ranging from consultations with client country governments, international and national CSOs, and the private sector, to participation of direct and indirect project beneficiaries. Stakeholders have been variously labeled “citizens,” “beneficiaries,” or “communities.” Although these terms are not equivalent, this evaluation mostly refers to “citizens” in keeping with the strategic framework and to “stakeholders” as per IFC’s preferred denomination.1 Citizen engagement activities have intensified over time, leading to a corporate commitment to mainstream citizen engagement in the Bank Group’s work. This was operationalized in the 2014 Strategic Framework for Mainstreaming Citizen Engagement in World Bank Group Operations (World Bank Group 2014a). The framework takes stock of the World Bank experience and identifies multiple entry points for engagement; elevates citizen engagement to a corporate priority; and aims to mainstream citizen engagement in Bank Group–supported operations.2 The two latest International Bank for Reconstruction and Development (IDA) replenishments, IDA17 and IDA18, have also introduced policy commitments related to citizen engagement.3 The Bank Group is a leading international financial institution in its commitment to mainstreaming citizen engagement, along with the Inter-American Development Bank and the U.S. Agency for International Development (USAID). The Norwegian Agency for Development Cooperation, the U.K. Department for International Development, and the International Fund for Agricultural Development are either updating or developing their strategies and guidelines. The Asian Development Bank will include engagement with civil society in its new corporate strategy. All agencies express similar motivations to engage with civil society—influencing development results, supporting society ownership and voice, and strengthening risk management. Some agencies, such as the Norwegian Agency for Development Cooperation, the Swedish International Development Cooperation Agency, and USAID, emphasize the human rights rationale at the basis of their commitment to engage citizens. Mainstreaming citizen engagement in operations appears to be a priority, especially for the multilateral agencies, such as the Asian Development Bank, the Inter-American Development Bank, and the African Development Bank—all of which are increasing their citizen engagement efforts. This evaluation aims to inform the Board of Executive Directors and the management of the Bank Group institutions on the effectiveness of the citizen engagement mainstreaming approach. Notably, the evaluation assesses the performance of the Bank Group in embedding citizen engagement in its various operations, drawing lessons from fiscal years (FY)11–16, which covers the three years before and three years after the introduction of an indicator on citizen and beneficiary feedback in the corporate scorecards.4 The evaluation informs the Board and management on the future directions and priorities of this agenda. The Landscape for Citizen Engagement and Scope of the Evaluation Resting on a long tradition of Bank Group stakeholder engagement, the 2014 strategic framework encompasses multiple entry points for engagement based on several commitments and operational policies. Entry points and activities have been identified at the operational (project), country, and 2 Engaging Citizens for Better Development Results | Chapter 1 the broader strategic and policy levels (table 1.1). Accordingly, this evaluation adopts the same broad perspective and assesses citizen engagement at multiple levels. This section briefly reviews how various streams of work evolved during the past 40 years to define the current priorities of the institution. Over the past four decades, the Bank Group has moved from a top-down, external, expert-driven to a more participatory approach to development. The concept of engagement has evolved from an initial focus on reputational risk management to more proactive interaction in operations, policies, and strategies. This process has happened at different levels: operational, country, and institutional. At the operational level, citizen engagement started within the framework of environmental and social (E&S) safeguards in response to civil society criticisms of harmful impacts of World Bank projects; over time, a participatory approach became more prominent in certain sectors. In 1982, consultations were introduced as part of the design of projects affecting indigenous peoples. Since the early 1990s, the World Bank has promoted collaboration with beneficiaries in implementing projects to better respond to local needs and enhance ownership. Water user groups and community forest management were the first examples. Other sectors followed in adopting participatory methods, including the education sector (for example, with parent-teacher associations) and the agriculture and rural development sector (for example, with farmer groups). These experiences evolved into what is now called community-driven development (CDD). At the country level, the emphasis on expanding consultative approaches in policy making started in the mid-1990s and expanded to most policy-level instruments. In 1996, the World Bank undertook a review of the social and political costs of the previous 15 years of structural adjustment operations in several countries, in consultations with nongovernmental organizations (NGOs; World Bank 2001). Around the same time, Bank Group President Wolfensohn launched a more strategic, participatory, and country-led engagement model. As part of this process, the World Bank began to support a country-driven analysis of poverty issues in the Poverty Reduction Strategy Papers, requiring NGO and stakeholder consultations, a process which was also followed for the preparation of country assistance strategies. Development policy financing (DPF), introduced in 2004, requires the client to conduct adequate stakeholder consultations during the preparation phase. Moreover, relevant analytical work underpinning the operation should be made available to the public as part of the consultation process.5 At the institutional level, the Bank Group incrementally introduced participation and stakeholder engagement as key principles in its strategic frameworks and promoted social inclusion, social accountability, governance, and anticorruption during the 2000s. In 1996, the World Bank published the Participation Sourcebook, a reference guide to participatory development that showcases participatory decision making.6 The 2002 Empowerment Sourcebook recognized that making the state and social institutions more responsive to poor people was fundamental to reducing poverty (Narayan 2002). The 2004 World Development Report highlighted the role of citizen engagement in improving service delivery (World Bank 2003). The report conceptualized two routes to accountability that would guide the World Bank’s engagement for the years to come: (i) a long, indirect route Independent Evaluation Group | World Bank Group 3 through high-level institutional reforms and policy dialogue and (ii) a short route, via direct contacts between beneficiaries and service providers, within the boundaries of projects. This commitment to citizen voice to improve governance was further crystalized in the 2007 governance and anticorruption strategy (World Bank 2007). IFC and the Multilateral Investment Guarantee Agency (MIGA) have E&S Performance Standards (PSs) as the systematic entry point for approaching stakeholder and community engagement. IFC adopted the E&S PSs in 2006 and revised them in 2012 (IFC 2012).7 The E&S PSs impose requirements for meaningful stakeholder engagement on all IFC and MIGA projects, with variation depending on project type and risk factors. These requirements are implemented by client companies under the supervision and monitoring of IFC and MIGA. The PSs call for information disclosure and engagement with local communities on matters that directly affect them; active management by clients of E&S risks; and TABLE 1.1 | Main Entry Points for Citizen Engagement Level and Instrument Citizen Engagement Entry Point Corporate • Dialogue with civil society organizations • Regional strategies • Partnership programs, including the Global Partnership for Social Accountability and the Open Government Partnership • Monitoring of the corporate indicators Country Consultations for the preparation of Systematic Country Diagnostics and Country •  Partnership Frameworks Identification of citizen engagement-related priority areas for inclusion in country •  operations Use of citizen engagement indicators in Country Partnership results frameworks •  Investment project Implementation of OPs and BPs requiring consultations, disclosure, and grievance •  financing and redress mechanisms: OP/BP4.01, 4.10, 4.12a Program-for-Results Inclusion of non-safeguards-related citizen engagement in projects, especially •  service delivery, natural resource management, governance, and community-driven development projects Development policy • Consultations based on OP 8.60 and related reporting; Poverty and Social Impact financing Analysis • Prior actions and benchmarking related to citizen engagement IFC and MIGA • Implementation of the Performance Standards Advisory Services • Citizen engagement in design, elaboration, and evaluation of knowledge products and Analytics • Technical assistance • Analytic work on specific aspects of citizen engagement Source: Adapted from World Bank Group 2014a, 22. Note: BP = World Bank Procedures; IFC = International Finance Corporation; MIGA = Multilateral Investment Guarantee Agency; OPs = operational policies. a. The new Environmental and Social Framework superseded these OPs/BPs, starting from October 2018. 4 Engaging Citizens for Better Development Results | Chapter 1 grievance redress mechanisms (GRMs) for affected communities, workers, and any people who may be affected by potential economic or physical displacement. The World Bank also adopted tools to engage more directly with civil society. The Small Grants Program was started in 1983 to promote dialogue and disseminate information about development and support activities related to civic engagement for the empowerment of marginalized and vulnerable groups. In 2012, it was replaced by the multi-donor-supported Global Partnership for Social Accountability (GPSA), whose goal is to provide direct strategic and sustained support to CSOs and governments for social accountability initiatives aimed at strengthening transparency and accountability. In addition, trust funds, such as the Japan Social Development Fund, the Nordic Trust Fund, and the State and Peacebuilding Trust Fund, contribute in important ways to the social accountability and citizen engagement agenda and to multilateral initiatives such as the Open Government Partnership.8 Three recent developments can open avenues for a more holistic approach to citizen engagement. First, the updated country engagement model introduced in 2014 aims to make the process of preparation and implementation of Bank Group country strategies more consultative and participatory (World Bank Group 2014b). Second, the new E&S Framework (ESF) makes important advances in areas such as transparency, nondiscrimination, social inclusion, public participation, and accountability (World Bank 2017c). One of the 10 standards is dedicated to stakeholder engagement and information disclosure, and it requires stakeholder consultation throughout the project cycle.9 Third, the World Development Report 2017: Governance and the Law emphasizes key constraints to effective policies, such as unequal power distribution among groups in society, that can undermine the core function of institutions (World Bank Group 2017). It recognizes the key role of nonstate actors, such as private sector, civil society groups, and individual citizens in addition to policy makers and bureaucrats. It encourages the Bank Group to strengthen the enabling environment for better governance by directly addressing power asymmetries between groups and fostering more equitable bargaining, contestation, and accountability spaces. Evaluating the Bank Group Support to Citizen Engagement Because of the diffused presence of citizen engagement across Bank Group activities—at the project, country, and corporate levels (table 1.1)—this evaluation had to be selective in defining its scope. The evaluation reviewed the following: (i) at the project level, citizen engagement activities in World Bank investment project financing (IPF) and the implementation of PS 1, requiring stakeholder engagement in IFC investment services; (ii) at the country level, citizen engagement elements in country strategies and consultations during Country Partnership Framework (CPF) and DPF preparation;10 and (iii) at the corporate level, the strategic approach to mainstreaming citizen engagement (including the corporate commitment and indicators) and the GPSA. The evaluation excluded Advisory Services and Analytics, Program-for-Results operations, stand-alone trust-funded activities, and partnerships other than the GPSA.11 It also excluded MIGA because the relationship between MIGA and final beneficiaries is highly intermediated by the guarantee holder and the project enterprise, which diffuses responsibility and complicates attribution.12 Independent Evaluation Group | World Bank Group 5 The overarching question for this evaluation is, “How effectively has the World Bank Group mainstreamed citizen engagement at the project, country, and corporate levels, and what is the evidence on how this process contributes to the achievement of development outcomes?” Four subquestions were defined to better specify the main evaluation question (see box 1.1). The evaluation tackled these questions in three steps summarized in figure 1.1, which represents the conceptual framework and provides the road map of the report. First, the evaluation assessed the progress of the institution in operationalizing citizen engagement mainstreaming toward four subobjectives defined by the strategic framework (chapter 2). These subobjectives are (i) scaling up context-specific citizen engagement across the World Bank Group client engagement spectrum; (ii) improving the quality and outcomes of safeguards-related mechanisms; (iii) achieving 100 percent citizen engagement in IPF projects with clearly identifiable beneficiaries; and (iv) improving monitoring and results reporting on citizen engagement (World Bank Group 2014a, 7). In this chapter, the evaluation also assessed the engagement elements in country strategies and DPF, and stakeholder engagement in IFC investment projects, based on PS 1. Second, the evaluation analyzed the quality of citizen engagement activities in operations and country strategies based on the criteria established by the strategic framework (chapter 3). The strategic framework posits that development outcomes depend on the quality of citizen engagement activities. It introduces five principles that define preconditions for influential mechanisms as those that are results focused; engage throughout the operational cycle; strengthen country systems; are context specific; are gradual, iterative, and scalable. Social inclusion is a sixth dimension, cutting across these principles. Fulfilling these characteristics does not guarantee success; but approaches that do not apply these principles are less likely to succeed and more likely to trigger unintended negative effects, such as elite capture. The evaluation used these quality dimensions to determine Four Lines of Inquiry Guiding the Evaluation Box 1.1 |  n What is the magnitude and nature of citizen engagement mainstreaming within  World Bank Group operations, and how have they changed over time? n What is the quality of design of citizen engagement activities? How much attention  is paid to social inclusion and closing the feedback loop with citizens? n What are the extent and quality of monitoring and evaluation of citizen engagement  activities, and what is the evidence of results (contribution to strengthening existing country systems for participation and to improved development outcomes)? n To what extent is the Bank Group corporate environment enabling citizen  engagement mainstreaming? 6 Engaging Citizens for Better Development Results | Chapter 1 whether mechanisms are well designed, effectively implemented, and work as intended to enhance development outcomes. Third, the evaluation documented changes at the behavioral, operational, and institutional levels that are associated with high-quality citizen engagement and can contribute to improved development outcomes (chapter 4). The strategic framework adopts a results-based approach to citizen engagement, which intends mainstreaming as a means to accelerating, sustaining, or enhancing development outcomes (box 1.2). A few studies have reviewed the existing evidence on whether citizen engagement improves development outcomes (Mansuri and Rao 2013; Fox 2015; Grandvoinnet, Aslam, and Raha 2015; Gaventa and Barrett 2010; O’Meally 2013) confirming the importance of the quality principles established by the strategic framework to trigger development impact. Nonetheless, the evidence remains mixed and extremely context specific. Contextual elements (including the country development stage; the client government’s buy-in and capacity; and the level of societal inequality, conflict, and fragmentation) and internal conditions (including management and staff buy-in; human, technical, and financial resources; and corporate incentives to pay attention to quality and results) have a strong influence on the process and its results (figure 1.1). The evaluation adopted a multilevel, mixed methods, and case-based design. To gauge the extent of mainstreaming across Bank Group instruments and assess trends over time, the evaluation team conducted portfolio reviews, interviews, focus group discussions, in-depth country-level and project- or mechanism-level case studies, literature reviews, a survey of World Bank staff and management, Objective of the Strategic Framework Box 1.2 |  “The overall objective of [the] Strategic Framework is to mainstream citizen engagement in Bank Group–supported policies, programs, projects, and Advisory Services and Analytics where such engagement can improve development results and, within the scope of these operations, to contribute to sustainable processes for citizen engagement with government and the private sector.” Source: World Bank Group 2014, 7. Independent Evaluation Group | World Bank Group 7 8 FIGURE 1.1 Road Map of the Evaluation Report | Mainstreaming that is well ... of citizen engagement ...can contribute to development implemented and well tracked in mechanisms that are of high outcomes by: the World Bank Group... quality ... World Bank Group scales up citizen engage- Engagement is “thick” and Triggering positive behavioral and ment mechanisms across WBG instruments results oriented relationship change (improved awareness, trust, voice, ownership, and answerability) Engagement is continuous Bank Group leverages multiple entry points throughout the operational cycle at project and country levels Triggering positive operational change and closes the feedback loop (improved community involvement, project Bank Group uses indicators that reflect Engagement ensures inclusion implementation, quality of services) progress in implementation of diverse voices Triggering positive institutional change Engaging Citizens for Better Development Results | Chapter 1 Bank Group tracks quality and results Engagement builds capacity of both (improved policies, mutual accountability, of engagement citizens and governments country systems) Bank Group teams’ capacity and expertise; Management and staff buy-in; corporate incentives to pay attention to quality and Repository of knowledge on what works, resources; tracking systems results; internal coordination and under what circumstances Internal conditions knowledge management Project implementation units’ capacity; Government’s ability to resolve issues; laws capacity and willingness of both state and Clients’ buy-in; networking with CSOs and cultural norms around participation; citizens actors for engagement; societal nature of political settlement norms around inclusion External conditions Source: Based on World Bank Group strategic framework for mainstreaming citizen engagement. Note: CSOs = civil society organizations. and an online consultation of CSOs. Appendix A lays out the methodological design, including sampling and selection, sources of evidence, data collection and analysis methods, and links among methodological components. 1 The strategic framework is framed in terms of “citizen” engagement. Citizens are the “ultimate client of government,  development institutions, and private sector interventions in a country” (World Bank Group 2014a, 7). They may be represented by civil society organizations and other interest groups. Citizens—sometimes very diverse in terms of identities, power, and goals—interact with each other within communities. Community-driven development is an approach that gives control of decisions and resources to community groups, recognizing the existence of complex dynamics among citizens. Beneficiaries are defined as a “subset of citizens directly targeted by and expected to benefit from a development project” (World Bank Group 2014a, 8). The word stakeholders refers to all those who have a stake in a matter, including organizations, agencies, firms, and the government itself—not just citizens. The Bank Group consults with stakeholders, as it reaches out to a broader set of actors beyond individual citizens. International Finance Corporation Performance Standards and the World Bank Environmental and Social Standards refer to “stakeholders” to indicate all those who have a stake in a project. 2 The corporate goal of increasing beneficiary feedback to 100 percent of projects with clearly identified beneficiaries  was first announced at the 2013 Annual Meetings by the Bank Group President Jim Yong Kim. 3 The IDA17 policy commitment was to expand the use of beneficiary feedback in IDA-supported projects and report  at IDA17 Midterm Review on the impact of these mechanisms. The IDA17 Results Measurement System (RMS) includes two citizen engagement indicators: (i) projects using beneficiary-oriented design; and (ii) client feedback on responsiveness and staff accessibility in FCSs and on collaboration with other donors in FCSs. IDA18 includes three policy commitments related to citizen engagement: (i) support projects in at least 10 IDA countries in the development and implementation of user feedback and/or enhanced GRMs for service delivery that ensure participation by women in these processes; (ii) support at least one-third of IDA countries to operationalize reform commitments toward the Open Government Partnership agenda to strengthen transparent, accountable, participatory, and inclusive governments; and (iii) plan for operationalization of World Development Report 2017, which calls for expanding access to the public policy arena for nonstate actors. 4 An indicator tracking the “percentage of projects with beneficiary feedback during implementation” had been added  for the first time in the FY14 corporate scorecard. Moreover, the indicator “projects using beneficiary-oriented design” was introduced in FY14 in the IDA17 results framework to track early progress in implementing the citizen engagement initiative. This reflects the emergence of citizen engagement as a corporate priority. (In FY13, a more narrowly defined indicator existed in the Corporate Scorecard Integrated Results and Performance Framework: “Bank operations with beneficiary feedback (percent): Investment lending operations that provide support to develop or use community- based monitoring systems/processes, community scorecards, or citizen scorecards to improve social accountability;” in April 2013 it was reported that operations with beneficiary feedback had increased significantly.) This evaluation recognizes, however, that in FY14 the Strategic Framework was not yet being implemented and that in FY14 the citizen engagement commitments applicable to IPF based on the Strategic Framework were still being defined. The commitment to increase beneficiary feedback to 100 percent of projects with clearly identified beneficiaries was announced by Bank Group’s President Jim Yong Kim at the 2013 Annual Meetings. The strategic framework was issued in 2014, followed, in the same year, by the guidelines for implementation. So, FY14 and part of FY15 are not post-Strategic Framework, but a corporate commitment on beneficiary feedback existed already in those years. The new corporate scorecards indicator “percentage of projects with beneficiary feedback indicator at design” replaced in October 2015 the previous indicator “percentage of projects with beneficiary feedback during implementation,” with a FY14 baseline. Recognizing that the Strategic Framework was fully operational in FY16, figures for FY16 are shown separately when relevant. 5  Operations Policy 8.60, 3, para. 5: The World Bank advises its clients to conduct stakeholder consultations and specifies that “key stakeholders include social groups directly affected by the operations, as well as public sector, Independent Evaluation Group | World Bank Group 9 private sector, and donor organizations relevant to the operation” (World Bank 2014d). The responsibility to initiate this process rests with the government. As per the operational policy, the program document should describe the consultative arrangements that are relevant to the operation and its outcome. In addition, relevant analytical work, particularly on poverty and social impacts and on environmental aspects, should be made available to the public as part of the consultation process. 6  he World Bank Participation Sourcebook built on the work of the Learning Group on Participatory Development, T which was launched in December 1990 with the task of examining the issue of stakeholders’ participation in development initiatives (World Bank 1996). 7 The Multilateral Investment Guarantee Agency followed a year later (2007 and 2013, respectively; MIGA 2013).  8 T  he Japan Social Development Fund was established in 2000 to support community-driven development and poverty reduction projects and build capacity to strengthen local communities, nongovernmental organizations, civil society organizations, and other key stakeholders in the development process. The Nordic Trust Fund, launched in 2009, is a knowledge and learning program that supports exchange of knowledge about human rights—how they relate to the World Bank’s sector and thematic work, as well as operational work, including strategy, planning and implementation. Since 2008 the State and Peacebuilding Trust Fund has been supporting innovative approaches to state and peace building in regions affected by fragility, conflict and violence. One of its goals is to improve governance and institutional performance in fragility, conflict and violence-affected countries. The Open Government Partnership (OGP) is a multilateral initiative that aims to secure concrete commitments from governments to promote transparency, empower citizens, fight corruption, and harness new technologies to strengthen governance. It was launched in 2011; as of January 2018, over 70 participating countries and 15 subnational governments have committed to uphold the principles of open and transparent government by endorsing the Open Government Declaration. As partner of the OGP, the Bank Group provides financial and technical assistance for the development and implementation of OGP national action plans and to facilitate knowledge exchanges among participating countries and adoption of successful open government practices. This evaluation does not assess any of these initiatives. 9 Environmental and Social Standard 10: Stakeholder Engagement and Information Disclosure. 10 Although development policy finances are individual financing instruments their preparation is country-led. 11  itizen engagement activities supported by trust funds and attached to projects were captured in the portfolio review C and in country case studies. The Global Partnership for Social Accountability was selected because it is more strictly related, in its goals and operational modalities, to the Bank Group citizen engagement agenda. The other tools and initiatives, though relevant, could not be analyzed because of limited resources. 12  n Expert Advisory Council was established at inception of the strategic framework to bring external expertise A and insight to the development and implementation of the framework. It is composed by representatives from civil society, academia, private sector, government, foundations, and donor organizations (including the Bank Group). The evaluation team has not assessed its role or effectiveness in the implementation of the citizen engagement agenda. 10 Engaging Citizens for Better Development Results | Chapter 1 2 highlights 1 The commitment to mainstreaming citizen engagement and tracking progress in the corporate scorecard mobilized senior management and received buy-in from staff. Meeting the Target 2 Engagement mechanisms not tied to safeguards are now more prevalent, denoting a move toward a more proactive approach. Through its updated Performance Standards, the International Finance Corporation (IFC) encourages its clients to engage more systematically with stakeholders throughout the project cycle. 3 Although the World Bank Group leverages its new country engagement model to consult with a wider range of stakeholders, it is not using the full range of entry points to involve citizens in strategic decision making and policy dialogue. 4 Monitoring of citizen engagement remains weak beyond tracking progress on corporate targets: indicators rarely track results of all mechanisms used and are not designed to capture whether the feedback loop with citizens is closed. IFC reporting provides poor information on implementation and results. 11 5 The corporate arrangements in place to oversee and coordinate the commitment work well, but dedicated staff time to operationalize the approach beyond ensuring compliance with the corporate target is limited and lack of resources may constrain implementation. 12 x W I T H T H E I N T R O D U C T I O N of the corporate commitment, the institution succeeded in raising awareness, especially internally, of the importance of citizen engagement in the Bank Group’s work. This chapter explores how the institution has responded to the commitment to mainstreaming citizen engagement; how well it is meeting the specific requirements of the corporate mandate (especially on integrating citizen engagement in all IPF) and operational policies requiring engagement in several Bank Group instruments, such as country strategies, DPF, and IFC investment projects; and what progress has been made to date in improving monitoring and reporting on results. The Bank Group has progressively embraced the mandate to mainstream citizen engagement, which opens opportunities and involves risks, as stressed by staff closely involved with this agenda. The corporate commitment is increasingly understood—especially by citizen engagement focal points and social development specialists—as creating an “authorizing environment” to go beyond mere compliance and step up the level and quality of engagement. Many staff applaud the fact that the Bank Group is setting itself up to better document, measure, and monitor citizen engagement activities;, however, several staff highlighted two main risks. A “check-the-box” attitude can seep in when cross-cutting themes are mainstreamed across a large portfolio of activities. Also, some believe that, at least in the short term, the strategic framework may not have stimulated more and different activities but simply prompted staff to better document what was already happening in the field. Evidence of both patterns appears in the portfolio of projects reviewed for this evaluation. Embedding Citizen Engagement in Investment Operations World Bank IPF The World Bank currently measures the progress of its corporate commitment using three process indicators. The first indicator tracks the share of investment projects with a “citizen-oriented design,” which means having at least one citizen engagement mechanism among the project activities. A target of 100 percent of all projects with identifiable beneficiaries has been set up for FY18.1 The second indicator (included in the World Bank Corporate Scorecards) tracks whether projects’ results frameworks incorporate at least one beneficiary feedback indicator, for which the target is also 100 percent. A third indicator measuring the percentage of projects that report on the beneficiary feedback indicator (or demonstrate credible progress toward reporting on this indicator) at least by the third year of implementation has been piloted but not rolled out yet.2 There is no clear-cut description of how citizen-oriented design and beneficiary feedback indicators should be formulated, which complicates the screening of projects and interpreting progress in the corporate indicators. As closing the feedback loop is critical to the definition of engagement, activities and indicators should be defined to ensure that the feedback loop with citizens has been or will be closed and to inform engagement during implementation, not simply at design. However, to date, no Independent Evaluation Group | World Bank Group 13 clear inclusion and exclusion rules are enforced for activities and indicators to meet good practices,3 thus fulfilling these characteristics is not necessary to “count.” The rather unselective nature of the current definition of citizen-oriented design, and of indicators capturing implementation and results, raises critical questions related to the quality of engagement, which will be tackled in the next chapter. The Independent Evaluation Group (IEG) portfolio review found that, adopting this loose definition, almost all World Bank investment project appraisal documents include a citizen-oriented design, that is, at least one planned mechanism for engaging citizens. The 92 percent coverage has not changed between FY11–13 and FY14–16 (the more recent figures provided by the Citizen Engagement Secretariat are 100 percent in FY17 and 98 percent in FY18). Engagement mechanisms mandated by safeguards continue to be the most prevalent; encouragingly, nonmandatory mechanisms significantly increased in the most recent period. In FY11–13, 86 percent of projects had safeguards-mandated engagement, a percentage that remained basically unchanged (81 percent in FY14–16).4 By contrast, the percentage of projects including at least one citizen engagement mechanism not linked to safeguards significantly increased, from 67 percent in FY11–13 to 76 percent in FY14–165 (84 percent in FY16).6 This is a promising result, likely reflecting the increased awareness and buy-in of the corporate commitment. The type of mechanisms embedded in projects have not changed since the introduction of the framework, except for a significant increase in the percentage of projects with GRMs. As shown in figure 2.1, the most common mechanisms in World Bank IPF projects remain consultations (in 86 percent of projects) and GRMs (in 67 percent of projects), both typically related to safeguards (71 percent of consultations and 53 percent of GRMs were mandated by safeguards). Engaging with beneficiaries at large—as opposed to only those likely to be negatively affected by the project—was one of the goals of expanding citizen engagement beyond safeguards. However, beneficiary surveys, focus groups, and interviews to collect the views of all citizens affected by a project can only be found in a third of projects and have not become more prevalent during the past three years or in FY16 alone. The most common mechanisms in World Bank IPF projects establish a “light” degree of engagement,7 consisting of “informing” and “consulting.” More intense forms of engagement, such as “collaborating” and “empowering,” remain much scarcer in the World Bank portfolio. Only 20 percent of projects have mechanisms for citizen monitoring and evaluation; a similar percentage support community collaboration. Projects that empower communities with resources and decision-making powers, mostly CDD, are about 8–9 percent (figure 2.1). IFC Performance Standard 1 IFC has leveraged its updated PS on E&S sustainability to prompt its clients to expand their stakeholder engagement. For IFC, the E&S PS are the systematic entry point for citizen engagement (or stakeholder engagement, as IFC refers to it). Even if IFC clients are not bound to the 100 percent corporate commitment, they are required to carry out stakeholder engagement in all investment projects based on PS 1.8 The 2012 E&S sustainability update continues to be a risk management 14 Engaging Citizens for Better Development Results | Chapter 2 FIGURE 2.1 Citizen Engagement Mechanisms Used | 86% Consultation 86% 41% Grievance redress 67% Collecting, recording, and reporting on 34% inputs received from beneficiaries 31% 24% Capacity building for engagement 23% Citizen/community collaboration 28% on planning and/or execution 20% of a policy, program or project Citizen monitoring, evaluation 16% and oversight 20% Empowering citizens/communities with 8% resources and decision-making power 9% 0 20 40 60 80 100 Percentage of projects with citizen engagement mechanisms FY11–13 FY14–16 Source: Independent Evaluation Group portfolio review of investment policy financing. Note: Percentages add up to more than 100 percent because each project can have multiple mechanisms. framework, which posits that riskier projects (for instance, those that trigger PS 5 on Land Acquisition and Involuntary Resettlement or PS 7 on Indigenous Peoples) are expected to promote a deeper engagement with affected communities. However, it does promote engagement with communities as a normal business practice (box 2.1). After the update in 2012, IFC client companies do indeed resort to a broader array of engagement activities—based on appraisal information analyzed by IEG for two samples of projects approved before and after the 2012 policy update (see appendix D for methodological information).9 As shown in figure 2.2, clients plan to engage with communities through a variety of mechanisms more frequently in FY15–17 than in FY07–11. GRMs and disclosure of information to stakeholders are now much more likely to be included in project design, with an increase of 30 percentage points between the two periods. Although plans for stakeholder identification were almost absent before 2012, they are now found in more than a quarter of projects. IFC progressively promoted forms of community engagement directly related to the client’s core business activity, or “project-oriented activity,” per PS 1. Prior to 2012, it was common for IFC to accept corporate social responsibility (CSR) activities (such as providing support to local schools, road maintenance, and training for women) as a type of community engagement, even if these activities were entirely unrelated to the company’s business. Since 2012, the share of CSR activities Independent Evaluation Group | World Bank Group 15  takeholder Engagement in International Finance Corporation Performance Box 2.1 | S Standards The International Finance Corporation (IFC) Performance Standards (PSs) highlight the importance of managing risks but also the need to identify impacts as a way of doing business in a sustainable way. PS 1 identifies stakeholder engagement as “the basis for building strong, constructive, and responsive relationships that are essential for the successful management of a project’s environmental and social impacts” (IFC 2012, 7). It envisages engagement as an ongoing process that may involve stakeholder analysis and planning, disclosure and dissemination of information, consultation and participation, grievance mechanisms, and periodic reporting to affected communities. PS 1 applies to all projects that have environmental and social risks and impacts. These are essentially all projects, although the nature, frequency, and level of effort of stakeholder engagement may vary considerably depending on the project’s risks and adverse impacts, and the project’s phase of development. Although the policy does not clearly specify which level of risk triggers which type of engagement modalities, riskier projects—those that trigger PS 5 on Land Acquisition and Involuntary Resettlement or PS 7 on Indigenous Peoples—are expected to promote a deeper engagement with affected communities. The previous 2006 PS 1 already required community engagement as an ongoing process involving the client’s disclosure of information. However, consultations and grievance redress mechanisms were required only in cases of anticipated risks of adverse impacts. Source: Based on IFC 2012. accepted as community engagement by IFC has dropped, as confirmed by interviews with IFC staff. Currently, a more typical stakeholder engagement may involve, for instance, an ongoing engagement with farmers who are part of the value chain to help increase the productivity of their cattle, as observed in the case of a Pakistani subsidiary of a leading global dairy processor. This specific IFC client also developed a GRM, confidential and monitored, while involving communities in health and hygiene awareness training in schools. In this example, the stakeholder engagement and inclusive business approaches were both used to engage with the community.10 In expanding its efforts, IFC also seeks to deepen its engagement. It has moved from promoting primarily reactive risk-mitigation activities, such as GRMs, to encouraging more proactive activities, in which clients also plan to inform stakeholders on their business activities and seek feedback 16 Engaging Citizens for Better Development Results | Chapter 2 FIGURE 2.2 Community Engagement Mechanisms in International Finance | Corporation Planned Investments 100 90% 90 83% 80 70 64% Projects (percent) 60 54% 53% 50 47% 44% 40 37% 33% 33% 30 27% 20% 20 17% 13% 10 7% 1% 0 GRM Disclosure of Consultation Focal Point CSR Interviews Stakeholder Feedback Information Mapping FY07–11 FY15–17 Source: Independent Evaluation Group portfolio review of International Finance Corporation investment projects. Note: The difference between the two periods is statistically significant at the 5 percent level for CSR, disclosure of information, GRM, and stakeholder identification and at the 10 percent level for firm-based focal point. CSR = corporate social responsibility; GRM = grievance redress mechanism. throughout the life of the project. Especially high-risk projects increasingly plan repeated interactions with communities—in FY15–17, 57 percent of projects did, doubling the 26 percent in FY07–11. While setting up mechanisms to respond to grievances, IFC clients also envisage involving communities more frequently in project design. PS 1 opened more avenues for engagement, but shortcomings remain. The policy is vague in defining the level of engagement expected—it applies to all business activities but in a way commensurate with the level of the project’s E&S risks and impacts, without clearly specifying what is required at each different risk level. Despite improvements, there are still projects without a GRM or a system for disclosing project information, both required by PS 1 particularly for projects that may have an adverse impact on local communities. Stakeholder engagement plans, also a requirement, are absent in most IFC projects approved between FY15 and FY17, per project documents. Crucially, weaknesses in monitoring and documenting results, as discussed later in this chapter, are prevalent. Bringing Citizens to the Table in Policy and Strategy Discussions Since the late 1990s, the Bank Group has carried out consultations for the preparation of country strategies, which have sometimes generated friction between the World Bank and civil society. A Independent Evaluation Group | World Bank Group 17 2013 review of World Bank engagement with CSOs reported that CSOs were consulted in 90 percent of country strategies approved by the Board during FY10–12 (World Bank 2013). Yet, CSOs often harbored a perception that the Bank Group “only talks to the usual suspects,” as confirmed by interviews and the online survey of CSOs. The 2014 country engagement model introduced a more inclusive and participatory consultation process, with the goal of enhancing the relevance of country programs to citizens’ needs and increasing their ownership of the strategy (World Bank Group 2014b). Bringing citizens to discuss their countries’ development challenges and how to achieve better development results are central tenets of the new model. In the spirit of establishing a dialogue and encouraging consistent follow-up, the new model requires two sequential consultation processes: first, at the time of preparation of the Systematic Country Diagnostic (SCD) and then during the preparation of the CPF.11 Responding to the new requirements, most country strategies are being prepared in close consultation with a more diverse range of stakeholders: local or subnational governments and relevant line ministries, civil society, development partners, and—more systematically than in the past—the private sector. Almost all CPFs reviewed for this evaluation (43 out of 46),12 refer to having consulted civil society. Indigenous groups, youth, and women groups were specifically referenced in 43 percent of the documents and consultations with local governments and local NGOs in 63 percent of the CPFs reviewed. DPFs generally comply with the requirement of conducting stakeholder consultations as laid out in operational policy (OP) 8.60 (World Bank 2014d). As with country strategies, several requirements for consultations are in place, with the objective of informing the design, implementation, and monitoring of reform programs. IEG’s review of a random sample of DPF operations approved between FY11 and FY16 found that almost all (57 of 60) included some reporting on the consultation and participation of stakeholders. However, most DPF consultations were conducted in the framework of the discussion of a country’s national development plan or poverty reduction strategy instead of focusing on the reforms at stake. Only one-quarter of the program documents included some discussion of the outcome of the consultations.13 Monitoring the Effort The share of projects with an indicator to track beneficiary feedback in their results framework (the second corporate-level indicator) is high and has increased quickly over the past two years. Before the introduction of the corporate scorecards indicator, 42 percent of projects had at least one beneficiary feedback indicator in their results framework. Afterward, the share of projects with at least one indicator increased to an average of 63 percent over FY14–16, with a steady annual increase up to 95 percent in FY16, based on the IEG IPF portfolio review. This trend testifies to a high level of compliance with the corporate requirement. At this initial stage, the corporate indicators may have surfaced numerous already-planned and implemented activities not measured by any indicator in 18 Engaging Citizens for Better Development Results | Chapter 2 the project’s results framework, thus fulfilling a basic objective of the strategic framework—namely to improve monitoring at the corporate level. However, this rapid upward trend hides three significant challenges. First, the level of reporting on the indicator is low. Second, the type of citizen engagement most commonly tracked is at odds with what is most frequently planned. Third, indicators are primarily process—or at best—output oriented and fail to capture the higher-level outcomes of engagement mechanisms. Only 57 percent of the projects approved in FY14–16 with engagement indicators in their results frameworks have reported on at least one of their citizen engagement indicators. This low ratio suggests a gap between intent at design and implementation. Focusing on the projects approved in FY14, whose implementation is more advanced, this percentage is 58 percent—still very low. A higher percentage (77 percent) is reported by projects approved in FY13; yet, it is a cause for concern that four years after approval, about one-fourth of projects are not reporting on the implementation of citizen engagement activities as per indicators included in their results frameworks. The corporate commitment to report on progress in implementation, which will start in FY19 for projects approved in FY16, may provide a much-needed incentive to improve reporting. A major discrepancy exists between what the organization tracks and the most frequently used citizen engagement mechanisms: consultations and GRMs. Out of 276 projects with at least one engagement mechanism, 237 (or 86 percent) include a consultation (figure 2.3). Yet, only 8 percent have an indicator in their results framework to monitor and report on consultations. Fifty-six percent of projects have a GRM, but only 19 percent have GRM-related indicators. The correspondence between mechanisms and indicators increased in FY16 for consultations and GRMs, but not for the other mechanisms. Thus, as an institution, the World Bank knows very little about the implementation of its main engagement mechanisms, let alone about the results achieved. In addition, the World Bank rarely relies on citizen-led monitoring and oversight as a source for tracking citizen engagement in its projects, results frameworks (18 percent of projects)—a missed opportunity, as this is where citizens could contribute their unique vantage point. The evaluation’s eight case studies of projects engaging citizens in oversight activities demonstrated the high value of their input, especially through rigorous community scorecard methodologies, such as those implemented in the multiple operations supporting the Dominican Republic’s conditional cash transfer system, the Cambodian Demand for Good Governance project, and the Ethiopian Promoting Basic Services program. Indicators embedded in results frameworks rarely measure the contribution of citizen engagement to project outcomes because most indicators are process or output oriented, and reporting in self- evaluation is scarce. Per this evaluation’s portfolio review, GRM was the only mechanism for which a majority (66 percent) of indicators included in results frameworks were outcome oriented (typically, the percentage of grievances addressed). Only 14 percent of indicators referred to specific service standards, such as resolutions within a specific time frame. For all other mechanisms, only a few or no indicators captured the effect of citizen engagement on the community or the project. For example, IEG found that most (76 percent) of the 26 indicators on consultation in the sample were Independent Evaluation Group | World Bank Group 19 FIGURE 2.3  iscrepancy between What Is Planned and What Is Tracked |D (FY11–16) Consultation 8% 92% 86% Grievance redress 19% 81% 55% Collecting, recording, and reporting on inputs received 56% 44% 33% from beneficiaries Capacity building for 18% 82% 24% engagement Citizen/community collaboration 32% 68% 24% Citizen monitoring, evaluation 4% 96% 18% and oversight Empowering citizens/ communities with resources 25% 75% 9% and decision-making power 0 10 20 30 40 50 60 70 80 90 100 Percentage of projects with citizen engagement mechanisms Share of projects with indicator to Share of projects with no indicator track mechanism to track mechanism Source: Independent Evaluation Group portfolio review of investment project financing. output oriented, capturing whether consultations had taken place but not their results. Similarly, of the 40 indicators tracking citizen collaboration mechanisms, only two measured outcomes (citizens’ satisfaction with the collaboration process); the rest tracked outputs, such as the number of farmers organized in interest groups or the share of user groups that completed community plans for water usage. CPFs that explicitly articulated the intended results of engaging citizens at the country level were rare (18 of 46); even rarer were those seeking to measure progress (11 of 46). Those that couched citizen engagement in a larger governance agenda sought to improve transparency and accountability or service delivery. In conflict-affected countries, engagement aimed to enhance social cohesion and trust between citizens and the state. A minority of CPFs tracked progress on these objectives through indicators in their results framework, most of which were linked to a specific project rather than trying to measure success at the level of country systems. As with country strategies, few and generic details on DPF consultations were provided. Forty percent of DPF projects reviewed did not specify which stakeholders were consulted, and among the rest, only a small subset specified who was consulted beyond very general categories such as “academia,” “CSOs,” and “private sector.” The positions or views of the stakeholders on the discussed reforms were mentioned in less than one-third of the program documents. 20 Engaging Citizens for Better Development Results | Chapter 2 Feedback mechanisms at project closing have not been consistently reported. The IEG evaluation of self-evaluation systems of the Bank Group found shortcomings in how Implementation Completion and Results Reports (ICRs) documented citizen engagement activities in IPF, such as poor reporting on activities mandated by safeguards and little integration of beneficiary survey results in the body of ICRs to support project’s rating (World Bank 2016a). IEG’s evaluation of self-evaluation systems found that just above one-third of self-evaluations provided some reporting on consultations, only a tenth described how citizens’ complaints had been addressed, and as little as 2 percent described how suggestions had been considered.14 As for DPF, only 4 of the 60 DPF projects reviewed for this evaluation mentioned engagement activities carried out at the end of the operation, including seminars or forums to discuss the progress of the reforms. Although these operations were not subject to the commitments of the strategic framework, the poor level of reporting at closing does not allow an understanding of the results of the OP, as highlighted in the DPF retrospective (World Bank Group 2015, 46). Reporting by IFC clients of stakeholder engagement activities is partial and largely driven by compliance rather than focused on outcomes achieved.15 High-risk projects (those triggering PS 5 on Land Acquisition and Involuntary Resettlement) more often provide detailed evidence on implementation of stakeholder engagement activities planned at design than non-high-risk projects, which was to be expected given the stronger emphasis of the 2006 PS 1 on risk management. However, only 5 percent reported on outcomes, such as changes in the number of complaints, measures taken by the firm to resolve issues raised by the community, and decrease in conflict at the community level, whereas 67 percent focused on compliance with requirements, providing information only on inputs or outputs (for example, whether consultations took place) rather than results. IFC interpretation of compliance is loose. Although IFC considered them in compliance, 26 percent of investments provided too little information to understand how the client engaged with the community. A similar percentage of high-risk projects, involving land acquisition or involuntary resettlements, also did not provide sufficient information on how the client engaged with the community. Implementing the Mandate: Organization, Resources, and Incentives By committing to include beneficiary feedback in 100 percent of investment projects with clearly identified beneficiaries, the strategic framework adopted a standardized approach to mainstreaming citizen engagement in the Bank Group and measuring its results. This choice has defined how citizen engagement is conceptualized internally and the incentives, motivations, and support devoted to the implementation of this agenda. But is the proper support in place—especially organizational arrangements, resources, and incentives—to meet the mandate? Over the past few years, senior management has consistently signaled that citizen engagement is a top priority, which has contributed to enhanced buy-in among staff. Most respondents (86 percent) to the IEG staff survey agree that engaging with citizens should be a responsibility of the Bank Group. Independent Evaluation Group | World Bank Group 21 Almost two-thirds indicated that they pay more attention to engaging citizens in their work because of the corporate priority. When asked for their underlying motivation, staff suggested that citizen engagement can improve design and iron out implementation issues, thus improving effectiveness of operations; strengthen accountability in service delivery; and help in mitigating risks and anticipating problems. As summed up by citizen engagement focal points, “The question on team leaders’ minds is no longer whether to engage citizens but rather how to engage.” This change in mind-set, whereby the value of engaging citizens is now accepted and encouraged in the Bank Group’s work, is a major achievement of the strategic framework. Some interviewees also highlighted that the mandate has provided a platform for “enterprising” social development and safeguards specialists to go beyond safeguards and—depending on the local needs and context, the country management unit’s support, and available resources—promote a deeper stakeholder engagement than strictly required by operational policies. This heightened, institution-wide endorsement of citizen engagement is an important success. At the same time, the emphasis on the corporate indicators and the corresponding target entails intrinsic risks. Staff signal that with its emphasis on quantity and compliance with the corporate commitment, as opposed to quality, the corporate indicators risk being perceived as yet another requirement and generating a check-the-box attitude rather than a genuine focus on how to improve the quality of citizen engagement activities. The fact that the Bank Group has very quickly “met the target” also created skepticisms among CSOs and observers about the nature of the commitment and its operationalization modalities. The existence of a potential trade-off between quantity and quality will be raised in the next chapters. The Citizen Engagement Secretariat implemented the monitoring of the corporate requirement with good results. Led by the Social, Urban, Rural, and Resilience and the Governance Global Practices (GPs), the secretariat is the “custodian” of the citizen engagement agenda and is tasked with monitoring the corporate requirement and providing a basic coordination of skills and resources. Supported by a team based in Chennai, the secretariat has worked on operationalizing definitions to screen activities and projects. It has not yet rolled out the guidelines to implement the third indicator (monitoring citizen engagement at implementation) and has not set up and enforced strict quality standards. Citizen engagement focal points, two for each Region and one for each GP and Global Theme Group, provide expertise and support to operational teams to meet the corporate goals. This system works better in the Regions—where most focal points are social development specialists—than in the GPs, where the type of commitment and expertise vary greatly. Social development specialists have the relevant skills and often the greatest motivation and enthusiasm to go beyond mere compliance but are overstretched, especially after the downsizing of the Governance GP. Regional focal points are the engines of the citizen engagement agenda; they have been very active in, for example, ensuring additional funds from country management units for activities and coordinating the production of stocktaking reports and road maps for individual regions and countries. 22 Engaging Citizens for Better Development Results | Chapter 2 Limited budget, time, and expertise to devote to citizen engagement activities are identified by survey respondents as the most important internal constraints and may be a threat to implementation. Limited funding from client governments is also considered as a top (external) constraint, second only to the reluctance of client governments to engage with citizens (figure 2.4). Many focus group participants and interviewees indicated that the mandate requires more systematic engagement without providing extra budget for it. Additional financial resources have been allocated for the corporate monitoring of citizen engagement, but their amount is barely sufficient. Activities required to comply with the corporate commitment (organizing clinics, producing guidelines, reviewing project appraisal documents, and reporting on indicators) have been supported by a limited annual allocation, which in FY17 was about $35,000 per Region, irrespective of the volume of regional IPFs approved. This budget was only made available well into the FY through the Citizen Engagement Secretariat. A steadier, more predictable and proportionate flow of resources could improve planning of regional activities. The Middle East and North Africa, Europe and Central Asia, and South Asia Regions added extra resources to strengthen the monitoring of citizen engagement activities. Individual Regions have supported region- and country-specific initiatives to provide guidance and gather knowledge of what works. The Middle East and North Africa Region was a leader in mainstreaming citizen engagement in operations by issuing the first guidance note in 2014 (Bousquet and Ross-Larson 2014; World Bank 2014b). A study conducted by the Africa Region in 2014 FIGURE 2.4 Main Constraints to the Implementation of the Agenda, According | to Staff (a) Internal constraints (b) External constraints Limited budget available 66% Reluctance of client 73% government to engage Limited time to dedicate 52% to this effort Reluctance of client govern- 58% ment to fund activities Limited availability 42% of expertise Clients worry projects 45% becoming too complex Limited training and guidance 35% Government/PIU lack 45% Limited evidence of capacity on effectiveness 25% Negative impact on Limited capacity of 39% World Bank/staff reputation 17% civil society organizations Limited support from Limited willingness of 12% 15% citizens to participate management Internal challenges 10% External challenges are 1% are negligible negligible Other 11% Others 7% 0 10 20 30 40 50 60 70 80 0 10 20 30 40 50 60 70 80 Percentage agree Percentage agree Source: Independent Evaluation Group staff survey. Note: Percentages add up to more than 100 percent because respondents could identify up to three constraints. Independent Evaluation Group | World Bank Group 23 assessed how and how often projects adopted citizen engagement, what worked, and which factors drove the results obtained (World Bank 2014c). In South Asia, country-level reviews of projects have been conducted for Afghanistan, Bangladesh, Pakistan, and Nepal to identify operational challenges and lesson learned. The publication of learning notes supports training and exchange of knowledge. Numerous very relevant social development how-to notes were published in the series “Dealing with Governance and Corruption Risks in Lending”16 before the strategic framework was adopted (for example, World Bank 2011a, 2011b). Nevertheless, project teams signal that they would benefit from more clarity and coordination. Even team leaders committed to the agenda are frequently not aware of available resources. Because modalities of assessing compliance and providing support are not standardized across Regions, they do not know exactly where and when to seek help. In the absence of adequate time and resources, team leaders prefer to consult first with safeguards experts to ensure compliance with operational and World Bank policies and only later, when there is little time for substantive work, to seek advice from the citizen engagement focal points. IFC increasingly provides training and support to clients to improve their stakeholder engagement, although the involvement of IFC specialists in stakeholder meetings is still infrequent. Based on a random sample of 30 projects, IFC specialists visited client sites in 25 cases (83 percent); in 22 cases, they met with the client’s key personnel; only in 8 cases did they meet with other stakeholders. Interviews with E&S specialists highlighted that IFC support shifted from checking compliance to providing assistance to clients. Clients receive increasing support not only on how to develop an E&S management system and stakeholder engagement plan but also training on how to communicate with communities. Moreover, IFC provides an oversight role by visiting affected communities, following up on communities’ concerns, and ensuring that clients perform to minimum standards and implement their plan—although the extent of this direct engagement is generally limited. 1 B  ased on the principles of the corporate framework, intended beneficiaries should be citizens, CSOs, farmers, or small enterprises, not ministries or implementing units. 2  he World Bank will report on this indicator during fiscal year 2019 to reflect citizen engagement activities T implemented in projects approved during fiscal year 2016. 3 A  ccording to the Citizen Engagement Secretariat, a “beneficiary-oriented design” includes an explicit description of at least one mechanism to engage with ultimate beneficiaries of the project (or affected population) and how the feedback loop will be closed (that is, establishes robust mechanisms to ensure that the feedback will trigger a response). Two criteria are proposed to identify beneficiary feedback: (i) reporting on whether and how the feedback loop was closed; and (ii) clearly capturing feedback from citizens or monitoring the degree of involvement in decision making that citizens have during implementation or oversight of projects. As for “closing the feedback loop” it can either mean informing citizens about what has been done with their feedback or acting on the feedback. In some cases, the definition includes both. 4  his is the percentage of projects in the Independent Evaluation Group portfolio review that triggered social T safeguards and required citizen engagement even before the introduction of the strategic framework. 5 “Significant” changes are statistically significant at the 5 percent level, when not otherwise specified. 24 Engaging Citizens for Better Development Results | Chapter 2 6 Several statistics have been calculated for FY16 to provide an indication of the latest trends, but – due to the small  sample size (39 projects), and the overrepresentation of SURR projects in the FY16 subsample – they should be interpreted with great caution. 7  Adopting the typology of citizen involvement in decision making laid out in the World Bank Group strategic framework, mechanisms aimed at informing and consulting are those implying a low level of interaction and minimal involvement in decision-making (World Bank Group 2014a, 8). 8 Performance Standard 1: Assessment and Management of Environmental and Social Risks and Impacts. 9 The Independent Evaluation Group reviewed a sample of investments approved between fiscal years 2007 and 2011,  and a smaller sample approved between fiscal years 2015 and 2017. 10 S  takeholder engagement as envisaged by PS 1 is distinct from the inclusive business approach implemented by IFC since 2010—although there may be a relationship between the two. Stakeholder engagement is primarily seen as a way to successfully identify and manage environmental and social risks, although the 2012 update of the E&S standards emphasizes the importance of an early, ongoing, and proactive engagement, which can help prevent as opposed to just mitigate risks as per the more traditional approach. Inclusive business is about integrating suppliers, distributors, retailers, or customers at the “base of the pyramid” as part of the IFC client’s core business model. This approach relies on the client’s knowledge of suppliers and beneficiaries in the value chain and the project’s impacts on these various stakeholders, which can be acquired through stakeholder engagement. The example of the Pakistani dairy processor illustrates how stakeholder engagement may be conducive to the inclusive business approach. The IFC’s experience with inclusive business has been assessed by an IEG meso-evaluation, forthcoming in 2018. 11  he Systematic Country Diagnostic aims to identify the most critical constraints to, and opportunities for, reducing T poverty and building shared prosperity sustainably, while considering the voices of the poor and the views of the private sector. The Country Partnership Framework is a joint (World Bank Group-government) strategy document that guides the focus areas for Bank Group support for the next 4 to 6 years given the key constraints and opportunities identified in the Systematic Country Diagnostic, the Bank Group’s comparative advantage in relation to other donors, and the country’s own development agenda. 12 This evaluation reviewed all CPFs approved between FY15 and FY17. 13  er the World Bank Group’s own assessment: “virtually all [development policy operations] reviewed in this P Retrospective discussed the country’s consultative and participatory process used in the formulation of the operation. However, less than half of the Program Documents described the outcomes of the process, making it difficult to assess to what extent the feedback that was gathered had an impact on the policy design” ( World Bank Group 2015, 46 ). The retrospective reviewed all operations approved between April 2012 and December 2014, which corresponds to 165 development policy operations and two supplemental financing operations. 14 T  he ROSES evaluation found that, out of 197 projects that closed in FY14, the majority included citizen engagement activities, especially consultations, triggered by safeguards policies. Yet, ICRs did not systematically report on these activities, at the level of either output or outcome. Moreover, 33 percent of projects (66 out of 197) mentioned beneficiary surveys in their ICRs. However, the results of those surveys were not discussed in the ICR or used to justify ICR ratings. Most often, they were relegated in appendixes. 15 Reporting has been analyzed only for the sample of projects approved in FY07–11. 16  ee the Governance and Anticorruption Strategy Knowledge and Learning Resources Database: https://www. S governanceknowledge.org/pro/Pages/Resources-Database.aspx. Independent Evaluation Group | World Bank Group 25 3 highlights Enhancing the 1 The World Bank Group has rightfully made the Quality of quality of citizen engagement a cornerstone of Citizen its mainstreaming commitment. The evaluation Engagement focused on four quality principles highlighted by the strategic framework: results orientation, continuity and feedback loop, social inclusion, and capacity enhancement. 2 Although closing the feedback loop with citizens is an intrinsic part of the Bank Group’s engagement, it is neither mainstreamed nor tracked. Inclusion of women and marginalized groups is not systematically scrutinized. 3 Numerous examples of high-quality mechanisms appear in the World Bank’s portfolio as well as some encouraging trends: projects that use multiple engagement mechanisms and pay attention to social inclusion are on the rise. 4 However, thin engagement remains the most prevalent and essential capacity building for engagement is limited. There is significant x 26 scope for pursuing opportunities for deeper engagement when the context allows. 5 The insufficient attention paid to quality principles points to a trade-off between meeting the mostly quantitative corporate targets and ensuring quality of engagement. Independent Evaluation Group | World Bank Group 27 T H E B A N K G R O U P strategic framework rightfully makes quality an integral part of the mainstreaming process. It sheds light on quality principles that should underpin the design and implementation of engagement activities. This chapter assesses the extent to which the Bank Group’s engagement incorporates four quality principles: First, aiming for results through a deep level of engagement by using multiple and innovative tools; second, supporting regular engagement and dialogue throughout the operational cycle and closing the feedback loop by providing a tangible response to citizens’ feedback, thereby meeting citizens’ expectations for change and justifying the opportunity cost of their engagement; third, ensuring inclusion of diverse voices, including women and vulnerable groups to avoid capture and reinforcing exclusion (which is treated by the strategic framework as a cross-cutting principle); and fourth, building engagement capacity of both citizens and governments. The evaluation could not systematically assess the impact of contextual factors, but their influence is noted in several case studies discussed. A final principle, concerning the modalities of the approach, which should be “gradual, iterative, and scalable” will be discussed in the next chapter when analyzing the corporate environment.1 This evaluation recognizes that, although some of these principles are essential for engagement to exist, others cannot realistically be mainstreamed in all projects. Closing the feedback loop and inclusiveness are intrinsic dimensions of engagement, but promoting “thick” engagement and building capacity at the country level are only possible under certain conditions (World Bank Group 2014a, 8). The scope and quality of citizen engagement depend on the country context (including the type and quality of its institutions, state legitimacy, and accountability systems, among other aspects), the commitment of the client government, and the World Bank’s ability to open opportunities to work with the client and provide adequate support. Nevertheless, various syntheses of the existing evaluation literature show that participation mechanisms that do not adhere to the quality principles highlighted in the strategic framework and analyzed in this chapter are less likely to positively affect development outcomes and may result in “civil society failures” that reinforce exclusion (Mansuri and Rao 2013; Ringold et al. 2012; Gaventa and Barrett 2010; Menocal and Sharma 2008; Grandvoinnet, Aslam, and Raha 2015; Fox 2015). The evaluation found that none of these quality criteria were embedded in how citizen-oriented design is defined or tracked at the corporate level. The two corporate indicators are silent on whether engagement is continuous or one-off, inclusive or not, system enabling, or project-centric. Consequently, the evidence underpinning this chapter goes beyond the Bank Group’s own metrics and relies on additional coding of project documents in the various portfolio reviews as well as case study evidence, triangulated with impact literature. As evidence on quality is essentially absent for IFC stakeholder engagement, this part is not covered in this evaluation, although the key findings of the evaluation may be relevant to strengthening IFC’s new approach to PSs. 28 Engaging Citizens for Better Development Results | Chapter 3 Aiming for Results Although the responsibility for engaging citizens in a broader country context lies with the government, the World Bank’s role is to facilitate processes of engagement through its operations and policy processes. For citizen engagement to affect projects and communities, deploying “thick” approaches that combine multiple tools to enable collective action and public sector responsiveness is the most promising strategy. Case studies conducted for this evaluation confirm their importance, which is corroborated by the impact evaluation literature (Fox 2015; Grandvoinnet, Aslam, and Raha 2015). The progress is encouraging, as the share of projects with thick engagement increased from 27 percent in FY11–13 to 38 percent in FY14–16, with significant growth in the most recent period. Archetypes of thick approaches include the Cambodia Demand for Good Governance project and its follow-on, the Dominican Republic’s citizen report card embedded in the national conditional cash transfer program and its equivalent in Morocco’s waste management sector, the participatory methodology to develop investment plans in the Palestinian Municipal Development Project, and the Ethiopia Promoting Basic Service Project. Yet, except in governance, social development projects, and formal CDD, the World Bank still favors “thin” approaches using only one or two mechanisms, which are short-lived and focus exclusively on gathering citizen feedback. Sixty-eight percent of the investment projects with citizen-oriented design in the evaluation sample (FY11–16) have thin engagement. These approaches have shown few results on their own and are based on overoptimistic assumptions (Fox 2015). Establishing thick engagement requires a good understanding of power dynamics and a long-term commitment by both the World Bank and the client government. In both Ethiopia and the Dominican Republic, the World Bank seized the opportunity opened by the political juncture (the 2005 postelection crisis in Ethiopia and the strong anticorruption stance of the Dominican society) to call the client to higher accountability standards by adopting institutional mechanisms for engagement. The Demand for Good Governance project in Cambodia used political economy analysis at entry to identify the leaders and state institutions that were likely to support social accountability approaches, which was an effective strategy (World Bank 2017a). Because it aims to change country systems for engagement, this process typically requires time and a long-term investment. It is instructive that the Cambodia and the Ethiopia projects were long-running well before the introduction of the strategic framework and have had multiple phases. The thick approach requires adaptation through continuous learning, which cannot be achieved through a one-off short-term intervention—and cannot be realistically pursued by every project in every country. When engaging citizens in the development of country strategies, teams are encouraged to use a variety of tools for stakeholder engagement,2 but they rarely tap into this range of options. By and large, country teams use traditional in-person consultation methods. Country opinion surveys were used in only 26 percent CPFs, despite being available in 87 percent of cases.3 Team leaders Independent Evaluation Group | World Bank Group 29 interviewed found the insights from the surveys to be of limited value for the CPF preparation, due to their high level of generality and their poor timing. Despite the proliferation of civic technology tools, such as social media or online consultation, only a few of the CPFs reviewed took advantage of these alternative models of engagement to reach a broader audience. This practice is still not widespread, except in the context of fragile and conflict- affected areas. For instance, the Bank Group strategically used social media in Myanmar, where it was reengaging after a long time, and in the Republic of Yemen, where it is striving to engage citizens despite the ongoing conflict and political uncertainties. There, the team held a series of online consultations with approximately 20,000 respondents drawn from different segments of the Yemeni civil society and envisages to use online consultations throughout the Country Engagement Note period to keep a pulse on Yemenis’ views. When complemented by traditional engagement mechanisms, civic technology platforms can be effective in projects. In the Pakistan Punjab province, the World Bank leveraged the high penetration of mobile phones among the poorest households to develop feedback data sent via SMS or call centers on service delivery. Almost 700,000 citizens provided feedback on the services of the police, urban development, land revenue, emergency response, and health and food departments. District- level scorecards were sent to the chief secretary for discussion on required actions, leading to almost 4,000 corrective actions based on the feedback. The early-stage qualitative evaluation of the Punjab model indicated that one of its most powerful features was its capacity to generate real-time data on service provision through a direct communication channel with citizens but also to link this bottom-up feedback with top-down accountability mechanisms (Grandvoinnet, Aslam, and Raha 2015). Continuous Engagement and Closing the Feedback Loop The strategic framework clearly establishes that one-off consultations are not sufficient for meaningful engagement and recommends engagement throughout the intervention cycle; however, this is not yet common practice. For instance, a recent Inspection Panel review of consultations and participation contends that these engagement modalities should foster periodic two-way communication from design to evaluation, keep people informed, and adequately respond to feedback from communities (World Bank 2017b). Yet, as informed by the panel case load, other reviews conducted by the citizen engagement community, and evidence from this evaluation’s case studies, it seems that reality is distant from these good practice principles. CSOs interviewed or solicited online during this evaluation highlighted the sporadic nature of their engagement with the World Bank. They mentioned being rarely involved beyond design and conceptual stages of strategies and projects, and not knowing whether and how their contributions were considered. CSOs continue to experience Bank Group consultations as a one-way channel for the organization to share information with them rather than as a dialogue or a collaborative process. In a few cases, the World Bank established a forum to get continuous feedback from civil society. Those forums were mentioned by World Bank teams as a regular channel used in Central America 30 Engaging Citizens for Better Development Results | Chapter 3 (Guatemala, Honduras). In other cases, “CSO sounding boards” are in place (Cambodia) or planned (Montenegro, Togo, Tunisia, Ukraine). Details on the feedback received by those consulted is not commonly included in CPF documents, despite being essential to understanding how the feedback loop was closed. Only 20 CPFs (43 percent) reported the CSOs’ feedback on the themes of consultations and their proposed areas of engagement. A dismal two CPFs (Tunisia and Uzbekistan) clearly documented how the feedback received was integrated. The Consultation Hub, an online one-stop shop for all consultations hosted by the Bank Group,4 is not systematically used to inform future SCD or CPF consultation processes or as a repository of the feedback solicited. Awareness of this resource is low among interviewed task team leaders. Although most teams said they prepared a standard online package describing the SCD or CPF process in English and translated into local language, less than a third reported having used the Hub. Many expressed a preference for using their own country website because they do not get additional support using the Hub. When IEG consulted the site in September 2017, out of all closed consultations within the past year that were supposed to have information there, only four did (Ethiopia, Kosovo, Peru and Ukraine). Ensuring that citizen engagement mechanisms close the feedback loop is a crucial element of what should count as citizen-oriented design in investment lending, yet there is some confusion about what this means and how it should be measured. For instance, the Citizen Engagement Secretariat’s guidance and the citizen engagement strategic framework provide two distinct definitions of the concept. The citizen engagement guidance on operationalization equates closing the feedback loop with “establishing robust mechanisms to ensure that the feedback will trigger a response.”5 This echoes the Operations Policy and Country Services 2014 Results Framework and M&E Guidance Note, which states that indicators “should report progress not only on efforts to engage with citizens, but also on whether there is a tangible response (…) in order to ‘close the feedback loop’” (World Bank 2013, 8).6 There is no clear distinction as to whether closing the feedback loop is solely associated with how feedback can influence World Bank-financed projects themselves, or more broadly the quality of services provided by the client government through the project. On the other hand, the strategic framework equates closing the feedback loop with “informing those engaged on how the information they provided has been used” (World Bank Group 2014a, 18). Multiple definitions of what “closing the feedback loop” means may generate different conclusions on the level of engagement. The evaluation team tested the two different formulations: (i) acting on the feedback of citizens (for instance, by using citizen feedback to improve project design or implementation or to improve the quality and timeliness of services delivered to citizens by the government) and (ii) informing citizens about the actions that have been taken because of their feedback. IEG reviewed the project appraisal document description of all the citizen engagement mechanisms (115) included in the 38 FY16 projects, which likely benefited from better guidance on how to select indicators capturing the closing of the feedback loop.7 Independent Evaluation Group | World Bank Group 31 At design, most of the mechanisms reviewed describe how they plan to take citizen feedback into account, but almost none signal whether they plan to inform citizens about the actions taken in response to their feedback. The IEG review found that around 70 percent of the 115 mechanisms under study met the first criteria, based on project description; compliance was higher for mechanisms mandated by safeguards (83 percent) than for non-safeguards-related mechanisms (67 percent). IEG also found that the compliance rate was high for mechanisms that embody approaches such as consultations (90 percent) but lower for other approaches such as “collecting, recording and reporting on inputs received from beneficiaries” (54 percent). Compliance with the second criteria was almost negligible: only 4 out of the 115 mechanisms elaborated on whether they planned to inform citizens about the actions taken because of their feedback. CDD, more than other types of projects, are likely to promote ongoing engagement and to embed feedback mechanisms on the quality of services received by the communities. Recurrent participation is germane to operations, from mobilizing a beneficiary base, to understanding local preferences and priorities, to adopting co-implementation modalities, such as participatory budgeting, monitoring, assessments, and reporting. In the sample of 36 CDD projects, 60 percent included consultative mechanisms that supported iterative feedback with the community throughout the project cycle.8 Participatory monitoring and evaluation mechanisms—when implemented at key stages in the project cycle—provided critical feedback to enhance project performance. More advanced methods for participatory monitoring, with indicators grounded in the local context and thus able to be more effectively internalized, have been rolled out in the Rural Livelihoods projects in India, with support from the Social Observatory—a part of the World Bank Research Group that works to make participatory projects more adaptive. Other projects, such as Gemi Diriya in Sri Lanka, invested heavily in social accountability tools that increased the transparency of budgeting and lead to more equitable benefit sharing. Relatively common citizen engagement mechanisms, such as GRMs, also have the potential to provide real-time and contextually sensitive feedback on project performance for timely course correction, but this is not yet the norm. Even CDD projects, despite their participatory designs, often lack GRMs that can effectively bring substantive service improvements. Only 45 percent of the CDD projects reviewed described the use of contextually sensitive and adaptive grievance systems, whereas 40 percent included standard (“cut and paste”) language. Some good practice examples of inclusive, citizen-focused systems are the Philippines National CDD Program and the Indian Uttarakhand Decentralized Watershed Development Project, which advertised the GRM using wall writing and radio programs and are run at the village level through interactive workshops.9 Ensuring Inclusion Ensuring that engagement mechanisms are inclusive—especially of vulnerable and marginalized groups—is a fundamental, albeit difficult, part of engaging citizens. A growing body of literature finds that participants in civic engagement activities tend to be wealthier, male, better educated, and with more social and political capital than nonparticipants, a phenomenon known as “elite capture” 32 Engaging Citizens for Better Development Results | Chapter 3 (Mansuri and Rao 2013). Engagement mediated through civic technology is not immune to the inclusion challenge, even if in some cases it can enhance participation (Peixoto and Sifry 2017). How can the Bank Group’s engagement activities avoid elite capture? Although many good practice examples of how to foster inclusive participation exist, notably in CDD projects, basic principles of inclusion are still not the norm. The share of projects describing how the participation of women and other groups would be ensured for at least one engagement mechanism increased from 40 percent in the FY11–13 sample to 58 percent in the FY14–16 sample (72 percent in FY16), showing encouraging progress. There is still, however, a sizeable share of mechanisms that do not pay attention to inclusion issues at the design stage. Attention to inclusion at the design stage may not materialize during implementation, as the monitoring of inclusion in projects remains weak. Less than a quarter of projects with beneficiary feedback indicators disaggregated at least one of them by gender or other social groups, although a positive trend arises. An increase occurred between the two periods, from 20 percent in FY11–13 to 29 percent in FY14–16 with significant increase in the latest year. In CDD, monitoring of vulnerable groups’ participation is more frequent; 40 percent of the projects reviewed adopted targets for representation of marginalized groups. However, indicators and measurement criteria remain vague. In projects with multiple vulnerable groups, participation is measured in the aggregate (for example, percentage of women, youth, and indigenous populations participating in decision making). Moreover, given the careful integration of marginalized groups in project design, opportunities for collecting disaggregated feedback about participation and assessing the influence of marginalized groups on decision making are underused. Only 20 percent of the CDD projects reviewed included this feature. These findings echo evidence previously presented by IEG (World Bank 2016b). The CDD projects supported by the Social Observatory are among the most advanced in promoting an inclusive approach to monitoring. The Social Observatory uses participatory methods to develop results indicators that are relevant to the local context, can be tracked and measured by communities themselves, and can be used in impact evaluations to assess long-term results. For instance, in India, poor women have developed empowerment indicators that genuinely reflect what they consider most important to achieve a power shift at a personal or household level. Such level of depth cannot be achieved everywhere, which creates an area of tension for a mainstreaming strategy. Projects using multiple facilitation tools and metrics more likely support meaningful engagement of marginalized groups throughout the project cycle. For example, in the Afghanistan’s Citizen Charter project, women’s participation is promoted through quotas, female staffing, provision of gender-related training, involving women in community planning and monitoring, and sensitizing the communities on the role of gender in development. Mobilizers’ and interlocutors’ role is often crucial for bridging power gaps among citizens, and between citizens and state officials and create credible interface (Fox 2015; Tembo 2012). The Lao People’s Democratic Republic forestry project sought to use facilitation processes to turn attendance into full participation and engagement of women and the most vulnerable, albeit with modest results based on IEG field visit (box 3.1). Independent Evaluation Group | World Bank Group 33  he Challenge of Inclusion in Lao People’s Democratic Republic SUFORD Box 3.1 | T Project The Sustainable Forestry and Rural Development Project (SUFORD) introduced the concept of participatory sustainable forest management (PSFM). ‘Participation’ in the Lao PDR context is an orchestrated process. The state manages the forest and the villagers living in and around natural forest have a limited role in decision making. Local forestry officials guide villagers through steps including boundary marking and forest inventory, culminating in the preparation of a forest management plan that allows for the sustainable harvesting of timber, with part of the revenue going to villagers. IEG found that, although most of the steps defined in the project operating manual had been followed, the villagers had not internalized PSFM principles. In seven villages that had prepared plans in 2004– 08, villagers had difficulty remembering the principles when interviewed by IEG in 2017. This seems to have been less the result of faulty project design than the consequence of a 2011 logging ban, the universal application of which removed the incentive for villagers to follow the PSFM model. IEG also found that PSFM had been implemented in a less inclusive way than the project design envisaged. Only three of the seven Village Forestry Committees had women representatives; the operating manual says there should be at least one woman per committee. The focus group discussions revealed that women helped prepare the Forest Management Plan in only one of the seven villages. Participation was limited by illiteracy and language barriers, partly aggravated by government’s slowness in producing and disseminating explanatory pamphlets in the local language. In all villages, women lacked basic reading and writing skills and in the ethnic minority dominant villages they did not speak the national language. The plans were drafted by provincial and district forest officers and then presented to villagers for their approval. There was no lengthy or active deliberation by villagers. Source: Independent Evaluation Group. The early implementation of the new country engagement model foreshadows a more inclusive process that reaches a broader set of stakeholders than in the past. For example, consultations used to take place mostly in the country capital, but teams now consult much more frequently with local governments and local NGOs. Although this effort is poorly documented in country strategies (29 of the 46 country strategies stated this specifically), nearly all teams interviewed confirmed that they had consulted outside the capital city. This extensive geographical outreach provided a space for local stakeholders to voice their issues with the World Bank for the first time. For example, teams consulted with monks in Bulgaria, youth in El Salvador, indigenous communities in Panama, people living in Serb majority municipalities in Northern Kosovo, and diasporas from fragile states, such as Myanmar and the Republic of Yemen. 34 Engaging Citizens for Better Development Results | Chapter 3 Partnering with CSOs also improved the outreach and quality of consultations, in the few cases where this occurred, according to team leaders of CPFs interviewed for the evaluation. In the Arab Republic of Egypt, Myanmar, and Tunisia, for instance, the Bank Group partnered with umbrella CSOs and jointly developed a framework for the consultation process, agreeing on timing, mechanisms, whom to invite, and how to document the process. Yet, lack of proper follow-up may hinder this collaboration (for example, Egypt). In other cases, the Bank Group has not applied basic good consultation practices. For example, only rarely were professional facilitators used to moderate the discussions; in most cases, little documentation or preparation materials were shared ahead of the meetings. Some of the concerns outlined by civil society are well documented in a guidance note to CSOs prepared by the Bank Information Center (BIC 2017). IFC’s newly defined role of leading CPF consultations with the private sector on behalf of the Bank Group broadened the range of stakeholders reached. Whereas in the past the World Bank consulted mostly with chambers of commerce or similar umbrella associations, the most recent CPFs have effectively reached out to a wide range of enterprises and small business owners, including nonclients and private sector entities that were outside of the World Bank’s established network. In some cases, these more extensive consultations during the CPF preparation paved the way for a longer-term engagement, as documented in a recent IEG evaluation (World Bank 2017d). Building Capacity to Engage Citizens and state actors’ capacity and willingness to engage represent major constraints to successful engagement per the literature and World Bank staff; yet projects seldom build engagement capacity.10 Systematic reviews of participatory projects found that they often fail to build resilient networks and organizations, and engagement mechanisms tend to dissolve when project incentives are withdrawn. Only when projects seek to build capacity, do they tend to improve collective action beyond the project cycle (Devarajan, Khemani, and Walton 2013; Mansuri and Rao 2013). The case studies and surveys conducted for the evaluation confirm these findings. Meanwhile, less than a quarter (23 percent) of projects reviewed that had at least one engagement mechanism included capacity enhancement activities in their design. Some of the most promising designs seek to build in tandem state and citizen’s capacity to engage. For example, the Santa Catarina Rural Competitiveness Project in Brazil deployed management and organization skill training for both the water resource management department and water river basin committees. Similarly, the Bihar Panchayat Strengthening Project capacity building component targeted both local governments’ members to strengthen their responsiveness to the community, and provided training to community members to participate in local governance and learn how to solicit the Panchayats to improve service delivery. Even in projects where capacity building is considered paramount at design, such as in governance, community management of common-pool resources, or CDD projects, these activities tend to be Independent Evaluation Group | World Bank Group 35 underfunded and under-monitored. In the Demand for Good Governance project in Cambodia, all components included some resources for capacity building, but these proved insufficient early in project implementation (World Bank 2017a). Implementing agencies are often not themselves sufficiently equipped to build local capacity, and sustain participation. In Vietnam, the Forest Sector Development Project successfully strengthened the forest management board but failed to provide communities living in or near special use forests with sufficient benefits to motivate their active management of biodiversity (World Bank 2018b). Project implementation units need incentives, skills, and resources to manage a well-running GRM; yet, building state capacity to address grievances collected through citizen engagement mechanisms requires more efforts than currently deployed. In the Ethiopian Promoting Basic Services Program GRM, there was insufficient attention to implementing units’ capacity. Conversely, when implementing agencies are committed and well-resourced, engagement is stronger. The case of the Kyrgyz Republic’s Community Development and Investment Agency is emblematic: the World Bank built its capacity through a long-lasting CDD project, enabling it to become an effective broker between the communities and local government agencies in many other development partners’ projects. These findings are in line with the literature indicating that the most effective mechanisms have built an interface between citizens and states in their design, along with institutional processes for addressing feedback (Björkman and Svensson 2009; Wetterberg, Brinkerhoff, and Hertz 2016; George 2003; Grandvoinnet, Aslam, and Raha 2015). Outside of projects, the Bank Group has very few channels to provide direct support to civil society; the GPSA is one of them. This partnership program builds capacity and provides grants to CSOs in support of strategic social accountability work aiming to create space for constructive engagement between the government and civil society. The support to CSO capacity building is also complemented by work at global level—building networks of practitioners through a Knowledge Platform and an annual Global Partners Forum that brings together various GPSA stakeholders. The Bank Group’s convening role and financial contribution to the GPSA were critical to kick start the program and mobilize donor support. To date, the GPSA has allocated $31.2 million to CSOs in countries that formally opted in to the GPSA.11 The program has 33 stand-alone recipient-executed projects in 25 countries. The projects are for three to five years in the amount of $0.5 million to $1 million and are supervised by World Bank staff (GPSA 2016). GPSA grantees report progress in achieving many of their expected outputs. The grants have been used to build the capacity of a wide network of CSOs and local community groups, for example in social monitoring or local budgeting. Many grantees also report constructive engagement with local government offices in charge of overseeing delivery of public services. Some of the piloted social accountability tools generated broad interest in countries. In Moldova, the GPSA spearheaded the introduction of school-level report cards and parents’ participation in education budgeting at the municipal level. These tools were thereafter adopted by the government to use across the country. Opportunities to leverage synergies between GPSA grants and World Bank projects taking place in the same country were not fully exploited, although there are some positive examples. The evidence 36 Engaging Citizens for Better Development Results | Chapter 3 available from the first round of projects analyzed by IEG suggests that better linkages would have helped the World Bank extend its engagement with CSOs in broader governance reforms, while improving the prospects of institutionalizing GPSA-supported social accountability mechanisms and sustaining the results. Projects are in some cases isolated pilots that do not mesh well with existing country systems for social accountability, as in the Dominican Republic and Tajikistan. There are positive dynamics in some countries. For example, the two GPSA grantees, CARE Bangladesh and the Manusher Jonno Foundation, participated in the design workshop for the third phase of the Bank Group’s supported Local Government Sustainability Plan to integrate the lessons learned from the GPSA pilots. Despite encouraging examples, sustainability and replication of even the most successful tools and mechanisms supported by the GPSA are still rare. The government of Indonesia showed some interest in adapting the approach to social accountability in maternal and child health service delivery developed by a grantee to its own program. In most cases, though, governments are reluctant to engage CSOs in the monitoring of national projects. Many grantee CSOs, often local branches of well-known international CSOs, are also unlikely to continue the work they have done with GPSA support without GPSA funding, partly because these CSOs usually have their own longer-term programs, often supported by bilateral partners, and treat the GPSA project as a one-off initiative. This also raises the issue of whether three years of GPSA support is sufficient for engaging civil society (that often relies on external funding) in a more sustainable way, and what strategies could be used to ensure more consistent and longer-term engagement, after the completion of the projects. After three rounds of calls for proposals for CSOs and limited resources, the program is spread thin across regions, countries, and sectors and is at a critical junction. A number of strategic issues may need to be considered by GPSA management to strengthen its relevance to the Bank Group and client countries, and the effectiveness and sustainability of its activities. One example is to ensure that the GPSA “call for proposal” model is fit for purpose for achieving the program’s objectives with its current resource envelope; another issue to address is understanding how well-suited the Bank Group is to manage such model, which can incur high transaction costs for the organization in terms of processing, approving and managing the grants (OECD 2012). More recent project- level collaboration with different external stakeholders (for example, USAID in Mexico) and inside the Bank Group (such as the Fragility, Conflict, and Violence Group) are valuable for leveraging program’s resources. Yet, these engagements need to be integrated in the overall strategic vision of the GPSA to represent a viable solution to the partnership’s sustainability issues. In the same vein, the collaboration between the GPSA and other global initiatives, such as the Open Government Partnership (OGP), which is already happening in few countries, can help the program to leverage its resources better and maximize its impact in the field. In the context of the new ESF, building country systems to collect and address citizen feedback becomes essential. The ESF sets out mandatory requirements for the borrower with a focus on E&S risk management. It places great emphasis on the use of existing borrower frameworks when they are materially consistent with the ESF, with the goal of building more sustainable institutions and increasing efficiency. Details on operationalization are not yet available. Although this entry point Independent Evaluation Group | World Bank Group 37 offers a clear opportunity to enhance the sustainability and outreach of engagement mechanisms embedded in projects, concerns arise when existing systems do not uphold the standards set out by the World Bank. 1 T  he strategic framework indicates that the approach should be “informed by evidence and lessons learned from internal and external impact studies and stocktaking and complementary regional initiatives” (World Bank Group 2014, 20) 2 Guidance for the Bank Group’s approach to Country Engagement paragraphs 18–20. 3  ountry surveys are designed to explore perceptions of the Bank Group’s work (speed, effectiveness, efficacy, C relevance, efficiency), and of the knowledge, responsiveness, attitudes, and engagement of Bank Group staff interfacing with stakeholders. Survey respondents include national and local governments, multilateral/bilateral agencies, media, academia, the private sector, and civil society. The goal is to survey nearly all Bank Group client countries every three years. The Bank Group’s External and Corporate Relations Public Opinion Research Group supports Country Partnership Framework or Performance and Learning Review task team lead in conducting a country survey. 4 http://consultations.worldbank.org/?map=1 5 Citizen Engagement website: https://raddrupalprod.worldbank.org/citizen_engage/operationalize 6  nnex C specifies that citizen engagement indicators “either capture citizen feedback (that is, reflect a two-way rather A than one-way interaction between citizens and governments or the private sector within the scope of Bank projects)” and “where relevant, they report on whether there is a tangible response to the inputs provided by citizens, and if so, how such inputs have been incorporated.” Indicators “include options to measure (i) improvements in processes to engage with citizens and (ii) contribution of citizen engagement activities to improve project effectiveness and project development outcomes” (22). Appendix C provides a list of potential citizen engagement indicators. Although they clearly track aspects of citizen engagement activities, not all these indicators capture the closing of the feedback loop; compare, for example, for consultations, the “change to project activities as a result of consultations” with “share of citizens (from women/vulnerable/marginalized groups) who participated in consultations.” Mere participation in consultations does not ensure that the feedback loop was closed, as it can be inferred from the Strategic Framework (for example “Consultations need to respect good practice principles, including providing adequate notice periods and closing the feedback loop more systematically” (World Bank Group 2014a, 3). 7  he assumption is that more recent projects received better guidance from the Citizen Engagement Secretariat, T Citizen Engagement focal points, monitoring and evaluation specialists and, more generally, benefited from the experience developed by older projects. 8  he evaluation reviewed a purposive sample of 36 out of 57 community-driven development projects that were T approved between FY14 and FY16. 9 C  omplaint tracking systems can play a key role in improving the effectiveness of GRMs. A recent evaluation of the MajiVoice system to handling complains by water service customers in Kenya demonstrates that effective oversight by the government to hold the service company accountable and efficient solutions that enable customers to submit and track the status of their complaints and service providers to process and handle complains following clearly defined workflows can sensibly improve the performance of GRMs (increasing resolution rates and decreasing resolution time). These success stories, however, tend to be a few isolated ones (Peixoto and Fox 2017; Belcher and Abreu Lopes 2017). 10  Outside of projects, recent regional efforts have been deployed to create and strengthen governments’ and implementing agencies’ capacity. For example, in FY16 the ECA Region established the Europe and Central Asia Citizen Engagement Capacity Building Series, which has since been systematically rolled out across the region. 38 Engaging Citizens for Better Development Results | Chapter 3 This initiative consists of training and capacity building activities tailored to Project Implementing Units based on a systematic review of citizen engagement in each country’s portfolio. To date, more than 70 PIUs in about 12 Europe and Central Asia countries have been trained in citizen engagement. Another example of a regional effort to build client capacity is the Collaboration for Development platform that the Middle East and North Africa Region is pursuing to allow client governments, CSOs and World Bank staff to share resources and experience online on citizen engagement, in their own language, sector and context. This evaluation has not reviewed the effectiveness of these initiatives. 11 C  ountry opt-in entails a formal commitment by government to the World Bank indicating their upfront, programmatic consent to each individual Global Partnership for Social Accountability grant operating in its country without later needing to provide specific consents for each grant proposal. Independent Evaluation Group | World Bank Group 39 4 highlights Engaging for 1 Engaging citizens in development interventions Results can have a range of positive effects: from improving trust between citizens and the state, to having a tangible effect on the quality of services. If sustained, it may lead to long-term institutional change. 2 The current emphasis on tools rather than results and the insufficient attention paid to quality principles risk undermining the World Bank Group’s objective to mainstream citizen engagement to improve development outcomes. 3 Several of the case studies show examples of citizen engagement mechanisms leading to positive behavior and relationship change. In fewer cases, citizen feedback strengthened operations by informing project design or triggering course corrections during implementation. x 40 4 When engagement mechanisms were sustained over multiple project cycles, they were more likely to become institutionalized and bring about durable change. 5 Achieving citizen engagement to improve development outcomes is an ambitious agenda that requires more effort, more focus on capacity building and learning, better tracking of resources, and better synergies with other relevant agendas, such as the new Environmental and Social Framework. 6 The institution will need to find the right balance between quantitative targets, quality requirements, and human and financial resources devoted to the effort. Independent Evaluation Group | World Bank Group 41 D O T H E B A N K G R O U P ’ S citizen engagement activities contribute to development outcomes? Like other empirical studies, this evaluation cannot provide a definite answer, especially because citizen engagement activities are not well documented or explicitly linked to projects’ core theories of change, results indicators are process-oriented, and citizen engagement’s results are not captured in self-evaluations. This chapter instead illustrates the type of change that can be achieved through the Bank Group’s support to citizen engagement. This evaluation’s empirical findings converge with the literature in demonstrating that if the conditions of high-quality design and implementation discussed in the previous chapter are met and activities are well-embedded in the local context, then impact on development outcomes is more likely. Citizen engagement contributes by triggering three types of change at the level of behaviors and relationships, operations, and institutions (table 4.1). Establishing a causal link between citizen engagement activities and development outcomes—such as improved learning, health, or shared prosperity—is fraught with methodological challenges, given the heterogeneity of both the intervention and the outcome space (Joshi 2013; Ringold et al. 2013). However, the evidence on proximate and intermediary outcomes is more robust (Devarajan, Khemani, and Walton 2013; World Bank 2017c; DPMG 2017). The case studies conducted by IEG have been used to document changes observed at these three levels—behavioral, operational, and institutional. Table 4.2 displays the systematic analysis of project case studies and the type of results observed. Although the findings of this chapter are supported by triangulated evidence across cases and aligned with the literature, only the most emblematic examples are reported and discussed. Behavior and Relationship Change The most common and proximate outcomes achieved by citizen engagement mechanisms, and observed across cases, are a transformation in awareness, relationships, and behavior of citizens and TABLE 4.1 | Path to Results: Typology of Proximate and Intermediary Outcomes Positive Behavior and Relationship Change Positive Operational Change Positive Institutional Change Knowledge of rights and •  Citizens’ priorities guide •  Improved policy framework •  obligations operational choices Transparency and accountability •  Self-confidence to voice concerns •  Efficiency in responding to •  Institutional mechanisms for •  Responsiveness to citizens •  citizens’ needs: citizens’ voice within country demands/needs – Better targeted budget systems Willingness to participate and •  allocations Inclusive and cohesive societies •  collaborate – Improved project and Trust between citizens and state •  service delivery actors management – Improved service quality 42 Engaging Citizens for Better Development Results | Chapter 4 service providers. Given that citizen engagement is often about power shift, empowering citizens to demand information, sanctions, or better public goods is critical. Equally important is changing the minds of public providers to improve their willingness to collaborate and avoid backlash (Grandvoinnet, Aslam, and Raha 2015). There is robust evidence that well-implemented citizen engagement mechanisms increase citizens’ knowledge of their rights, service standards, and channels for voicing grievances (Fox 2015; Gaventa and Barrett 2010). For instance, in Cambodia, the project’s active sensitization of citizens to the role and services of the Ombudsman and Arbitration Council resulted in enhanced awareness of corruption cases and avenues for tackling them, and enhanced confidence in the usefulness of these formal mechanisms to deal with grievances. Effective engagement also modifies service providers and government officials’ attitude toward citizens’ feedback, a prerequisite for fruitful and sustained engagement, as observed in case studies where engagement mechanisms are combined with traditional oversight; what Fox (2015) calls the “sandwich strategy.” For the facilitators and community mobilizers interviewed during IEG’s site visits, a change in attitude from service providers is a real victory.” For example, the use of citizen engagement in both IPF and DPF meshes well with the context of Morocco’s constitutional reform that seeks to bolster accountability and citizens’ voice. The community report cards embedded in municipal solid waste management, consultations for safeguards, and participatory planning embedded in the pilot coastal natural resource management initiative are examples of mechanisms that have enhanced residents’ sense of agency and made government officials more receptive to citizens’ ideas on how to deliver better services. Mechanisms that jointly sensitize citizens and service providers, and provide an interface between the two are also more likely to reduce conflicts, enhance trust, and strengthen involvement in the projects (Grandvoinnet, Aslam, and Raha 2015). This can result in increased payments of fees, dues, taxes, or voluntary contributions to project costs, ultimately resulting in more reliable service delivery as observed in the Dominican electricity project, or the water sewerage projects in Pakistani Punjab, West Bank, and the Kyrgyz Republic. For example, in Punjab, the evaluation team found that capacity building and guided facilitation provided to water user associations were key to solving perennial conflicts over the maintenance of watercourses and outlets. The new collaborative practice of joint submissions for annual funding requests and more participatory practices in oversight diluted previous sources of conflicts. Similar patterns emerged from well-designed school-based management projects observed in Pakistan and the Philippines, where teachers, principals, and parents interviewed during field visits credited participatory mechanisms with improving the relationship among all members of the school community. They also felt more empowered to get involved in school affairs. Various streams of evidence, including IEG case studies, show that rigorous participatory methodologies that are embedded in durable institutional systems and bring resolution to citizens’ feedback can improve trust in public services (Menocal and Sharma 2008; World Bank Group 2016). For example, in the Dominican Republic’s national cash transfer program, the improved self-confidence and satisfaction of beneficiaries with project services following citizen monitoring is evident from a survey conducted by Gallup and confirmed by the IEG process tracing study. The survey shows that beneficiaries in program areas where participatory monitoring had taken place Independent Evaluation Group | World Bank Group 43 TABLE 4.2 | Observed Results of Citizen Engagement Mechanisms in Project Case Studies 44 Country Voice & Accountability Index 2016 52 52 51 40 33 33 29 29 29 29 18 16 10 9 4 4 4 Dominican Republic (Electricity) Dominican Republic (PROSOLI CCT) Philippines (Basic Education) Mali (Rural Development) Kyrgyz Republic (VIP) Kyrgyz Republic (Water Mgmt.) Morocco (Solid Waste) Pakistan (Punjab Agriculture) Pakistan (Punjab Cities Governance) Pakistan (Punjab Education) Cambodia (Demand Good Gov) West Bank-Gaza (Municipal Development) Vietnam (Forest Sector Devt) Ethiopia (Promoting Basic Services) Laos (Forestry) Laos (Secondary Education) Laos (Poverty Reduction) Behavioral & Relationships Change Awareness/knowledge increased Awareness l l l l Trust enhanced Conflict/tensions reduced Trust Collaborative resolution increased l l l l l l l l l Citizen influence strengthened Voice Participation growth l l l Community ownership Ownership Service provider responsiveness Answerability l l l l improved Operational Change Community active - in plans, l l l l l supervision, problem solving Community Engaging Citizens for Better Development Results | Chapter 4 Community control on quality of Involvement l l l materials enhanced Citizens contribute resources l l l Project management and efficiencies Efficiencies l l l l l l l l Improved services - enhanced Services l l l l l l quantity and quality Institutional Change Enabling regulations, constitutional Policies l l reforms and formal policies Accountability strengthened (e.g., on budgets, institutionalization, responsive government) Mutual Accountability Complaints addressed, corrective actions taken Agency coordination improved l l Institutional- Replication of citizen engagement ization l l l l approach Note: The Country Voice and Accountability Index is based on the Worldwide Governance Indicators’ dimension of Voice and Accountability for 2016, which “captures perceptions of the extent to which a country’s citizens are able to participate in selecting their government, as well as freedom of expression, freedom of association, and a free media.” (Kauffman, Kraay, and Mastruzzi 2010) were more positive about the possibilities offered to voice their concerns, report problems, and increase cohesion in the community than in control areas (Aston and Cavatore 2015). These changes should not be undervalued: they signal a sense of citizen empowerment and a willingness of service providers to solve existing bottlenecks. However, they may be insufficient without the capacity and authorizing environment to make operational transformations. Only the projects in which increased citizen demand is met with sufficient resources and staff capacity to respond can realistically sustain citizens’ participation and boost their confidence in demanding better services and accountability. In Ethiopia, the engagement activities embedded in the Promoting Basic Services Program resulted in increased citizen and service provider knowledge of standards, quality deficits, and core processes. For instance, the share of the population who ignored how public budgets work decreased from 91 percent in 2009 to 34 percent in 2016, based on an impact assessment (BDS Center for Development Research 2017). It also became apparent from the focus groups conducted in eight districts (woredas) that trust in government processes increased, resulting in increased sense of ownership of the infrastructure and services and a willingness to contribute. As a beneficiary mentioned, “taxes are finally coming back to the woredas, and so we are ready to pay.” Operational Change Stakeholder consultations for the most part confirmed preidentified priority areas of attention and only sporadically triggered a change in CPF priorities, based on analysis of CPF documents and interviews with CPF task leaders. CSOs interviewed for this evaluation pointed to limitations in how the process was organized and facilitated and how priorities were identified. Participants referred to merely being asked to vote or rank priority areas and complained about the insufficient information received prior to the meetings, the absence of translation into local languages, or the short time frame to properly prepare. Yet, citizen feedback can improve the Bank Group’s operations by changing the portfolio composition to better serve citizens’ needs. This is the rationale for strengthening engagement during the preparation and implementation of the CPF. When this happened, it was prompted by feedback provided by the private sector. One such example is Bosnia-Herzegovina, where growth limitations associated with restrictions on the labor code, highlighted by the private sector, motivated changes in the strategy design. Adding a sector focus in country strategies following suggestions from civil society happened occasionally. In Uzbekistan, the Bank Group team made agriculture modernization a key pillar of the CPF after citizens highlighted the salience of the issue for them. Similarly, in Egypt, citizens strongly voiced that education should be at the forefront of the Bank Group’s engagement, which was not initially a priority area in the strategy. Attuned to citizens’ feedback, the CPF integrated the sector through technical assistance and lending.1 There is little evidence that citizen feedback is used to strengthen operations by informing project design or triggering course correction during implementation. This is true even for more participatory projects such as CDD—a disappointing result since this is where community-induced change would Independent Evaluation Group | World Bank Group 45 have been expected. In the sample of 36 CDD projects studied for this evaluation, only one case clearly showed that information collected through participatory feedback mechanisms and GRM was used for course correction to strengthen citizen engagement. When ownership of the participatory process by clients and operational teams is clear and a well- functioning feedback mechanism exists, significant operational changes can take place, resulting in better implementation. For instance, the Palestinian Municipal Development Project, started in 2009, integrates a rigorous participatory planning methodology, which has been iterative, with continuous adaptation based on citizen feedback. This resulted in several operational changes, such as infrastructure grants awarded to the municipalities based on citizen needs and performance, both expressed through participatory means. Furthermore, the Palestinian Authority requires donors working with local governments to align with the participatory process in place, enhancing coordination among donors and alignment with community needs. Likewise, the participatory monitoring embedded in the Dominican Republic’s conditional cash transfer program triggered a wide range of tangible operational changes locally, from adjusting clinic opening hours to ensuring teacher attendance and adjusting prices in local shops. Ample evidence shows that many challenges identified and actions proposed were solved during the cycle (box 4.1). These concrete and direct solutions to specific localized problems, however small they may be, make a difference in the perception and satisfaction of beneficiaries with public services. At times, they also translate in better services, with, for example, higher teacher and health worker attendance. Institutional Change The literature is clear on the importance of institutional change to sustaining citizen engagement. Changes at this level include enhanced institutional capacity, improved oversight mechanisms, and protection of the essential freedoms of association, expression, and media, which are key to enhance state responsiveness and the prospects of sustained influence (Fox 2015). Moreover, citizen feedback and institutional change may positively reinforce each other, as is well documented in the literature. For example, Gaventa and Barrett’s (2010) rigorous meta case study analysis of 100 in- depth cases maps significant outcomes of citizen engagement; many are institutional changes, such as strengthening citizenship and reinforcing state accountability. The Bank Group has committed to building country systems for citizen engagement and, so far, has primarily resorted to an indirect and bottom-up approach to fostering institutional change through investment projects, implying that more can be done to more systematically exploiting other, more direct, channels. Evidence from the structured reviews and case studies shows that the Bank Group can prepare the ground for institutional change through three main channels: it can (i) convene multiple voices in policy dialogues crystalized in the CPF; (ii) trigger reforms that create an enabling institutional environment for citizen-state collaboration through DPF prior actions; and (iii) use investment projects to spark institutional changes that are initially localized, sector specific, or time bound but with the potential for reinforcing country systems through scale-up or replication. The 46 Engaging Citizens for Better Development Results | Chapter 4 Concrete Operational Changes in a Conditional Cash Transfer Program Box 4.1 |  Conversations with service providers in the Dominican Republic elicited rich examples of operational change brought about by the Reportes Comunitarios (RCs) conditional cash transfer program: n I  n Don Juan, there was no health center; the health authorities knew this was an issue for the community but did not have the land to build an additional unit. Through the interface meeting, this land issue surfaced and the citizens committed to provide the land needed for the health center; they also contributed to its construction. Based on this precedent, the health authorities proposed the same ideas in other places where the lack of land was the bottleneck for the proper delivery of health services. n efore the RC, medicine inventory management was a real challenge for health B centers. Beneficiaries who feared shortage would run after the delivery truck to obtain medicines that they did not necessarily need. Through the dialogue introduced by the RC, health providers could explain to citizens the way medical stocks were managed and that medicines cannot be prescribed in the absence of symptoms. The process facilitated inventory management and diminished shortages. political economy and the readiness of the client to work with the Bank Group at any of these levels are preconditions for institutional change to happen. Country strategies in which governance is highlighted as an important cross-cutting theme tend to promote reforms for greater transparency and accountability and reinforce systems for citizen engagement. In Mali, the FY16–19 CPF governance pillar lays out a strategy for working with civil society, enhancing transparency and participation in policy making, and monitoring and formulating budget (World Bank 2015). CPF consultations identified strengthening civil society and community- based organizations, including in the peace and securitization process, as priorities. The portfolio is pivoting toward embedding more citizen engagement activities. The Bank Group can also mobilize its convening power to bring citizens to the policy and institutional reform discussion table outside of the CPF and DPF consultations via complementary analytical work, policy dialogue and consultations. One example in which the World Bank was successful in building a platform for engagement, taking into account the local political economy context, was the Dominican Republic’s Iniciativas Participativas Anticorrupción (Participatory Anti-Corruption Initiative; IPAC).2 Under the leadership of its country manager in Santo Domingo, the World Bank identified reform- minded champions within the government and civil society, brought together a coalition of donors,3 and facilitated a long-term policy dialogue on the perennial issue of corruption. The World Bank also Independent Evaluation Group | World Bank Group 47 provided training to civil society actors to partake in the conversations. The initiative was successful because it had teeth through a Social Observatory led by 14 CSOs. According to a GPSA-funded evaluation, the initiative led to greater institutional integrity and enhanced CSOs’ collective action (Kaufmann, Gallina, and Senderowitsch 2015). IPAC recommendations were used as a road map by many line ministries and have been the basis for the government’s entry in the OGP. The IPAC methodology was replicated in at least two other participatory proreform coalitions’ initiatives with different degrees of success. DPF prior actions and triggers, which are mechanisms with “teeth,” are rarely used to support policy reforms that could foster citizen engagement. There are exceptions, such as in the Middle East and North Africa Region, that indicate that this could be a useful entry point under the right conditions. Since the early 1990s, the number of prior actions coded by Operations Policy and Country Services as having directly sought to enhance citizen engagement and participation4 amounts to just 83 prior actions in 59 distinct DPFs. In the peak year of 2006, this type of prior action was still only 1.8 percent of all prior actions. Looking more specifically at the most recent period (FY11–16), 20 DPFs included a prior action that directly sought to promote participation. One such example is the Tunisia Governance and Opportunity DPF, which established a participatory process for systemic monitoring of the performance of public services by civil society and citizens. Prior Actions can also be used to indirectly promote citizen engagement through two channels: (i) supporting citizen capacity to act through improved access to law and justice or introduce conflict prevention mechanisms; or (ii) enhancing state capacity to respond to citizens’ demand, through decentralization and judicial or civil service reforms.5 Taken together, these types of prior actions only represent 10 percent of all DPF prior actions since 1990. Looking more specifically at FY11–16, 340 prior actions were found in 169 DPFs. Of those, the majority (79.4 percent) supported reforms that enable state action, either by seeking to make the state more accountable or by reinforcing the state capacity to respond to citizens. The Africa the Region has the highest share of prior actions supporting state action (90 percent), and Latin America is the only Region with a balance of prior actions enabling state action and enabling citizen action. A more granular analysis is presented in appendix C. Promoting reforms that enable citizen engagement is rarely an objective of DPFs, but the IEG case studies offer some insight on how DPFs can be used for this purpose. Responding to the Arab Spring events, the World Bank’s Middle East and North Africa Region is seeking to embed participation in its portfolio of DPFs. Morocco, with its large DPF portfolio, is emblematic of this effort. The solid waste and water DPF series mobilizes a range of mechanisms to ensure transparency and accountability in the sector. The DPF resorts to citizen report cards, introduces financial eligibility criteria, and fosters a three-way communication network in the sector between citizens, local governments, and private sector entities. Evidence from the ICRR corroborated by interviews conducted for this evaluation showed an improved institutional environment in the sector with a win-win scenario for all. As citizens became more aware of contractual arrangements with private sector operators for collection and landfill activities, their trust in the process grew, and they became more willing to pay collection fees; 48 Engaging Citizens for Better Development Results | Chapter 4 in turn, private sector entities became more resolute to invest in the sector and more transparent. Local governments’ revenue increased, enabling them to cut payment delays to service providers. The World Bank can also foster institutional change through the indirect and bottom-up route of investment projects, especially when there is a deliberate attempt to create or strengthen existing country systems for engagement. The practice of creating country road maps for citizen engagement that has emerged in a handful of countries, notably in the Europe and Central Asia Region, is promising. These road maps outline a set of priority areas and projects where citizen engagement could be strengthened for greater effect at a systemic level. They seek to deepen citizen engagement in specific sectors (such as in Kosovo); establish regular roundtables with CSOs to monitor CPF implementation (such as in Montenegro); or establish informal civil society advisory groups to raise awareness among CSOs of upcoming important institutional reforms (such as in Ukraine). Although these road maps are promising approaches to identify priorities at the country level, it remains to be seen how they are implemented and funded. Genuine political commitment and sustained Bank Group engagement with the client are both needed to generate lasting institutional change. For example, the follow-on project to Cambodia’s Demand for Good Governance resulted in a nationwide Social Accountability Framework embedded in the decentralization reform. Similarly, the third phase of the Kyrgyz VIP led to the institutionalization of citizen engagement activities in local government. The Promoting Basic Services Program in Ethiopia generated more trust and communication between citizens and service providers, but the limited government response in terms of budget reallocations following citizens’ input, systematic enforcement of quality standards, and visible improvements in the efficiency of the GRM may jeopardize the positive achievements at the level of attitudinal change, especially in a context characterized by high political conflict and tight government control on civil society. Recognizing that operational and institutional changes are slow processes that require strategic long-term engagement, the Bank Group has remained engaged for a long time. Currently, the Bank Group supports service delivery through a Program-for-Results (ESPES), following three phases of the Promoting Basic Services Program. ESPES will build capacity and strengthen government and service providers’ systems, thus complementing civil society capacity building activities provided by the third phase of a multidonor trust fund (ESAP). Supporting a Higher Level of Corporate Ambition Mainstreaming citizen engagement across the Bank Group’s operations to achieve better development outcomes is an ambitious commitment. It requires a gradual approach, as stressed in the strategic framework, and it needs to combine scale and quality to achieve results. The evidence from the literature is clear: a one-off, bottom-up engagement, such as a beneficiary survey, a consultation, or even a GRM, is rarely sufficient to significantly influence the trajectory of a project so that its development outcomes can be improved. Strategic, continuous, and inclusive engagement matched with vertical accountability channels is necessary. Despite the powerful guidance provided by the strategic framework (box 4.2), IEG found that achieving this type of engagement to scale Independent Evaluation Group | World Bank Group 49 requires more effort than the Bank Group has currently allocated to this agenda, and it entails focusing on activities that do not currently receive sufficient attention. The institution is not sufficiently monitoring key elements identified in the strategic framework as essential to ensure successful integration of citizen engagement in projects, but there are interesting exceptions. Overall, the quality of engagement is poorly tracked; crucially, the organization pays insufficient attention to documenting whether and how the feedback loop was closed, as evidenced by the IEG portfolio review. Not only does this generate inaccurate reporting of progress, it provides little incentive to teams to “raise the bar” and deprives the institution of the systematic information essential to learn and build on successes. The Quality Index rolled out in the Europe and Central Asia Region in FY17–18 is a promising effort to measure the quality of citizen-oriented design and the quality of beneficiary feedback indicators along multiple dimensions. With the strategic framework, the organization has set ambitious goals, but there is little or no investment in generating robust evidence of what works, where, and why. As interviewees close to the topic have stressed, the World Bank is presenting citizen engagement as necessary to improving development results but does not have a unified research program on citizen engagement or an institutional mechanism to coordinate and support the existing fragmented research initiatives, such as the Development Impact Evaluation governance program, the Social Observatory, and the Information and Communication Technologies program. No systematic investment is made in promoting transformational approaches, including using new tools, learning from best practices, and deriving lessons for scaling up. Building on the Strength of the Strategic Framework Box 4.2 |  The 2014 citizen engagement strategic framework usefully underpinned the corporate commitment. It comprehensively laid out the multiple entry points for citizen engagement in World Bank Group instruments and rightfully made quality of engagement a cornerstone of the strategy. The high-level principles laid out in the framework are evidence based and pay due attention to context specificity. However, the strategic framework does not constitute a corporate action plan for citizen engagement and leaves a number of key issues unresolved: how to combine universal mainstreaming in investment project financing with rigorous application of quality principles; how to ensure that adequate human, technical, and financial resources are mobilized to design and implement citizen engagement activities in projects and track results; and how to establish a strong research agenda and knowledge management system to assist teams in adopting successful approaches and piloting new ones. Source: World Bank Group 2014a. 50 Engaging Citizens for Better Development Results | Chapter 4 Internal training is another critical area requiring urgent attention. The IEG portfolio review detected very few capacity building activities (23 percent of projects with a mechanism) to strengthen clients’ and project implementation units’ capacity to engage. Limited capacity and training also emerge among the most relevant internal and external constraints in the IEG staff survey findings (see figure 2.4). Focal points and social development specialists confirmed in interviews that training is essential to ensure sustainability and the Bank Group is not investing enough in it. The institution has not yet succeeded in creating a critical mass of experts who can regularly provide capacity building activities for teams and clients. Channels to support civil society are also quite limited, besides the GPSA. The role that communication specialists in country offices may have in spearheading the citizen engagement process should be recognized and strengthened. Citizen engagement requires proper “development communications” to be deployed and maintained in the field. Survey respondents, interviewees, and IEG’s own field visits pointed to the fundamental role of communications officers in engaging with civil society during the preparation of country strategies and in supporting operations. In many ways, they are the custodians of the ongoing relationship with the national networks of CSOs and other counterparts. Yet, they are outside the existing network of focal points and feel somewhat left out. Perceptions of managers and staff differ regarding the shortage of resources in relation to the current level of engagement (figure 4.1); needs for resources, however, can only be quantified in relation to the level of intensity of the agenda. This evaluation has tried to quantify how many resources are currently spent, but no budget code or other system is in place to track how much projects spend for citizen engagement. Moreover, this evaluation highlighted essential aspects that need to be strengthened, such as improving monitoring; promoting and tracking quality; providing training and capacity building for the Bank Group’s teams, clients, and implementing agencies; testing transformational approaches and what works in which context; and improving knowledge management. These activities will require a coordinated effort at the corporate level, a stronger and more empowered secretariat to advance the mainstreaming process, and additional resources. Although the corporate target has been a deciding factor in increasing the internal buy-in, a trade-off emerges between the current mainstreaming effort and ensuring the quality of citizen engagement activities. Even among staff closely involved with the agenda, there is no universal agreement on the target—barely two-thirds of those who responded to the IEG survey agree that including citizen feedback in all projects with clearly identifiable beneficiaries is a meaningful target. The institution will need to find the right balance between quantitative targets, quality requirements, and human and financial resources devoted to the effort. Opportunities to establish synergies with other agendas of the institution to improve quality, create efficiencies, and mobilize additional resources are not obvious to those consulted through the survey and interviews. Although staff closely linked to the citizen engagement strategy tend to have a good understanding of the strategy and mainstreaming goal, one-quarter remain unclear about these objectives. Managers are more likely than other staff to state that they have a clear vision of how citizen engagement connects with the other Bank Group commitments, such as inclusion, gender, and governance, and recent reforms, such as the new ESF and country engagement model. Independent Evaluation Group | World Bank Group 51 FIGURE 4.1  taff and Management Perception of Resource Adequacy for |S Type of Engagement (a) Staff opinion (b) Managers’ opinion 3% 13% 15% 16% 36% 21% 49% 46% My manager encourages me to go beyond I encourage my teams to go beyond corporate requirements and provides me corporate requirements but often I cannot with the necessary resources provide necessary resources My manager encourages me to go beyond I encourage my teams to go beyond corporate requirements but may not provide corporate requirements even if it requires me with the necessary resources mobilizing additional resources My manager ensures that I comply with I ensure that my teams comply with corporate requirements on Citizen corporate requirements on Citizen Engagement and stakeholder consultations Engagement and stakeholder consultations My manager pays little attention to I pay little attention to Citizen Engagement; Citizen Engagement we have many other competing priorities Source: Independent Evaluation Group staff survey. Operational teams are less likely to see the connections between the various corporate goals and more likely to perceive each commitment as an additional requirement to comply with. The new ESF embeds several elements of the citizen engagement agenda and presents a unique opportunity to bridge the two approaches and integrate what many have described as a “fragmented social agenda” (box 4.3). The ESF moves from a pure risk management perspective to a more proactive engagement with stakeholders, with the potential to absorb many of the quality elements discussed in this and the previous chapter. The ESF agenda is given more staff and resources than the citizen engagement agenda, and it is in the process of being strengthened through new hiring and a massive capacity building effort and has clear enforcement mechanisms. Linking the two agendas could support more substantial work through pooling of resources, capacity, and experience. 1  he World Bank had been negotiating a major education operation with the Government of the Arab Republic of Egypt T for the past 3–4 years. A large operation to support the education sector was eventually launched in 2018. 2 Detailed information on the initiative, its recommendations by working group are available at http://www.ipacrd.org/  3  he U.S. Agency for International Development, the Inter-American Development Bank, United Nations Children’s Fund, T the Spanish Cooperation, and the European Union were strong supporters of the initiative, but many others contributed. 52 Engaging Citizens for Better Development Results | Chapter 4  ynergies and Differences between the New Environmental and Social Box 4.3 | S Framework and the Citizen Engagement Agenda After extensive consultations, in August 2016, the World Bank approved a new Environmental and Social Framework (ESF). The 10 Environmental and Social Standards included in the ESF enhance the World Bank’s requirements for transparency and stakeholder engagement. Environmental and Social Standard 10 (ESS10) sets standards for engagement and information disclosure that all investment projects must follow. It requires stakeholder identification and analysis; consultation, engagement, and reporting to stakeholders; information disclosure; and addressing and responding to grievances. ESS10 broadens the definition of stakeholders, which now includes not only the individuals or groups who are affected or likely to be affected by World Bank’s projects (as per traditional safeguards approach), but also other individuals or groups that may have an interest in World Bank’s projects (other interested parties). Stakeholder engagement, as outlined in the ESF, is well aligned with the citizen engagement framework, despite key differences between the two. The ESF sets out mandatory requirements for the borrower with a focus on environmental and social risk management. Citizen engagement is a “softer” entry point, which relies on a broader set of tools and approaches. Yet, both ultimately aim to strengthen government’s accountability toward its citizens. The ESF has strong similarities with citizen engagement as it calls for an early, open, meaningful, and continuous engagement with stakeholders, providing opportunities for stakeholders’ views to be considered in project design, implementation, and monitoring; requires grievance redress mechanisms in all projects; emphasizes the use of existing borrower frameworks, with the goal of building more sustainable institutions and increasing efficiency. The new ESF is expected to roll out in October 2018 and be implemented with an ambitious program of staff training and guidance to borrowers. Though the details of how the ESF will be operationalized are not yet available, raising awareness and building capacity among staff and borrowers are essential elements of its success. Source: World Bank 2017c, 98–100. 4 These are the prior actions captured by theme 57 “participation and civic engagement.” See appendix C for details. 5 P  rior actions have been classified based on the transmission channels they use to improve the enabling environment for citizen engagement (see appendix C, box C.4). Only one category directly refers to citizen engagement (“participation and civic engagement”), whereas other categories have a more indirect relationship, while still aiming at enabling citizen or state action. Several other prior actions, for example on public financial management, open budgeting, and public procurement, may support engagement of citizens in public policy, but they have not been “counted” for this exercise to minimize errors of inclusion. Independent Evaluation Group | World Bank Group 53 5 T H E B A N K G R O U P has a long tradition of seeking the participation of citizens in its work. This culminated in a 2013 corporate commitment to mainstreaming citizen engagement, underpinned by a 2014 strategic framework to operationalize Conclusions and this commitment. The Bank Group notably committed to scale Recommendations up engagement activities across its lending, policy, and advisory work; put quality of engagement and development outcomes at the center of its mainstreaming effort; and track progress and results of citizen engagement activities. The mainstreaming commitment has been instrumental in shedding institutional light on an important part of the Bank Group’s activities formerly seen as restricted to specific lines of work, such as CDD or safeguards. In most staff and managers’ minds, citizen engagement evolved from something that was marginal or specialized to an integral part of the work. The commitment benefits from strong senior management support and robust buy-in from staff. The introduction of corporate indicators has been a powerful tool to promote mainstreaming and track its progression. Indeed, the Bank Group is currently well advanced in this area when the achievement is measured in percentage of projects with at least one engagement mechanism. The nature of the engagement has also evolved. Although in the past almost all engagement activities originated from safeguards policy requirements, during the past three to four years a significant increase in noncompulsory beneficiary feedback mechanisms arose. In IFC, the 2012 policy update of the PSs also triggered a shift in how IFC promoted stakeholder engagement to its clients, moving away from a peripheral activity, mostly perceived by clients as CSR, to a “core of business principle.” The Bank Group has been promoting more active participation of citizens in the preparation of country strategies by leveraging its reformed country engagement model to increasingly consult with a more diverse group of stakeholders and, to a lesser extent, reach beyond the well-established CSOs that are used to work with 54 international organizations. IFC’s renewed role in leading consultations with the private sector on behalf of the Bank Group has broadened the network of private sector stakeholders consulted by the Bank Group. Yet, the Bank Group is not using the full gamut of instruments and entry points at its disposal, identified in the strategic framework, to facilitate citizens’ participation, notably in the policy-making process. DPF consultations remain focused on broad government strategy rather than the specific reforms at hand. They are also poorly documented, with scant evidence of impact on the reform process. Additionally, there are limited examples of the Bank Group leveraging its convening power to broker policy dialogues between government and citizens. This timidity of engagement in policy processes continues to present a widespread image of the Bank Group as first and foremost the governments’ partner, reluctant to assume a more active role in leveraging its influence and creating spaces where a broad spectrum of voices can engage. The Bank Group has rightfully made quality of engagement a cornerstone of its citizen engagement commitment. For engagement to be transformational and contribute to development outcomes it needs to be thick and strategic, that is, use multiple and innovative tools to improve the country’s authorizing environment; it needs to ensure inclusion so that diverse voices are heard, especially those of traditionally marginalized groups; it needs to build citizens’ and governments’ capacity to engage so that change can be long-lasting; and, crucially, it needs to be regular and continuous throughout the project cycle to ensure that citizen feedback is properly taken into account. However, quality principles are currently neither mainstreamed nor systematically tracked at the corporate level. The exclusive focus on quantity of engagement to the detriment of quality principles in the scale-up effort and the drive to meet the mostly quantitative corporate targets, risk overshadowing the necessity to pay close attention to fundamental principles of quality, in both design and implementation, jeopardizing the mainstreaming objective. The only quality principle given some attention in calculating the corporate indicator that tracks progress is related to closing the feedback loop. So far, though, no clear definition of closing the feedback loop has been put forward or enforced at the corporate level. IEG found that a rigorous definition of closing the feedback loop and its enforcement in monitoring the mainstreaming of citizen engagement could lower the rate of compliance but may increase the confidence of identifying genuine engagement. The evaluation found that the mandate as it is currently formulated and monitored may fuel a check- the-box attitude. There are some signs that this is a real risk. For example, teams increasingly include engagement indicators in results frameworks, but, in the aggregate, indicators do not measure the results of the most common mechanisms used, such as consultations. Also, indicators are mostly process rather than outcome oriented; that is, teams report on whether the project complied with what it committed to do, but not on the results that engaging citizens generated. Quality dimensions, as previously discussed, are not generally monitored. Although the E&S policy update and the updated PS 1 prompted IFC’s clients to plan for more regular and proactive forms of stakeholder engagement, there is little evidence of implementation and results achieved. Reporting of IFC’s clients is frequently partial, if not totally insufficient to understand the type, progress, and results of engagement; even projects that report on activities planned focus Independent Evaluation Group | World Bank Group 55 only on the process and not on its outcomes. Given this scarcity of evidence for IFC clients, even more opacity is easily predictable for the clients of financial intermediaries, which represent a large part of the IFC portfolio (and were excluded from the evaluation due to the expected impossibility of assembling evidence). Engaging citizens in development operations can have a range of positive results, from improving relationships and trust between citizens and state actors, to having tangible effects on the quantity and quality of services delivered. When sustained over the long-run citizen engagement activities can even be transformational at the level of institutions, enhancing transparency and accountability well beyond the project boundaries. These direct and indirect contributions to development outcomes tend to take place when engagement is deep, multifold, and sustained over time; and above all, when they are matched with vertical accountability channels that hold public and private service providers responsible for addressing citizens’ grievances and suggestions. The lean corporate arrangements put in place to support the commitment have worked quite well to monitor and track the corporate scorecard indicators and share experience among a restricted group of committed professionals. Given the very limited budget specifically allocated to the secretariat and its affiliated focal points, the number and quality of guidance notes, stocktaking exercises, and tools produced is commendable. However, given the Bank Group’s ambitious objective of mainstreaming citizen engagement to enhance and sustain development outcomes, the overall amount of dedicated human, technical, and financial resources are not fit for purpose. At this stage, a mechanism-driven approach to tracking compliance and the lack of enforcement of quality requirements do not do justice to the rationale that guided the Bank Group in developing its citizen engagement agenda: engaging citizens is important because it leads to better development outcomes. Although managers and teams are committed to the agenda to improve development outcomes, there is currently no tool that allows the institution to test this assumption and measure the added value of engagement. This is not to say that many of the activities of the Bank Group are not generating better development results thanks to more and better stakeholder engagement. However, the World Bank has not yet formulated the corporate requirements to provide incentives to track results and to learn from failures and successes. IEG found that achieving engagement to a scale that improves development outcomes requires more effort than the Bank Group has currently committed. Moving forward, the organization should consider carefully the tensions between ensuring “quality” engagement that can lead to development outcomes and a “universal” approach that may remain superficial for many activities. The findings of this evaluation confirm that mainstreaming citizen engagement is not only technically ambitious, it is also politically challenging. By embracing the commitment to mainstream citizen engagement, the Bank Group should recognize three fundamental paradoxes. First, although the Bank Group needs to remain apolitical, promoting and supporting citizen engagement requires it to be conscious of the political environment where citizen engagement takes place and to meet client governments’ demand. Second, engagement that generates positive and lasting impact on development outcomes needs to produce institutional transformation in country systems for accountability; yet the 56 Engaging Citizens for Better Development Results | Chapter 5 tools available to the Bank Group are mostly timebound, solution oriented, and sector specific. Third, engaging citizens is about understanding societal complexity, avoiding elite capture, and changing power balance, habits, and norms, which are context-specific processes; yet, the current mainstreaming approach does not sufficiently recognize that properly accounting for country-specific factors affects the scope and time frame for mainstreaming. Explicitly recognizing these paradoxes, as well as the technical constraints to mainstreaming, will be essential in defining the next phase of this agenda. Based on its findings, the evaluation offers the following five recommendations: Recommendation 1. As it defines future corporate priorities for citizen engagement, the World Bank should reflect in those priorities the need to achieve greater depth and quality of the citizen engagement activities it supports. This will entail building on the progress achieved under the strategic framework to promote a more strategic approach that incorporates deeper citizen engagement where opportunities arise. These future corporate priorities will need to be clearly communicated to staff and stakeholders. Recommendation 2. The World Bank should encourage and support efforts of its Region, country, and GP teams to establish, where appropriate, “thick” citizen engagement that is regular and continuous, uses multiple tools, and is embedded in country systems. This could be achieved by more systematically using existing channels of dialogue and stakeholder engagement (such as Systematic Country Diagnostics, CPFs, DPF) and applying tools (such as roadmaps and indexes) to plan, monitor, and assess results achieved at the various levels (Region, country management unit, Global Practice). This would entail better synergies between World Bank operations and other programs that support citizen engagement, including GPSA-financed activities; mobilizing adequate expertise to support World Bank teams; investing in training for staff and stakeholders; and improving knowledge management. Recommendation 3. The World Bank should strengthen the monitoring of its citizen engagement activities by systematically adopting results framework indicators that are results oriented. This will entail tightening the alignment between the citizen engagement mechanisms used in projects and the indicators that measure their quality and results, with special emphasis on indicators that show how feedback loops were closed and how diverse groups of stakeholders were included. Recommendation 4. The World Bank should seize the opportunity of the implementation of the ESF to leverage citizen engagement mechanisms—beyond consultations and GRMs—to reach the objectives of managing social risks, strengthening country systems, and promoting social inclusion. This will entail better drawing on the existing expertise on citizen engagement and making citizen engagement more prominent in the ESF trainings, both for staff and for project implementation units. Recommendation 5. IFC should ensure that its clients’ stakeholder engagement activities required by PS 1 in projects with affected communities are carried out during appraisal and supervision of the projects and systematically documented. This will entail mobilization of Independent Evaluation Group | World Bank Group 57 references Aston, T., and adequate M. Cavatore. expertise support general 2015. Sistematización to systematically clients del and proceso de implementación reporting a escala nacional that is comprehensive del and results monitoreo focused. participativo de servicios sociales mediante “Reportes Comunitarios” en el Programa Progresan- do con Solidaridad. London: Care International UK. BIC (Bank Information Center). 2017 Influencing World Bank Projects Before They Happen: A Beginner’s Guide to Advocacy on Country Partnership Frameworks. Washington, DC: Bank Information Center. http://www. bankinformationcenter.org/wp-content/uploads/2017/08/CPF-primer-July-2017-FINAL.PDF. BDS Center for Development Research. 2017. Impact Assessment on Financial Transparency and Accountability Implementation. Submitted to Ministry of Finance and Economic Cooperation in July 2017. Addis Abeba: BDS, Center for Development Research. Belcher, M., C. Abreu Lopes, F. M. Sjoberg, and J. Mellon, J. 2017. MajiVoice Kenya: Better Complaint Manage- ment at Public Utilities. In Civic Tech in the Global South: Assessing Technology for the Public Good, edited by T. Peixoto and M. L. Sifry. Washington, DC: World Bank and Personal Democracy Press. Björkman, M., and J. Svensson. 2009. “Power to the People: Evidence from a Randomized Field Experiment on Community-Based Monitoring in Uganda.” The Quarterly Journal of Economics 124 (2): 735–769. Bousquet, F., and B. Ross-Larson. 2014. Piloting Citizen Engagement in Projects: A Guidance Note for World Bank Staff Working in the Middle East and North Africa Region. Washington, DC: World Bank Group. https://hubs.worldbank.org/docs/ImageBank/Pages/DocProfile.aspx?nodeid=19684765. Devarajan, S., S. Khemani, and M. Walton. 2013. “Can Civil Society Overcome Government Failure in Africa?” The World Bank Research Observer 29 (1): 20–47. doi: 10.1093/wbro/lkt008. DPMG (Development Portfolio Management Group), University of Southern California. 2017. Open Government Partnership: Mid-Term Review. Final Report. Washington DC: Open Government Partnership. http://www. opengovpartnership.org/sites/default/files/OGP_MTR-Report_Final-Jan26-2016.pdf Fox, J. A. 2015. “Social Accountability: What Does the Evidence Really Say?” World Development 72: 346–361. doi:http://dx.doi.org/10.1016/j.worlddev.2015.03.011. Gaventa, J., and G. Barrett. 2010. “So What Difference Does it Make? Mapping the Outcomes of Citizen Engage- ment..” IDS Working Papers 2010 (347): 1–72. doi:10.1111/j.2040–0209.2010.00347_2.x. George, W. W. 2003. Authentic Leadership: Rediscovering the Secrets to Creating Lasting Value. San Francisco, CA: Jossey-Bass. GPSA (Global Partnership for Social Accountability). 2016. GPSA in Review. May 2016 http://www.thegpsa.org/ sa/Data/gpsa/files/field/documents/gpsa_in_review.compressed.pdf. Grandvoinnet, H., G. Aslam, and S. Raha. 2015. Opening the Black Box: The Contextual Drivers of Social Ac- countability. Washington, DC: World Bank. IFC (International Finance Corporation). 2012. Performance Standards on Environmental and Social Sustainability. Washington DC. Bank Group. http://www.ifc.org/wps/wcm/connect/115482804a0255db96fbffd1a5d13d27/ Performance Standard_English_2012_Full-Document.PDF?MOD=AJPERES. Joshi, A. 2013. “Do They Work? Assessing the Impact of Transparency and Accountability Initiatives in Service Delivery.” Development Policy Review 31 (1): 29–48. doi:10.1111/dpr.12018. Kaufmann, D., A. Gallina, and R. Senderowitsch. 2015. A Collective Action Approach Against Corruption: The Case of the Dominican Republic. GPSA Working Paper 3. Washington, DC: World Bank Group. http://gpsa- 58 Engaging Citizens for Better Development Results | Section knowledge.org/wp-content/uploads/2015/03/A-Collective-Action-Approach-against-Corruption-the-Case- of-the-Dominica....pdf. Lührmann, A., S. I. Lindberg, V. Mechkova, M. Olin, F. Piccinelli Casagrande, C. Sanhueza Petrarca, and L. Saxer. 2017. V-Dem Annual Report 2017. Democracy at Dusk? V-Dem Institute. Gothenburg: University of Go- thenburg. https://www.v-dem.net/media/filer_public/b0/79/b079aa5a-eb3b-4e27-abdb-604b11ecd3db/v- dem_annualreport2017_v2.PDF. Mansuri, G., and V. Rao. 2013. Localizing Development: Does Participation Work? Washington, DC: World Bank. Menocal, A. R., and B. Sharma. 2008. Joint Evaluation of Citizens’ Voice and Accountability: Synthesis Report. London: DFID. https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/3425.PDF. MIGA (Multilateral Investment Guarantee Agency). 2013. Performance Standards on Environmental and Social Sustainability. Washington, DC: World Bank Group. https://www.miga.org/Documents/MIGA_Performance Standards_October-1–2013_CleanUp_to_date.PDF. Narayan, D., ed. 2002. Empowerment and Poverty Reduction: A Sourcebook. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/15239. OECD (Organisation for Economic Co-operation). 2012. Partnering with Civil Society: 12 Lessons from DAC Peer Reviews. Paris: OECD. http://www.oecd.org/dac/peer-reviews/12%20Lessons%20Partnering%20with%20 Civil%20Society.pdf. O’Meally, S. C. 2013. Mapping Context for Social Accountability: A Resource Paper. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/293491468151492128/Mapping-context-for-social-ac- countability-a-resource-paper. Peixoto, T., and J. Fox. 2017. “When Does ICT-Enabled Citizen Voice Lead to Government Responsiveness?” In Civic Tech in the Global South: Assessing Technology for the Public Good, edited by T. Peixoto, and M. L. Sifry. Washington, DC: World Bank and Personal Democracy Press. Peixoto, T., and M. L. Sifry. 2017. Civic Tech in the Global South: Assessing Technology for the Public Good. Washington, DC: World Bank and Personal Democracy Press. https://openknowledge.worldbank.org/han- dle/10986/27947. Ringold, P. L., J. Boyd, D. Landers, and M. Weber. 2013. “What Data Should We Collect? A Framework for Iden- tifying Indicators Of Ecosystem Contributions to Human Well-Being.” Frontiers in Ecology and the Environ- ment 11 (2): 98–105. doi:10.1890/110156. Tembo, F. 2012. Citizen Voice and State Accountability: Toward Theories of Change that Embrace Contextual Dy- namics. London, UK: Overseas Development Institute. https://www.odi.org/sites/odi.org.uk/files/odi-assets/ publications-opinion-files/7602.PDF. Wetterberg, A., D. Brinkerhoff, and J. Hertz, eds. 2016. Governance and Service Delivery: Practical Applications of Social Accountability Across Sectors. Research Triangle Park, NC: RTI International Press. World Bank. 1996. World Bank Participation Sourcebook. Environment Department Papers 19. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/650481468781551648/World-Bank-participa- tion-sourcebook. ———. 2001. Adjustment from Within: Lessons from the Structural Adjustment Participatory Review Initiative. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/386221468332448978/Adjust ment-from-within-lessons-from-the-Structural-Adjustment-Participatory-Review-Initiative. Independent Evaluation Group | World Bank Group 59 ———. 2003. World Development Report 2004: Making Services Work for Poor People. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/5986. ———. 2007. Strengthening Bank Group Engagement on Governance and Anticorruption. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/426381468340863478/Main-Report. ———. 2011a. The Practice of Grievance Redress. Vol. 2 of Feedback Matters: Designing Effective Grievance Re- dress Mechanisms for Bank-Financed Projects. Social Development How-To Series. Washington, DC: World Bank. http://siteresources.worldbank.org/EASTASIAPACIFICEXT/Resources/GRMP2-Final.PDF. ———. 2011b. How, When, and Why to Use Demand-Side Governance Approaches in Projects. Social Develop- ment How-To Series. Washington, DC: World Bank. https://www.governanceknowledge.org/pro/Resourc- es/How percent20to percent20Use percent20DFGG percent20in percent20Operations.PDF. ———. 2013. Results Framework and M&E Guidance Note. Washington, DC: World Bank. http://siteresources. worldbank.org/PROJECTS/Resources/40940-1365611011935/Guidance_Note_Results_and_M&E.pdf. ———. 2014a. Global Review of Grievance Redress Mechanisms in World Bank Projects. Washington, DC: World Bank. Retrieved from https://openknowledge.worldbank.org/handle/10986/20117. ———. 2014b. Piloting Citizen Engagement in Projects: A Guidance Note for World Bank Staff Working in the Mid- dle East and North Africa Region. Washington, DC: World Bank. ———. 2014c. Listening to Citizens: Learning from Projects in Africa. Washington, DC: World Bank. ———. 2014d. Operational Policies OP 8.60. World Bank. http://siteresources.worldbank.org/OPerformance Stan- dardMANUAL/112526–1124459412562/23585629/OP8.60July1-clean.PDF. ———. 2015. Mali—Country Partnership Framework for FY16–19. Washington, DC: World Bank. ———. 2016a. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/902331469736885125/Behind-the-mirror-a- report-on-the-self-evaluation-systems-of-the-World-Bank-Group. ———. 2016b. Women’s Empowerment in Rural Community-Driven Development Projects. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/917631485554140658/Womens-empowerment-in-ru- ral-community-driven-development-projects. ———. 2017a. Cambodia Demand for Good Governance Project. Independent Evaluation Group, Project Perfor- mance Assessment Report 116799. Washington, DC: World Bank. http://documents.worldbank.org/curat ed/en/987321500652281850/PDF/116799-PPAR-P101156-PUBLIC.PDF. ———. 2017b. Consultations, Participation, and Disclosure of Information. Inspection Panel, Emerging Lessons Series 4. Washington, DC: World Bank. http://ewebapps.worldbank.org/apps/ip/Pages/Emerging-Lessons. aspx. ———. 2017c. The World Bank Environmental and Social Framework. Washington, DC: World Bank. http://docu- ments.worldbank.org/curated/en/383011492423734099/The-World-Bank-environmental-and-social-frame- work. ———. 2017d. World Bank Group Country Engagement: An Early-Stage Assessment of the Systematic Country Diagnostic and Country Partnership Framework. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/scd-cpf.PDF. 60 Engaging Citizens for Better Development Results | References ———. 2018a. IFC’s Experience with Inclusive Business: An Assessment of IFC’s Role, Outcomes, and Potential Scenarios. Independent Evaluation Group. Meso Evaluation. Washington, DC: World Bank. https://ieg .worldbankgroup.org/sites/default/files/Data/reports/ ———. 2018b. Vietnam Forest Sector Development Project. Independent Evaluation Group, Project Performance Assessment Report 123895. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/ files/Data/reports/ppar_vietnamforest.pdf. World Bank Group. 2014a. Strategic Framework for Mainstreaming Citizen Engagement in World Bank Group Operations: Engaging with Citizens for Improved Results. Washington, DC: World Bank Group. http://docu ments.worldbank.org/curated/en/266371468124780089/Strategic-framework-for-mainstreaming-citizen-en gagement-in-World-Bank-Group-operations-engaging-with-citizens-for-improved-results. ———. 2014b. World Bank Group: A New Approach to Country Engagement. Washington, DC: World Bank Group. http://documents.worldbank.org/curated/en/940631468324888939/pdf/878460BR0R2014050Box 385206B00OUO090.pdf. ———. 2015. 2015 Development Policy Financing Retrospective: Results and Sustainability. Washington, DC: World Bank Group. http://pubdocs.worldbank.org/en/420441457100264616/DevelopmentPolicyRetrospec tive2015.pdf. ———. 2016. Open Government Impact and Outcomes: Mapping the Landscape of Ongoing Research. Washing- ton, DC: World Bank. http://opengovimpact.org/img/og-impact-full-report.PDF. ———. 2017. World Development Report 2017: Governance and the Law. Washington, DC: World Bank. https:// openknowledge.worldbank.org/handle/10986/25880 License: CC BY 3.0 IGO. Independent Evaluation Group | World Bank Group 61 A PPE NDI X E S Engaging Citizens for Better Development Results AN INDEPENDENT EVALUATION Independent Evaluation Group | World Bank Group 63 Appendix A. Methodological Approach Evaluation Questions The evaluation’s objective was to assess how effectively has the World Bank Group mainstreamed citizen engagement and what evidence exists on how this process contributes to the achievement of development outcomes. This overarching objective inspired four lines of inquiry that guided the collection and analysis of data and the framing of findings and recommendations (box A.1). Methodological Design Broadly speaking, the evaluation design was made-up of eight evaluation components: literature reviews, portfolio reviews, assessment of global partnership, survey of World Bank staff, Interviews and Focus Group Discussion with Bank Group staff, Questionnaire administered to civil society organizations (CSOs), case studies at the country level, case studies of citizen engagement mechanisms within projects (detailed in table A.1). In collecting and analyzing data throughout these components, the evaluation adopted a multilevel and mixed methods design. Multilevel Design The evaluation covers the main evaluation questions at three levels: total portfolio, country, and citizen engagement mechanism levels. Given the nature of the evaluation (a mainstreaming strategy), the primary level of analysis is the overall portfolio of the Bank Group. The evaluation ensured coverage of the multiple entry points for citizen engagement in various Bank Group instruments, as laid out in the strategic framework and in chapter 1, with the exception of Advisory Services and Analytics and Program-for-Results, which were outside the scope of the evaluation (World Bank Group 2014, 22). Five structured reviews, were conducted by main Bank Group instrument: (i) investment project financing (IPF), (ii) development policy financing (DPF), (iii) International Finance Corporation (IFC) investments, (iv) Country Partnership Framework (CPF); (v) Global Partnership for Social Accountability. Four BOX A.1.  Lines of Inquiry Guiding the Evaluation What is the magnitude and nature of mainstreaming of citizen engagement within n  World Bank Group operations, and has it changed over time? What is the quality of design of citizen engagement activities? How much attention n  is paid to social inclusion and closing the feedback loop with citizens? What is the extent and quality of monitoring and evaluation of citizen engagement n  activities and what is the evidence of results (contribution to strengthening existing country systems for participation and to improved development outcomes)? To what extent is the Bank Group corporate environment enabling of citizen n  engagement mainstreaming? 64 Engaging Citizens for Better Development Results | Appendix A At the country level, the evaluation covered the complete universe of CPF (46) that had been prepared by July 2017, to assess the level of citizen engagement activities throughout the development of country strategies. Through eight country-level case studies, the evaluation team interviewed government officials and members of civil society to provide more depth of understanding. At the citizen engagement mechanism level, the evaluation covered all the types of citizen engagement mechanisms identified in the strategic framework and listed in box A.2 (World Bank 2014, 24) through portfolio reviews and case studies. The portfolio review, although not representative at the level of each mechanism, provided useful trend information, that were studied in detail through in-depth mechanism-level case studies across outcome areas and country contexts. Figure A.2 at the end of the appendix recapitulates the evaluation design by level of analysis and evaluation question. Ensuring Depth of Understanding and Generalizability at Multiple Levels The design balanced the trade-off between breadth of coverage (as a basis for generalizability) and depth of analysis (as a basis for understanding contextual factors), in the following way: For each level and each evaluation question, the design combined approaches that enable broad coverage (for example, portfolio reviews, surveys), with approaches that enable depth of understanding (for example, semistructured interviews, case studies). To support generalizability of our findings, we applied the three principles of external validity: (i) ensuring representativeness of the sample (through a combination of stratified random sampling and clustered purposive sampling for all evaluation components); (ii) systematically looking for convergence across sources, as well as evaluation components, often aided by content analysis Types BOX A.2.  of Citizen Engagement Mechanisms Consultation n  Grievance Redress n  Collecting, recording and reporting on inputs from citizens n  Collaboration in decision making n  Citizen-led monitoring and evaluation or oversight n  Empowering citizens with resources and authority on their use n  Building citizen capacity for engagement n  Source: World Bank Group 2014, 24. Independent Evaluation Group | World Bank Group 65 software such as NVIVO; (iii) checking whether emerging findings resonated with the broader evaluation literature on social accountability and citizen engagement. Mixed Methods Design This evaluation design combined quantitative and qualitative approaches in all stages of the evaluation design. The purpose of mixing methods was twofold: to enhance internal and external validity of the findings. First, the evaluation team combined quantitative and qualitative approaches, both within and across evaluation components. Using methods that tapped into different dimensions of citizen engagement allowed for a more comprehensive understanding of citizen engagement interventions. For example, several mechanism-level case studies used a combination of quantitative and qualitative data; this was the case in Mali, the Lao People’s Democratic Republic, and Vietnam Project Performance Assessment Reports, and the Dominican Republic process tracing exercise. Portfolio reviews consist of a systematic content analysis of documents to generate descriptive statistics, which were the basis for more in-depth inquiry through interviews with task team leaders and case study evidence at the level of specific mechanisms. To improve construct and internal validity, the team used methods that offset each others’ biases, with a triangulation purpose. First, triangulation was applied by cross-checking sources of evidence within a given methodological component. For instance, within case studies, evidence stemming from interviews and focus group with CSOs were compared with evidence from interviews with government officials and World Bank staff on the same topic. Second, triangulation was applied across evaluation components. For each component, the team generated quantitative data (through survey or structured portfolio reviews) that were corroborated/qualified with information from interviews and focus groups. Evaluation Components Literature Reviews Three types of literature reviews were conducted to inform this evaluation: (i) a review of the literature on the impact of social accountability and citizen engagement interventions; (ii) a review of the Bank Group’s institutional history of integrating citizen engagement; (iii) a review of other development partners’ practice and institutionalization of citizen engagement. The three types are briefly described below: Review of the Evaluative Literature The evaluation team conducted a structured literature review to synthesize the evidence on citizen engagement activities implementation and effectiveness in three outcome areas: (i) public service delivery; (ii) social inclusion and empowerment; and (iii) natural resource management. The review was structured around a set of guiding questions: What are the objectives of citizen engagement mechanisms? What is the evidence on the results of these interventions? What are the enabling 66 Engaging Citizens for Better Development Results | Appendix A Table A.1. Evaluation Components Evaluation Components Description Literature reviews • Structured review of the academic, evaluation, and gray literature on citizen engage- ment’s contribution to improved service delivery, accountability, and natural resource management • Historical review of the integration of participatory approaches into the Bank Group’s work • Benchmarking of Bank Group’s citizen engagement practice with those of other multilateral and bilateral development organizations • Review of World Bank’s regional stocktaking exercises on Citizen Engagement Portfolio reviews • Systematic desk review and assessment of World Bank Investment Lending; Policy Lending; and International Finance Corporation investments • Systematic desk review of Country Partnership Frameworks and country engage- ment notes followed by in-depth interviews with their task team leaders • Structured review of the World Bank’s community-driven development projects Assessment of Review of the Global Partnership for Social Accountability, including synthesis of exist- partnership ing evaluative evidence Survey of World Bank Survey addressed to staff across the World Bank whose role or function make them staff particularly instrumental to citizen engagement mainstreaming Interviews and focus Semistructured interviews with staff and three focus group discussions with citizen group discussion with engagement focal points Bank Group staff Questionnaire An online short consultation form was published and disseminated, and collected administered to civil responses from 76 CSOs society organizations Case studies of citizen Case study of the World Bank’s support to reforms and systems for citizen engage- engagement at the ment and voice at the country level in eight countries (Dominican Republic, Ethiopia, country level Mali, Morocco, West Bank, the Kyrgyz Republic, Pakistan, and Lao PDR) Case studies of citizen In-depth analysis of specific citizen engagement mechanisms embedded in World engagement mecha- Bank projects with a balance of service delivery, natural resource management, and nisms within projects governance projects. Evidence stemming from the analysis of 19 project-level cases using a mix of PPAR, project case and process tracing Note: PDR = People’s Democratic Republic; PPAR = Project Performance Assessment Report. contextual factors? What is known about the effectiveness of specific types of mechanisms? A search strategy with keywords and search strings and a list of sources was established, along with inclusion and exclusion criteria. The literature review became the backbone of the other evaluation components, informing the selection of case studies, as well as the development of templates for the portfolio review and mechanism-level case study. The process tracing study in the Dominican Republic was used to test formally the theoretical framework emerging from the literature review. Independent Evaluation Group | World Bank Group 67 A BOX A.3.  Structured Literature Review to Understand Citizen Engagement in Public Service Delivery The Independent Evaluation Group conducted a structured literature reviewed through a backward snowballing approach. This approach implied the identification of seed articles and books and a selective review of their references to answer three questions: (i) what is the rationale for citizen engagement interventions in public service delivery? (ii) what is the evidence on the results of these interventions and what are the enabling contextual factors? (iii) what is known about the effectiveness of specific types of mechanisms? The literature review found that citizen engagement activities in service delivery can have impacts on three analytical categories: (i) states; (ii) state-society relationships; and (iii) social actors. Some examples of impacts on states are reduced corruption, responsive public officials, better policy design, and overall good governance. Impacts on state-society relationships include the creation of institutional channels for interaction and increase citizen-state trust and legitimacy. Finally, examples of impacts on social actors include improved provision of public goods, empowered citizens, increased social cohesion, and inclusive social norms. With respect to evidence on outcomes, the review found that most of the research has been centered around three outcomes: (i) reduced corruption, (ii) more responsive states, and (iii) improved provision of public goods. The literature indicates that these outcomes are achieved through strengthened accountability, which requires increased transparency as a necessary but not sufficient condition. In addition, the literature also identifies the contextual drivers of accountability, thus shedding light on the circumstances under which citizen engagement interventions are more likely to lead to desired outcomes. In contrast, the review found considerably less research and evidence on the impact on more intangible outcomes such as improved social cohesion. This is not surprising: since measuring these concepts is fraught with difficulties, and lack of consensus on their definition. Finally, the review assessed the effectiveness of specific types of mechanisms and found evidence that, each of them, have different underlying theories of change about impact and that not all of them are expected to deliver impact on the three identified analytical dimensions: (i) states; (ii) state-society relationships; and (iii) social actors. Overall, the review found mixed evidence on impact and that the context, particularly the political context in which these initiatives are carried out, matters. Source: World Bank Group 2014, 24. 68 Engaging Citizens for Better Development Results | Appendix A Retrospective on the World Bank Group Citizen Engagement Practice To put the recent Bank Group citizen engagement efforts into historical perspective, the Independent Evaluation Group (IEG) commissioned a background note whose purpose was to provide an overview of the key activities, policies, research, and milestones related to citizen engagement at the Bank Group since the 1980s. Overall, the note found that the strategic framework builds on and brings together three streams of work in the Bank Group that have been incrementally evolving over several decades (World Bank Group 2014). The first stream dates to 1980 onwards with the integration of stakeholder consultation in Bank Group policies and operational instrument. The second stream starts on 1993 onwards with the inclusion of collaboration with beneficiaries in implementation to ascertain demand, increase responsiveness to local needs, and enhance beneficiary participation and ownership. Finally, starting in 1996, the World Bank started including consultation with stakeholders in the formulation of Bank Group policies and strategies and macro- lending instruments, with the purpose of sharing information and obtaining feedback. Overall, the background note shows a clear evolution in the inclusion of the voice of citizens and civil society that went from reputational risk management to operational engagement to stakeholder influence on policies and strategies at the macrolevel. Review of Other Development Partners’ Practice To position the Bank Group’s citizen engagement efforts within those of other development agencies, the evaluation team conducted a benchmarking of Bank Group’s citizen engagement practice with other multilateral and bilateral development organizations. The IEG team took a purposive sampling approach for selecting the agencies used for the exercise. The agencies chosen are typical comparators to the World Bank and/or are known to have commitments to citizen engagement issues. The agencies chosen were the following: African Development Bank, Asian Development Bank, Inter-American Development Bank, International Fund for Agricultural Development, Norwegian Agency for Development Cooperation, the Swedish International Development Cooperation Agency, the United Kingdom’s Department for International Development and the U. S. Agency for International Development. The IEG team conducted the benchmarking exercise along the following lines of inquiry: (i) nature of the formal guidelines and corporate mandates in place; (ii) terminology used to describe citizen engagement efforts; (iii) organizational arrangements and leadership to support the efforts; (iv) motivations to carry out the efforts; and (v), where available, the nature of results documented. Portfolio Review of IPF on a Population Proportion The review of World Bank lending was guided by the following specific questions: ƒƒ To what extent are citizen engagement indicators and mechanisms present in World Bank IPF? Is there a difference in the number and nature of citizen engagement activi- ties before and after the issuance of the citizen engagement strategic framework and its associated corporate targets? ƒƒ To what extent are citizen engagement indicators included in results frameworks at appraisal reported throughout project implementation? Independent Evaluation Group | World Bank Group 69 ƒƒ What is the extent of implementation of citizen engagement mechanisms? ƒƒ To what extent reporting of citizen engagement has a social inclusion dimension embedded? ƒƒ To what extent are citizen engagement mechanisms in the IPF portfolio driven by safe- guards policies that require mandatory citizen engagement? ƒƒ What are the types of citizen engagement approaches more prevalent in the IPF portfolio? To answer the above questions, the IEG team conducted a structured desk review of a random sample of IPFs. The population under study were International Bank for Reconstruction and Development (IBRD), International Development Association (IDA), and Grants IPFs approved between FY11 and FY16. Two purposive exclusions from the population should be noted. First, the team removed grants equal or below $5 million since these projects are not processed following IBRD / IDA procedures and, for this reason, lacked the standard program documents needed for the structured review. Second, the team removed additional financing projects from the population. However, when conducting the review, the team reviewed the additional financing documents of the sampled projects. After these two exclusions, the population yielded 1,326 IPFs. Using available online sample calculators, the team determined that it needed a sample size of 299 IPFs to be able to draw inferences about the population with a 95 percent confidence level and a 5 percent margin of error. Also, to be able compare the two-time periods FY11–13 versus FY14–16, the team further sampled these two periods following their proportions in the population. This stratified random sampling approach yielded 147 and 152 projects for FY11–13 and FY14–16 respectively.1 A coding template was developed, using the Bank Group strategic framework and other resources developed by the Citizen Engagement Secretariat as its conceptual reference. Coding was distributed among four coders, supervised by the portfolio review coordinator. Training was organized and intercoder reliability was ensured through a piloting phase as well as periodic quality assessment and spot-check by the portfolio review coordinator. Portfolio Review of Development Policy Financing on a Population Proportion In the DPF review, the evaluation team analyzed whether and how the World Bank has leveraged DPF as an entry point for fostering an environment for citizen engagement in its client countries. The following two questions guided this review: ƒƒ To what extent has the World Bank supported reforms that create an enabling environ- ment for citizen engagement and participation through DPF prior actions and triggers? ƒƒ How effective has the World Bank been in promoting a dialogue between the govern- ment and a wide range of stakeholders to support the reform process funded through DPF? 70 Engaging Citizens for Better Development Results | Appendix A The evidence base for the study came from three primary sources: ƒƒ A synthesis of information contained in the 2015 DPF retrospective and IEG work on Managing Social and Environmental Risks in Development Policy Loans; World Bank 2015) ƒƒ An analysis of the prior actions database between FY90 and FY17 to identify broad trends about the prevalence of prior actions that promote reforms of countries’ system and processes for citizen engagement. The prior actions database compiles more than 25,000 prior actions, organized by thematic codes. A trend analysis was carried out on the full population of prior actions classified by thematic codes, to identify broad patterns of prevalence of certain types of prior actions that can, directly and indirectly, enable citizen engagement ƒƒ A structured review of a random sample of DPF program documents (including pro- gram documents, Legal agreements, Implementation Completion and Results Reports and Implementation Completion and Results Report Reviews as available) to assess reporting on stakeholder consultations. There were 374 DPF approved between FY11 and FY16 (the period of the evaluation). The team reviewed a random sample of 60 DPF which allowed drawing conclusions about the population of DPF with a 95 percent confi- dence interval and 11.5 percent margin of error. Portfolio Review of IFC Investments on a Population Proportion To assess the extent of mainstreaming in IFC, IEG analyzed a sample of the portfolio of investment projects at two points in time. First, the evaluation team analyzed all projects evaluated by IEG that were approved between FY07 and FY11. Projects with IEG ratings of “No Opinion Possible” and “Not Applicable” were not included in the data set. Following the criteria outlined above, a sample of 137 real sector investment projects were selected for analysis. IEG compared this early portfolio with newer, post-2012 projects to capture any difference in design after the Performance Standard (PS) policy change of 2012. For this set of projects, IEG randomly selected a small sample (30) of real sector projects approved in FYs 2015 to 2017, to allow enough time for these new PSs to be rolled out. Finally, 16 interviews with IFC investment officers were conducted by the IEG Environmental and Social specialist and analyzed to provide more qualitative insights. IEG used the Expanded Project Supervision Report as the source of information for creating and populating citizen engagement categories. In the Expanded Project Supervision Report, IEG looked at the Stakeholder Engagement, PS 1, and, where relevant, PS 4, 5, and 7 under the Environmental and Social Mitigation Measures and the Environmental and Social Action Plan sections. Finally, IEG used the information under the heading Overview of IFC’s Scope of Review to categorize how IFC provided support to the client and reviewed the client’s application. Independent Evaluation Group | World Bank Group 71 Structured Review of Community-Driven Development Projects A structured document review of 36 recently approved community-driven development (CDD) projects was undertaken to assess their alignment with the strategic framework on citizen engagement and its guiding principles for quality engagement. The project documents and, when available, the Implementation Completion and Results Report Reviews and Project Performance Assessment Reports of CDD projects were consulted. The sample of CDD projects reviewed was purposefully built from the following sources: ƒƒ Fourteen CDD projects available from the randomly derived portfolio of investment proj- ects approved in FY14–16 described above ƒƒ Fifteen “full” CDD projects identified through the World Bank CDD Database2 ƒƒ Seven CDD projects from parallel work undertaken by IEG in the forest sector, approved since FY14 Structured Review of Country Strategies To analyze how citizen engagement was included in country strategies, we conducted a desk review of the universe of the CPFs and Country Engagement Notes (CENs) conducted between FY15 and FY17, 49 CPFs and 8 CENs in total. Given the heterogeneity in reporting and documentation on participatory processes/engagement efforts across CPF and CEN documents, the desk review was complemented with 40 semistructured interviews with key informants (the task team lead of the CPF and CEN, country managers and country representatives, and communication officers who participated in the process) and information collected in country visits. The analysis also benefited from the underlying data collected for the IEG’s evaluation of the new Bank Group country engagement that reviewed the first 22 strategies conducted in countries with a full engagement cycle. Assessment of Global Partnership for Social Accountability The program was assessed using IEG’s evaluation framework (World Bank 2007) for assessing global and regional partnership programs. This framework is based on the Organisation for Economic Co-operation Development Assistance Committee criteria of evaluating development assistance adapted to evaluating global partnership programs. It should be noted that the review of GPSA aimed to inform this evaluation and therefore, it is not a full “top-down” review of the program. A more comprehensive external evaluation of GPSA would provide broader perspective on the relevance and effectiveness of the program against its all stated objectives. The assessment is based on a desk review of key program and project documents for the period of 2012–17, external evaluation of the Global Partnership for Social Accountability (GPSA; Robinson 2015), interviews with the program management, team leaders, program donors as well as CSOs executing the GPSA grants. 72 Engaging Citizens for Better Development Results | Appendix A Twelve projects in 11 countries were selected for in-depth desk review. Because the GPSA project portfolio is very new (only one project was closed as of July 2017) the projects with highest disbursement rates were selected for the desk review, on the assumption that these would have made the most progress toward their intended outcomes. These projects were primarily from the first round of GPSA’s grants that became effective in FY14 and a few from the second round (Ghana and Democratic Republic of Congo).3 Semistructured Interviews and Focus Group Discussions with Bank Group Staff Between May and October of 2017, the evaluation team conducted 230 semistructured interviews with Bank Group staff and management to assess the enabling environment for citizen engagement mainstreaming in the organization. IEG conducted the following interviews: ƒƒ 40 interviews with senior-level World Bank staff and management in the Global Practic- es, cross-cutting solutions areas, and World Bank Regions selected to obtain cross-cut- ting perspectives ƒƒ 40 interviews with staff involved in the preparation of Systematic Country Diagnostics (SCDs) or CPFs to inquire on the nature of consultations ƒƒ 3 Focus Group Discussions with 12 citizen engagement focal points involved in the day-to-day operationalization of the World Bank’s mainstreaming strategy. The evalua- tion team also participated in three meetings of the citizen engagement focal points, for participatory reflections and observations ƒƒ 18 interviews with task team leads of World Bank operations that were studied in the portfolio review to identify lessons learned and challenges ƒƒ 104 interviews with World Bank staff were conducted during case study missions ƒƒ 16 interviews with IFC investment officers to provide details and reflections on the trends identified in the portfolio reviews The evaluation team took detailed, written notes for each interview and systematically coded and analyzed them using a content analysis software (NVivo) to derive themes and key messages from the interviews that could be triangulated with each other and with other information sources (notably survey responses). Survey of World Bank Staff The evaluation team conducted a survey of the World Bank staff members and managers. The objective of the survey was twofold: (i) To test on a broader scale some of the emergent findings from interviews with staff; (ii) to elicit information and opinions on how the World Bank staff has operationalized the citizen engagement in their work and the challenges they face. Independent Evaluation Group | World Bank Group 73 Sampling Strategy The target population of the survey were staff members and managers who are the most directly concerned with the corporate mandate on citizen engagement by their role or function. Jointly with the Citizen Engagement Secretariat, it was decided that the survey should be administered to the following group of staff: country directors and managers, country representatives and country program coordinators, practice managers, social development specialists, governance specialists (excluding financial sector specialists and private sector specialists), communication specialists based in country offices, and citizen engagement focal points. A sampling frame was reconstructed based on a title search in the Human Resources iSearch page, and completed with an updated list of all citizen engagement focal points compiled by the secretariat. The sampling frame had 1,081 members. A stratified random sample was then drawn from the sampling frame to ensure a 95 percent confidence and 10 percent margin of error. The total number of staff contacted was 417, the final sample of respondent was 211. Questionnaire Design The design of the questionnaire was iterative and participatory with several rounds of review and revision, building on expertise from citizen engagement focal points. A focus group discussion was organized with citizen engagement focal points and members of the secretariat to discuss the relevance and wording of the questions. The revised questionnaire was then piloted with the full network, as well as one key informant in each of the targeted subgroup. Data Collection A web-based survey, through SurveyMonkey, followed up by phone recall, were implemented over two-time periods. The first period ran from July 6, to August 2, 2017. The second period ran from September 15, to October 5. As follow-up contacts are found to increase response rates, we sent two reminder emails (see Munoz-Leiva et al. 2010). For each period, two email reminders were sent to non- respondents, followed up by a phone reminder. In total, 211 people answered the questionnaire, with a response rate of 50 percent. Online Satisfaction Survey of Civil Society Organizations A short online satisfaction survey was open to global and national CSOs. The online tool was used for two purposes: (i) gather feedback on the Bank Group’s engagement strategy as well as suggestions for improvement; and (ii) identify some of the citizen engagement and social accountability initiatives of the Bank Group that resonated most with CSOs. The feedback from 76 CSOs was collected. Although this sample is not representative of the range of CSOs the Bank Group tends to interact with, their input was used as a source of information, which was then triangulated with feedback received from the 245 members of CSOs interviewed during field visits for case studies. 74 Engaging Citizens for Better Development Results | Appendix A Case Studies of Citizen Engagement at the Country Level The complex nature of the evaluand warranted an in-depth view that was best served by a case- based design. Case selection at the country level was an iterative process, with input provided by the Citizen Engagement Secretariat, considering the following conditions: ƒƒ Sufficient diversity of country context along the important dimensions of Voice and Ac- countability, as well as fragility; ƒƒ Representation of all six Regions; ƒƒ Diversity of substantive citizen engagement activities across outcome areas; ƒƒ Countries highlighted by key informants as having prominent citizen engagement pro- grams with some maturity, as well as countries considered to not have that. Eight countries were selected for the country-level analysis and represent a broad spectrum of enabling environment for Citizen Engagement as illustrated in figure A.1. A structured protocol for case studies was developed, pilot tested in the Dominican Republic and refined before being deployed in the other cases. The methods included: desk review of country strategies; portfolio review of interventions and Advisory Services and Analytics with a citizen engagement component; interviews of World Bank staff in country management units, government officials, representative of international and local CSOs; and focus group discussion with CSOs FIGURE A.1. Voice and Accountability Index Dominican Republic Mali* Kyrgyz Republic Morocco Pakistan West Bank and Gaza* Ethiopia Lao PDR 0 10 20 30 40 50 60 Percentile ranking Note: Asterisks denote fragility, conflict, and violence-affected countries Independent Evaluation Group | World Bank Group 75 involved and not involved in the dialogue around the country engagement (including local CSOs based outside the capital city). The four main questions that guided the country-level cases were: ƒƒ How effectively has the Bank Group integrated Citizen Engagement throughout its coun- try engagement cycle? ƒƒ How effectively and inclusively has the Bank Group consulted citizens during the prepara- tion and monitoring of the current strategy, as well as the evaluation of the previous one? ƒƒ How strategically does the Bank Group build on and strengthen existing (non)govern- mental systems for citizen engagement and existing mechanisms for collective actions? ƒƒ What notable results has the Bank Group achieved in institutionalizing citizen engage- ment in the country over the past ten years? For each case, a narrative was drafted and coded using NVIVO as a basis for synthesizing evidence across cases and feeding into the overall evaluation report. Case Studies of Citizen Engagement Mechanisms within Projects The evidence generated through the portfolio review of a sample of IPF provided information on all the types of citizen engagement mechanisms used by the Bank Group. Given the high number of cases of consultations, grievance redress, and beneficiary feedback mechanisms found in the portfolio, findings on these mechanisms are established with a high level of confidence. Moreover, consultations were also studied more in-depth via the review of the SCD/CPF. Three types of mechanisms were less common in the portfolio: (i) collaboration in decision making and empowerment of citizens with resources; (ii) citizen-led monitoring; and (iii) capacity building for citizens. Other evaluation components were used to mitigate this low coverage. The review of CDD was performed to better cover the first; the second was over-sampled in the case study, and the third was covered in-depth through the review of the GPSA. However, given the limited reporting on citizen engagement activities in project documents, empirical analysis was necessary to answer the key evaluation questions. The focus of the case studies of specific citizen engagement mechanisms within projects was on the quality of design (with a specific attention to social inclusion), the extent and challenges of implementation, the quality of monitoring, the extent to which the feedback loop was closed, and the contribution to development outcomes. In-depth inquiry into specific citizen engagement mechanisms was deployed with the view to cover the typical mechanisms across the main outcome areas highlighted in the strategic framework: service delivery, governance, and natural resource management. Table A2 summarizes the case- base design at the mechanism level. Data collection was either embedded within Project Performance Assessment Report missions or missions organized for country-level cases (described above). It consisted of in-depth interviews with project implementation units and other government entities involved in project implementations (N = 203), CSOs or community mobilizers (N = 252) directly involved or familiar with the projects, as well 76 Engaging Citizens for Better Development Results | Appendix A TABLE A.2. Cases of Citizen Engagement Mechanisms Outcome Country Area (N = 11) Project (N = 19) Citizen Engagement Mechanism (N = 40) collaboration Citizen mon- engagement and decision Empowering reporting on citizens with Community Building for itoring and Collecting, Grievance resources Citizen or feedback oversight Capacity redress power Kyrgyz Village investment X X Republic Project (2002–2015) Water Management X Project West Bank Cash Transfer Program X and Gaza Southern West Bank X X X X Solid Waste Management Morocco Initiative Nationale X X de Développement Humain Solid Waste DPL X X Ethiopia Promoting Basic X X X X Services (2006–2018) Laos Poverty Reduction X X Fund Public Service Delivery Philippines Support to Basic Edu- X X cation Sector Pakistan Punjab Education X Sector Project Dominican ProSoli conditional X X Republic cash transfer Electricity Distribution X X and Grid Modernization Morocco Integrated Coastal X Zone Management Natural Resource Vietnam Forest Sector Develop- X X ment Project Management Laos SUFORD (2006–2017) X X Pakistan Punjab irrigated X agriculture and barrages improvement West Bank Municipal Development X X X and Gaza Project (2009–2017) Governance Mali Rural Community X X Development Cambodia Demand for Good X X X X Governance Total 11 countries 19 projects 6 6 9 8 5 6 cases Note: DPL = development policy loan; ProSoli = Progresando con Solidaridad; SUFORD = Sustainable Forestry and Rural Development Project. Independent Evaluation Group | World Bank Group 77 Process BOX A.4.  Tracing of Citizen Engagement in the Dominican Republic The evaluation team piloted an in-depth causal analysis method called process tracing in the case of the Reportes Comunitarios of the national conditional cash transfer of the Dominican Republic. Process Tracing was used to assess the impact of embedding a participatory monitoring in the conditional cash transfer and to evaluate the significance of the World Bank’s contribution. Process Tracing is a rigorous method of within-case causal inference that relies on Bayesian updating logic to transparently assess the probative value of pieces of evidence provided to justify specific contribution claims (Befani and Mayne 2014; Befani and Stedman-Bryce 2016). The team followed the basic steps of process tracing: (i) Laying out a very thorough theory of change, which spells out a series of testable contribution claims; (ii) establishing sources of evidence that have either disconfirmatory or confirmatory power; (iii) collecting the evidence during a week of in-depth inquiry with the support of the operational team from the Social Cabinet; (iv) testing the evidence collected through four formal tests of inference. These tests are “smoking gun” (confirmatory); “hoop test” (disconfirmatory); “doubly decisive” (both); “straw in the wind” (neither); (v) Updating the overall confidence in the contribution claims and explanation of impact. as focus group discussions with citizens involved in the specific citizen engagement mechanisms. In the case of the Community TABLE A.3. Recap Scorecard of Sampling and embedded Selection the Dominican infor Republic conditional Main Evaluation Components cash Sampling Period Source of evidence Strategy Covered Target Sample Sampling Frame Portfolio of IPF Stratified FY11–16 Approved IPFs 299 1,326 Random Portfolio of DPL Random FY11–16 Approved DPLs 60 374 Portfolio of CPF Universe FY15 –17 CPFs and CEN N/A 49 CPFs and 8 CEN Portfolio of real sector Random FY07–11 IEG ENS Reviews 137 930 IFC investments EV-Notes XPSR Portfolio of real sector Random FY15–17 ENS Review Summary 30 481 IFC investments Board Documents Portfolio of CDD Purposive FY14–16 Approved IPFs 36 57 GPSA Purposive 2012–2017 Approved Grants 12 32 Survey Stratified 2017 World Bank Group Staff 417 1,071 Random (211 respondents) Note: CEN = citizen engagement note; CPF = Country Partnership Framework; DPL = development policy loan; ENS = Environmental and Social; EV-Note = evaluation note; IEG = Independent Evaluation Group; IFC = International Finance Corporation; IPF = investment project financing; XPSR = Expanded Project Supervision Report. 78 Engaging Citizens for Better Development Results | Appendix A transfer program (PROSOLI), an in-depth causal analysis using process tracing was performed, as described in box A.4. Recapitulation of Samples and Selection The findings and conclusions reached by this evaluation should be interpreted in light of both the strengths and limitations of the evidence gathered. Broadly defined, there are three sets of limitations. The first is the result of conscious choices about scope given the necessary trade-off between depth and breadth of inquiry. The second has to do with sampling issues and the third relates to data availability. Limitations Scope Delimitations ƒƒ The team made a necessary trade-off between breadth and depth of analysis. Some topics were left outside the scope of the evaluation, such as citizen engagement in advisory services, SCDs, PforR, and in Multilateral Investment Guarantee Agency guarantees; other themes were covered more superficially than others, such as citizen engagement in DPF. ƒƒ The evaluation’s objective was to provide timely feedback to the Board and manage- ment on its ongoing effort to mainstream citizen engagement, thus deliberately taking a forward look and focusing on relatively recent developments, notably the period of FY11 to FY17. Consequently, many operations (projects, and CPF) are still not closed, with limited information on results achieved. Sampling and Selection ƒƒ Given the nature of the evaluand (a mainstreamed concept), the evaluation team could not rely on a neatly defined portfolio of activities to anchor its assessment. It thus relied on probability sampling principles to select a set of activities (IPF, DPF and IFC investments) to assess. For the IPF portfolio, results obtained can only be extrapolat- ed to the two periods under analysis (FY11–13 and FY14–16) with a known confidence and margin of error. Estimations on population proportions are not representative at the level of the Global Practices (GPs), the Regions, or citizen engagement mecha- nisms. ƒƒ Countries studied more in-depth for case studies were selected in dialogue with the Citizen Engagement Secretariat to represent an array of contexts in which the Bank Group engages. However, the team purposefully centered the case study selection model on countries in which the Bank Group had a deliberate country-level approach to citizen engagement to gauge effects at a more systemic level. This purposive sampling of countries is not representative of the total population of countries in which the Bank Group is active. Independent Evaluation Group | World Bank Group 79 ƒƒ Similarly, project-level cases were purposefully selected to represent the range of out- come areas in which citizen engagement takes place (natural resource management, service delivery, and governance) and represent the full gamut of citizen engagement mechanisms used by the World Bank. However, there was a deliberate decision to cen- ter on interventions that had multiple citizen engagement mechanisms or mechanisms that required in-depth engagements, as well as projects that were relatively mature, to trace contributions to development outcomes. This purposive sampling of interventions is not representative of the total population of projects in which the Bank Group applied citizen engagement mechanisms. ƒƒ Survey respondents did not represent the overall population of World Bank staff for several reasons. First, the survey frame intentionally focused on a subset of World Bank staff who, by their role or position, were seen by the evaluation team and the Citizen Engagement Secretariat as having a specific stake in the citizen engagement agenda. Second, within these purposefully defined clusters, survey participants were randomly selected. However, there was an unavoidable response bias inherent in online voluntary surveys, with a 50 percent response rate. ƒƒ Although the evaluation team attempted to reach a broad set of stakeholders, benefi- ciaries, and CSOs through interviews and focus groups in case studies, IEG had to rely on World Bank staff’s networks and referrals to identify stakeholders to engage, with inevitable response biases. Data Availability ƒƒ The team faced important challenges in collecting information on implementation and results of citizen engagement activities. Given that citizen engagement activities are rarely “full components” of projects, the design, implementation, and results are poorly documented in project and program documents. When engagement is mandatory (for example, in safeguards and development policy loan preparation), reporting can be formulaic. When it is not mandatory, reporting is uneven. Information on engage- ment may be captured at best in operational manuals that are not easily accessible and assessable at scale. Consultation records are poorly documented and stored on individual drives. 80 Engaging Citizens for Better Development Results | Appendix A  verall BOX A.5. O Evaluation Design Data collection & analysis methods Evaluation Questions Ensuring coverage Ensuring depth IPF Portfolio Review Interviews with TTL Corporate DPL Portfolio Review Environment Interviews with WB managers IFC Portfolio Review and staff Focus Group Discussions with GPSA Structured Review CE Focal Points Survey of CSOs Interview with IFC Staff Magnitude and Overall portfolio level nature of Survey of Staff & Management mainstreaming Interview with Government Stakeholders in 8 Countries Interview and Focus Group Discussions Structured Review of SCD/CPF with Civil Society in 8 Countries Quality of citizen engagement Interview with Staff in 8 CMUs Country level IPF Portfolio Review Quality of monitoring and CDD Structured Review Project-Level Case Studies in contribution to 11 Countries Covering 19 Projects develoment Level Citizen outcomes Mechanism Engagement Structured Literature Review Independent Evaluation Group | World Bank Group 81 ƒƒ In the period of our evaluation, none of the GPSA projects were closed, so no self-evaluation reports were available. ƒƒ There is very limited data on the cost of citizen engagement activities. 1  Given these samples sizes for each period, the team will be able to make inferences with a 95 percent confidence and 7.1 percent margin of error for FY11–13 and a 7.04 percent margin of error for FY14–16. 2  Per the database, “full” community-drive development (CDD) refers to projects that are entirely CDD projects as opposed to projects that only have a CDD components. The database, curated by the CDD Anchor in the Social Development Department, can be found by visiting http://sdweb.worldbank.org/cdd/index.cfm?Page=home. 3  angladesh (2 projects), Ghana, Indonesia, Malawi, Mozambique, Philippines, Tajikistan, Kyrgyz Republic, Moldova, B Democratic Republic of Congo, and Dominican Republic. references Befani, B., and J. Mayne. 2014. “Process Tracing and Contribution Analysis: A Combined Approach to Generative Caus- al Inference for Impact Evaluation.” IDS Bulletin 45 (6): 17–36. doi:10.1111/1759–5436.12110. Befani, B., and G. Stedman-Bryce. 2016. “Process Tracing and Bayesian Updating for impact evaluation.” Evaluation 23 (1): 42–60. doi:10.1177/1356389016654584. Robinson, M. 2015. Program Evaluation: Global Partnership for Social Accountability. Washington, DC: World Bank. World Bank. 2007. Sourcebook for Evaluating Global and Regional Partnership Programs: Indicative Principles and Standards. Independent Evaluation Group. Washington, DC: World Bank. https://openknowledge.worldbank. org/handle/10986/6601. World Bank Group. 2014. Strategic Framework for Mainstreaming Citizen Engagement in World Bank Group Operations: Engaging with Citizens for Improved Results. Washington, DC: World Bank Group. http:// documents.worldbank.org/curated/en/266371468124780089/Strategic-framework-for-mainstreaming- citizen-engagement-in-World-Bank-Group-operations-engaging-with-citizens-for-improved-results. World Bank. 2015. Managing Social and Environmental Risks in Development Policy Financing. Washington, DC: World Bank. 82 Engaging Citizens for Better Development Results | Appendix A  ortfolio Review of Citizen Appendix B. P Engagement Activities in Investment Project Financing Introduction The overall objective of the portfolio review was to assess the coverage and quality of citizen engagement mechanisms and indicators in World Bank investment project financing (IPF). More precisely, the review was guided by the following specific questions: ƒƒ What is the coverage of citizen engagement mechanisms and indicators in World Bank IPF? Has there been a difference in coverage before and after the issuance of the citizen engagement strategic framework and its associated corporate targets? ƒƒ What are the types of citizen engagement mechanisms included in projects? ƒƒ What is the quality and implementation rate of citizen engagement mechanisms? ƒƒ What type of citizen engagement approaches does the World Bank monitor through indicators in results frameworks? What is the quality of these indicators? ƒƒ What is the reporting rate of citizen engagement indicators? To answer the above questions, the Independent Evaluation Group (IEG) team conducted a structured desk review using a coding protocol that was developed based on the main elements of the strategic framework and the guidance note.1 This review follows therefore the Citizen Engagement Secretariat and Operations Policy and Country Services definition of what constitutes a citizen engagement indicator and a beneficiary-oriented design. An indicator is classified as a citizen engagement indicator when it “clearly captures feedback from citizens or monitors the degree of involvement that citizens have in the design, implementation, or oversight of projects” (World Bank 2014, 8). This review also considers to be citizen engagement indicators those that track World Bank support to capacity building for engagement either on the demand side (citizen side) or the supply side (government side). These indicators usually track World Bank support to institutionalize systems to engage with government agencies and other stakeholders and / or build capacity of citizens and government to engage. Finally, projects with a “beneficiary-oriented design” are those that declare the intention to engage with citizens during implementation and that explicitly describe at least one citizen engagement mechanism in the project appraisal document or the project paper for additional financing. Sample IEG applied the coding protocol to a random sample of IPFs approved between fiscal year (FY)11 and FY16.2 The IPFs under study included International Bank for Reconstruction and Development Independent Evaluation Group | World Bank Group 83 (IBRD), International Development Association (IDA), and Grants. Two purposive exclusions from the population should be noted. First, IEG removed grants equal or below $5 million since these projects are not processed following IBRD / IDA procedures and, for this reason, lack standard program documents needed for the structured review. Second, IEG removed additional financing projects from the population. However, when conducting the review, the team reviewed the additional financing documents of the sampled projects. After these two exclusions, the population yielded 1,326 projects. Using available online sample calculators,3 IEG determined that it needed a sample size of 299 IPFs to be able to draw inferences about the population with a 95 percent confidence level and a 5 percent margin of error. Also, to be able to make comparisons between FY11–13 and FY14–16, the team further sampled these two periods following their proportions in the population. This stratified random sampling approach yielded 147 and 152 projects for FY11–13 and FY14–16.4 Statistics have been also calculated for FY16 separately to assess the latest trend, although – due to the small sample size, and the overrepresentation of SURR projects in the FY16 subsample – they should be interpreted with great caution. Once the sample was randomly chosen for each period, the IEG team applied a coding protocol to project documents to identify whether they included citizen engagement mechanisms and indicators. Project documents reviewed included project appraisal documents or project papers, implementation supervision reports, additional financing and restructuring papers, aide memoires and, when available, implementation completion reports. The tables below present descriptive statistics of the sample and population under analysis and show that distributions by year, Global Practice, Region, and type of agreement are roughly similar, thus confirming that the randomization process was adequate. The distribution of projects by years in the sample shows that the highest share is in FY11 and FY14, whereas the lowest is in FY13 and FY16 (table B.1). With respect to Global Practices, the highest share is in Social, Rural, Urban and TABLE B.1. Sample and Population, by Fiscal Year Sample Population Fiscal Year  Number Percentage Number Percentage 2011 59 19.7  247 18.6  2012 46 15.4  177 13.3  2013 42 14.0  211 15.9  2014 61 20.4  263 19.8  2015 52 17.4  242 18.3  2016 39 13.0  186 14.0  Total 299 100.0  1,326 100.0  Source: Independent Evaluation Group calculations based on data retrieved from Business Warehouse 84 Engaging Citizens for Better Development Results | Appendix B Resilience (19.7 percent) and Transport and ICT (15.1 percent) and the lowest in Poverty (0.3 percent) and Macroeconomics and Fiscal Management (1 percent). In terms of Regions, the highest share is in Africa (31.8 percent) and the lowest in GGSVP (2 percent; table B.3). With respect to agreement types, table B.4 shows that almost 80 percent of the projects in the sample belong to IBRD / IDA followed by recipient-executed trust funds (RETF) (14 percent) and Global Environmental Facility (GEF; 5.4 percent) projects; less than 1 percent of projects are under special fund and Montreal protocol agreements. TABLE B.2. Sample and Population by Global Practice Sample Population Global Practice  Number Age Number Age AGR 27 9.0  130 9.8  EDU 19 6.4  135 10.2  EAE 30 10.0  143 10.8  ENR 19 6.4  113 8.5  FAM 9 3.0  42 3.2  GOV 14 4.7  72 5.4  HNP 22 7.4  91 6.9  MFM 3 1.0  8 0.6  POV 1 0.3  11 0.8  SPL 17 5.7  73 5.5  URS 59 19.7  180 13.6  TAC 7 2.3  45 3.4  TIC 45 15.1  162 12.2  WAT 27 9.0  121 9.1  Total 299 100.0  1,326 100.0  Source: Independent Evaluation Group calculations based on data retrieved from Business Warehouse. Note: AGR = Agriculture; EDU = Education; EAE = Energy; ENR = Environment; GOV = Governance; HNP = Health, Nutrition and Population; MFM = Macro Fiscal Management; POV = Poverty; SPL = Social Protection and Labor; URS = Urban Rural and Social; TAC = Trade and Competitiveness; TIC = Transport and Infrastructure; WAT = Water Independent Evaluation Group | World Bank Group 85 TABLE B.3. Sample and Population by Vice Presidential Unit (VPU) Sample Population VPU  Number Age Number Age AFR 95 31.8  456 34.4 EAP 55 18.4  238 17.9  ECA 50 16.7  185 14.0 GGS 2 0.7 2 0.2 LAC 32 10.7  167 12.6 MENA 17 5.7  73 5.5  SAR 48 16.1  205 15.5 Total 299 100.0  1,326 100.0  Source: Independent Evaluation Group calculations based on data retrieved from Business Warehouse Note: AFR = Africa; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; EAP = East Asia and Pacific; MENA = Middle East and North Africa; SAR = South Asia. TABLE B.4. Sample and Population by Agreement Type Sample Population Agreement Type Number Age Number Age GEF 16 5.4  76 5.7 IBRD 85 28.4  379 28.6  IDA 154 51.5 662 49.9 Montreal Protocol 1 0.3 3 0.2 RETF 42 14.0  198 14.9 Special Fund 1 0.3 8 0.6  Total 299 100.0 1,326 100.0 Source: Independent Evaluation Group calculations based on data retrieved from Business Warehouse. Note: GEF = Global Environment Facility; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association. 86 Engaging Citizens for Better Development Results | Appendix B Results: Citizen Engagement Mechanisms Time Trend The percentage of World Bank projects that explicitly describe at least one citizen engagement mechanism in their project appraisal document is 92 percent (N = 299) for the period FY11–16. A comparison between FY11–13 and FY14–16 shows that these high percentages have not changed over time with 92 percent for period FY11–14 and 93 percent for period FY14–16. Although the percentage of projects with at least one citizen engagement mechanism has not changed, the percentage of projects with at least one non-safeguards-related citizen engagement mechanism has increased over time. The portfolio review classified projects based on whether they only had safeguards-related citizen engagement mechanisms or at least one non-safeguards-related mechanism. This was done by tagging whether each mechanism within a project was related to a safeguard requirement or not. For instance, the Jilin Hunchun Railway Project (P122321) included only consultations and a grievance redress mechanism (GRMs) due to the project having triggered the following safeguard policies: environmental assessment (operational policy [OP] 4.10, category A), indigenous populations (OP 4.10) and involuntary resettlement (OP 4.12). In contrast, the National Highways Authority of India Technical Assistance Project (P121515) introduced a scorecard to collect feedback from road users, who were in this way involved in monitoring the performance of the National Highways Authority of India. The portfolio review found that the percentage of projects that describe at least one non-safeguards-related citizen engagement mechanism has increased from 67 percent (FY11–13) to 76 percent (FY14–16),5 likely in response to the citizen engagement strategic framework and its associated targets (figure B.1). FIGURE B.1. Projects with Citizen Engagement Mechanisms 100% 92% 93% 80% 76% 67% 60% 40% 20% 0% At least one CE mechanism At least one non-safeguard related mechanism FY15–11 (N = 147) FY14–16 (N = 152) Source: Independent Evaluation Group structured desk review. Note: CE = citizen engagement. Independent Evaluation Group | World Bank Group 87 FIGURE B.2. Percentage of Projects with “Thick” Citizen Engagement Approaches 100% 80% 60% 40% 38% 27% 20% 0% Percentage of projects with “thick approaches” FY13–11 (N = 135) FY14–16 (N = 141) Source: Independent Evaluation Group structured desk review. Finally, within the pool of projects with at least one citizen engagement mechanism (N=276), the portfolio review found that the percentage of projects with a “thick approach” increased over time, from 27 percent in FY11–13 to 38 percent in FY14–16, with a clear increase to 58 percent in FY16. Projects with a “thick approach” have more than one mechanism and include at least one of the following approaches: citizen/community collaboration, citizen monitoring, evaluation and oversight, empowering citizens/communities, and capacity building for engagement. In contrast, the definition of a “thin approach” included projects with only one citizen engagement mechanisms or projects that only have consultation, grievance redress, and beneficiary feedback approaches. Mechanisms Consultations are the most frequently adopted citizen engagement mechanism (86 percent) followed by GRMs (55 percent) and collecting, recording and reporting on inputs received from beneficiaries (33 percent). Other important, though less frequently included, mechanisms are citizen / community collaboration (24 percent), citizen monitoring (18 percent) and capacity building for citizen engagement (23 percent). Finally, empowering citizens / communities with resources and decision- making powers is described as a mechanism in 9 percent of the projects (table B.5). The percentage of projects with any given mechanism remained stable over time, except for projects adopting GRMs, whose percentage increased from 41 percent in FY11–13 to 67 percent in FY14–16. Using the number of mechanisms as the unit of analysis, the IEG review found that consultations and GRMs are primarily driven by safeguards requirements whereas, for the remaining mechanisms, the relation to safeguards is only marginal. Table B.6 breaks down mechanisms by type of approach and shows the relation of each of these approaches to safeguards requirements. 88 Engaging Citizens for Better Development Results | Appendix B TABLE B.5.  rojects with at Least One Citizen Engagement Mechanism, by P Approach Type, FY11–16 (N=276) Type of Citizen Engagement Approach  FY11–13 FY14–16 FY11–16 N  % N % N %  Consultation 116 86  121 86  237 86  Collecting, recording and reporting input from beneficiaries 46 34  44 31  90 32  Grievance redress 55 41  94 67  150 54  Citizen/community collaboration 38 28  28 20  66 24  Citizen monitoring, evaluation and oversight 22 16  28 20  50 18  Empowering citizens/communities with resources and decisions 11 8  13 9  23 9  Capacity building for engagement 32 24  32 23  65 23  Source: Independent Evaluation Group structured desk review. Note: Percentages add up to more than 100 because a project can have multiple mechanisms.  itizen TABLE B.6. C Engagement Mechanisms by Type of Approach and Relation to Safeguards Requirements Safeguards Related Nonsafeguards Mechanisms Type of Approach (percent) Related (percent) (no.) Consultation 71  29  302 Collecting, recording and reporting on inputs 1  99  97 received from beneficiaries Grievance redress 53  47  155 Citizen/community collaboration on planning and/ 4  96  70 or execution of a policy, program or project Citizen monitoring, evaluation and oversight 6  94  50 Empowering citizens/communities with resources 8  92  25 and decision-making powers Capacity building for engagement 7  93  70 Source: Independent Evaluation Group structured desk review. Independent Evaluation Group | World Bank Group 89 Quality Closing the Feedback Loop The World Bank Group has no single and clear-cut definition of what is meant by “closing the feedback loop.” The corporate mandate states that each project going to the Board for approval must do as follow: “Include an explicit description of at least one mechanism to engage with ultimate beneficiaries of the project (or affected population) and how the feedback loop will be closed (that is, establish robust mechanisms to ensure that the feedback will trigger a response).”6 In turn, the citizen engagement strategic framework states that “closing the feedback loop” is about “informing those engaged how the information they provided has been used” (World Bank Group 2014, 18, para. 42). This review tested both definitions: (i) acting on the feedback of citizens (for instance by using this feedback to improve project design or implementation); and (ii) including a plan to inform citizens of the actions taken in response to their feedback. To investigate the extent to which citizen engagement mechanisms included in projects meet these two criteria, the team reviewed the project documents of all projects in the sample approved in FY16 (N=38), that likely benefited from better guidance. The team identified first all the citizen engagement mechanisms included in the projects (115 mechanisms in 38 projects). The team then analyzed all mechanisms to assess whether they met the definitions. IEG found that for most mechanisms (70 percent), project documents mentioned how they plan to act in response to the feedback but almost none of them (4 percent) mentioned a plan to inform citizen about the actions taken in response to their feedback. With respect to the first definition, the IEG review found that compliance is higher for safeguards related (83 percent) than for non- safeguards-related mechanisms (67 percent); it is also higher for consultations (90 percent) but lower for collecting, recording and reporting on inputs received from beneficiaries (54 percent). Inclusion Half of the projects with at least one citizen engagement mechanism (N=276) include at least some discussion in their project appraisal documents of how the voice or participation of women or marginalized groups will be ensured. Box B1 provides examples of projects that discuss how they will address inclusion of the most vulnerable groups as part of the description of the citizen engagement mechanism. The percentage of projects including this discussion increased between the two periods under study from 40 percent to 58 percent,7 with a clear increase to 72 percent in FY16, thus reflecting an improvement in the design of citizen engagement mechanisms or, at least, an increased awareness of the importance of tracking the extent of engagement with diverse and vulnerable groups (table B.7). 90 Engaging Citizens for Better Development Results | Appendix B TABLE B.7.  rojects with at Least One Citizen Engagement Mechanism P Discussing Inclusion Projects with at Least One CE  Projects with at Least One CE Projects with at Least One Mechanism with an Inclusion Mechanism with an Inclusion Period CE Mechanism (no.) Discussion (no.) Discussion (percent) FY11–13 135 54 40 FY14–16 141 82 58 FY11–16 276 136 49 Source: Independent Evaluation Group structured desk review. Note: CE = citizen engagement. Implementation  nclusion BOX B.1. I Discussion in Citizen Engagement Mechanisms n The Republic of Yemen Labor Intensive Public Works Project (P122594) included community involvement in subprojects selection, preparation, operation and maintenance. The project documents note that subproject identification would entail standardized, gender disaggregated consultations to ensure that women’s preferences for new subprojects are reflected in selection outcomes. n The Guinea-Bissau: Emergency Electricity and Water Rehabilitation Project (P120910) triggered OP 4.01 Category B and required mandatory consultations. The project appraisal document notes that consultations with key stakeholders involving the private sector, the public sector, the civil society at large, and the most vulnerable groups (for example, female-headed households, the elderly, disabled and youth) had been conducted. n The project appraisal documents for the Georgia Third Regional Development Project (P150696) notes that consultations with citizens were an input to overall project design and they would have continued throughout project implementation for the selection and design of subprojects. The document notes that consultation would have incorporated the points of view of men, women, and vulnerable groups, including ethnic minorities, internally displaced persons, and disabled individuals. Moreover, it also notes that if ethnic minorities had been present in a subproject location, communications and consultation efforts would have considered the languages spoken by these individuals. Source: Independent Evaluation Group structured desk review. Independent Evaluation Group | World Bank Group 91 To assess progress in implementation of the citizen engagement mechanisms specified at appraisal, IEG reviewed all the closed projects included in the sample. Overall, the sample comprised 40 closed projects out of which 31 included at least one citizen engagement mechanism, as per project appraisal documents. In total, these 31 projects included 85 citizen engagement mechanisms. The review found that 90 percent of the projects (28 out of 31 projects) presented at least some evidence on implementation, that is for at least one of the citizen engagement mechanisms included. However, the picture changes when the analysis is conducted using mechanisms as the unit of analysis. In this case, the review found evidence of implementation for only 62 percent of the 85 mechanisms included in the 31 projects with at least one citizen engagement mechanism at appraisal. Implementation rates differ based on the type of citizen engagement mechanisms. Mechanisms that engage citizen to collaborate in the planning or execution of a policy had the highest implementation rate – 87.5 percent of them were implemented. In contrast, grievance redress and citizen monitoring mechanisms had the lowest implementation rates (46.2 percent and 33.3 percent). Table B.8 below illustrates the reporting rates by type of citizen engagement approach. TABLE B.8.  itizen Engagement Mechanisms with Evidence of Implementation C by Type of Citizen Engagement Approach Type of Citizen Mechanisms Mechanisms with Evidence  Mechanisms with Evidence Engagement Approach (no.) of Implementation (no.) of Implementation (percent) Citizen/community 8 7 87.5  collaboration Capacity building for 5 4 80.0  engagement Consultation 33 24 72.7  Empowering citizens/ 3 2 66.7  communities with resources and decision-making powers Collecting, recording and 17 8 47.1  reporting on inputs received from beneficiaries Grievance redress 13 6 46.2  Citizen monitoring, 6 2 33.3  evaluation and oversight Total 85 53 62.4  Source: Independent Evaluation Group structured desk review. Note: N = 40 closed projects. 92 Engaging Citizens for Better Development Results | Appendix B Results: Citizen Engagement Indicators Time Trend The citizen engagement strategic framework and associated targets have led to an increase in the number of projects with at least one citizen engagement indicator in their results framework, from 42 percent in FY11–13 to 63 percent in FY14–16 (table B.9),8 (with a steady increase up to 95 percent in FY16). This increase indicates that operational teams responded well to the corporate mandate. This review also assessed the extent to which projects include at least one citizen engagement indicator with an inclusion dimension in its formulation and found that, overall, only 20 percent of the sampled projects have this trait which increases over time from a 14 percent in FY11–13 to 26 percent in FY14–16 (48 percent in FY16). Measurement There is discrepancy between what remain the most common citizen engagement mechanisms in project appraisal documents and the citizen engagement indicators included in projects’ results frameworks. Table B.10 illustrates this paradox. For instance, consultations and grievance redress constitute the most prevalent citizen engagement mechanisms within the pool of project with at least one citizen engagement mechanism (N = 276) since 86 percent and 55 percent of these projects include at least one of these mechanisms in their project appraisal documents. However, only 7 percent and 10 percent of these projects include at least one indicator to report on the implementation and outcomes of the consultations and GRMs. In contrast, other not so prevalent mechanisms are included in results frameworks at a higher rate. For instance, 33 percent of the projects within this pool include in their project appraisal documents a description of a citizen engagement mechanisms that are about collecting, recording and reporting on inputs received from citizens. Almost one-fifth of these projects (18 percent), include at least one indicator to track the implementation or report on the results of these mechanisms. TABLE B.9. Projects with at Least One Citizen Engagement Indicator Projects  Projects with at Least One  Projects with at Least One CE Indicator with an Period (no.) CE Indicator (percent) Inclusion Dimension (percent) FY11–13 147 42  14  FY14–16 152 63  26  FY11–16 299 53  20  Source: Independent Evaluation Group structured desk review. Independent Evaluation Group | World Bank Group 93  rojects TABLE B.10. P Including Citizen Engagement Mechanism and an Indicator to Track the Mechanism (percent) Projects with at Least One CE  Projects with at Least One CE Mechanism Mechanism Indicator to Track Mechanism Consultation 86  7  Collecting, recording and reporting on inputs 33  18  received from beneficiaries Grievance redress 55  10  Citizen/community collaboration on planning and/or 24  8  execution of a policy, program or project Citizen monitoring, evaluation and oversight 18  1  Empowering citizens/communities with resources 9  2  and decision-making powers Capacity building for engagement 24  4  Source: Independent Evaluation Group structured desk review. Note: N = 276. CE = citizen engagement. Quality Consultation IEG identified 26 indicators tracking consultations in the analyzed sample and conducted a qualitative analysis that included developing a typology along two criteria.9 The first analytical criterion distinguishes whether an indicator is output oriented versus outcome oriented. Output-oriented indicators only track participation without any reference to the outcome of that participation. In contrast, outcome-oriented indicators include a reference to the outcome of citizen participation in their formulation. The second analytical criterion identifies whether an indicator included a reference to the inclusion of women or marginalized groups in their formulation. Overall, the indicators the World Bank used for tracking consultations are not meeting best practice principles. The IEG analysis found that 73 percent of the 26 indicators are output oriented and, within this pool, 63 percent are formulated in a way that captures whether consultations included women or other vulnerable groups. In contrast, only 27 percent of the indicators were outcome oriented. These outcome indicators tracked inclusion in 43 percent of the occasions. Box B.2 provides examples from each of these categories extracted from the sample under analysis. Beneficiary Feedback This review found 128 indicators tracking beneficiary feedback in the sample under study and classified them per three general types.10 First, this review identified output-oriented indicators 94 Engaging Citizens for Better Development Results | Appendix B Typology BOX B.2.  of Consultation Indicators Formulation with an Inclusion Dimension Yes No Output oriented Percentage of affected households Local stakeholder workshops having woman attended consultation held in each beneficiary country activities on compensation/resettle- (number) (P117170). ment (P123961) Number of people participating in project consultation (number) (P120234) Outcome oriented Project-supported organization(s) Changes to project activities publish reports on inputs and effect because of consultations (Yes/No) of consultation on project includ- (P148129) ing gender disaggregated impact (P147499). Participants in consultation activi- ties during project implementation (female) and Proportion of funds directed to priority sectors increased based on citizen consultations in Balochistan (P126425) Source: Independent Evaluation Group structured desk review. that simply measure the implementation of feedback mechanisms or participation in feedback mechanisms. This represent 9 percent of the indicators. Second, this review identified indicators that either: (i) collect socioeconomic data / behavior data; or (ii) measure satisfaction with quality of services or capture beneficiaries’ views. These indicators represent 87 percent (or 110) of the 128. Within this pool, the majority measures satisfaction with quality of services (91 percent) and only a minority collect socioeconomic / behavior data (9 percent). Overall, 28 percent of these indicators have an inclusion dimension in their formulation (31 out of 110). Finally, this review found a minority (5 out of 127) of indicators that measure feedback use or changes in project activities because of citizen feedback. Only one of these 5 indicators included an inclusion dimension in its formulation (box B.3). Grievance Redress IEG identified 44 indicators tracking GRMs in the portfolio sample under study and classified these indicators following two analytical categories.11 The first analytical category distinguishes whether the indicator is outcome or output oriented. Outcome-oriented indicators report on the number of grievances received and the numbers of grievances redressed or addressed. This is commonly referred to as the “fix rate.” In contrast, output-oriented indicators do not refer to the “fix rate.” The Independent Evaluation Group | World Bank Group 95 Typology  BOX B.3.  of Indicators on Collecting, Recording, and Reporting on Inputs Received from Beneficiaries Output oriented n Implementation of Feedback Mechanisms: Customer surveys completed (P128950); Number of citizen scorecard exercises/sessions conducted in the project districts (P126130). n Participation on Feedback Mechanisms: Percentage of customers providing feedback on the use of prepaid meters installed under the project (P144573). n Satisfaction with quality of services / Capturing beneficiaries’ views n Citizen perception of quality of public services (percent of satisfied users) (P121231)—No inclusion dimension. n Proportion of line 3 users that rate the service as satisfactory or better disaggregated by gender and bottom 40 percent income group (P128919)—With inclusion dimension. n Percentage of beneficiaries reporting improvements in project results and processes (P151416)—No inclusion dimension. Feedback use or changes in project activities because of citizen feedback n Number of primary health care facilities that developed and implemented action plans because of citizen feedback (P126130)—No inclusion dimension. n Social Accountability Implementing Partners woredas that have developed joint action plans for service quality improvement based on interface meetings between service users and providers (P128891)—No inclusion dimension. n Number of beneficiary feedbacks reviewed and addressed (disaggregated by gender) (P147521)—With inclusion dimension. n Strategic Environmental Impact Assessment prepared based on the feedback from all stakeholders (P147348)—No inclusion dimension. Source: Independent Evaluation Group structured desk review. 96 Engaging Citizens for Better Development Results | Appendix B second analytical distinction distinguishes whether indicators refer to any service standards such as resolving grievances within a time frame. Box B.4 below present examples of indicators classified per these two analytical distinctions. The IEG review found that two-thirds (66 percent) of indicators have a formulation that is outcome oriented and that only 14 percent included a reference to service standards in their formulation. Likewise, inclusion is rarely included in the formulation of GRMs. This review only found one indicator capturing this dimension in its formulation.12 Citizen / Community Collaboration IEG identified 40 indicators tracking citizen collaboration mechanisms in the sample under study.13 These indicators were classified along two analytical categories. The first analytical category distinguishes whether indicators are output or outcome oriented. Output-oriented indicators either track citizen / community collaboration in planning or implementation or its participation in decision- making bodies. These types of indicators do not refer or report on the effect of collaboration or participation on project / program / policies. In contrast, outcome-oriented indicators either report on the citizen / community satisfaction with the collaboration process or on the effects of collaboration on project / program / policies. The second analytical category distinguishes whether citizen / community collaboration included the participation of women and other disadvantaged groups. Using the above categories for guiding the analysis, this review found that almost all the citizen / community collaboration indicators are output oriented (38 out of the 40). Out of the two outcome- oriented indicators identified, both track citizen / community satisfaction with the collaboration process. The review did not find outcome indicators tracking the effects of collaboration on project / program / policies such as “project-supported organization(s) publish reports on effect of Typology BOX B.4.  of of Indicators on Grievance Redress Monitoring and Reporting on Service Standards for GRM Yes No Output oriented n  rievances Redress System oper- G n ocial Transfer Program griev- S ational with registry of complaints ance mechanism implemented and recording of response times (Yes / No) (P150430). (Yes / No) (P145347). n ercentage of upazilas where P grievance redress system is functioning (P118701). Outcome oriented n roportion of cash transfer P n  rievances registered related G component related to grievances to delivery of project benefits resolved within three months of that are addressed (percentage) being recorded in the GRM data- (P127079). base (P143588). Source: Independent Evaluation Group structured desk review. Independent Evaluation Group | World Bank Group 97 collaboration on project/program/policies (Yes/No).”14 Finally, IEG also found that only 27 percent of the indicators (11 out of 40) refer to inclusion in their formulation. Box B.5 provides examples extracted from the pool of indicators reviewed classified using the previously proposed analytical distinctions. Typology  BOX B.5.  of Indicators Citizen or Community Collaboration Output Oriented—Collaboration in Planning n Participatory Strategic Plans completed for target River Basins (P118540) n Percentage poor community members who participate in planning, decision making (P128832) n Percent of participants in planning and decision-making meetings who are women (P128832) Output Oriented—Collaboration in Implementation n Number of Joint Patrols for Monitoring, Control, and Surveillance between Navy, District and community-based groups (P127813) n At least 75 percent of targeted villages or user groups completed and successfully implementing micro-plans with conservation benefits (P088520) n Citizens and/or communities involved in planning/implementation/evaluation of integrated ecosystem management plans (Yes/No) (P151102) n Men and women engaged in subprojects supporting sustainable harvesting and marketing (P130474) Output Oriented—Establishment of decision-making bodies for collaboration n Number of operational water user associations (P126440) n Number of operational water user associations created and/or strengthened (P144497) n Number of Farmers organized into Farmers Interest Groups (FIGs) (P131235) Outcome Oriented—Satisfaction n Percentage of beneficiaries satisfied with community involvement in design and implementation of the irrigation rehabilitation works (P122235) n Percentage of beneficiaries that perceive their role has increased in decision making (disaggregated by gender) (P146970) Source: Independent Evaluation Group structured desk review. 98 Engaging Citizens for Better Development Results | Appendix B Citizen Monitoring Overall, this review found eight indicators reporting on citizen monitoring, evaluation and oversight. Three out of these eight indicators monitor whether citizen monitoring bodies have been established whereas four-track participation in social monitoring exercises. Only one indicators monitor whether citizen monitoring reports have been published. Finally, three out of the eight indicators refer to inclusion in their formulation. Box B.6 provides examples of these indicators extracted from the sampled projects. Empowering Citizens or Communities with Resources and Decision-Making Powers This review had a 19 indicators reporting on empowering citizens / communities with resources and decision-making power in its sample15. Most of the indicators (63 percent) track community engagement in project planning, implementation, and postproject operations and maintenance (12 out of 19) whereas 21 percent report on citizen participation in community-based decision making and management structures (4 out of 19). Only a minority of indicators report on the establishment of community-based decision making and management structures (2 out of 19) and on funds / resources delegated for management by citizen groups or households. Finally, this review also found that 21 percent of indicators include an inclusion dimension in their formulation (4 out of 19) and, that these indicators, usually fall under the category of “participation in community-based decision Typology BOX B.6.  of of Indicators on Grievance Redress Inclusion Yes No Establishment of citizen n ystem of detailed performance S n ercent of villages with a P monitoring bodies evaluation with citizen engagement functional community oversight (both gender) in plan (number of team (P128832). sites) (P148129). Tracking participation n ercentage of members of the P n  umber of Participating Dis- N in social monitoring community oversight team who tricts Using Beneficiary Monitor- exercises are women (P128832). ing Mechanisms (P132750). Publishing citizen moni- n  ot applicable. N n nnual Expert Panel environ- A toring reports mental and social monitoring report disseminated and made publicly available—annual expert panel reports/RPIU progress reports (P075941). Source: Independent Evaluation Group structured desk review. Independent Evaluation Group | World Bank Group 99 making and management structures.” Box B.7 provides illustrations of each of the identified type of indicators within this approach. Capacity Building for Engagement This review identified 16 indicators that report on this approach to citizen engagement in its sample16. Within this pool, two types of indicators can be distinguished. Half of the indicators report on support delivered for the institutionalization of channels for citizen participation whereas the other half reports on training or skills development for citizen engagement. Within this second half, the review found four indicators with an inclusion dimension in their formulation whereas none were find under the first half. Box B.8 provides illustrative examples for each type. Typology  BOX B.7.  of Empowering Citizens or Communities with Resources and Decision-Making Powers Indicators Participation in Community-Based Decision Making and Management Structures n Percent of participating households in the Gram Sabha meetings; representatives; and (ii) percent of which are female (P131235). n Representatives in community-based decision making and management structures that are from the vulnerable or marginalized beneficiary population (Percentage) (P145196). n Community Engagement in project planning / implementation / postproject operations and maintenance n Percentage of Community subprojects operating one year after completion (P127200). n Subprojects with postproject community engagement or operation and maintenance arrangements (percentage) (P151077). n Establishment of Community-Based Decision Making and Management Structures n Local Community Associations (LCAs) established: LCAs trained and organized under the project (P126498). n Community-based organizations formed or reinforced and still operational one year after receiving funding (number) (P152822). n Funds/resources delegated for management by citizen groups or households n Number of communities receiving grants (P124134). Source: Independent Evaluation Group structured desk review. 100 Engaging Citizens for Better Development Results | Appendix B Typology  BOX B.8.  of Indicators on Capacity Building for Citizen Engagement Institutionalization of channels for citizen participation n Community Information and Recourse Centers established at the regional and district level in the Western regional (P120005). n Establishment of pilot beneficiary feedback mechanisms to provide feedback on municipal services (P147521). n Trainings / skills for engagement n Government and CSOs staff trained in new accountability, citizens engagement mechanisms, and PFM (20 percent of whom are women) (number) (P155121). n Community-based natural resources management institutions provided with access to improved management practices for sustainable landscape management (Number) (P148183). Source: Independent Evaluation Group structured desk review. Reporting IEG assessed the level of reporting on indicators for all FY14 projects in the sample and found that the level of reporting is 58 percent. To make this assessment, IEG used the planned reporting timeline that is indicated in project documents, more precisely in the results framework, as the benchmark for judging whether a project reported on an indicator. Using this criterion, four scenarios can be distinguished. First, indicators for which data are reported as planned. Second, indicators for which data are being collected. These are indicators for which no reporting has yet happened but for which there is no default with the schedule. Third, indicators for which data are not being reported. These are indicators for which reporting is in default with the proposed reporting schedule at appraisal. Finally, indicators that have been dropped during implementation. 1  The team relied mainly on the citizen engagement strategic framework and Citizen Engagement Secretariat and Operations Policy and Country Services guidance for developing the coding protocol. 2  Given its different nature, the Independent Evaluation Group reviewed Development Policy Lending in separate desk review using a different protocol. 3 http://www.raosoft.com/samplesize.html 4  Given these samples sizes for each period, the team will be able to make inferences with a 95 percent confidence and 7.1 percent margin of error for FY11–13 and a 7.04 percent margin of error for FY14–16. However, estimations on population proportions will not be representative at the level of global practices, regions and citizen engagement mechanisms. Independent Evaluation Group | World Bank Group 101 5  Statistically significant difference per two-sample z-test to compare sample proportion (one tailed test, z-value 1.7; P-value 0.0423). 6  Forum on Citizen Engagement / Operationalization, https://raddrupalprod.worldbank.org/citizen_engage/ operationalize. 7  Statistically significant difference per two-sample z-test to compare sample proportion (one tailed test, z-value −2.9; P-value 0.00139). 8  The increase is statistically significant per two-sample z-test to compare sample proportion (one tailed test, z-value −3.1; P-value 0.0009). 9 The Independent Evaluation Group identified 25 projects with this type of indicator. 10 The Independent Evaluation Group identified 87 projects having this indicator. 11 The Independent Evaluation Group identified 39 projects with this type of indicator. 12  Percent of registered project related grievances (disaggregated by gender) responded to within stipulated service standards for response times (P146199). 13 The Independent Evaluation Group identified 34 projects with this type of indicator. 14  Operations Policy and Country Services guidance note on indicative citizen engagement indicators that can be adapted for inclusion in investment project financing results frameworks. 15 The Independent Evaluation Group identified 17 projects with this type of indicator. 16 The Independent Evaluation Group identified 13 projects with this type of indicator. references World Bank. 2014. OPCS (Operations Policy and Country Services) Investment Project Financing Project Preparation Guidance Note. Washington, DC: World Bank. World Bank Group. 2014. Strategic Framework For Mainstreaming Citizen Engagement in World Bank Group Operations: Engaging with Citizens for Improved Results. Washington, DC: World Bank Group. http://documents.worldbank.org/curated/en/266371468124780089/Strategic-frame- work-for-mainstreaming-citizen-engagement-in-World-Bank-Group-operations-engag- ing-with-citizens-for-improved-results. 102 Engaging Citizens for Better Development Results | Appendix B  itizen Engagement in World Bank’s Appendix C. C Development Policy Lending Background In the citizen engagement strategic framework, development policy loans (DPLs) are presented as one of the several entry points for the World Bank Group to promote citizen engagement by country clients, notably through the two main channels. First, under the operational policy that regulates the design of DPLs (Operations Policy [OP] 8.60) several requirements for consultations to take place in the process of DPL preparation are in place, notably with the objective of mitigating the risks of negative social or environmental effects. Under this policy, the World Bank advises its clients to conduct stakeholder consultations and specifies that “key stakeholders include social groups directly affected by the operations, as well as public sector, private sector, and donor organizations relevant to the operation.” Although the responsibility to initiate this process rests with the government, the World Bank has a role to play as a facilitator, and adviser of the government and the OP specifies that the World Bank’s program document should describe the consultative arrangements that are relevant to the operation. Box C.1 summarizes the dedicated paragraph of the OP. Second, one of the five key principles for citizen engagement in Bank Group work outlined in the citizen engagement strategic framework is “strengthening country systems for citizen engagement within the scope of World Bank Group operations” (see box C.2 with specific text from the strategy). DPL operations can be used to facilitate the adoption of reforms, including nationwide legislation that build the foundations for enhanced citizen engagement. Through the negotiations and implementation of prior actions and triggers, the World Bank can leverage its influence to create an enabling environment for citizen engagement and participatory approaches by government. Consultations  BOX C.1.  and Participation Rules under Operational Policy 8.60 In carrying out dialogue with a Member Country, the World Bank advises it to consult with and engage the participation of key stakeholders in the country in the process of formulating its development strategies. For a development policy operation, the Member Country draws on this process of strategy formulation to determine, in the context of its constitutional and legislative framework, the form and extent of consultations and participation in preparing, implementing, and monitoring and evaluating the operation. The World Bank’s program document (“Program Document” or “PD”) describes the Member Country’s arrangements for consultations and participation relevant to the operation, and the outcomes of the participatory process adopted in formulating the Member Country’s development strategy. Relevant analytic work conducted by the World Bank, particularly on poverty and social impacts and on environmental aspects, is made available to the public as part of the consultation process, in accordance with the World Bank’s Policy on Access to Information (OP 8.60, 3). Independent Evaluation Group | World Bank Group 103 Development  BOX C.2.  Policy Lending as an Entry Point to Strengthen Country Systems for Citizen Engagement “Strengthening country systems for citizen engagement within the scope of Bank Group operations. To facilitate sustainable development outcomes, including those of engagement processes, Bank Group–supported operations aim to support and strengthen government systems for engaging with citizens. The scope of such support needs to be agreed with client governments, and it varies by type of operation. For example, a development policy lending operation can facilitate the adoption of national legislation on participatory budgeting or procurement monitoring, whereas an investment project financing operation can contribute to building effective feedback and recourse mechanisms to improve service delivery in specific sectors or empower citizens at the local level to participate in the planning, implementation, and monitoring of development interventions.” Source: World Bank 2014, 19. In this review, we seek to analyze to what extent and how the World Bank has leveraged DPL as an entry point for fostering an environment for citizen engagement in its client countries. The following specific questions guided this study: ƒƒ To what extent has the World Bank supported reforms that create an enabling environ- ment for citizen engagement and participation through DPL prior actions and trigger? ƒƒ How effective has the World Bank been in promoting a dialogue between the govern- ment and a wide range of stakeholders to support the reform process funded through DPL? ƒƒ What are the challenges in using DPL as an entry point for citizen engagement in World Bank Operations and how can these challenges be overcome? The evidence base for this review stems from multiple sources, as laid out below: ƒƒ A synthesis of information contained in the 2015 DPL retrospective and prior Indepen- dent Evaluation Group (IEG) work on managing social and environmental risks in DPL (World Bank 2015; World Bank Group 2015) ƒƒ An analysis of the prior actions database between 1990 and 2017 to identify broad trends about the prevalence of prior actions that promote reforms of countries’ system and processes for citizen engagement 1 ƒƒ A structured review of a random sample of DPL program documents (including program documents, legal agreements, Implementation Completion and Results Reports, and 104 Engaging Citizens for Better Development Results | Appendix C Implementation Completion and Results Report Reviews as available) to assess the extent of reporting on stakeholder consultations2 Consultations and Participation in Reforms Supported by DPL Discussion of Consultations on General Reforms As laid out in the citizen engagement strategy, in the framework of reforms supported by DPL, the World Bank advises governments to hold consultations, and enable the participation of a large array of key stakeholder groups—including social groups directly affected by a reform—with the following objectives: ƒƒ Informing the design of the reform program ƒƒ Improving implementation effectiveness ƒƒ Improving the monitoring and evaluation of the reform The review of DPL program documents conducted for this evaluation sought to review the evidence on the consultation process and its outcome through looking for information about the following five simple dimensions: 1. Whether the program documents include a description of the consultation process; 2. Whether the program documents specify which stakeholders were consulted; 3. Whether the program documents discussed the opinion of stakeholders consulted about the reform; 4. Whether the program documents described the outcome of the consultations; 5. Whether the program documents describe feedback mechanisms at closing. The review found that almost all DPL in the sample (57 out of 60) have included some reporting on the consultation and participation of stakeholders that took place in the broadly defined context of the reforms supported by the DPL. However, the degree of precision with which consultations are described is very limited. Forty percent of the sample of DPL reviewed did not specify which stakeholders were consulted, and among the rest, only a small subset specified who were consulted beyond very generic categories such as “academia, civil society organizations, and private sectors.” The positions or views of the stakeholders on the discussed reforms were mentioned in less than one-third of the program documents Most consultations described were conducted in the framework of the discussion of a country’s national development plan or poverty reduction strategy. The number of DPLs that evoke Independent Evaluation Group | World Bank Group 105 consultations held to inform the specific reforms or sector strategies supported by the DPLs are much rarer, whether the DPL were part of a PRSCs series or not. Most importantly, only one-quarter of program documents reviewed include some discussion of the outcome of the consultations, which is less than what the Bank Group 2015 DPL retrospective found.3 In the retrospective, the following was found regarding consultation and participation: “[virtually all development policy objectives] reviewed in this retrospective discussed the country’s consultative and participatory process used in the formulation of the operation. However, less than half of the program documents described the outcomes of the process, making it difficult to assess to what extent the feedback that was gathered had an impact on the policy design” (World Bank Group 2015, 46). Feedback mechanisms at closing are hardly ever reported. Only 4 of the 60 DPLs reviewed mentioned having some activities to engage stakeholders at the end of the operation, including seminars or forums to discuss the progress of the reforms.  pecificity FIGURE C.1. S of Description of Consultations in the Framework of Development Policy Loans, FY11–16 Reporting on consultations Type of stakeholders consulted Stakeholders’ opinion No reporting, 3 Some reporting, 17 No reporting, Some reporting, 25 35 Some reporting, 57 No reporting, 43 Outcome of consultations Feedback at closing Some reporting, 4 Some reporting, 18 No reporting, 42 No reporting, 56 106 Engaging Citizens for Better Development Results | Appendix C Specific Consultations on Environmental and Social Issues Program documents tend to describe in generic terms consultations that took place in the framework of broad development plans. However, they rarely describe explicit consultations that occurred to discuss specific social or environmental risks. The IEG Learning Products on managing environmental and social risks in DPLs found that the implementation of consultation as part of the due diligence requirements for DPL was very low for prior actions for which there was a potential environmental risk. As shown in figure C.2, only 11 percent of the prior actions that the World Bank identified as bearing a potential environmental risk reported having a specific consultation on the matter. With regards to social risk, 70 percent of all prior actions that the World Bank identified as presenting social risks were discussed in consultations (the percentage is lower if we consider the prior actions that IEG considered as bearing social risks, 39 percent). The graphs below illustrate this point. We know very little about specific consultations on environmental or social risks and even less on their outcome, which are almost entirely absent from documents. Yet, there is a clear demand from nongovernmental actors to be more involved and consulted in the framework of development policy financing, both at a more general level as well as on specific risks, as attested through the  vidence FIGURE C.2. E of Consultation with Nongovernment Stakeholders on Negative Environmental and Social Risks Associated with Specific Prior Actions, 2005–14 70% Identified by Bank 11% 39% Identified by IEG 6% 0 10 20 30 40 50 60 70 80 Percent Consultations for PA with social risks Consultations for PA with environmental risks Source: World Bank 2015. Note: The sample reviewed included a randomly selected sample of 70 development policy objectives approved between 2005–14 covering 692 prior actions. The percentage reported are in terms of number of prior actions. For example, 70 percent of all prior actions that the World Bank identified as bearing a potential social risk were discussed in consultations with nongovernmental stakeholders. Independent Evaluation Group | World Bank Group 107 from the Feedback on Consultation Gathered for the Insight BOX C.3.  Development Policy Financing Retrospective 2015 n There was appreciation expressed for the role that various stakeholders have played in the design of development policy financing (DPF); at the same time, there were requests for more effective overall coordination and upstream engagement of citizen groups in consultations at the design stage. n The importance of ensuring transparency and accountability around DPF was stressed, specifically on assessing governance risks, developing strong monitoring and evaluation systems and engaging communities and civil society to support accountability. n There were calls for more information on DPF to be made more available to citizens throughout the reform cycle, which can be done by collaborating with academics and other institutions. n Suggestions were made for the provision of more resources to member countries for building the capacity of civil society, parliament, and other important stakeholders. Source: World Bank Group 2015. consultation held for the 2015 development policy financing retrospective, a summary of which is presented in box C.3. Typology of Consultation Outcomes In 17 of the 60 DPL reviewed there were some indications of the outcome of the consultations and participation of civil society in the reform process, albeit rather allusive. From this rather limited number of examples, it is nonetheless possible to sketch a typology of the results achieved through consultations, which we illustrate with examples in table C.1. Prior Actions as an Entry Point for Institutionalizing Citizen Engagement? DPLs tend to support reforms in public sector governance and in the development of financial market. The breakdown of prior actions developed for the 2015 retrospective and reproduced in figure C.3 provides a good overview of the types of reforms promoted through budget support operations (World Bank Group 2015). DPLs may promote reforms that directly or indirectly foster an enabling environment for a constructive dialogue between the state and citizens. In the next section we build on the conceptual framework 108 Engaging Citizens for Better Development Results | Appendix C TABLE C.1. Emerging Typology of Results Achieved through Consultations Type of Results Example of Consultations and Associated Results Mitigating (Mongolia, P117421) The operation was accompanied by intensive dialogue, consensus political risks building, and information dissemination to garner support for complex and politically sensitive reforms, with intensive outreach to Mongolia’s powerful parliament conducted directly and indirectly through the media. There were intensive efforts at consensus building and dialogue between key stakeholders across government and information dissemination, ultimately leading to support for EITI compliance reforms, making Mongolia one of the first countries to fully comply with EITI requirements. Garnering (Armenia, P116451) Outreach efforts can be effective in overcoming opposition to crucial popular support reforms. The government used it effectively while trying to get the National Assembly to for reforms adopt the law on ‘conflict of interest.’ These outreach efforts were targeted at beneficiaries (poor and middle class), parliamentary commissions and CSOs and finally led to the adoption of the law. Refining a reform (Morocco, P115659) The government’s reform of the urban transport sector has included numerous consultations with stakeholders including the government of large cities, researchers, transport operators and CSOs. Through multiple conferences, a wide diversity of stakeholders was consulted on broad priorities. More targeted consultations were also held on specific aspects of the reform, including issues and solutions to improve the accessibility of persons with limited mobility, which were not anticipated prior to the participatory process. Permitting (Paraguay, P117043) The reform program of State-Owned Enterprises Oversight was the effective outcome of extensive consultations with stakeholders, including members of congress, the implementation media. In the process of implementation continuous consultation took place including with of a reform civil society organization to focus on the social monitoring of SOEs service delivery indicator which reinforced a demand-side request for sustaining the reform process. Empowering (Uganda, P097325) The World Bank has worked with the GOU to set up and build the institutions capacity of institutions to fight corruption cases, through a Data Tracking Mechanisms fighting against aimed at tracking corruption trends and provide evidence-based data to promote corruption anticorruption reform and deepen the public dialogue about corruption with various stakeholders. The mechanism was set up through a consultative process and recent developments have confirmed that the parliament and citizens ‟voices through the media are stronger with evidence-based data, leading to the stepping aside of three ministers and the resignation of one minister, and paving the way for further investigation into high corruption cases. (Unintended) (Vietnam, P111183) The process leading to the government-led Socioeconomic Slowing down Development Plan included serious efforts to gather feedback from different groups of the reforms Vietnamese society. A series of participatory research exercise involved local experts and nongovernmental organizations and helped gather feedback from poor communities in 17 sites across the country. CSOs also participated actively in the analytical work underlying the series of DPLs. A well-functioning mechanism was also put in place to consult with business associations. Extensive consultation has led to a process that builds popular support for policy decisions that otherwise might have been difficult to obtain. However, this process also played a role in slowing down the decision-making process on reforms and legislation. Source: Program documents of specific interventions. Note: CSO = citizen society organization; DPL = development policy loan; EITI = Extractive Industries Transparency Initiative. Independent Evaluation Group | World Bank Group 109  verall FIGURE C.3. O Thematic Breakdown of Prior Actions, Q4 FY12–Q2 FY15 Environmental and Economic Natural Resources Management Management 4% 6% Rural Development 4% Social Protection and Risk Management 11% Public Sector Governance Trade and Integration 36% 3% Urban Development 1% Human Development 8% Social Development, Gender, and Inclusion 3% Rule of Law 3% Financial and Private Sector Development 21% Source: World Bank Group 2015, thematic breakdown presented in appendix 11. developed by Grandvoinnet, Aslam, and Raha (2015) to characterize the transmission channels to improve social accountability and citizen engagement that can be supported through DPL. Conceptual Framework In the framework by Grandvoinnet, Aslam, and Raha (2015), a number of factors need to come together to foster a meaningful interface between the state and the citizens. Both parties need to have some capacities to act on the dialogue, but for the dialogue to take place, citizens need to have a sufficient level of information, and some degree of civic mobilization needs to occur. In box C.4, we categorize the various transmission channels to improving the enabling environment for citizen engagement, both direct (those captured by theme 57 “participation and civic engagement”) and indirect. We mapped all of the prior actions that broadly fall within this conceptual framework to the specific transition channels to Citizen Engagement. Trend Analysis Figure C.4 shows the evolution of the number of prior actions that broadly seek to create an enabling environment for citizen engagement. Since 1990, about 2,587 prior actions fall in this broader 110 Engaging Citizens for Better Development Results | Appendix C Transmission  BOX C.4.  Channels to an Enabling Environment for Citizen Engagement Enabling environment for Citizen-State interface Enabling Citizen Action Enabling State Action Enabling Civic cohesion Citizen rights Enabling State Enabling State and mobilization accountability capacity to answer demands • Participation and • Access to law and • Accountability and • Decentralization Civic engagement justice anti-corruption • Administrative and • Social risk mitigation • Personal and • Judicial and other civil service reforms • Conflict prevention property rights dispute resolution • Gender mechanisms • Indigenous people • Social inclusion  volution FIGURE C.4. E of Enabling Environment Prior Actions over Time 300 25 250 20 Ratio of EE prior actions (%) 200 Number of prior actions 15 150 category, representing 10 percent of all of the prior actions taken in that period of time. About 10 100 of these prior actions sought reforms that would enable citizen action and 55 percent 45 percent were pursuing reforms that were more targeted toward improving state action. 5 50 0 0 19 0 19 1 19 2 19 3 19 4 95 19 6 19 7 98 20 9 00 20 1 20 2 20 3 20 4 05 20 6 20 7 20 8 20 9 20 0 11 20 2 20 3 20 4 20 5 16 9 9 9 9 9 9 9 9 0 0 0 0 0 0 0 0 1 1 1 1 1 19 19 19 20 20 20 Enabling environment prior actions Ratio of enabling environment PA Independent Evaluation Group | World Bank Group 111 FIGURE C.5. Distribution of Types of Prior Actions Over Time 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 19 19 19 19 19 19 19 19 19 19 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 Enabling citizen action Enabling state action Note: N = 2,587 enabling environment prior action. If we take a closer look at DPLs during our evaluation period (2011–16), we identify 328 prior actions that can be broadly defined as accompanying reforms that create an enabling environment for citizen engagement under one of the transmission channels laid out in our conceptual framework. These 340 prior actions were found in 169 distinct DPLs. The vast majority of the 340 prior actions (79.4 percent) support reforms that enable state action, either by seeking to make the state more accountable or by reinforcing the state capacity to respond to demands from citizens. A Regional breakdown might be instructive (figure C.7). We find that Africa is the region where the ratio of DPL with citizen engagement prior actions out of all DPL implemented during the evaluation period is the highest (54 percent) and East Asia and Pacific is the lowest (36 percent). When considering the balance between prior actions that enable state versus citizen action, most Regions privilege reforms that would more directly enable state action (figure C.8). Latin America is an exception, with a balance of prior actions enabling state action and enabling citizen action. It is rare that DPLs combine multiple types of prior actions to strengthen holistically the enabling environment for citizen engagement. Notably, none of the DPL included a combination of prior 112 Engaging Citizens for Better Development Results | Appendix C  ype FIGURE C.6. T of Prior Actions to Enhance Enabling Environment for Citizen Engagement, 2011–16 Enabling civic cohesion and mobilization 17% Citizen rights 1% Enabling state capacity to answer demands 36% Enabling state accountability 46% Social inclusion 6 Social Risk mitigation 0 Indigenous people 2 Gender 21 Conflict prevention 4 Participation and civic engagement 22 Personal and property rights 2 Access to law and justice 1 Judical and other dispute resolution mechanism 4 Accountability and anti-corruption 148 Decentralization 8 Administrative and civil service reform 110 0 20 40 60 80 100 120 140 160 Note: N = 328. actions promoting changes both through enabling citizen and state actions at the same time (figure C.9). The most common combination of prior actions to enable state action are prior actions that seek an administrative or civil service reform and other measures to improve accountability and fight corruption. For example, in Morocco, the DPL on transparency and accountability (P154041) included prior actions that established a national public contracts committee with a mandate on oversight, Independent Evaluation Group | World Bank Group 113  atio FIGURE C.7. R of DPL with Citizen Engagement Prior Actions Out of all DPLs, by Region, 2011–16 60% 54% 50% 47% 46% 45% 40% 36% 30% 24% 20% 10% 0% AFR EAP ECA LCR MNA SAR Source: Independent Evaluation Group analysis based on prior actions database. Note: AFR = Africa; DPL = development policy loan; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; EAP = East Asia and Pacific; MENA = Middle East and North Africa; SAR = South Asia.  ype FIGURE C.8. T of Prior Actions, by Region, 2011–16 100% 80% 53.2% 60% 89.9% 81.5% 81.6% 84.6% 100% 40% 20% 46.8% 18.5% 15.4% 18.4% 10.1% 0% 0.0% AFR EAP ECA LCR MNA SAR Enabling citizen action Enabling state action Source: Independent Evaluation Group analysis based on prior actions database. Note: n = 328 prior actions. AFR = Africa; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; EAP = East Asia and Pacific; MENA = Middle East and North Africa; SAR = South Asia. 114 Engaging Citizens for Better Development Results | Appendix C  istribution FIGURE C.9. D of DPLs by Regions and Type of Support, 2011–16 Enabling citizen action Enabling state action SAR MNA LCR ECA EAP AFR 70 60 50 40 30 20 10 0 0 10 0 0 20 30 40 50 60 70 Source: Independent Evaluation Group analysis based on prior actions database. Note: n = 328 prior actions. AFR = Africa; DPL = development policy loan; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; EAP = East Asia and Pacific; MENA = Middle East and North Africa; SAR = South Asia. complaints handling and which included nonstate actors as core members. The DPL also included measures of decentralization of power to the regions. Prior Actions with a Direct Link to Enhanced Citizen Engagement The World Bank has defined theme codes to categorize prior actions by the type of reforms that they support. Code 57 captures prior actions that areare directly related to participation and civic engagement. These specific prior actions have been very rare in the history of the World Bank’s engagement, as shown in figure C.10. Since the early 1990s the number of prior actions that have directly sought to enhance citizen engagement and participation is very limited, amounting to 83 prior actions in 59 distinct DPLs since 1990. When compared with the overall number of prior actions, the rarity of this type of prior action is even more striking, never representing more than 1.8 percent of all prior actions, in the peak year of 2006. Looking more specifically at the period of reference for this evaluation (FY11–16) 20 DPLs included a prior action that directly thought to promote participation. A few examples are listed in box C.5. Outcomes Very little is known about the efficacy of DPL in promoting reforms that enable citizen engagement. The table C.2 provides a few examples of outcomes achieved through including prior actions that trigger participatory mechanisms for civil society and were self-reported in Implementation Completion and Results Reports. Independent Evaluation Group | World Bank Group 115  volution FIGURE C.10. E of “Participation and Civic Engagement” Prior Actions since 1990 12 2.0% 1.8% 10 1.6% Ratio of CE prior actions 1.4% Number of prior actions 8 1.2% 6 1.0% 0.8% 4 0.6% 0.4% 2 0.2% 0 0.0% 19 0 19 1 19 2 19 3 19 4 19 5 19 6 19 7 19 8 20 9 20 0 20 1 20 2 20 3 20 4 20 5 20 6 20 7 20 8 20 9 20 0 20 1 20 2 20 3 20 4 20 5 16 9 9 9 9 9 9 9 9 9 9 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 19 Participation and Civic Engagement prior actions Ratio of Civic Engagement PA Source: Independent Evaluation Group analysis based on prior actions database. Note: N = 83 prior actions. Examples  BOX C.5.  of Prior Actions Seeking to Directly Enhance Participation and Civic Engagement n In the “strengthening social resilience to climate change” (Mexico 2011), three prior actions sought to enhance participation of civil society, through (i) signing bilateral cooperation agreements between major cooperatives in the forest industry; (ii) the establishment of a national and three regional multistakeholder consultative REDD+ councils; n In the Governance and Opportunity development policy loan, (Tunisia 2011), the prime minister established a participatory process for systemic monitoring of the performance of public services by civil society, citizens and service providers, notably for the social sectors, with the objective; n In the Solid Waste Development Policy Operation III (Morocco 2013), The National Commission for the National Municipal Solid Waste Management Program has approved the introduction of a citizen feedback tool on quality and adequacy of municipal solid waste services (Citizens Report Cards), as a key component of the National Municipal Solid Waste Management Program to support local governments in building demand-side governance in the municipal solid waste service delivery. n In the “Social Protections System DPL” (Rwanda 2016), the government has developed and implemented a curriculum for targeted staff in Districts and Sectors to strengthen the engagement of citizens in the delivery of social protection programs. 116 Engaging Citizens for Better Development Results | Appendix C TABLE C.2.  xamples of Participatory Mechanisms Embedded in Prior Actions E and Their Associated Results Example of Participatory Mechanisms Embedded in Prior Actions, and Type of Results Associated Results Ensuring sustainability of (Mexico, P120170) The second prior action aims to increase civil society reforms participation in policy making related to forests and climate change, and the REDD+ agenda, by creating a national consultative stakeholder council, as well as state-level multistakeholder consultative technical councils. The results of the participatory nature of the reform were mixed, depending on the region. In the State of Chiapas, the model of dialogue between state agencies and social organizations stands out. Actions are planned to strengthen the capacity of rural and indigenous population respecting their culture and building on traditional knowledge for the sustainable use of natural resources. Disaster risk management is also incorporated at the community level through effective participatory mechanisms. However outside of Chiapas, no state included monitoring measures for social resilience and some plans had no participatory mechanisms. Overall, progress have occurred on increasing participation in the implementation of REDD+ both at the local and national level (ICR, p.14). Improving results achieved (Morocco, Solid Waste-Water DPL series) Linking demand-side governance, through CRCs, to financial eligibility criteria under the PNDM incentivized the private sector to strong implementation performances. This fundamental change to implementation of the policy framework achieved a three-way communications network within the waste sector (that is, citizens to local governments to private sector), which built confidence for the private sector to invest and perform. Citizens also became fully aware of each contract signed with private sector operators for both collection and landfill activities. As demand-side governance improved, it also helped LGs increase their aggregate municipal incomes and decrease the payment delays (arrears) to private operators (ICR, p. 7). Increasing efficiency (Brazil, 3rd mining Gerais DPL) Subsequently the government strategy evolved to a focus on citizen participation, called Estado em Rede (Management for Citizenship) with the aim of consolidating the previous phases, increasing efficiency and building a permanent constituency through citizen participation while ensuring the model’s sustainability, through devolution of performance management to individual secretariats. This third phase of the state’s reform program was supported by the Third Minas Gerais Partnership for Development DPL in 2012 (P121590) of $450 million as well as, technical assistance. Independent Evaluation Group | World Bank Group 117 1  The Prior Actions database compiles more than 25,000 prior actions. Trend analysis were carried out on the full population of prior actions that are classified by thematic codes, to identify broad patterns of prevalence of certain types of prior actions that are directly and indirectly linked to enabling citizen engagement 2  A total of 374 development policy loans (DPLs) were approved between FY11 and FY16 (the period of the evaluation). We reviewed a random sample of 60 DPLs, which allows us to draw conclusion about the population of DPLs with a 95 percent confidence interval and 11.5 percent margin of error. 3  he 2015 development policy financing retrospective found that “less than half of the Program Documents described T the outcomes of the process” (World Bank Group 2015, 46). references Grandvoinnet, H., G. Aslam, and S. Raha. 2015. Opening the Black Box: The Contextual Drivers of Social Accountability. Washington, DC: World Bank. World Bank. 2015. Managing Environmental and Social risks in Development Policy Financing. Indepen- dent Evaluation Group. Washington, DC: World Bank. World Bank Group. 2015. 2015 Development Policy Financing Retrospective: Results and Sustainability. Washington, DC: World Bank Group. 118 Engaging Citizens for Better Development Results | Appendix C  nalysis of Citizen Engagement Appendix D. A Activities in International Finance Corporation Investment Projects Overview This note analyses community engagement performance of investment projects from the International Finance Corporation (IFC). IFC requires clients to adhere to a set of eight Performance Standards (PSs) covering multiple environmental and social sustainability issues; community engagement activities fall mostly under PS 1—Assessment and Management of Environmental and Social Risks and Impacts. IFC PS 1 calls for information disclosure and effective engagement of local communities on matters that directly affect them; active management by clients of social and environmental risks, and grievance redress mechanisms (GRMs) for affected communities, workers, and any people affected by displacement. IFC oversees clients’ implementation of the PSs. The Independent Evaluation Group (IEG) found that IFC community engagement activities improved significantly since 2012, despite performance have been uneven across projects. Driven by the PS update in 2012, clients’ perception of community engagement improved significantly both in the type of engagement (that is, moving from corporate social responsibility [CSR] to project-oriented activities) as well as in the quality (that is, moving from more reactive activities such as a grievance mechanism to being actively engaged in the community with consultations and focal point).1 Clients have been generally responsive, and the share of projects engaging communities repeatedly during their life has increased. Despite these improvements, the extent of stakeholder engagement varied greatly across projects (and not necessarily because of different levels of project risk). Interviews with IFC staff highlighted that client ownership and IFC early access to the project are important elements to a successful engagement. They also suggested that, at times, client performance on citizen engagement drops post the construction phase and is linked to economic performance. In one case, the environmental and social (E&S) specialist suggested that sound citizen engagement can help small and medium projects to improve their performance. Methods IEG analyzed community engagement performance in 137 IEG-evaluated IFC investment projects approved between FY07 and FY11, using IEG evaluations of IFC self-project evaluations (Expanded Project Supervision Report Evaluation Notes) as main source of information. This sample of projects allowed drawing conclusions about the population of IFC investment projects of 930 with a 95 percent confidence interval and 7.8 percent margin of error. IEG also included newer, post 2012 projects, to capture any difference in design after the change of PSs on Environmental and Social Sustainability in 2012. For this set of projects, IEG randomly selected a small sample (30) of real sector projects in FY15–17. This sample of projects allowed drawing conclusions about the population of IFC investment projects of 481 with a 95 percent confidence interval and 17.4 percent margin of error. Further, IEG used a set of key IFC categories for community engagement as a Independent Evaluation Group | World Bank Group 119 starting point in the coding phase of the portfolio, and complemented it by defining four further categories to capture extra details in how projects designed their community engagement (table D.1). Boxes D.1 and D.2 provide an example for these newly created categories. Sixteen interviews were conducted by the IEG E&S specialist of projects approved between FY07 and FY11 and analyzed to provide more qualitative insights. These projects were subject to 2006 PSs. The 2012 update on policy and PSs on E&S sustainability significantly raised engagement standards.2 PSs were significantly updated in FY12, and this policy update is reflected in improved design of project engagement activities.3 Although the client is still required to establish and maintain an E&S Management System, which includes community engagement, objectives and implementation have changed.4 The formulation of objectives in PS 1 was significantly improved, the emphasis moving from reacting with avoidance/minimization actions to adopting mitigation actions, grievance response and management mechanisms, and providing means for adequate engagement. The updated policy formulation is clearer, includes more details on each action and when each citizen engagement action needs to be triggered, as well as what are the requirements for the client in cases when citizen engagement responsibilities fall on the government. Higher attention to PSs and citizen engagement TABLE D.1. Citizen Engagement Categories Category Description Stakeholder consultation IFC client consulted with stakeholders over time sharing with them information about the projects and receiving their concerns. Grievance management IFC client established a grievance mechanism that captures and responds to stakeholders’ concerns related to the project’s operations. Corporate social This variable identifies activities as those activities unrelated to the project responsibility scope or objective, conducted with the objective to improve community relations or brand perception. Disclosure of information IFC client reported back to project-affected groups and other stakeholders on the outcome of prior consultations / reached a wider, multistakeholder audience through sustainability reporting. Stakeholder mapping IFC client identified stakeholders and assessed their interests and concerns. Interviews This variable captures those projects where the IFC client interviewed members of the community, local leaders or used focus groups to capture issues more in detail than those arising from public consultations. Feedback This variable identifies the instances where the IFC client reported back to the community the outcomes from previous consultations with the communities, and how the client planned to address them. Focal Point* This variable captures the instances in which the IFC client identified an employee or unit responsible for monitoring and following up on community engagement activities. 120 Engaging Citizens for Better Development Results | Appendix D Corporate  BOX D.1.  Social Responsibility in International Finance Corporation Projects Corporate social responsibility (CSR) activities typically involve an interaction with the community unrelated to the project at hand. Despite CSR was never intended as a mechanism of stakeholder engagement, before 2012 IFC clients often relied on CSR to conduct citizen engagement. Stakeholder engagement in IFC projects is now increasingly project oriented; CSR may be included as an additional measure. Some examples of the earlier, pre-2012 projects: An n  IFC client in the business of producing chemicals in India was expanding its caustic soda plant and provided support for schools, road maintenance, watershed and drinking water programs and promoted training for women. A n  Turkish manufacturer of home textile was funding a high school for 900 students, as well as a foreign language school even though it was not engaging with the community (because the plant was in an industrial zone). A n  Russian Federation subsidiary investing in the forest sector did not have a community engagement plan, but provided “several actions every year on an ad hoc basis, following requests from the community, for example providing funds to the local schools and clinics, providing computers to schools.” An n  Indian pharmaceutical ingredients manufacturer was undertaking social initiatives such as “eye diagnostic and treatment camp for the elderly, AIDS awareness for staff and community through street plays, first aid training for local school children, donations to local schools, etc.” Some examples of the later, FY15–17 projects include the following: An n  Ethiopian flower manufacturer and marketer provided access to healthcare and education facilities to all community members by funding a full-service hospital and engaging in family planning, nutrition and HIV activities. It also instituted a mechanism to ensure understanding of environmental and social impacts and risks by community members and address their concerns. A n  Pakistani subsidiary of a leading global dairy processor developed a grievance mechanism, which is confidential and registered while regularly undertaking “outreach activities (such as tree plantings, health awareness training in schools, and the use of fire extinguishers in households) involving nearby communities.” Independent Evaluation Group | World Bank Group 121 Stakeholder  BOX D.2.  Engagement in New International Finance Corporation Projects International n  Finance Corporation (IFC) projects post-2012 typically include one or more consultations with the local community, information disclosure in the local language and an appropriate grievances mechanism. Some projects went further and identified a dedicated focal point between the company and the community, as well as setup interviews and focus groups with the community and other stakeholders. A few clients went further and established continuous contact with stakeholder to ensure the communities are informed about the progress of project implementation and issues. A n  client developing a wind power plant in Pakistan organized public consultation in in several villages between 5 km and 15 km, including meetings with individuals, and with institutional and nongovernmental stakeholders. The client also planned to maintain an extensive social complaint register at the sites to document all complaints received from the local communities or any other stakeholder (including date of the complaint, particulars of the complainant, description of the grievance, actions to be taken, the person responsible to take the action, follow-up requirements and the target date for the implementation of the mitigation measure and the actual measures taken to mitigate these concerns). The client planned to disclose the Initial Environmental Examination (IEE) report at specific sites in accordance with the national legislation (PEPA 1997). In addition, the executive summary of the IEE, Supplemental IEE and Faunal Baseline Environmental Study were to be translated into Urdu language (and Sindhi language if necessary), and made available to the affected communities. Another n  IFC client building a gas power plant in Bangladesh performed a Social Impact Assessment, which included five formal public consultations and meetings with over 100 members of the communities near the project site, a detailed grievance mechanism (process and procedures for collection, collation and timely resolution within 15 days of receipt) and monitoring. The client also planned to implement corporate social responsibilities activities around health, education and sanitation, and livelihood support, as well as maintenance and improvements of common property resources (that is, roads and mosques). A bus n  operator concessionary is one of the operators new integrated mass public transport system in Cartagena, Colombia. The client arranged a consultative process, which included formal and informal meetings, focus groups and so on, and placed a dedicated team to manage the implementation of the stakeholder engagement plan. The client also plans to develop a grievance mechanism, and advertise it on their website, by radio and other outreach materials. The client noted that it considered “stakeholder engagement, communications and outreach programs as critical to the overall success.” 122 Engaging Citizens for Better Development Results | Appendix D activities can also be noted in a longer, more detailed reporting in project documents. Stakeholder engagement is now a separate section in the Environmental and Sustainability Report Summary, and each PS is addressed in a separate subheading on the IFC disclosure website.5 Design of Quality and Quantity of Stakeholder’s Engagement Improved, but Project Performance with Respect to PS 1 Has Been Uneven After the policy update, communities are more involved in project design, and resolution mechanisms are more frequently set up.6 When looking at activities by project, there has been a significant improvement in citizen engagement activities in newer projects (figure D.1).7 In FY07–11, 40 percent or more projects included consultations, GRMs, and CSR activities and information disclosure. In FY15–17, the frequency of citizen engagement activities at design has increased, except for CSR. Interviews with communities and “feedback mechanisms” have tripled, 8 GRM and disclosure of information have also increased by over 30 percent. Stakeholder mapping has significantly improved from being performed in 1 percent of the projects to about 23 percent, and stakeholder engagement plans have tripled from 5 percent to 16 percent of the projects, but there is still significant room for improvement: 60 percent of the projects approved between FY 15 and FY 17 do not explicitly reference to a stakeholder engagement plan or mapping being conducted.  itizen FIGURE D.1. C Engagement Activities, by Project 100 90 80 Percentage of projects 70 60 50 40 30 20 10 0 CSR Disclosure of GRM Stakeholder Consultation Interviews Feedback Focal point information identification FY07–11 FY15–17 Note: CE = citizen engagement; CSR = corporate social responsibility; GRM = grievance redress mechanism. Independent Evaluation Group | World Bank Group 123 Interviews with IFC specialists on evaluated projects revealed that citizen engagement was often dependent on client ownership and investment performance, as well as IFC early access to the project. IFC PSs requirements and reputation were the main drivers behind citizen engagement activities, according to the interviewees. Despite these improvements, compliance with IFC regulatory framework has been uneven. GRM and disclosure of information are still not achieved in all projects as required,9 and stakeholder mapping is well below what would be expected, based on IFC definitions. IFC requirement for stakeholder mapping and analysis is easier to trigger than the one for consultations.10 Despite this difference, consultations happen in about 60 percent of projects, whereas stakeholder mapping only in 27 percent of projects. Driven by the 2012 policy update, the design of citizen engagement activities has increasingly shifted to approach stakeholder’s engagement as part of the investments core business.11 Since 2012, IFC clients have moved from planning community engagement activities mainly as CSR, not necessarily related to an investment’s objective, to using citizen engagement as integral part of their business (figure D.2). CSR is not viewed by clients as the main community engagement tool anymore, and no project was designed with only CSR as their engagement strategy. Sixty-four percent of projects approved between 2007 and 2011 included CSR in their engagement strategy, with 14 percent of the investments conducting only CSR. In FY15–17, there were no stand-alone CSR projects, and project-oriented activities doubled their share, to about 60 percent of the total. Between 5 percent and 7 percent of projects did not perform any community engagement.  itizen FIGURE D.2. C Engagement Activities, by Orientation CE activities by “orientation” and fiscal year n = 137 n = 30 100 No activities Informationdisclosure No activities 90 CSR only 80 Both 70 Percentage of projects 60 50 Both 40 30 Project oriented 20 Project oriented 10 0 FY07–11 FY15–17 Note: CE = citizen engagement; CSR = corporate social responsibility. 124 Engaging Citizens for Better Development Results | Appendix D The nature of stakeholder engagement has improved as investments plan to engage with communities more often. Since 2012, IFC has moved from promoting primarily “reactive” and risk-mitigation activities, such as GRMs, to more “proactive” activities where clients plan to inform communities on their business activities and seek their feedback throughout the life of the project. Projects which foresee repeated interactions with the communities in their design have increased by about 44 percent, from 26 percent in FY07–11 to 37 percent in FY15–17 (figure D.3).12 This increase shows a particularly positive trend, and IFC specialists noted that, in case of smaller companies and postconstruction, clients tend to conduct less citizen engagement activities.13 The combination of reactive activities (GRM and information disclosure or CSR) has also increased, from 21 percent to 37 percent of projects, and remains the largest share of how projects conduct their community engagement. The share of investments designing single interactions with communities (such as consultations at the start of the projects) has instead decreased from 19 percent to 13 percent of total projects. Projects that envisage CSR activities as their only engagement with communities have disappeared, and there was one project approved between FY15–17 that was classified by IEG as CSR only, although it was providing a both CSR and information disclosure.14 Projects that were only disclosing information were about 6 percent of the total between FY07 and FY11, although there are none in the FY15–17 cohort (figure D.4). Investments post 2012 design to engage communities in more ways when compared with investments approved between 2007 and 2011. Investments that plan to engage communities with five or more activities15 more than doubled, from 13 percent to 37 percent, and those envisaging between three and four stakeholder engagement activities have increased by 14 percent. This trend  itizen FIGURE D.3. C Engagement Activities, by Aggregated Type CE activities by “aggregation” and fiscal year, (%) n = 137 n = 30 100 No activities No activities 90 Only disclosure of information Only CSR Only GRM 80 Only CSR Percentage of projects 70 Reactive activities Only GRM 60 Reactive activities 50 One-off CE activities 40 One-off CE activities 30 20 Repeated Repeated interactions 10 interactions 0 FY07–11 FY15–17 Note: CE = citizen engagement; CSR = corporate social responsibility; GRM = grievance redress mechanism. Independent Evaluation Group | World Bank Group 125  agnitude FIGURE D.4. M of Stakeholder Engagement, by Risk and Approval FY FY15–17 7 High risk FY07–11 49 FY15–17 23 Low risk FY07–11 88 O 10 20 30 40 50 60 70 80 90 100 Percent of projects Less than 3CE activities Three and four activities Five and more is particularly strong when looking at high-risk projects, which triggered PS 5 on land acquisition and involuntary resettlement, with five and more stakeholder activities increasing from 18 percent to 57 percent of projects, although investments with less than three stakeholder activities decreased from 49 percent to 14 percent of projects (figure D.5).  nternational FIGURE D.5. I Finance Corporation Support and Appraisal of Clients since 2012 IFC support and appraisal of client, proj. FY15–17 (%) 100 90 80 Percentage of projects 70 60 50 40 30 20 10 0 Field Explicit Environmental Stakeholder Meeting key appraisal compliance assessment meeting client personnel (or similar) 126 Engaging Citizens for Better Development Results | Appendix D In new projects, IFC support to client was extensive, but stakeholder meetings are still infrequent.16 When looking at projects approved between FY15–17, IFC provided an explicit statement of compliance in 29 projects over 30 (figure D.6). IFC specialists visited client sites in 25 cases (83 percent); in 22 cases (73 percent), they met with the client’s key personnel; and only in 8 with other stakeholders (27 percent). An environmental assessment was prepared in 16 projects out of 30 (53 percent) and community engagement activities were planned in 24 projects out of 30 (80 percent). Analysis of Community or Stakeholder Engagement Results for Evaluated Projects IEG analyzed the design and E&S results of all 137 evaluated investments and found that IFC did not systematically report on stakeholder engagement in its projects, and, in cases in which it does, it focuses on compliance issues. This reporting is based on the risk that IFC assigns to the project itself. This problematic reporting significantly limits the ability to arrive at a strong conclusion, insofar as some activities could have been not reported on but still performed. For the purpose of this section, IEG examined evaluative notes of Expanded Project Supervision Report (XPSR) and IEG project E&S reviews.  ctivities FIGURE D.6. A that Evaluated Projects Planned and Reported On Status of CE activities at evaluation 100 90 80 Percentage of projects 70 60 50 40 30 20 10 0 Stakeholder Consultations Interviews Feedback GRM CSR Focal Information identification point disclosure CE activities in compliance CE activities not in compliance CE activities at evaluation that were not reported on Note: CE = citizen engagement; CSR = corporate social responsibility; GRM = grievance redress mechanism. Independent Evaluation Group | World Bank Group 127 Reporting on citizen engagement activities at evaluation was often partial or incomplete, despite IFC claims that 105 projects were in compliance with IFC E&S. Reviewing the evidence presented, IEG found that only 87 investments were in compliance and 8 were partly in compliance (69 percent) with IFC E&S standards and IEG previous reviews. A closer look indicates that only 7 investments of the 137 analyzed provided exhaustive reporting on stakeholder engagement (5 percent) and 32 investments (23 percent) were considered to have fully reported on key PS 1 standards.17 Of the remaining, 98 projects, 28 (20 percent) provided only partial reporting remaining 70 investments (51 percent) provided inadequate or no reporting at all.18 The evidence reported is often limited to accountability (such as listing outputs or inputs) and not results. Ninety-five projects (69 percent) reported on accountability issues, 6 investments also included some outcomes and only 1 investment was fully outcome oriented (5 percent).19,20 Thirty-five investments (26 percent) provided too little information to even support accountability claims. Compared with appraisal at design, community engagement activities have been positively reported sporadically (figure D.7). Consultation, feedback, and GRM are the activities most commonly reported on (at 67 percent, 70 percent, and 74 percent), and stakeholder identification and interviews are the two activities least reported on (at 33 percent and 22 percent). Reporting is typically overwhelmingly related to achievement of outputs planned at design, as opposed to reporting not achievement.  ate FIGURE D.7. R of Unplanned Citizen Engagement Activities Compared with All Evaluated Activities Rate of unplanned or unreported activities at design per activity at the time of evaluation 100 90 80 70 Percentage of projects 60 50 40 30 20 10 0 Stakeholder Consultations Interviews Feedback GRM CSR Focal Information identification point disclosure PS1 PS5 Note: CE = citizen engagement; CSR = corporate social responsibility; GRM = grievance redress mechanism; PS = Performance Standard. 128 Engaging Citizens for Better Development Results | Appendix D Many investments have conducted community engagement activities that were not planned or recorded at design. For example, 18 investments conducted stakeholder mapping (8 planned it at design), 13 investments had a feedback mechanism back to the community (only 10 planned one at design) and 17 investments developed a community grievances mechanism (74 planned at design). These activities represent a high share of projects, particularly for those activities that were less planned at design (such as stakeholder identification or feedback). This trend is true for low and high- risk projects (figure D.7). This lack of good reporting makes more difficult to understand what progress has been achieved in stakeholder engagement. As shown in table D.2, commitments to stakeholder engagement activities are not generally followed through, except for stakeholder mapping and feedback mechanisms. That said, the high number of projects that do not report results prevents us from drawing firm conclusions as some activities could have gone unreported. Finally, IFC stakeholder engagement performance and reporting generally show signs of improvement when resettlement is involved and PS 5 is triggered (figure D.8). In our portfolio there are 49 investments for which PS5 is triggered; the performance of engagement at evaluation is higher for those than the rest of the projects, despite reporting issues persist. Most of the differences in figure D.8 are small and statistically not significant (except for partial compliance, statistically significant at the 10 percent level) TABLE D.2. Stakeholder Activities at Design and Evaluation Committed at Performed at Not Performed Activates Design Evaluation at Evaluation Not Reported Stakeholder mapping 9 19 9 109 Consultation 60 48 8 81 Interviews or focus groups 23 11 6 120 Feedback 10 20 5 112 GRM 74 65 21 51 CSR 88 66 3 68 Community focal point 18 27 0 110 Information disclosure 73 59 13 65 Note: CE = citizen engagement; CSR = corporate social responsibility; GRM = grievance redress mechanism. Independent Evaluation Group | World Bank Group 129  ommunities FIGURE D.8. C Engagement of Projects at Evaluation PS5 n = 49 PS1 n = 88 O 10 20 30 40 50 60 70 80 90 100 Percent of projects Poor or no reporting Partial compliance Full compliance Beyond requirements Note: PS = Performance Standard. Policy Update, 2012 Current community engagement and PS 1 are more detailed and their triggers are more clearly identified. In the current formulation of PS 1, stakeholder engagement is organically built as a cohesive set of actions to rigorously engage with a community to address their concerns “on project risks, impacts and mitigation measures,” and provide them with periodic reports on implementation. Stakeholder engagement is defined as “the basis for building strong, constructive, and responsive relationships that are essential for the successful management of a project’s environmental and social impacts.” Furthermore, stakeholder engagement elements are explicitly spelled out: (i) stakeholder analysis and planning, (ii) disclosure and dissemination of information, (iii) consultation and participation, (iv) grievance mechanism (GRM), and (v) ongoing reporting to affected communities. Stakeholders are defined as “local communities directly affected by the project (the affected communities),” and as well as those stakeholders not directly affected by the project but that have an interest in it (that is, national/local governments, NGOs, neighboring communities and so on).21 Stakeholder analysis is required when the project include elements “likely to generate adverse environmental and social impacts to affected communities;” disclosure of information is required and includes description of the grievances mechanism; consultations are triggered when communities are “subject to identified risks and adverse impacts from a project,” in order for the communities “to express their views on project risks, impacts and mitigation measures, and allows the client to consider and respond to them;” for projects with “potentially significant adverse impacts on affected 130 Engaging Citizens for Better Development Results | Appendix D communities, the client will conduct an Informed Consultation and Participation (ICP),” which requires an iterative, in-depth exchange of views and information and the incorporation of the affected communities on matters that affect them directly by the client into their decision-making process the views. A GRM is required for every project22, and where there are affected communities its objective is “to seek to resolve concerns promptly, using an understandable and transparent consultative process that is culturally appropriate and readily accessible, and at no cost and without retribution to the party that originated the issue or concern.” Finally, ongoing reports are required if there are affected communities, including reporting on implementation of plans on ongoing issues and issues that were identified through the GRM. Community engagement and PS 1 were relatively limited and bland in the 2006 policy formulation. The policy in effect before 2012 was approved in 2006 and had the following objectives for community engagement: “(i) To avoid, or where avoidance is not possible, minimize, mitigate, or compensate for adverse impacts on workers, affected communities, and the environment;” (ii) “To ensure that affected communities are appropriately engaged on issues that could potentially affect them.” Social and environmental impact assessments were required for “projects with potential significant adverse impacts that are diverse, irreversible, or unprecedented,” whereas project with more limited impacts could conduct a narrower assessment, and “projects with minimal or no adverse impacts” were excluded from this requirement.23 Community engagement was defined as “an on-going process involving the client’s disclosure of information.” It required a consultation with affected communities only when “local communities may be affected by risks or adverse impacts from a project,” and its objective was to “to build and maintain over time a constructive relationship with these communities.” Free, prior and informed consultation was required only for projects with significant adverse impacts on affected communities.24 A grievance mechanism was identified as “respond to communities’ concerns related to the project,” and required only if the client “anticipated ongoing risks to or adverse impacts on affected communities.” Finally, the client was required to disclose the action plan to the affected communities, along with provide at least annual, periodic reports that describe progress with implementation of the action plan on issues that involve ongoing risk to or impacts on affected communities, and on issues that the consultation process or grievance mechanism has identified as of concern to those communities. Portfolio Selection and Analysis Methods Portfolio Selection IEG sampled two sets of investments to understand how IFC clients perform citizen engagement activities over time. Both the early and recent portfolios focused on real sectors, and as such excluded any investment in Finance and Insurance, Collective Investment Vehicle, Financial Markets and Trade Finance. For the early portfolio, IEG selected only IFC investments that (i) were evaluated by IEG between calendar years 2012 and 2015; and (ii) excluded investments with IEG evaluation ratings for the PS Community Engagement PS 1 subindicator, both at appraisal and at evaluation, Independent Evaluation Group | World Bank Group 131 corresponding to “No Opinion Possible” and “Not Applicable.” A final sample of 137 real sector investment projects were selected for analysis. A small sample of 30 investments was deemed appropriate to see whether substantial changes occurred. IEG started by considering the IFC investment portfolio between FY15 and FY17.25 Twenty- eight projects were excluded because there was no lending amount recorded (either in project size, original commitment or net commitment). Then, 548 investments were excluded because they fell in nonreal sectors or industries.26 A further 82 investments were excluded because they were rights issues, swaps, equity investments, loan increases or special purpose vehicles. Then, IEG stratified the investments by sector and fiscal year,27 and replicated that distribution in an ideal portfolio distribution. As a second step, the distribution was rounded to obtain 30 investments. Once the stratification was completed, IEG used an excel function (RAND) to generate a random number to use as a starting point and then randomly selected the portfolio composition for each fiscal year and sector,28 and identified the 30 investments. Finally, three investments were replaced because they were rated as “confidential” or “strictly confidential.” Portfolio analysis IEG used the Environmental and Social Review Summary, IEG Environmental and Sustainability Review (E&S), and Expanded Project Supervision Report as the source of information for creating and populating citizen engagement categories, starting from IFC own categories (see table D.1). In these documents, IEG looked at the Community or Stakeholder Engagement, PS 1 and, where relevant, PS 4, 5, and 7 under Environmental and Social Mitigation Measures and Environmental and Social Action Plan sections. Finally, IEG used the information under the heading Overview of IFC’s Scope of Review to understand and categorize how IFC provided support to and reviewed the client’s application. These data was then aggregated and reframed in useful categories to highlight changes in citizen engagement design before and after the policy reform of 2012, and to show what support was provided to the clients in newer projects (for more information see table D.1and footnotes 11 and 12).29 International Finance Corporation nomenclature for citizen’s engagement activities changed in 2012 from community 1  engagement to stakeholder engagement. These two terms are used interchangeably in this document. The following analysis is based on the 2006 International Finance Corporation’s Performance Standards on 2  Social & Environmental Sustainability, and the 2012 IFC Performance Standards on Environmental and Social Sustainability. Available online at http://www.ifc.org/wps/wcm/connect/ac3381804886593bb892fa6a6515bb18/ IFC%2BPerformance%2BStandards.pdf?MOD=AJPERES&attachment=true&id=1322803957411 and http://www.ifc.org/wps/wcm/connect/ac3381804886593bb892fa6a6515bb18/ IFC%2BPerformance%2BStandards.pdf?MOD=AJPERES&attachment=true&id=1322803957411. The following analysis is limited to Performance Standard 1, which more directly calls for community engagement. 3  Other elements were (i) Social and Environmental Assessment; (ii) management program; (iii) organizational capacity; 4  (iv) training; (vi) monitoring; and (vii) reporting. 132 Engaging Citizens for Better Development Results | Appendix D 5 See project disclosure page at https://disclosures.ifc.org/#/landing 6  Figure D.1 shows citizen engagement activities by fiscal year per project and are not mutually exclusive. Therefore, the sum across the activities would exceed 100 percent. 7  The * indicates that the difference in values is statistically significant per two-sample z-test to compare sample proportion (one tailed). Corporate social responsibility: p<0.005, z-value: 3; disclosure of information: p<0.005, z-value: 3.7; grievance redress mechanism: p<0.005, z-value: 3.7; Stakeholder mapping: p<0.005, z-value: 3.1; interviews: p<0.05, z-value: 2.1; feedback: p<0.05, z-value: 2.1; focal point: p<0.005, z-value: 3.1 8  Including focus groups. 9  Ninety percent of projects explicitly refer to instituting a formal or informal grievance redress mechanisms and about 80 percent of projects explicitly refer to a disclosure of information system. The International Finance Corporation disclosure of information through its website is not considered for the purpose of disclosure of information. 10 For more information, see Policy Update, 2012 section, para 1. 11  Project oriented: this variable identifies citizen engagement activities developed based on project needs; corporate social responsibility (CSR) only: this variable identifies projects in which citizen engagement activities consisted of CSR (see box D.1 for more details), Both: this variable identifies projects in which citizen engagement activities consisted of both CSR and project-oriented activities. The reason for a slight discrepancy with figure D.3 is that in that figure  a project approved between FY15–17 is classified as “CSR only” despite providing information disclosure and for ease of reading. 12  Reactive activities: this variable captures any combination of corporate social responsibility, disclosure of information or grievance redress mechanism, where the client does not take a first initiative in engaging the community; Repeated interactions: this variable refers to citizen engagement activities that were performed multiple times during the project life (that is, ongoing, repeated consultations; or interactions with a specially designed focal point); One-off citizen engagement activities: this variable captures activities such as consultations or interviews performed only once, typically without returning to the community to present the results and follow-up actions. 13  According to the results of Independent Evaluation Group interviews of International Finance Corporation environment and social specialists 14  The reason for a slight discrepancy with figure D.2 is that in figure D.3 a project approved between FY15–17 is classified as “corporate social responsibility only,” despite providing information disclosure as well for ease of reading. 15 The activities are described in table D.1. 16  The International Finance Corporation Environmental and Sustainability analysis is based on several variables, including explicit statement of compliance, site visits, explicit statement that an environmental assessment was performed, meeting with stakeholders, meetings with company executives/employees 17 These requirements are consultations, grievance redress mechanisms, and information disclosure. 18  Reporting is considered substantive if it provides enough information to understand what the project performed in addition to the required activities, such as information disclosure, grievance mechanism and corporate social responsibility. If the project reported on all required activities but not the others, reporting is considered partial. If the project reported only on some of those activities its reporting is considered inadequate. 19  Examples of these outcomes are community engagement resolved and reduced complaints on client’s operations; certain citizen engagement activities improved community engagement overall; inadequate citizen engagement led to opposition among villagers and threats to the project; 20 Three investments (7 percent) did not report any information on community engagement. 21 See 2012 IFC Performance Standard 1, para 1. Independent Evaluation Group | World Bank Group 133 22  The grievance redress mechanism is defined as follows: “Clients will implement and maintain a procedure for external communications that includes method (i) receive and register external communications from the public; (ii) screen and assess the issues raised and determine how to address them; (iii) provide, track, and document responses, if any; and (iv) adjust the management program, as appropriate.” 23  Impacts defined as “few in number, generally site-specific, largely reversible, and readily addressed through mitigation measures.” 24  Informed participation involves organized and iterative consultation, leading to the client’s incorporating into their decision-making process the views of the affected communities on matters that affect them directly, such as proposed mitigation measures, the sharing of development benefits and opportunities, and implementation issues. 25  The portfolio consists in 1,139 projects, including 42 projects that didn’t have an approval or commitment year recorded 26  Nonreal sectors and industries are Finance and Insurance, Collective Investment Vehicle, Financial Markets and Trade Finance. 27 To do this, the Independent Evaluation Group looked at each sector’s share of the portfolio by fiscal year 28 This is using each sector distribution as the basis to calculate a step to identify the sample. 29 Appendix E 134 Engaging Citizens for Better Development Results | Appendix D  itizen Engagement in the Bank Appendix E. C Group’s Country Engagement Model The World Bank Group engages with stakeholders to better inform the design of its country programs, improve their implementation and ultimately enhance the sustainability of World Bank Group’s action in the country by enhancing ownership of a wide range of stakeholders. As per the guidance available to teams, the objective of stakeholder consultations is to seek the views, feedback, and possible collaboration of those affected by the Bank Group’s strategy in client countries. The expectation is that stakeholder’ consultations will increase the sustainability of country programs. This appendix is based on a desk review of the universe of Country Partnership Frameworks (CPFs) and Country Engagement Notes (CENs) approved between FY15 and FY17 (46 and 8 respectively). The desk review was complemented with 36 semistructured interviews with key informants.1 The review also benefited from the analysis conducted by the Independent Evaluation Group (IEG) of the Bank Group’s new country engagement model that reviewed the first 22 countries with a full engagement cycle. Evolution of Stakeholder Engagement in Country Strategies The country assistance strategy (CAS) process began to include consultations with key stakeholders with the start of public disclosure in the late 1990s.2Already in FY01 a review of participatory processes in CASs documented positive trends in the level and quality of stakeholder participation.3 A 2013 review of World Bank engagement with civil society organizations (CSOs) reported then that 82 percent of CAS related documents approved by the World Bank’s Board during fiscal 2010–2012 show evidence of civil society participation and that CSOs were consulted in 90 percent of the CAS reviewed (World Bank 2013). Since 2011, the Bank Group commissions Country Opinion Survey to systematically measure and track perceptions of Bank Group’s clients, partners, and other stakeholders in client countries. Country Surveys are designed to explore perceptions on the Bank Group’s work (speed, effectiveness, efficacy, relevance, efficiency), and on the knowledge, responsiveness, attitudes and engagement of Bank Group staff in the field. Survey respondents include national and local governments, multilateral/bilateral agencies, media, academia, the private sector and civil society. The goal is to survey nearly all Bank Group’s client countries every three-year cycle.4 The introduction of a new country engagement model in 2014 raised expectations on how consultative and participatory the development of Bank Group country strategies should be to make country programs better informed and more sustainable. Including citizens’ perceptions of their own country development challenges and identifying key citizen engagement bindings constraints to improve development results are central elements of this effort. Independent Evaluation Group | World Bank Group 135 Citizen Engagement in the New Bank Group Country Engagement Model The Bank Group aims to enhance stakeholder engagement at the country level through two key instruments: the Systematic Country Diagnostic (SCD) and the CPF. The former aims to identify the most critical constraints to, and opportunities for, reducing poverty and building shared prosperity sustainably, while considering the voices of the poor and the views of the private sector.5 The latter aims to discuss focus areas for Bank Group support. Areas of Bank Group support are expected to (i) be selected to address the key constraints and opportunities identified in the SCD; (ii) consider the Bank Group’s comparative advantage in relation to other donors; and, (iii) be aligned with the country’s own development agenda.6 A CEN outlines a short-term country strategy when a government and the Bank Group are unable to develop a medium-term program.7 Citizen engagement is embedded in SCD and CPFs. Both the SCD and the CPF (and, to the extent possible, the CEN) are expected to be prepared in consultation not only with national authorities, but should involve local or subnational governments and relevant line ministries, the private sector, civil society, development partners, and other stakeholders in the country.8 As per guidelines, the SCD is the reference point for client consultations and should help focus the Bank Group’ support on the areas with the greatest potential to achieve the Bank Group’s twin goals of eradicating extreme poverty and promoting shared prosperity.9 SCD consultations maybe limited to those stakeholders who can provide input to the World Bank’s diagnostics and its interpretation. CPF consultations are broader by nature, as they focus on Bank Group’s engagement and aim to elicit views from a wide range of stakeholders, ideally all possible interest groups in a country. The guidance to teams notes that the separation of the SCD and the CPF, not just in the scope and content of the documents but also in terms of the discussions and processes by which the documents are produced, is critical to the success of the Bank Group’s new country engagement model.10 ƒƒ In addition, an important innovation of the new country engagement model was defining the role of the International Finance Corporation (IFC) in leading the private sector en- gagement. IFC teams are expected to be fully engaged in consultations with the private sector in general, not limited to IFC activities as was done in the past. Quality of Design and Implementation of Consultation Processes Guidelines available to teams outline key principles of good consultation processes: openness, access to information, accountability, transparency, visibility and accessibility.11 Additional good practices are outlined in the citizen engagement strategic framework, and previous assessment of participatory process in country strategies. IEG assessed the quality of design and implementation of the CPF consultation process taking into account these various guidelines and using the following criteria:12 ƒƒ Geographic outreach and inclusion (level of intragovernmental participation; extent and nature of civil society involvement, consultations with poor and vulnerable groups; private sector) 136 Engaging Citizens for Better Development Results | Appendix E ƒƒ Continuous engagement (existence of feedback mechanisms for follow-up during and after strategy formulation) ƒƒ Innovations in mechanisms used ƒƒ Evidence on closing the feedback loop Geographic Outreach and Inclusion Most of the country strategies reviewed were prepared in close consultations with a broad range of stakeholders. All the CPFs reviewed for this evaluation except three refer to having consulted civil society. Indigenous groups, youth groups, women groups were referred in 43 percent of the documents. Interviews with task team leaders provided valuable additional information about the process, not captured in CPF documents. The new country engagement model promotes much broader engagement with stakeholders than in the past. The process of reaching out to citizens outside of capital cities was much more intensive than what country teams used to do before (teams referred to being often on the road to reach a more inclusive group of citizens and reported having consulted with monks in Bulgaria, youth in El Salvador, indigenous communities in Panama, people living in Serb majority municipalities in Northern Kosovo, and diasporas from fragile states, such as Myanmar and the Republic of Yemen). An important outcome of CPF consultation was the acquisition of direct knowledge and awareness of local needs and priorities by the government and Bank Group’s teams. Teams reported having used this knowledge for the preparation of projects and in policy dialogue with the government. For many teams, the new, broader consultation process was eye-opening. In addition, teams consulted with local governments and local NGOs much more frequently than before (63 percent of the CPF reviewed mentioned this in the CPF document but conversations with task team leaders suggest that many more did). In many cases this extensive geographical outreach provided a space for local stakeholders to voice their issues with the World Bank for the first time. In a few selected countries, the World Bank partnered with umbrella CSOs very early on and agreed on a framework for the consultation process. In Tunisia and the Arab Republic of Egypt, for instance, the World Bank and CSOs agreed on the terms of engagement (timing, mechanisms, whom to invite, and how to document the process) based on the expected duration of the CPF process and its key decision points. In Myanmar, CSO inputs were also sought to design the CPF consultations; and, in Cambodia, NGO networks were invited to identify CSO participants to consultations. Governments were generally invited to the consultations with other stakeholders and in most cases participated. This was always the case with local governments whose role was pivotal in identifying who should be invited. Although this approach has clear advantages, in some cases it resulted in proforma consultations “captured” by the government. Independent Evaluation Group | World Bank Group 137 IFC leadership of private sector consultations broadened the range of stakeholders the World Bank was used to engage with. The most recent CPFs were very effective in reaching out to a wide range of enterprises and business owners (often including nonclients or stakeholders outside the usual World Bank network), as opposed to just heads of chambers of commerce or similar umbrella associations as many country strategies previously did. This seems to suggest a learning curve as World Bank teams become more used to the new country model. Findings from the recent IEG evaluation on the new country engagement model suggest that in countries where IFC led private sector consultations, the SCD and CPF provided a platform for engaging with the country’s private sector over the long term (World Bank 2017, appendix G, 116). Task team leaders interviewed for this evaluation confirmed the above-mentioned finding. Continuous engagement? Citizen engagement in country strategies is supposed to be a continuous process, not limited to CPF formulation. Engagement processes for Performance and Learning Review (PLR)13 and Completion and Learning Review (CLR) are expected in the new country engagement model, although they may be less extensive than those for the formulation of the CPFs. Engagement at these stages is expected to help determine progress achieved and the effectiveness of the Bank Group’s program and draw lessons for future engagement. Most of the staff consulted for this evaluation, however, did not envision a major engagement for the PLR or CLR adducing lack of budget and sufficient time to do this well. Many did not feel that it was important to do so. Most small country offices could not mobilize a dedicated person for engaging citizen or the person available was not trained for this task. Although budgets did not seem to be a problem for engaging stakeholders during the process of formulation of country strategies, they may not be sufficient to support a continuous engagement. Teams reported having incurred extra costs to cover additional staff time and travel, and translation and dissemination of draft documents. Consultations are still seen by many CSOs as a one-way channel for the World Bank to share information rather than a collaborative engagement process. This sporadic engagement was confirmed by CSOs consulted during IEG country visits. CSOs voiced the importance of promoting greater interaction between government, partners, and citizens in 10 percent of the CPFs reviewed. In a few cases, the World Bank has established a ‘forum’ to get continuous feedback from civil society. Those forums were mentioned as a regular channel used in Honduras and Guatemala). In other cases, a sort of sounding boards of CSO are in place (Cambodia) or planned (Montenegro, Togo, Tunisia). Innovations in mechanisms Country teams are encouraged to use a variety of tools for stakeholder consultation and engagement (World Bank 2014, para 18–20) but the use of more collaborative or innovative approaches is either not reported or not happening; traditional in-person consultation, focus group meetings and public forums remain the preferred way to engage. 138 Engaging Citizens for Better Development Results | Appendix E Most country teams had country opinion surveys to complement the information obtained through interviews or broader consultations but there is little evidence of their use. Eighty-seven percent of the CPFs and 50 percent of the CENs had a recent survey, but for only 26 percent of the CPF there is evidence of its use. CPF task team leaders interviewed for this evaluation mentioned having used the information obtained through the surveys as an additional source to inform their programming but many referred to the limited usefulness of survey findings, which they considered to be generic and untimely. Yet, some teams went well beyond requirements. As part the CPF consultation process in El Salvador, the Bank Group launched an outreach campaign through different media channels to gather citizens’ views on the country development priorities. Participants were invited to send quotes and images completing the phrase: “El Salvador needs...”14 The team not only summarized the responses received which informed the CPF proposal but they also sought other opportunities to disseminate more broadly the feedback received. The team partnered with a local museum and showcased hundreds of drawings from fifth-grade students across the country representing their country without violence. The World Bank team saw this as a powerful way to keep citizens’ priorities high on the government agenda. Consultations with civil society took place in fragile and conflict situations, even though they are not mandatory. Seventy-five percent of the CEN reviewed reported having consulted CSOs. Teams contacted for this evaluation referred to the difficulties in reaching out to civil society or citizens in conflict or fragile situations. The majority said that engaging with civil society in such situations may create tensions with the government. Thus, CEN consultations are in many cases not reported on or not as open due to security or political issues. The Republic of Yemen illustrates a notable exception where the World Bank is trying to keep the engagement with the citizens despite not recognizing the incumbent government. To overcome difficulties associated with security, the team preparing the Republic of Yemen CEN held a series of online consultations. Approximately 20,000 respondents, drawn from different segments of Yemeni civil society, including academia, youth, women activists, think tanks, CSOs, and media responded to the online consultations. Given the success of the consultation process, the World Bank team envisages the use of online consultations throughout the CEN period to keep a pulse on Yemeni views. This review found that country teams used little innovation with ICT, social media or online consultation methods. Only a quarter of the CPF and one CEN have used web-based platforms. When asked the reasons for this choice, World Bank teams referred to the lack of capacity or time among local office staff to set these systems up, or to monitor, and moderate them regularly. World Bank teams also referred to government sensitivities. Despite not being widespread among teams, social media proved useful for a wider outreach in cases where the World Bank was reengaging after a long time (Myanmar), when the political situation Independent Evaluation Group | World Bank Group 139 made broader engagement difficult or impossible (the Republic of Yemen), and when the preparation of the country strategy faced time limitations (Kosovo). Closing the feedback loop Based on the new country engagement model, the preparation of the CPF provides the opportunity to “close the feedback loop with key stakeholders,” that is to take into account citizens’ feedback gathered at the time of the SCD. Yet, to close the feedback loop the same stakeholders need to participate in both consultations; although this is always the case for the government, it may not happen for CSOs. In very few cases World Bank teams went back to stakeholders consulted to discuss how the feedback they received was integrated (or not) in the strategy document. In the cases this was done, it was not through a formal channel. When explicitly asked, World Bank teams often referred to disclosure as a way of closing the feedback loop (for the strategic framework, sharing information is not yet engagement). In all cases the country strategies are publicly disclosed in the World Bank’s website, oftentimes having a condensed/more ‘reader-friendly’ version for public distribution in English and the national language. The CSOs feedback on the consultation process suggests space for improvement. First, it is not common for the Bank Group to request CSOs a feedback on the consultation process; this happened only in Myanmar and the Arab Republic of Egypt. For this evaluation, IEG requested and received feedback on the process in Cambodia, Ethiopia, Mali, Peru, Tunisia, Uganda, Uzbekistan, Vietnam, and the Republic of Yemen. The main complaints have to do with participants not having sufficient information in advance of the consultation process, the information not being available in local languages, and participants not having sufficient time to provide input. Several issues were raised also regarding how the process was facilitated and how priorities were identified. Monitoring and Reporting on the Consultation Process and the Inclusion of Citizen Engagement Indicators in CPF Results Frameworks The documentation on the extent and nature of engagement processes used in CPF was found to be very limited. Although the citizen engagement framework refers to the important guiding principle of documenting consultations and closing the feedback loop, nothing in the World Bank guidelines to CPF teams outlines how these processes should be documented. Although all CPF documents had at least a paragraph that broadly reports on how consultative the process was and often lists the mechanisms used to elicit stakeholder participation, only a few (17 out of 46 CPF) included substantive details about the process in an annex. Civil society feedback on key themes and proposed areas of engagement was reported in only 39 percent of CPFs (18). A dismal 2 CPFs clearly documented how the feedback received from CSOs was integrated in the CPF (Tunisia and Uzbekistan). Only one document described how the CSOs invited to the consultations were identified (Cambodia) and no document made reference to a CSO stakeholder mapping. The Consultation Hub is not being systematically used to inform future SCD/CPF consultation processes nor as a repository of the feedback solicited.15 Awareness of this resource if low among 140 Engaging Citizens for Better Development Results | Appendix E interviewed task team leaders. Although most teams said that they prepared a standard online package describing the SCD/CPF process in English and translated into national language, less than a third reported having used the Hub. Many have expressed a preference for using their own country website as they do not get additional support for using the Hub. When IEG consulted the site in September 2017, only four out of 23 closed consultations within the past year had some information published on the consultations (Ethiopia, Kosovo, Peru and Ukraine). Thirty-seven percent of the country strategies (17) explicitly identified the objectives of engaging citizens. Improving transparency and accountability, and providing citizens access to information and feedback channels on service delivery are the most frequent outcomes found in country strategies where governance was highlighted as an important cross-cutting issue. Building trust between citizens and the state is often the expected outcome of citizen engagement activities included in conflict-affected countries. Sixty percent of the CPFs that have an explicit reference to citizen engagement also included a citizen engagement indicator in the CPF results framework. CPF indicators of citizen engagement are mostly project-level indicators of user satisfaction with services, resolution rates of grievance redress mechanisms, and transparency measures. Contributions to Development Effectiveness Changing country strategies Stakeholder consultations for the most part confirmed preidentified priority areas of attention and only sporadically determined a change in CPF priorities. When this happened, it was most often prompted by feedback provided by the private sector. In Bosnia-Herzegovina, for instance, constraints to GDP growth associated with labor code restrictions highlighted by the private sector motivated changes in the proposed country strategy. In Myanmar, the World Bank broadened its approach to foreign direct investments as there was a lot of skepticism among the private sector around the potential negative impact it could have had on the domestic landscape. Adding a sector focus in country strategies following suggestions from civil society was less common but occasionally happened. In Uzbekistan, the Bank Group team made agriculture modernization a key pillar of the CPF because it was a key issue raised by citizens. Similarly, in Egypt, citizens voiced many times that education should have been a key priority area in the CPF. Initially, the World Bank had not included education in the CPF; however, after receiving feedback from citizens, the World Bank was requested to engage in the sector through technical assistance and projects aimed at improving the quality of education (the World Bank had been negotiating a major education operation with the Government of Egypt for the past 3–4 years. A large operation to support the education sector was eventually launched in 2018.) Influencing Country Systems Citizen Engagement Country Roadmaps were found in Kazakhstan, Kosovo, Kyrgyz Republic, Montenegro, Tajikistan, Ukraine, and Uzbekistan and CPFs in the Europe and Central Asia Independent Evaluation Group | World Bank Group 141 Region; the Guinea-Bissau CPF refers to plans for developing one. Central Asia initiated the citizen engagement country road map in the Kyrgyz Republic by including it in the 2016 PLR. These roadmaps outline a set of priority areas and projects were citizen engagement could be strengthened. In Uzbekistan, although there is no formal road map, an appendix to the CPF document outlines the constraints and opportunities for strengthening citizen engagement. The Kosovo country road map is illustrative of a particularly ambitious and thick approach to citizen engagement at the country level. It aims to improve its quality of design and implementation, deepen citizen engagement in key reforms and sectors (energy, natural resource management and health insurance), and explore opportunities for technical assistance and Advisory Services and Analytics geared toward strengthening local and national-level trust-building mechanisms for citizen feedback and state-nonstate dialogue. It aims to improve local governance and service delivery, reaching excluded young men and women, and to contribute to building effective national mechanisms for engagement by working with local stakeholders, including community and religious leaders and local governments. In Montenegro, the CPF highlights the results of previous efforts to include beneficiary feedback in project preparation, implementation, and evaluation that translated into better project quality and outcomes and into increased awareness of stakeholders’ rights and responsibilities. The country road map aims to build on these past achievements and ensure CSOs engagement in monitoring the CPF through regular roundtables. In Ukraine, the country road map includes plans for raising awareness among CSO of upcoming important institutional reforms (land, energy, pensions) and creating dialogue spaces between civil society and the government around them. The road map also refers to plans for third party monitoring of World Bank operations; the creation of citizen feedback mechanisms at local service delivery; and, an informal civil society advisory group to enhance the quality of engagement across World Bank operations. Many CPF documents include plans for citizen engagement–relevant activities such as third party monitoring (Bangladesh, Ukraine), enhanced information disclosure through open data (Uganda, Egypt) and social accountability measures (Chad, Guinea-Bissau, Mozambique), strengthening and mobilizing different members of the community (Haiti, Mali), participatory budgeting programs and information and communication technologies to provide feedback loop between beneficiaries, service providers, and policy makers (Côte d’Ivoire), tools for citizens to monitor service delivery Afghanistan, Ethiopia, Indonesia), and promoting dialogue between citizens and government (Kosovo). The team interviewed the task team leader of the Country Partnership Framework and Country Engagement Note, 1  relevant senior communication officer, EXR, country opinion survey team. For a detailed review of the evolution of country assistance strategies see World Bank 2017, appendix A. 2  First, an increased level of intragovernmental participation in preparing country assistance strategies with participation 3  142 Engaging Citizens for Better Development Results | Appendix E of parliamentarians, line ministries and local governments. Second, an increased effort to actively engage civil society upstream during country assistance strategy formulation and to use them as facilitators in the process. Third, an increased complementarity with existing in-country participatory processes. Fourth, a shift away from traditional downstream information disclosure to more upstream information sharing. http://siteresources.worldbank.org/ INTPCENG/1143251-1116574134357/20509179/partprocessCAS2001.pdf 4  The Public Opinion Research Group surveyed 29 countries in FY12 (July 2011–June 2012), 41 countries in FY13 (July 2012–June 2013), 42 countries in FY14 (July 2013–June 2014), and 35 countries In FY15 (July 2014-June 2015). In FY16, surveys were conducted in 45 countries. Surveys are available at http://countrysurveys.worldbank.org/ 5  The Systematic Country Diagnostic would be prepared upstream from the Country Partnership Framework, as it is supposed to be independent from it. 6  The Country Partnership Framework (CPF) guides the World Bank Group’s support to member countries’ development programs. As per the Bank Group’s country-driven model, the CPF builds on the country’s vision of its own development goals and strategy. The Bank Group and the country then further draw on the Systematic Country Diagnostic, and together develop the CPF objectives, which will reflect the Bank Group’s comparative advantage and align with the twin goals. When the conditions do not allow for a CPF, support is guided by a Country Engagement Note. 7  Country Engagement Notes are used in countries facing conflict or political crises, or where the World Bank Group is reengaging after a prolonged hiatus. As per guidelines, Country Engagement Note might also be used when a country is going through an unusually uncertain period (for example, preelection, social crisis, natural disaster) that prevents the formation of medium-term objectives. 8  Guidance on Consultations and Stakeholder Engagement from http://intresources.worldbank.org/INTOPCS/ Resources/380831-1403553684257/CPFGuidanceJanuary162015JL.pdf. 9  Systematic Country Diagnostic guidance, para 5; see https://policies.worldbank.org/sites/ppf3/ PPFDocuments/8f7c06ff182246ba974bf1c6c985a03b.pdf. 10  Systematic Country Diagnostic guidance, para 8; see https://policies.worldbank.org/sites/ppf3/ PPFDocuments/8f7c06ff182246ba974bf1c6c985a03b.pdf. 11  See http://consultations.worldbank.org/Data/hub/files/documents/world_bank_consultation_guidelines_oct_2013_0.pdf. 12  Similar criteria were used in a FY01 review of participatory processes in country assistance strategies, http:// siteresources.worldbank.org/INTPCENG/1143251-1116574134357/20509179/partprocessCAS2001.pdf 13  Performance and Learning Reviews are Country Partnership Framework midpoint assessments to update the Board of Executive Directors on progress and make any necessary adjustment to the program. 14 https://www.youtube.com/watch?v=grYqGQSfPa4 15 http://consultations.worldbank.org/?map=1 references World Bank. 2013. World Bank–Civil Society Engagement: Review of Fiscal Years 2010–12. World Bank: Washington, DC. ———. 2014. World Bank Group: A New Approach to Country Engagement. Washington, DC: World Bank. ———. 2015. Mali—Country Partnership Framework, FY16–19. Washington, DC: World Bank. Independent Evaluation Group | World Bank Group 143 Appendix F. Staff Survey Background The evaluation team conducted a survey, with eight primary questions, of sampled World Bank staff members and managers. The objective of the survey was twofold: (i) to test on a broader scale some of the emergent findings from interviews with staff; (ii) to elicit how the World Bank staff has operationalized the citizen engagement in their work and the challenges they face. The survey’s sample targeted 417 staff members and managers whose work most intersects with the corporate mandate on citizen engagement by their role or function (with 211 respondents and a 51 percent response rate). The sampled staff groups included country directors and managers, country representatives and country program coordinators, practice managers, social development specialists, governance specialists (excluding financial sector specialists and private sector specialists), communication specialists based in country offices, and citizen engagement focal points. Table F.1 below describes the population, sampling strategy, and response rates. TABLE F.1. Sampling Strategy and Response Rates Number after Assigning the Nonidentifiable Response Responses Sample Respondents 19 Respondents Rate Groups (no.) Sizea (no.) (no.) (no.) (percent) Country directors and country 105 51 20 22 43 managers Representatives and CPC 41 29 12 13 45 Practice managers 151 59 14 16 27 Social development specialists 204 65 33 36 55 TTL who took projects to the 275 72 34 37 51 Board in FY17 Governance specialists 135 57 31 34 60 (excluding financial sector and private sector specialists) Communication specialists 127 55 31 34 62 based in country offices Citizen engagement focal 33 31 17 19 61 points Nonidentifiable 19 Total 1,071 419 211 Note there were two Actual 417 211 211 51 duplicate Sustainable sample Development and focal points Source: Independent Evaluation Group sample size calculations. Note: TTL = task team leader. a. 95 percent confidence, 10 percent margin of error. 144 Engaging Citizens for Better Development Results | Appendix F Survey results provided below include whole group findings, as well as data disaggregated by group for some questions where the disaggregated findings were particularly revealing. Not all disaggregated charts produced are provided given their volume. The percentages and numbers in the charts and tables provided below include “no opinion” or “I don’t know” items (depending on question) or “nonresponses.” Main Survey Findings Staff overwhelmingly agree (87 percent strongly agree or agree) that there is strong evidence that engaging citizens can contribute to development outcomes. The same percentage agree that engaging citizens should be the responsibility of the World Bank Group. But when asked if they have a good understanding of the Bank Group’s strategy and goals for mainstreaming citizen engagement in operations, while still highly positive, the percentage of staff agreeing or strongly agreeing drops to 73 percent (see figure F.1). Those identifying as managers (90 percent) or citizen engagement focal points (85 percent) reported having a good understanding on the Bank Group’s strategy and goals for mainstreaming citizen engagement in operations, followed by those part of a task team (73 percent). But for those not identifying with any of these categories, only 50 percent had a good understanding (see figure F.2). The level of understanding varies by regional affiliation, with South Asia (80 percent) and Middle East and North Africa (79 percent) being the highest, followed by staff in East Asia and Pacific (73 percent), Europe and Central Asia percent (72 percent), and Africa (68 percent), with Latin America and the Caribbean having the lowest percentage (62 percent; see figure F.3). FIGURE F.1. Understanding the Strategy and Beliefs on Citizen Engagement Question 1. To what extent do you agree with the following statements? I have a good understanding of the WBG;s strategy and goaks for mainstreaming Citizen Engagement 73% 24% 3% in operations Engaging citizens should be a responsibility of the WBG 87% 13% 1% There is strong evidence that engaging citizens can 87% 8% 6% contribute to achieving development outcomes O 10 20 30 40 50 60 70 80 90 100 Percent N = 208 (3 responses missing) Strongly agree + Agree Strongly disagree + Disagree No opinion Note: World Bank Group = World Bank Group. Independent Evaluation Group | World Bank Group 145 the Strategy and Beliefs on Citizen Engagement, Understanding FIGURE F.2.  by Respondent Role Response to “I have a good understanding of the World Bank Group’s strategy and goals for mainstreaming citizen engagement in operations.” Yes, I am a Citizen Engagement 33% 52% 7% 7% Focal point (N = 27) Yes, as part of a task team (N = 91) 16% 57% 22% 1% 3% No, none of the above (N = 46) 7% 43% 35% 11% 1% No, but as a manager I supervise these 25% 65% 8% 2% activities (N = 40) O 10 20 30 40 50 60 70 80 90 100 Percent 7 respondents did not provide relevant information Strongly agree Agree Disagree Strongly disagree No opinion Seniority is less associated with seeing the World Bank as responsible for engaging with citizens. Those with the lowest tenure of two years or less were most in favor of engaging citizens (93 percent) contrasted with those with the highest tenure of 20+ years (70 percent) (see figure F.4). Notably, 11 percent of those who were part of task teams indicated that they disagreed or strongly disagreed with the premise of citizen engagement contributing to achieving development outcomes. Although, not surprisingly, those with roles as citizen engagement focal points (96 percent) and managers (95 percent) were most favorable to engagement (see figure F.5). There is regional variability in the extent to which staff see the World Bank as responsible for engaging with citizens. The highest favorability came from those in Latin America and the Caribbean (96 percent), and Middle East and North Africa (93 percent). Those in South Asia (75 percent) and Africa (77 percent) provided lower favorability (see figure F.6). The best way to contribute to development outcomes is by making projects more responsive to beneficiaries, according to 89 percent of respondents when asked to choose from a list of Citizen Engagement activities. The other activities that followed were: by enhancing government transparency and accountability to citizens (82 percent); by mitigating risks (81 percent); by improving inclusiveness and social cohesion (77 percent); by building capacity of citizens and civil society organizations to engage with the government (75 percent); and by building government’s capacity to respond to citizen’s feedback (73 percent) (see figure F.7). 146 Engaging Citizens for Better Development Results | Appendix F Understanding FIGURE F.3.  the Strategy and Beliefs on Citizen Engagement, by Regional Affiliation Response to “I have a good understanding of the World Bank Group’s strategy and goals for mainstreaming citizen engagement in operations.” AFR (N = 59) 50% 23% 5% 2% EAP (N = 26) 54% 19% 4% 4% ECA (N = 33) 48% 24% 3% LCR (N = 24) 54% 33% 8% MNA (N = 14) 43% 7% 7% 7% SAR (N = 20) 70% 10% 5% 5% No affiliation 92% 23% 8% (N = 13) O 10 20 30 40 50 60 70 80 90 100 Percent 22 respondents did not provide relevant information Strongly agree Agree Disagree Strongly disagree No opinion Note: AFR = Africa; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; EAP = East Asia and Pacific; MENA = Middle East and North Africa; SAR = South Asia. Engaging FIGURE F.4.  Citizens Should Be the Responsibility of the World Bank Group, by Respondent’s Years Working at the World Bank Response to “Engaging citizens should be a responsibility of the World Bank Group.” Less than 2 years (N = 15) 33% 60% 7% 2–5 years (N = 31) 39% 39% 16% 3%3% 6–10 years (N = 43) 35% 49% 9% 5% 2% 11–20 years (N = 87) 47% 44% 8% 1% More than 20 years (N = 20) 35% 45% 20% 0 10 20 30 40 50 60 70 80 90 100 Percent 15 respondents did not provide relevant information Strongly agree Agree Disagree Strongly disagree No opinion Source: Independent Evaluation Group survey analysis. Independent Evaluation Group | World Bank Group 147 Engaging FIGURE F.5.  Citizens Can Contribute to Achieving Development Outcomes, by Respondent’s Role in Citizen Engagement Activities Response to “There is strong evidence that engaging citizens can contribute to achieving development outcomes.” Yes, I am a Citizen Engagement 44% 52% 4% Focal point (N = 27) Yes. as part of a task team (N = 91) 44% 40% 1% 10% 5% No, none of the above (N = 46) 37% 46% 2%4% 13% No, but as a manager I supervise these activities (N = 40) 55% 40% 5% 0 10 20 30 40 50 60 70 80 90 100 Percent 7 respondents did not provide relevant information Strongly agree Agree Disagree Strongly disagree No opinion Engaging FIGURE F.6.  Citizens Should Be the Responsibility of the World Bank Group, by Regional Affiliation Response to “Engaging citizens should be a responsibility of the World Bank Group.” AFR (N = 59) 45% 32% 17% 3% 3% EAP (N = 26) 54% 35% 12% ECA (N = 33) 30% 58% 6% 3% 3% LCR (N = 24) 29% 67% 4% MNA (N = 14) 50% 43% 7% SAR (N = 20) 35% 40% 20% 5% No affiliation 54% 46% (N = 13) 0 10 20 30 40 50 60 70 80 90 100 Percent 22 respondents did not provide relevant information Strongly agree Agree Disagree Strongly disagree No opinion Source: Independent Evaluation Group survey analysis. Note: AFR = Africa; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; EAP = East Asia and Pacific; MENA = Middle East and North Africa; SAR = South Asia. 148 Engaging Citizens for Better Development Results | Appendix F How FIGURE F.7.  and to What Extent Are Citizen Engagement Activities Contributing to the World Bank’s Overall Effectiveness, by All Respondents Question 2. In your opinion, how and to what extent are citizen engagement activities contributing to the World Bank’s overall effectiveness? By improving inclusiveness and 31% 46% 15% 3%4% social cohension (N = 206) By enhancing government transparency and accountability 31% 51% 11% 4% 2% to citizens (N = 205) By building government’s capacity to respond 25% 48% 19% 4% 3% to citizens’ feedback (N = 206) By building capacity of citizens and Civil Society Organizations to engage with the government 28% 47% 16% 5% 4% (N = 207) By making projects more responsive to beneficiaries 48% 41% 7% 2%2% (N = 206) By mitigating risks (N = 205) 26% 55% 12% 1% 6% O 10 20 30 40 50 60 70 80 90 100 Percent To a large extent To some extent To a negligible extent Not at all I don’t know Source: Independent Evaluation Group survey analysis. Staff feel confident overall in understanding how their role contributes to the Bank Group strategy to mainstream Citizen Engagement in operations (with 80 percent strongly agreeing or agreeing). And 60 percent of respondents pay more attention to engaging citizens because of the strategic framework and corporate commitment (but with another 34 percent disagreeing with this). Despite a more holistic strategy in place on Citizen Engagement, the view was largely held that safeguard requirements largely drive citizen engagement (67 percent). Finally, there are mixed views on whether including beneficiary feedback in 100 percent of projects with clearly identifiable beneficiaries is a meaningful target for the organization (67 percent agreeing, 26 percent disagreeing; see figure F.8). Managerial Incentives Employed: Comparing Views from Staff and Managers Respondents were asked to choose the top managerial incentives used in citizen engagement, with largely the same questions asked of (i) nonmanagerial staff, and (ii) managers and directors. There was not strong alignment in how the two groups viewed the use of managerial incentives. Managers took a more optimistic view of their roles in overseeing and providing incentives for citizen engagement. Independent Evaluation Group | World Bank Group 149 Statements FIGURE F.8.  of Agreement on Citizen Engagement Approaches, by All Respondents Question 3. To what extent do you agree with the following statements? Including beneficiary feedback in 100% of project with clearly 25% 41% 19% 7% 8% identifiable beneficiaries is a meaningful target for the organization (N = 204) Despite a more holistic strategy on Citizen Engagement, the WBG activities on engaging with citizens 21% 46% 17% 5% 10% remain driven by safeguards requirements (N = 205) I understand how my role contributes to the WBG strategy to 3% 3% mainstream Citizen Engagement 28% 52% 13% in operations (N = 206) I pay more attention to engaging citizens in my work because of the strategic framework and corporate 20% 40% 27% 7% 6% commitment to Citizen Engagement (N = 205) 0 10 20 30 40 50 60 70 80 90 100 Percent Strongly agree Agree Disagree Strongly disagree No opinion Source: Independent Evaluation Group survey analysis. Note: WBG = World Bank Group. Although nonmanagerial staff see the managerial incentives as being more compliance driven (49 percent of 160 respondents), only 15 percent (of 39 respondents) of managers see this as their most commonly used managerial incentive. Another, 16 percent of nonmanager say that their manager pays little attention to Citizen Engagement. Although only 3 percent of managers held this same view. There are other nonmanagers who see their managers doing more, with 34 percent finding managers encourage staff to go beyond corporate requirements. But this 34 percent is broken into 21 percent of nonmanagers who find their managers may not be able to provide additional resources, whereas 13 percent do enjoy the necessary resources from managers (see figure F.9). Most (82 percent) managers indicated that they encouraged staff to go beyond corporate requirements, with 36 percent unable to provide additional resources, and 46 percent indicating they encourage this even when it requires mobilizing additional resources (see figure F.10). 150 Engaging Citizens for Better Development Results | Appendix F Internal and External Challenges Staff fault limitations on money (66 percent), time (52 percent) and expertise (42 percent) as the top three internal challenges to citizen engagement, when respondents were asked to identify (up to three each) internal and external challenges to successfully engage citizens in World Bank operations. These were followed by limited training and guidance (35 percent), limited evidence of effectiveness (25 percent), risks of negative feedback (17 percent), limited support from management (15 percent), and other lesser factors (see table F.2). Government reluctance related to political reasons (73 percent) and funding (58 percent) were the main reasons given on external challenges to engage citizens. Other external challenges noted included the potential for adding complexity to the project (45 percent), government/PIU capacity to conduct citizen engagement activities (45 percent), and limited civil society organization capacity for constructive dialogue (39 percent; see table F.3). Nonmanager FIGURE F.9.  Descriptions of Managerial Incentives Provided for Citizen Engagement How would you describe the level of incentives that your manager provides to meaningfully engage citizens in your work? My manager encourages me to go beyond corporate requirements in engaging citizens and 13% provides me with the necessary resources 16% My manager encourages me to go beyond corporate requirements in engaging citizens but may not provide me with the necessary resources 21% My manager ensures that I comply with corporate requirements on Citizen Engagement and stakeholder consultations My manager pays little attention to Citizen 49% Engagement N = 160 167 people who self-identified as “non-manager/directors” were supposed to answer this question. Source: Independent Evaluation Group survey analysis. Independent Evaluation Group | World Bank Group 151 Manager FIGURE F.10.  Descriptions of Managerial Incentives Provided for Citizen Engagement How would you describe the level of incentives that you provide to your teams to meaningfully engage citizens in their work? Please select the most accurate statement 3% I encourage my teams to go beyond corporate requirements in engaging citizens but often I cannot provide them with the necessary resources 15% I encourage my teams to go beyond corporate requirements in engaging citizens, even if it 36% requires mobilizing additional resources I ensure that my teams comply with corporate requirements on Citizen Engagement and stakeholder consultations 46% I pay little attention to Citizen Engagement; we have many other competing priorities N = 39 Source: Independent Evaluation Group survey analysis. TABLE F.2. Internal Challenges to Citizen Engagement Internal Challenge Identified Number Percent Limited budget available 134 66 Limited time to dedicate to this effort 105 52 Limited availability of expertise on citizen engagement at the World Bank 85 42 Limited training and guidance 72 35 Limited evidence on the effectiveness of citizen engagement activities 51 25 Risk of negative feedback from citizens or government that can affect World Bank 35 17 staff reputation/career Limited support from management 31 15 Others 23 11 Internal challenges are negligible and do not impact our capacity to mainstream 20 10 citizen engagement in our operations Total respondents 203 Source: Independent Evaluation Group survey analysis. 152 Engaging Citizens for Better Development Results | Appendix F TABLE F.3. External Challenges to Citizen Engagement External Challenges Identified Number Percent Reluctance of client government to engage citizens for political reasons 147 73 Reluctance of client government to fund activities related to Citizen Engagement 117 58 and participation Clients worry that engaging citizens can make the project overly complex 91 45 Government/PIUs lack capacity to conduct Citizen Engagement activities 91 45 Limited capacity of for a constructive dialogue with the government 79 39 Limited willingness of citizens to participate 24 12 Others 15 7 External challenges are negligible and do not impact our capacity to mainstream 2 1 Citizen Engagement in our operations Total respondents 202 Source: Independent Evaluation Group survey analysis. Independent Evaluation Group | World Bank Group 153 Appendix G. Partnership Review of the Global Program for Social Accountability Program Background on the Global Partnership for Social Accountability The World Bank established the Global Partnership for Social Accountability (GPSA) in 2012 to provide grants to civil society organizations (CSOs) and advance knowledge on social accountability. It is a global partnership program supported by multiple donors.1 The program is governed by a Steering Committee, which is composed of a broad and balanced representation of CSOs, donors and governments. The World Bank’s Governance Global Practice hosts the small team that runs the GPSA program, and the World Bank chairs GPSA’s Steering Committee. The stated purpose of GPSA is to bridge the so-called feedback loop—or accountability gap—between what citizens want and what the governments do, enhancing citizens’ voice and, supporting the capacity of governments to respond effectively to their voice. The GPSA is based on constructive engagement between governments and civil society to create an enabling environment in which citizen feedback is used to solve fundamental problems in service delivery and to strengthen the performance of public institutions. To achieve its goal, GPSA provides strategic and sustained support to CSOs and governments for social accountability initiatives aimed at strengthening transparency and accountability. Further, GPSA builds on the World Bank’s direct and ongoing engagement with public sector actors as well as with a network of Global Partner organizations to achieve the World Bank’s twin goals of eliminating extreme poverty and boosting shared prosperity. More specifically, GPSA’s work involves two main components: Funding and Knowledge. Through the funding component, grants are available to CSOs (and networks of CSOs) working in countries that have “opted in” to the GPSA. The grants focus on the institutional development of CSOs working on social accountability, and on knowledge generation and dissemination activities. The knowledge component offers a global space for facilitating the advancement of knowledge and learning on social accountability. Produced through this component are resources (working papers, think pieces and learning notes), events (forums, conferences, roundtables, brown bag lunches) and an online Knowledge Platform. A formal opt-in to the GPSA by the national governments is required for the CSOs to be eligible for GPSA grants. By opting in, a national government grants an umbrella, upfront consent to each individual GPSA grant operating in its country (GPSA 2013). The opt-in aims to use the World Bank Group’s comparative advantage of “having strong continuous engagement with client country governments” to ensure commitment and ownership by the government to be more responsive to citizen feedback (World Bank 2012, iv). Some of the World Bank’s development partners, however, were worried that an opt-in would open room for unnecessary government interference in the CSO selection process. Unlike other global partnership programs promoting transparency and good 154 Engaging Citizens for Better Development Results | AppendixG governance, such as the Open Government Partnership and the EITI, GPSA opt-in countries do not commit to implement specific reforms or action plans. The program has allocated $31.2 million for its global and country work as of June 30, 2016 (figure G.1).2 GPSA has issued three rounds of global calls for proposals allocating about $24.5 million to build CSO capacity in 33 projects in 25 out of 52 opt-in countries. The projects were selected through competitive mechanisms using country calls for proposals. A total of 1,152 applications were received (table G.1). The grants are stand-alone recipient-executed activities ranging from 3 to 5 years and in the amount of $0.5 to $1 million, supervised by assigned Bank Group task team leaders and embedded in the sectors and areas where the World Bank has an active engagement, such as health, education, public financial management, and local governance (GPSA 2016). FIGURE G.1. GPSA Grant Allocation by Region and Sector, FY13–16 Water & SAR Agriculture MNA 1,493,106 Social protection 1,640,000 1,520,000 5% AFR 7% 1,400,000 5% 8,234,000 6% Health & Education 26% 10,465,955 42% LAC 6,832,755 22% EAP 2,900,000 9% Global ECA 6,698,378 3,555,788 Governance and 22% 11% Public Finance 11,029,697 45% Note: AFR = Africa; ECA = Europe and Central Asia; GPSA = Global Partnership for Social Accountability; LAC = Latin America and the Caribbean; EAP = East Asia and Pacific; MENA = Middle East and North Africa; SAR = South Asia. TABLE G.1. GPSA Grants, FY13–16 Participating Call for Proposal (CfP) Selected Proposals Number of Applications Countries 1 CfP (2013) 12 216 12 2 CfP (2014) 12 420 33 3 CfP (2015) 8 516 42 Source: GPSA website Independent Evaluation Group | World Bank Group 155 GPSA Main Findings The summary findings in this section are drawn from the data sources described in the Methodology section. GPSA’s goal and strategy to use social accountability approaches tailored to local context are highly relevant to the Bank Group citizen engagement strategic framework. The citizen engagement strategic framework sees GPSA as one of the key trust-funded instruments complementing Bank Group–supported operations, that can help build adequate citizen/CSO capacity to use social accountability mechanisms for monitoring public service delivery and thus, enable better citizen engagement (World Bank Group 2014, 60). At the operational level, however, the synergies between the GPSA grants and the citizen engagement agenda are not fully exploited. As such, the GPSA’s contribution to the World Bank’s Citizen Engagement agenda is limited, and this impairs GPSA’s potential impact in the field. However, GPSA has room for better exploiting the World Bank’s comparative advantages to meet the objective of bridging feedback loops. The Bank Group’s convening role and financial contribution to GPSA were critical to kick start the program and mobilize donor support. Donors expect that World Bank’s engagement would allow to scale up social accountability work, lead to greater impact, and tap in the World Bank’s convening power to engage constructively with governments (Robinson 2015, 44). CSOs have similar expectations. The formal opt-in by the governments does not lead to a constructive engagement by itself. A stronger role for the World Bank during the implementation of the projects would help close the feedback loop and help nurture and institutionalize the feedback loops after the grants are closed. Improving the linkages of GPSA’s social accountability projects with the World Bank operations is critical. The evidence from countries shows that GPSA pilot projects, at least those from the 1st round, have weak links to Bank Group operations. This has implications for the uptake and institutionalization of the results. In some cases, the initiatives are isolated pilots that do not mesh well with the World Bank projects or existing country systems for social accountably (see, for example, the projects in the Dominican Republic and Tajikistan). Some projects, such as those in Indonesia, Mozambique, and the Democratic Republic of Congo, despite being stand-alone, are better linked to sector operations, often through having the same task team leader. The latter demonstrate better potential of being adapted and used in larger initiatives. Designing the pilots in such way that the linkages with relevant larger projects are established at the outset or the grants are embedded in larger sector operations, would ensure more synergies, so when the tools and approaches or the capacities built through the pilot work well, they can easily be replicated and integrated. In the sampled grant projects, some countries have shown interest in learning and using GPSA project experience in their national programs. In Moldova, GPSA-supported tools used in the education project (school-level report cards and citizen participation in education budgeting at municipal level) are adapted by the government to use countrywide. In Indonesia, the World Vision’s local branch executing the grant has adapted World Vision’s Citizen Voice and Action 156 Engaging Citizens for Better Development Results | AppendixG approach to social accountability that aims to improve service delivery in maternal and child health. The government has shown some interest in adapting the approach to strengthen the social accountability aspect of its long-term frontline service delivery program, KOMPAK, which is carried out in partnership with Australia. Despite examples of possible uptake from the national governments, the potential for sustainability and replication of even the most successful tools and mechanisms is vague due to lack of clear strategies for exit and sustainability of the project results.3 Building CSO capacity and helping establish and test social accountability mechanisms are critical, but clearly not sufficient for the uptake and institutionalization of these mechanisms, after the closure of the projects. In most cases, the governments are reluctant to engage CSOs in the monitoring of the national projects. Many grantee CSOs, which often are local branches of well-known international CSOs, such as Oxfam, CARE and World Vision, have strong capacity and often are well-resourced. However, none of the interviewed CSOs intended to continue the work that GPSA helped to kick start after the closure of the project without receiving additional financial support (it is even less likely that the local CSO networks that international CSOs partnered with will be able to do that). GPSA grants are often treated as one-off initiatives. GPSA is yet to carry out a systematic evaluation of how the grants contribute to its main objective. GPSA amassed a wealth of information on the progress of its grants thanks to its rigorous grant making, management and reporting processes. The reports of the grant recipients provide evidence on the progress made in their expected outputs and short- or medium-term (for the duration of the grants) improvements in delivery of services, such as in health and education or increased participation of citizens at local level budgeting. The program still needs to demonstrate how these results add up and contribute to its two program-level outcomes,4 that is, how the projects have helped “to increase constructive engagement between civil society actors and government decision makers” and “to improve collaboration of CSOs with state accountability institutions.” The grantees seem to have made less progress in engaging with oversight or state accountability institutions. GPSA efforts in gathering and sharing knowledge since its inception is impressive. It is using a wide range of knowledge sharing and learning mechanisms. It has established a Knowledge Platform and rolled out number of e-courses, e-forums and webinars, peer-to-peer learning, thematic learning events, commissioned think pieces by lead researchers in the field. At the same time, although the GPSA Knowledge Platform and Partners Forum have provided valuable knowledge sharing benefits, they could be exploited more. The program still needs to make the knowledge available through the Knowledge Platform more systematized to be useful for different target groups, including the Bank Group. The program has also yet to develop indicators to measure the reach and significance of knowledge and learning activities (Robinson 2015). Even more critical for future of the Knowledge Platform and the Partner Forum is how GPSA will reposition itself and whether and in what form the program will continue its country-level engagement, if its large- scale grant program for CSOs will be shrunk or discontinued due to limited resources. Independent Evaluation Group | World Bank Group 157 GPSA’s impact on introducing and promoting new approaches and tools of social accountability inside the Bank Group through its pilots, broader partnership base and Knowledge Platform is limited so far. Because of weak connections with Bank Group operations, the GPSA-supported projects do not seem to inform World Bank operations systematically. The potential of learning from CSOs’ experience, many of whom have developed their own strategies and tools and adapted locally, is high, if the linkages with the World Bank operations improve. With its limited resources GPSA would need to rethink the strategies and types of activities to support to achieve its goals. After three rounds of call for proposals, it is not clear whether there will be another call for proposals for country-level CSO work with the current resource envelope.5 At its inception, there was an expectation that GPSA would mobilize $75–125 million over the next 5–7 years, maintain its support to CSOs and scale-up. This, however, did not materialized for various reasons.6 The Bank Group contribution of $20 million for FY13–16 so far is the largest contribution to the program, followed by support from private foundations and bilateral donors bringing program’s total funds to about $29 million. Recent diversification of GPSA activities are valuable. However, program’s overall strategy needs to be revised to properly reflected those changes. GPSA complements its work with cross-cutting initiatives, such as a collaboration with fragility, conflict and violence groups supported by the State and Peace Building Fund, aiming “to empower citizens through social accountability interventions to improve the delivery of IDA18 programs in Guinea, Niger, Nepal, and Tajikistan through alternative models of oversight.” The program also partnered with the U.S. Agency for International Development in Mexico to help strengthen transparency and accountability in Mexico’s federal public administration. Although these may be good leveraging of program’s limited resources, these engagements seem to be rather opportunistic, given the lack of an overall strategic vision of the program that reflects the new realities. The assessment of the Independent Evaluation Group (IEG) indicates that in programmatic partnerships the mismatch between the program’s resources and the goals they want to attain, and the dependence on earmarked support often lead to unfulfilled expectations and diversion from strategic objectives. A series of steps could help improve GPSA’s relevance to the Bank Group and its client countries, its effectiveness and the prospects of sustaining social accountability mechanisms that were effective. Such steps could include better alignment with the Bank Group citizen engagement agenda and strengthening the linkages between the GPSA grants and the World Bank operations; adjusting GPSA’s design (strategy and key activities, such as call for proposals) to be consistent with the program’s available resources; thinking about exit strategies and better linking of GPSA-supported initiatives to country-led social accountability mechanism or longer-term initiatives supported by other development partners. Forging external alliances with partnership programs with similar objectives would improve the credibility of the program and amplify its impact. For example, a closer collaboration with the Open Government Partnership, which has complementary objectives and common stakeholders, can be 158 Engaging Citizens for Better Development Results | AppendixG mutually beneficial. GPSA could help build the CSO capacity in the development, implementation, and monitoring of Open Government Partnership action plans at the country level. The synergy of both initiatives would help build trust and improve the constructive space for government-CSO interaction. Methodology This review of the GPSA aimed to inform IEG’s evaluation of Bank Group citizen engagement on the strategic relevance and effectiveness of this key partnership in the area of social accountability. It is not intended to be a self-standing evaluation of the GPSA. GPSA was assessed as an in-depth case study using IEG’s evaluation framework for assessing global and regional partnership programs (World Bank 2007). This framework is based on the Organisation for Economic Co-operation Development Assistance Committee criteria of evaluating development assistance adapted to evaluating global partnership programs. Following IEG’s decision in 2013 to mainstream partnership program evaluations, this review does not assess all the aspects of GPSA, such as its organizational effectiveness, but rather focuses on its results particularly relevant to the main evaluation. This GPSA assessment is based on a desk review of key program and project documents for the period of 2012–17, GPSA’s external evaluation (Robinson 2015), interviews with the program management, team leaders, program donors as well as CSOs executing the GPSA grants. Where possible, country visits for the main evaluation were used to provide evidence on program’s contribution. Twelve projects in 11 countries were selected for desk review. Since GPSA project portfolio is very new (only one project is closed as of July 2017) the projects with the highest disbursement rates were selected for the desk review, assuming that these would have progressed most toward their intended outcomes. These included projects primarily from the first round of GPSA’s grants that became effective in FY14 and few from the second round (Ghana and the Democratic Republic of Congo).7 As with many trust-funded partnership programs of the Bank Group, GPSA uses its own templates. This makes the assessment of GPSA projects challenging since the concept notes and progress reports of GPSA projects are not publicly available. For some of the projects, Implementation Status and Results Reports are used for project monitoring, aside the GPSA’s own progress reporting tools, which are supplemented with other fit-for-context sources to obtain in-depth project updates. The World Bank is the largest donor followed by the United States Agency for International Development (USAID), Ford 1  Foundation, Open Society Foundation, Aga Khan Foundation U.S.A., Bertelsmann Stiftung, and the governments of the United States, Finland, Mexico, and the Dominican Republic. Global work includes expenses of the GPSA secretariat, the Knowledge Platform and number of other initiatives. 2  For example, improving the links of GPSA-supported initiatives with existing country system of social accountability or 3  with major longer-term South Asia initiatives supported by other development partners. Independent Evaluation Group | World Bank Group 159 4 See Global Partnership for Social Accountability theory of change and results framework in GPSA 2015. 5  Calls for proposals can incur high transaction costs for donors in terms of processing the project proposals, decision making, and responding to candidates (OECD 2012, 33). 6  Some of the reasons for Global Partnership for Social Accountability low resource mobilization were the initial skepticism to the provision of government opt-in. Also, some major potential donors, such as U.K. Department for International Development and U.S. Agency for International Development (USAID) have their own large programs in the field with well-established social accountability tools and approaches, such as Making All Voice Count, supported by USAID. In addition, there was a skepticism that the World Bank is well-equipped to handle such small grants programs and work with civil society organizations directly. 7  Bangladesh (2 projects), Ghana, Indonesia, Malawi, Mozambique, Philippines, Tajikistan, Kyrgyz Republic, Moldova, Democratic Republic of Congo, and Dominican Republic. references GPSA (Global Partnership for Social Accountability). 2013. Global Partnership for Social Accountability: Operations Manual. (Updated June 19, 2014). https://www.thegpsa.org/sa/Data/gpsa/files/field/ documents/gpsa_operations_manual_updated_08252014.pdf ———. 2015. Global Partnership for Social Accountability: Results Framework. Washington, DC: World Bank. https://www.thegpsa.org/sa/Data/gpsa/files/field/documents/gpsa_revised_results_ framework_10december2015.pdf ———. 2016. GPSA in Review. May 2016 http://www.thegpsa.org/sa/Data/gpsa/files/field/documents/ gpsa_in_review.compressed.pdf OECD (Organisation for Economic Co-operation). 2012. Partnering with Civil Society: 12 Lessons from DAC Peer Reviews. Paris: OECD. http://www.oecd.org/dac/peer-reviews/12%20Lessons%20Partner- ing%20with%20Civil%20Society.pdf Robinson, M. 2015. Program Evaluation: Global Partnership for Social Accountability. Washington, DC: World Bank. https://www.thegpsa.org/sa/Data/gpsa/files/field/documents/gpsa_evalua tion_report.pdf World Bank. 2007. Sourcebook for Evaluating Global and Regional Partnership Programs: Indicative Principles and Standards. Independent Evaluation Group. Washington, DC: World Bank. https:// openknowledge.worldbank.org/handle/10986/6601. ———. 2012. Board Paper: Global Partnership for Social Accountability and Establishment of a Multi- donor Trust Fund. http://documents.worldbank.org/curated/en/772371468325192991/pd- f/675810BR0REVIS0Official0Use0Only090.pdf World Bank Group. 2014. Strategic Framework for Mainstreaming Citizen Engagement in World Bank Group Operations: Engaging with Citizens for Improved Results. Washington, DC: World Bank Group. http://documents.worldbank.org/curated/en/266371468124780089/Strategic-frame work-for-mainstreaming-citizen-engagement-in-World-Bank-Group-operations-engag ing-with-citizens-for-improved-results. 160 Engaging Citizens for Better Development Results | AppendixG The World Bank 1818 H Street NW Washington, DC 20433