Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00005234 IMPLEMENTATION COMPLETION AND RESULTS REPORT < TF18164> ON A SMALL GRANT IN THE AMOUNT OF USD 0,8 MILLION TO The Catholic Organization for Relief and Development Aid CORDAID FOR THE Reinforcing Social Accountability of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu Project (P150874) August 2022 Health, Nutrition & Population Global Practice Western and Central Africa Region Regional Vice President: Hafez M. H. Ghanem Country Director: Jean-Christophe Carret Regional Director: Amit Dar Practice Manager: Francisca Ayodeji Akala Task Team Leader(s): Hadia Samaha ICR Main Contributor: Karamjit Chohan Confidential ABBREVIATIONS AND ACRONYMS CSO Civil society organizations CBO Community based organization ReCO Community health workers (Relias communautaires) CSC Community Scorecards DRC Democratic Republic of the Congo DGOGSS Directorate for Primary Health Care DALY Disability-adjusted life years GPSA Global Partnership for Social Accountability GDP Gross domestic product GNI Gross national income HCC Health Care Committees HF Health facility CODESA Health Facility Committees HSSS Health System Strengthening Strategy HZ Health zone ECZ Health zone management teams (Equipe Cadre de Zone de Sante) IRI Intermediate Results Indicator IDP Internally displaced persons IRC International rescue committee MSP Ministry of Health (Ministere de la Sante Publique) PNDS National Health Sector Development Plan (Plan National de Développement Sanitaire) OiC Officer in Charge OOP Out-of-pocket PBF Performance-based financing PDO Project Development Objectives DPS Provincial health directorate (Direction Provincial de la SanteÌ?) Q Quarter RETF Recipient Executed Trust Fund SSA Sub-Saharan Africa SCD Systematic Country Diagnostic ToA Theory of action VG Vulnerable groups Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) TABLE OF CONTENTS DATA SHEET ....................................................................... ERROR! BOOKMARK NOT DEFINED. I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 4 II. OUTCOME .................................................................................................................... 16 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 32 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 41 V. LESSONS LEARNED AND RECOMMENDATIONS .............................................................. 50 SUPPORTING DOCUMENTS .................................................................................................. 54 Page 3 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name Reinforcing SAcc of health services by supporting health P150874 committees and the community diagnosis in Bas Congo and South Kivu Country Financing Instrument Congo, Democratic Republic of Investment Project Financing Original EA Category Revised EA Category Not Required (C) Not Required (C) Organizations Borrower Implementing Agency The Catholic Organization for Relief and Development The Catholic Organization for Relief and Development Aid CORDAID Aid CORDAID Project Development Objective (PDO) Original PDO The objective of the Project is to improve access and quality of health care services in Targeted Regions through the strengthening of Health Facility Committees ("HFCs').   Page 1 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) FINANCING FINANCE_T BL Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) Donor Financing TF-18164 800,000 796,329 796,329 Total 800,000 796,329 796,329 Total Project Cost 800,000 796,329 796,329 KEY DATES Approval Effectiveness Original Closing Actual Closing 18-Nov-2014 18-Nov-2014 18-Nov-2018 18-Feb-2019 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 11-Oct-2018 0.54 Change in Loan Closing Date(s) Change in Implementation Schedule KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Moderately Satisfactory Modest RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 22-Mar-2016 Moderately Satisfactory Moderately Satisfactory 0.12 02 07-Apr-2017 Satisfactory Satisfactory 0.36 03 10-Jul-2018 Satisfactory Satisfactory 0.54 Pag Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) ADM STAFF Role At Approval At ICR Regional Vice President: Makhtar Diop Victoria Kwakwa Country Director: Jan Walliser Jean-Christophe Carret Director: Amit Dar Amit Dar Practice Manager: Trina S. Haque Francisca Ayodeji Akala Task Team Leader(s): Josef S. Trommer Hadia Nazem Samaha ICR Contributing Author: Karamjit Singh Chohan Pag Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. Context at Appraisal Country Context 1. The Democratic Republic of the Congo (DRC) has experienced considerable economic growth and stability for over a decade, although it remains one of the poorest countries in the world. The DRC has become one of the fastest growing economies in Africa. Macroeconomic indicators such as inflation rates are stable, and average gross domestic product (GDP) annual growth was 6.1 percent between 2008 and 2012. 2. The country remains fragile after decades of violence and conflict. Inequality remains high and more than 35 percent of the population was living in poverty in 2016. Gross national income (GNI) per capita in the DRC has remained low compared with the Sub-Saharan Africa (SSA) average, and the country had a human capital index score of 0.37 in 2018 – ranked 147th out of 157 countries. Poverty is rife in the DRC, particularly in rural areas which accounted for 55 percent of the population in 2019. In 2020, 43 percent of households had access to drinking water (69 percent in urban areas, 23 percent in rural areas) and only 20 percent had access to sanitation. The highly urbanized country also faces growing demands from a youthful population. Lack of social inclusion and existing disparities and inequalities between urban and rural areas, and the North and South of the country, are key drivers of fragility. Weak infrastructure and lack of transportation also contribute to provincial disparities. 3. The DRC is still recovering from a series of conflicts which broke out in the 1990s. The World Bank reengaged with the DRC in 2001, after nearly a decade of suspension of its activities, gradually shifting from emergency assistance to a sustainable development strategy. The second Congo war formally ended in July 2002 with the signing of the Pretoria Agreement, but the eastern part of the country has remained beset by a high level of violence and insecurity. In the provinces of North and South Kivu many armed groups are still active. Civil society actors in the DRC face challenges including insecurity, aid-dependency, and legal recognition. 4. The country’s complex demographics and the presence of many internally displaced persons (IDP) present significant challenges for public sector systems and host communities . According to the most recent Demographic and Household Survey, more than 200 ethnic groups live in the country, the majority of which are Bantu peoples. The DRC shares borders with 9 countries and the humanitarian situation remains among the most complex crises in the world. According to United Nations High Commission on Refuges in 2020, there were 527,114 refugees and asylum-seekers, largely from Burundi, the Central African Republic, Rwanda, and South Sudan.1 In addition, the DRC has above 5.5 million IDPs, the largest IDP population in Africa. This speaks to the importance of setting up representative and inclusive management and accountability structures from provincial down to community levels. 5. Political instability, poor governance, and the resulting lack of basic service delivery all contribute to the country’s limited development and persistent poverty. The DRC’s turbulent history has prevented the 1 https://reporting.unhcr.org/drc Page 4 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) establishment of stable institutions, sustained elite capture of resources and contributed to endemic corruption. According to the Worldwide Governance Indicators in 2018, the DRC’s percentile rank on voice and accountability was 7.39 (where 0 corresponds to the lowest rank and 100 to the highest), 5.77 on government effectiveness, 5.77 on regulatory quality, 2.88 on the rule of law and 3.85 on control of corruption. All of these ratings are significantly below SSA averages. Instability and poor governance are important contributing factors to the lack of service delivery across sectors. Health Sector Context 6. By the end of the war in 2002, the health sector had almost completely collapsed. Revitalization of the health sector began in 2005 with the adoption of the Health System Strengthening Strategy (HSSS) by key national actors. In 2010, a five-year National Health Sector Development Plan (Plan National de Développement Sanitaire (PNDS) 2011-2015) was set up and aligned with the orientations of the HSSS. 7. The health sector faces major challenges as evident by the population’s poor health outcomes. Almost half of all children under the age of five are malnourished. The seven principal causes of disability-adjusted life years (DALY) are malaria, diarrhea diseases, protein-energy malnutrition, lower respiratory tract infections, HIV, preterm birth complications and tuberculosis. Non-communicable diseases associated with epidemiological transitions account for 21 percent of the national burden of disease, although this proportion is rising. Mental health and the consequences of violence are also major public health challenges. Fertility rates (6.1 children per woman) are higher than the SSA average (4.8), and maternal mortality rates have remained stubbornly high despite government efforts to train and implement midwives and to improve access to and quality of health services. 8. Inequities in access to and quality of health care are rife across the DRC and can be partly attributed to how the health sector has been financed, including high out-of-pocket payments and low government funding (approximately USD 1 per capita per year on healthcare) – one of the lowest rates in the world .2 Fees in public health care facilities are relatively high and are comparable to the fees in the private sector. Households account for the majority (42 percent) of total health expenditure, followed by external aid (35 percent), and government (14 percent). Low government investment in public health services and facilities, and significant out-of-pocket (OOP) payments also continue to limit the range and availability of services and exacerbate the cycle of poverty and ill health for much of the poor. In 2014, only 4 percent of GDP was spent on health in the DRC (below the SSA average) and this remained stable until 2018 when it fell to 3.3 percent. Of this total health expenditure, only 15.1 percent was spent by government in 2018. External aid is fragmented and accounted for 35 percent of total health expenditures in 2018, more than double the SSA average of 12 percent. 9. Beyond financing, the health sector in the DRC faces many challenges that limit access and quality of health services. Health facilities have limited health infrastructure and equipment. Health workers have insufficient training and low motivation due to the challenging environment in which they work. Lack of trust in the country’s health system prevents many people from using services when needed. Furthermore, the country has limited supply of quality drugs outside of the capital. Overall, actors at all levels of the health system operate with limited accountability mechanisms in place. 2 National Health Accounts, 2016 Page 5 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) 10. The DRC health system is decentralized, which presents opportunities for improved service delivery and program implementation through focused investments. The health system is organized in three levels: ï‚· Health Zones (district level): a health zone management team manages a network of health centres and a district hospital. Health zone management teams (Equipe Cadre de Zone de Sante (ECZ)) consist of polyvalent professionals responsible for development and operation of all district health services. They share the responsibilities of supervision, follow-up and support for all health facilities in the district. Health zones typically cover a population of 100,000 to 200,000 ï‚· Health Provinces (intermediate/provincial level): these correspond to each of the 26 administrative provinces in the DRC. Each province is managed by its own provincial health directorate (Direction Provincial de la SanteÌ? (DPS)), which is responsible for technical and logistic support. ï‚· Ministry of Health (Ministere de la Sante Publique (MSP)) (central level): the DRC MSP has a normative role, which includes for example development of health standards and national programmes for the control of specific diseases. At the national level, the Directorate for Primary Health Care (DGOGSS) leads coordination efforts for community health. 11. Health facilities have some degree of legal and de facto autonomy in management, which was set up through the process of centralization, ongoing insecurity in certain areas of the country, and initiation of performance- based financing (PBF). PBF has been scaled throughout multiple regions of the DRC. The World Bank Health System Strengthening for Better Maternal and Child Health Results (P147555 – US$714.50 million) implements PBF in over 3000 health facilities in the country in 12 provinces, amounting to approximately 40 percent of the population. 12. Health Facility Committees (HFCs) provide social accountability mechanisms through which citizens can interact with health authorities and service providers to improve access, quality and utilization of health services at the local level. To enable these improvements, HFCs should enhance the responsiveness of services to local needs and improve awareness of available care among the community. 13. A strong body of literature outlines the major role that HFCs play to improve health service delivery . In their analysis of countries including the DRC, Lodenstein et al. (2017) categorize HFCs as performing two critical sets of activities to improve health service provision. First is supporting the functioning of health facilities and objectives of providers (as an extension of service providers, HFCs engage in community outreach, the co- management of health centre resources and the facilitation of repairs and fundraising). Second is supporting voice of users and citizens and “bottom-upâ€? integration of community preferences in decision-making in service delivery. In a systematic review, McCoy, Hall & Ridge (2012) found “some evidence that HFCs can be effective in terms of improving the quality and coverage of health care, as well as impacting on health outcomes. However, the external validity of these studies is inevitably limited. Given the different potential roles/functions of HFCs and the complex and multiple set of factors influencing their functioning, there is no 'one size fits all' approach to community participation for health via HFCs, nor to the evaluation of HFCs.â€? Page 6 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) 14. The DRC has had a rich and long experience with HFCs, yet the potential benefits have not yet been fully realized. HFCs were first introduced in the DRC in 1979 and expanded upon in 1982 and 2006. There was explicit commitment to community participation in the DRC’s National Health Sector Development plan in 2010, whereby the MSP proposed to transform the ineffective or non-existent health management committees into multidisciplinary, multi-sectoral health development committees (8,504 planned for 8,504 health areas). In 2012, there were 8,126 HFCs attached to primary health care centers in the DRC.3 These HFCs, called CODESAs, are strongly associated with volunteer community health workers (Relias communautaires (RECOs)) who play an important role in health care in the DRC given the shortage of trained health professionals.4 Each village has a CHW representative in the CODESA, and CODESAs include at least 10 members who are elected to represent community health workers (chair), civil society and the health centre Officer in Charge (OiC). The OiC cannot be a member of the CODESA executive board. 15. CODESAs help to ensure community participation in health facility management. CODESA members collect information that can be shared and discussed with health facility staff and the community. According to the MSP (2012), CODESAs “have the capacity to develop co-manage and mobilise local resources for the revitalization of health services, but also to strengthen community capacity in the mobilisation of local resources.â€? In practice, the CODESAs are largely responsible for the co-management of their primary health- care facilities, specifically: (i) technical co-management: mostly informing staff about the health situation of the population, as well as planning; (ii) administrative co-management: price monitoring, liaison with administration, inventories, etc; (iii) financial and strategic co-management: review and action plans; (iv) co- management of human resources; and (v) promotion of the health facility within the community. 16. A subset of CODESAs in South Kivu have previously been supported in two notable projects by Cordaid, which included the use of community scorecards and charters. The first 2012-2014 project, CSF/CODESA, was funded by the World Bank (Civil Society Fund) and implemented in four pilot health zones of South Kivu. Set up in close collaboration with provincial health authorities, the project consisted of training the CODESAs and equipping them with two innovative tools: (i) an early version of Community Scorecards (CSCs) to identify and resolve problems, and (ii) CODESA charters. Over a year, improvements in CODESA organization, accountability, and management were observed. However, it was unclear whether the CODESA mechanism benefited the poorest, as poorer people had limited access to the CODESA and were frequently denied primary health care. Cordaid and the MSP then implemented a second project, CODESA II, which covered the same areas and sought to consolidate the results of CODESA, as a core finding suggested that regular support was instrumental to the success of the CODESAs. The project also experimented with matching grants which would be further developed in the current Project. 17. The initial Cordaid initiatives were assessed upon completion. A qualitative study by Ho et al. (2015) analyzed the effects of community scorecards on improving the health system in Eastern DRC and found the most salient changes were related to increased transparency and community participation in health facility management, 3 Health centers (HCs) and health facilities (HFs) are used interchangeably in this report. 4 RECOs may be grouped into village-level, multisectoral community outreach units named Cellules d’Action Communautaire (CAC). CACs promote community participation in community health. RECOs and CAC focus on health promotion, hygiene, and sanitation while CODESAs are tasked with the co-management of the health facility. Page 7 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) increased access to services, and improved quality of care. Changes occurred through different mechanisms including provider response to information, pressure from community representatives or supervisors, and improved collaboration and joint action by HFCs and providers. The report found that when scorecards identified solutions to problems, users and healthcare providers were able to work together to implement mutually acceptable solutions that improve quality of health services and make them more responsive to community needs. Overview of the Project in South Kivu and Kongo Central5 18. The Project was financed through a Recipient Executed Trust Fund (RETF) Grant from the Global Partnership for Social Accountability (GPSA) Trust Fund, housed within the World Bank. The Project’s context was set within the Government of the DRC decision to opt-in to the GSPA6, established by the World Bank in 2012. The country’s opt-in status into the GPSA allowed civil society organizations (CSOs) with a legal presence in the country to apply to the program’s second global call for proposals, launched in January 2014. Grants were intended to provide strategic and sustained support to CSO projects with the following objectives: (i) addressing critical governance and development problems through social accountability processes that involve citizen feedback and participatory methodologies geared to helping governments and public sector institutions solve these problems; and (ii) strengthening civil society's capacities for social accountability by investing in CSOs' institutional strengthening and through mentoring of small, nascent CSOs by well-established, larger CSOs with a track record on social accountability.7 19. GPSA selected Cordaid from among 428 submissions from CSO's and CSO-networks. Cordaid’s mission is to focus on fragility, foster local ownership, and link disaster relief and development to achieve inclusivity, social justice, and gender equality.8 The proposal was selected after undergoing a technical review by the GPSA’s roster of experts and a subsequent eligibility review by the Country Management Unit. The proposal was also shared with the DRC government for comment and then published for public comments. 20. In November 2014, Cordaid received a four-year total grant of USD 800,000 from the GPSA Trust Fund to implement the Reinforcing Social Accountability of Health Services by Supporting Health Committees and the Community Diagnosis in Bas Congo and South Kivu (P150874) Project in the DRC (also referred to as the GPSA/CODESA project). Specifically, Cordaid, along with the provincial ministries of health and CSOs would strengthen CODESAs in South Kivu and Kongo Central. Cordaid aimed to reactivate a total of 190 CODESAs (190 catchment areas) across nine health zones in the province of South Kivu and three health zones in Kongo Central. These health zones cover over 2.2 million people – a third of the population of South Kivu, and a tenth 5Bas-Congo was renamed (now Kongo Central). 6 The World Bank Global Partnership for Social Accountability (GPSA) supports collaborative social accountability between governments and civil society, where citizen feedback is used to solve problems in service delivery and to strengthen the performance of public institutions . GPSA was established in 2012 with the purpose of bridging this gap between what citizens want and what governments do by enhancing citizens’ voice, including concerns and preferences, and supporting the capacity of governments and providers to respond effectively to their voice. To ensure sustainability, GPSA utilizes approaches which address longer term institution building, with strong emphasis on engaging citizens to build political support, promote social cohesion and strengthen resilience. 7 See GPSA Second Call for Proposals, available at https://dev.bluetundra.com/wp-content/uploads/2021/07/Second-Call-for- Proposals.pdf 8 About Cordaid, available at: https://www.cordaid.org/en/who-we-are/about-us/ Page 8 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) of the population of Kongo Central. The Project's target was later revised to 182 HFCs across eight health zones in South Kivu and three health zones in Kongo Central. 21. The Project increased the scale and scope of Cordaid’s existing pilot, which had been funded by the World Bank CSF. World Bank funding for the initial project ended in 2013, but the pilot was kept open as a result of a EUR 140,000 donation from the Dutch Ministry of Foreign Affairs to fund through February 2014. Cordaid DRC provided strategic guidance, training and support to local communities, and managed all operational aspects of the project, including monitoring and evaluation. 22. In 2018, Cordaid received additional funding from the World Bank Nordic Trust Fund to explore the relevance of the CODESA approach in an urban context. This complementary pilot project (Pilot on Revitalizing the Community System by Strengthening Social Accountability Mechanisms) was implemented in two additional CODESAs in two health zones of Kinshasa province for a period of three months during the first quarter of 2019 (new total of 184 targeted HFCs). 23. Bank support aimed to increase social accountability using: (i) community scorecards (CSCs), (ii) new terms of reference (local charters) for CODESAs. These tools aimed to enhance CODESA’s organisation and ability to hold health facilities to account. As such, the Project supported a bottom-up approach to accountability and developed capacity of local grassroot groups for planning. However, rather than focusing solely on bottom-up grassroots action, the Project also sought to "close the loop" between state-society interactions by encouraging the government to respond to citizens and civil society actors (with respect to citizens' preferences for public service delivery), through charters and contracts with health authorities. These social accountability processes were particularly relevant to South Kivu, where the persistence of conflict significantly affected community relations, and to Kongo Central where there was a lack of government follow up on community health management. It was anticipated that these social accountability tools and processes could help to restore relations and dialogue between and within communities and their health facility. 24. The Project was originally implemented in four different contexts (Table 1): (i) The 79 CODESAs of four of the health zones of South Kivu (Walungu, Miti-Murhesa, Katana, and Idjwi), which received support since 2012 as part of CSF/CODESA. The novelty of the CODESA/GPSA project introduced in these zones was (i) the introduction of community matching grants; and (ii) the development of a pilot approach to make the CODESAs more inclusive of and beneficial to the poorest. (ii) The CODESAs of four additional health zones of South Kivu (Kalehe, Lemera, Mubumbano, and Uvira) , which were less stable (with ongoing severe security issues in two of them) than the first four, and where the CODESAs had not been supported by any other partner until this Project. (iii) The CODESAs of three health zones of Kongo Central (Boma-Bungu, Muanda, and Kitona), which also had not been supported by any other partner until this Project, but which are located in a context far less tense than South Kivu. Cordaid relied on a partner (AAP Kongo Central) for project implementation and had less intense contact with health authorities in this area. Page 9 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) (iv) The CODESAs of two health zones of Kinshasa are located in a semi-urban area. Pilot activities there lasted three months. Table 1: Areas of intervention 25. Excluding project management and implementation costs, most of the Project budget was allocated to three sets of activities: (i) training and retraining of CODESAs; (ii) matching grants; and (iii) CODESA subsidies. Matching grants were for new, one-off, CODESA-led projects at the facility level while CODESA subsidies were for routine CODESA activities. 26. The project had both direct and indirect beneficiaries. Direct beneficiaries included health service users in targeted rural communities particularly, vulnerable groups (including women, children, and Indigenous Peoples) and the medical staff working in health facilities. Indirect beneficiaries included all citizens of DRC. The strategy behind the scale up of the intervention was to re-launch the role of the Health Facility Committees at the national level and improve the health care system nationally. In addition to the citizens living in the 184 catchment areas covered by the Project, beneficiaries and key stakeholders of the Project are outlined in Table 2. Page 10 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) Table 2: Project Beneficiaries by Health System level Level Beneficiary ï‚· PartiCom (community participation department): develops guidelines and ensures compliance with National legal and technical provisions in the field of community health ï‚· Cordaid country office ï‚· DPS: technically and logistically assisted the health zones in the Project. As such, DPS assisted with regulation, development, and adaptation of approaches to the local contexts. Provincial health Provincial departments enjoy relative autonomy, particularly in the east of the country (e.g., South Kivu). ï‚· Cordaid (Bukavu office, South Kivu) ï‚· AAP Kongo Central (Cordaid partner for Kongo Central operations) ï‚· Partner country offices ï‚· The ECZ have a regulatory function and provide leadership and governance within health zones. In Health the Project, teams supported CODESAs, CAC and all RECOs with supervision to improve their skills, Zone and closely monitored CODESA activities. ï‚· CODESAs ï‚· Community RECOs and CACs ï‚· Health facility staff ï‚· CBOs: contracted out to verify the functionality and performance of CODESAs and health facilities, and CODESA implementation of matching grant projects; also assisted CODESAs in planning community activities and helped to manage the Project in hard-to-reach areas. Rationale for Bank Support 27. At time of appraisal, the project aligned with pillar one of the DRC’s five-year National Health Sector Development Plan (PNDS 2011-2015) - to develop health districts. The Project builds on the Country Assistance Framework (2007-2010) which stressed issues of local governance, improving access to basic social services, and reducing vulnerability. The Project also aligned with and supported the following strategic objectives of the Country Assistance Strategy 2013-16. The Project also built upon experience, evidence, and tools generated by earlier Cordaid pilots and a Bank project involving CODESAs. This Project complemented a series of larger programs and ongoing reforms supported by the World Bank in the DRC, including: (i) The USD 714.5M ‘Health Systems Strengthening for Better Maternal and Child Health Results Project, which seeks to improve utilisation and quality of maternal and child health services, with a strong focus on the good management and governance of health facilities. (ii) The USD 5M Public Financial Management and Accountability Project for Democratic Republic of Congo’, which seeks to “enhance the credibility, transparency, and accountability in the management and use of DRC's central and selected sub-national public finances.â€? Page 11 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) (iii) The USD 133M Eastern Recovery Project, which seeks “to improve access to livelihoods and socio-economic infrastructure in vulnerable communities in the eastern provinces of Democratic Republic of Congo (DRC).â€? It has a strong focus on community resilience and inclusive community participation and has strengthened the capacity of local development committees. Project Development Objectives (PDOs) 28. The PDO was to improve access and quality of healthcare services in targeted regions through the strengthening of Health Facility Committees. Targeted Regions refer to the provinces of South Kivu and Kongo Central in the DRC. Unpacked, the project development objectives were: Objective 1: Improved access to health services; Objective 2: Improved quality of health services. Key Expected Outcomes and Outcome Indicators 29. The PDO-level results indicators were: (i) 190 Health Care Committees (HCCs)9 strengthened Unit of measure: Number of HFCs trained/recycled. (ii) 190 Social contracts between citizens, HCCs and medical staff signed Unit of measure 1: Number of signed and evaluated contracts of HFCs and medical staff (contracts refreshed each year). Unit of measure 2: Median score of performance of HFC and medical staff. (iii) Increased access to healthcare facilities for vulnerable groups Unit of measure: Rate of identified vulnerable people (according to specific criteria) who have access to healthcare facilities (based on project impact study). Components 30. The project consisted of four components. Each component is referenced as presented in the Legal Agreement, and is supplemented with further information on implementation. During the first two years, the Project focused on building organizational capacity of HFCs, clarifying their roles and responsibilities, and making them known to different actors within the health system. This required building common grounds to enable actors to effectively communicate and work together. In the latter year, the Project sought to reinforce accountability by helping HFCs to understand and use information and data from health centers and the community. It was anticipated that this would lead to the delivery of more data-driven health services and communities through micro-projects. In this process, provincial parliamentarians were identified as potential stakeholders to engage. To promote understanding of the project’s design, the following table summarizes the key focus of component activities, and key elements as further planned by the Project. This includes information obtained on implementation from project documents, the final evaluation, and the Results Framework. 9 The term “HCCsâ€? is used interchangeably with “HFCsâ€? in project documentation and this report. Page 12 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) Table 2: Summary of Components Component, objective, Focus of activities Key elements and estimated cost 1. Reinforcing the health Capacitating for ï‚· Training & retraining HFC members across levels of facility committee improved functioning health system (e.g., roles, responsibilities, skills). system and effectiveness of ï‚· Strengthening supervision & support system (with HFCs (including health authorities and CSOs). (39% of estimated increased utilization of ï‚· Introducing contracting mechanism for monitoring and costs) health services). subsidizing HFC activities, through health authorities and CSOs. ï‚· Dissemination of social accountability tools (e.g., CSCs, new TOR). 2. Re-appropriation of Empowering CODESAs ï‚· Use and application of social accountability tools (e.g., the health facility to improve health CSCs, new TOR). through community services and address ï‚· Matching grants (financial support for rehabilitation of actions and projects issues prioritized by the HFs & community micro-projects), including community served. implementation of a system for matching grant. (9% of estimated execution costs) ï‚· CODESA subsidies issued through contracting mechanism (financial support for CODESA operations). ï‚· Non-financial support for meaningful engagements. ï‚· Hotlines to include service users in most remote areas. 3. Integrating the Empowering CODESAs ï‚· Training CODESAs to identify and promote inclusion of poorest and most and Vulnerable Groups VGs in HFCs (e.g., creating and updating VG lists). vulnerable fringes of (VGs) to improve access ï‚· Implementation of quota system for participation and the population in the and quality of services election of VGs in HFCs. HFC and decisions on for the vulnerable. ï‚· Support & training to representatives of VGs in HFCs healthcare services ï‚· Pilot of social (activation of social commissions) and financial (matching grants for VG-focused projects) (7% of estimated interventions to promote access and inclusion of VGs costs) in HF decision-making; and impact evaluation. 4. Knowledge & Building a coalition of ï‚· Training CSOs on social accountability. Learning and Project actors for social ï‚· Integration of social accountability approach in all Management: health accountability and HFCs health-related interventions of Cordaid-DRC. facility committees across levels of the ï‚· Sharing lessons & best practices through quarterly are fully integrated health system, including newsletters and online portal for Project stakeholders, into the health the national level, and through online communities of practice. system and beyond through exchange and ï‚· National level working sessions with DRC MSP and advocacy. provincial health authorities to strengthen awareness (44% of estimated of and commitment to HFCs (advocacy and exchange). costs; excluding Page 13 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) project management ï‚· Dissemination of new national guidelines and and implementation refreshed framework for HFCs. costs: 2.5%) ï‚· Ensuring system for development of HFCs exists at provincial levels (e.g., provincial steering committee and technical commission meetings). ï‚· Project management (including M&E). Theory of Change 31. Core elements of the GPSA approach were integrated into the design and structure of the Project, as indicated by the Project components, including collaborative governance, flexible funding, sustained non-financial support and establishment of compacts between civil society and government at the local level. The GPSA’s theory of action (ToA) and Results Framework are key building blocks of its learning for results work. The latest iteration of the Results Framework builds on lessons from GPSA-supported operations between 2012 and 2019 as well as the growing body of evidence about social accountability, governance and development. According to this most recent GPSA ToA (as of September 2020) and Results Framework (July 2021), GPSA supports a new generation of collaborative social accountability processes, which engage citizens, communities, civil society groups, and public sector institutions in joint, iterative problem solving to tackle poverty and improve service delivery, sector governance, and accountability.10 As such, GPSA blends (i) flexible funding for civil society-led coalitions to work with public sector institutions to solve problems that local actors have prioritized with, and (ii) sustained non-financial support to meaningful engagements, including implementation support, capacity building, facilitation, and brokering. The aim is to contribute to country-level governance reforms and improved service delivery through more sustainable and effective civil society organizations that will support collaborative social accountability initiatives for addressing implementation gaps, beginning in the frontline. GPSA-supported coalitions develop capacities to engage meaningfully and collaboratively in policy-making, implementation, and service delivery processes.11 These core elements were also integrated into the structure of the Project, as indicated by the Project components. 32. A theory of change was constructed for this Implementation Completion Report to help structure the assessment of the original expected outcomes. The reconstructed Theory of Change presents the project components and their associated activities, outputs, outcomes and long-term outcomes. 10(Guerzovich and Poli 2020a) 11 The Global Partnership for Social Accountability: Theory of Action. 2020., available at https://documents1.worldbank.org/curated/en/425301607358292998/pdf/The-Global-Partnership-for-Social-Accountability- Theory-of-Action.pdf Page 14 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) Figure 1: Theory of Change Longer term impact may include: Increased efficiency of health system Intermediate results enumerated here to align with Project components Improved responsiveness of health system Improved health (levels & equity) Other systems Page 15 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) B. Significant Changes during Implementation Extension of project closing date 33. In quarter (Q) 4 2018, the Project’s closing date was extended for a period of ninety (90) days to February 18, 2019, after a detailed action plan with adjusted activities and implementation timelines was provided. This extension did not trigger the need for any policy waiver. The project did not have overdue audits at the time. The rationale for this extension was: (i) to compensate for initial delays of between seven to nine months in the project launch due to holdups in the signing of agreements with Regional Government entities; and (ii) to allow time for the completion of Knowledge Management products, including the impact study to show results in improving access to health services of poor and vulnerable populations, as well as the project’s contribution to improving social cohesion and self-care. II. OUTCOME A. Relevance of PDOs 34. At the time of project closure in February 2019, the PDO was relevant to the achievement of the country’s development goals and consistent with the World Bank’s strategy and guidance for the DRC . The project remained relevant to the CAS 2013-2016 and the 2018 Systematic Country Diagnostic (SCD). The 2018 SCD for the DRC, used to inform the development of the new Country Partnership Framework that is currently under development, stressed topics closely linked to the Project, such as building inclusive institutions and strengthening governance. The diagnostic encouraged voice, empowerment, and accountability interventions to help support inclusive institutions and organizations. Furthermore, to enhance the stock and quality of human capital and help build resilience and sustainable growth, the SCD recommended that policies should increase access to good-quality health services, while avoiding impoverishment because of catastrophic health expenditures. 35. The Project and its key objectives were also strongly aligned with more recently established government strategies and priorities for the health sector. The DRC PNDS for 2016-2020 aimed to increase the coverage and use of quality health care and services by the population with equity and financial protection. The PNDS emphasized the role of community health systems in ensuring the health of all citizens. B. Achievement of PDOs (Efficacy) 36. The PDO of improving access to healthcare services was vaguely worded regarding the types of access to be addressed. As explicitly addressed in the Results Framework, reduced socio-economic barriers to accessing care for vulnerable groups was one type of access to be explicitly addressed by HFCs through financial and social interventions implemented through component three. Discussion of such outcomes will be supplemented by other types of access addressed through project activities, such as availability of appropriate services through service improvements and, on the demand-side, awareness of services in the community, including information Page 16 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) on availability, benefits and importance, and how to access. While geographic access is not explicitly addressed in the Project components, it may also be addressed by HFCs through matching grants or other initiatives. 37. Three PDO-level indicators and ten intermediate result indicators were agreed upon to monitor and evaluate the achievement of the two PDOs. The indicators identified as PDO-level indicators provide some insight into the achievement of PDOs, but they do not necessarily provide a specific or complete account of all outcomes promised in the PDO statement. As such, the achievement of PDO-level and intermediate outcomes (reflecting component outcomes) are discussed using available project data and information from project documents, reports, and evaluations and is not limited to Results Framework indicators. Given significant M&E framework and data limitations, indicators and outcomes are discussed together and relative to each intermediate outcome, followed by a final discussion of achievement of each of the PDOs. The report draws largely on qualitative data in addition to the project results framework. Intermediate Result 1 (Component 1): Reinforced health facility committee system 38. The Project sought to reinforce the HFC system by building capacity of CODESAs and stakeholders to improve the functioning and effectiveness of CODESAs, including improved use of the CODESA system by members and healthcare providers. Key elements included training and retraining (roles, responsibilities, and skills), establishment of a supervision and support system and contracting mechanism, and dissemination of social accountability tools. By reinforcing organizational structures and processes, this component aimed to improve the responsiveness of health services, directly and indirectly, and ultimately for improvements in service access and quality over time. As summarized by the theory of change, outcomes of component one activities (creation of structures and processes) supported implementation and contributed to outcomes of components two and three. 39. By Project closing, records for Intermediate Results Indicator (IRI)-1 (1 (number of HCCs with members who know their roles, according to pre- and post-training tests) indicated successful delivery of training and initial adoption of roles and responsibilities, and records for IRI-2 (HCCs which are regularly supervised, monitored and evaluated by Health Zone (HZ) teams; 79 HCCs at baseline) indicated adequate and regular supervision, monitoring and evaluation by HZ teams. For IRI-1, the Project trained 105 HCCs in the first year and 140 HCCs in the second year, as a result of significant funding delays (described further in Section III). However, assuming 185 HCCs was the correct target number of HFCs, targets for both indicators were eventually achieved in year 3 (originally planned for year one). According to the final evaluation, all surveyed health facilities (164 chief nurses) had a CODESA and recalled a recently completed training of CODESA members (within the last two years of the survey and Project). It is important to note that four measures were proposed in the Results Framework for assessing IR-1, but only two were reported on. 40. The final evaluation indicates that the project improved CODESA operations. Specifically, the frequency of CODESA involvement in RECO and HF management activities, internal and external engagement, as well as the establishment of CODESAs at the health zonal level in South Kivu. ï‚· CODESA involvement in RECO and HF management activities: RECO activities were executed more often than HF management activities, differences existed by province, and greater activity in both areas may have Page 17 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) positioned CODESAs as more effective mechanisms for social accountability, without necessary trade-offs for CODESA members. CODESAs in South Kivu were more often active in all core activities. The results may indicate that frequency of activities may be associated with level of CODESA decision rights, as demonstrated in the case of South Kivu versus Kongo Central. Notably, CODESAs in South Kivu had more decision rights and were more active in both RECO (community-focused) and HF management activities. Interviews and this data indicate that RECO and HF activities complemented each other, and that there may not be a necessary trade-off for members becoming more involved in both HF management (linked to activities for social accountability) and RECO activities across contexts. As such, CODESAs in South Kivu appear to have been better positioned to collect information from the community, meet HF staff, and jointly make decisions (decision rights) on HF management, ultimately for social accountability and improved outcomes. While many CODESAs in Kongo Central appeared to be less involved in HF management, this was not necessarily because they had devoted more time and resources to RECO activities but was likely due to the lack of decision rights described previously. ï‚· Internal CODESA engagement: The final evaluation also sought to establish levels of ‘internal’ CODESA engagement from the 157 CODESA presidents. Specifically, the evaluation aimed to assess the frequency of meetings between CODESA members, and of CODESA meetings with HF staff, in each of the three contexts. No major differences were found, although results suggested that CODESA members in South Kivu met with each other more often than in Kongo Central, and that meetings between CODESA and HF staff were held more often (as a core operation of the CODESA) than meetings between CODESA members (mostly monthly) across contexts. These meetings were likely used for information sharing and collective decision making in most CODESAs of South Kivu and some of Kongo Central, as previously described (insufficient information was available on outcomes of these meetings). ï‚· External CODESA engagement: The evaluation also sought to establish levels of ‘external’ CODESA engagement with other parties. The evaluation assessed the number of CODESA meetings with RECOs, the community, local leaders, and high-level leaders (above village-level). The concept of meetings was applied in the broad sense as, for example, CBO reports suggested that most meetings with community constituted a home visit or informal gathering (comprehensive and formalized general meetings are rare). According to responses from CODESA presidents, CODESAs interacted with RECOs in their official capacity once a month, on average. Interestingly, meetings with the community and local leaders appeared to be more frequent in ‘new’ intervention areas of South Kivu and Kongo Central, which may serve to highlight how CODESAs utilized relationships with local leaders to support CODESA activities. As might be expected, meetings with high-level leaders were more frequent in South Kivu than in Kongo Central. Qualitative interviews suggested that interactions with leaders were primarily led by the HF president. A similar overall picture of relative engagements among CODESAs was given by chief nurses, except that chief nurse reported more frequent CODESA meetings than the CODESA presidents themselves, particularly in meeting RECOs and the community. ï‚· Establishment of Health Zone CODESAs in South Kivu. HZ CODESAs comprised representatives from each CODESA in a health zone (a committee of committees) and were supported and sometimes set up by Cordaid. HZ CODESAs were not originally planned as part of the Project but were expanded upon and provided an additional mechanism for upward accountability (uniting CODESA interests across catchment Page 18 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) areas), influence, and sharing of experiences. HZ CODESAs also served as useful intermediaries for monitoring CODESA activities in health zones and maintaining support to CODESAs in unreachable areas during the Project. The setup, use, and activities of HZ CODESAs are described further in Section III. Intermediate Result 2 (Component 2): Re-appropriation of the health facility through community actions and projects. 41. As part of component 2, the Project sought to empower CODESAs to improve health services (including access and quality) and to address issues prioritized by the community served. Key elements included (i) use and application of social accountability tools for improved decision rights, information flows and, ultimately, social accountability; (ii) financial support, including issuance of CODESA subsidies, and setting up and issuance of matching grants for community micro-projects; (iii) non-financial support for meaningful engagements; and (iv) hotlines to include service users in most remote areas. As previously outlined, the implementation and impact of component one activities are key for understanding the level of achievement the second intermediate result. Through collaborative trust-building activities, including community micro-projects, CSC processes, component 2 also aimed to promote awareness, use, and trust of CODESAs and health services in the community. Positive demand-side outcomes, such as improved community awareness intended to lead to access and quality gains, and ultimately promote utilization of services. Additionally, mutual associations between the second and third intermediate results are important, as the structures (e.g., matching grant mechanisms, social commissions), activities (e.g. non-financial support, inclusion activities, CSCs, planning), and organization (e.g. decision rights and information flows from effective use of social accountability tools) required may be described as synergistic for achieving outcomes across both components. As such, component 3 may be viewed as an extension of component 2 (empowering CODESAs) with an appropriately specific focus on ensuring the needs of vulnerable groups were integrated into the CODESA approach. Accordingly, intermediate result 2 and 3 are associated with improved responsiveness of health services and increased social and financial risk protection in the community, ultimately for improved quality and access of services. 42. The Project was to ensure a new Terms of Reference (new guidelines and framework) was available to HFCs by the second year (IRI-3), following reflection and drafting in the first year. It is unclear when this result was achieved, although it is assumed this was completed on time by Cordaid based on reporting on PDOI-2 (social contracts were signed from year one). According to IRI-4, by the second half of year two, 95 of 185 targeted HCCs had completed at least one CSC process following significant Project delays (79 assumed at baseline given ‘old’ intervention areas). The extent to which this measure represents restored trust is questionable. IRI-5 was not reported on, except that 76 activities to improve health services were reported for year 2, despite that matching grants had not yet been fully implemented. However, based on the final evaluation, the Project exceeded its total target of 150 matching grants: 148 matching grants were approved in South Kivu and 81 matching grants in Kongo Central. It is important to note that while three indicators and measures were established in the project Results Framework for measuring progress toward the second intermediate result, only two were reported on (once) in project reports. 43. The project’s final evaluation determined that in South Kivu, CODESAs were described by Project evaluators as catalysts for collective community action. This was largely due to the CODESA’s active planning at the HF and in the community. For example, in some cases, CODESA members were seen as upskilled community organizers for distribution of insecticide treated bed nets, census activities, and community micro-projects. Page 19 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) Project evaluators described the catalytic role of CODESAs in facilitating other external projects with aid actors (e.g., UNICEF), facilitating new local community activities and projects as the nexus between the community and local institutions, and helping to train and share lessons with local organizations for other community-level projects. CODESAs were trusted by other key local actors such as churches, leaders, and local media (e.g., community radio). This was unsurprising given the maturity of CODESAs, Cordaid relationships, community participation, and the aid environment in South Kivu. Local leaders were key for unlocking opportunities for CODESAs and access to the community through promotion of local communications and information sharing, for example. From interviews, CODESAs were commonly described by HZ management teams as changing the local narrative by promoting local solutions and self-sufficiency in a context where aid-dependency has been a major concern for local officials. CODESAs in Kongo Central showed similar signs of progress toward this vision, as the provincial chief-doctor in Kongo Central explained, “CODESAs have understood that local initiatives must come from themselves – they do not have to wait [for someone] to bring them something. They, themselves, can have small initiatives.â€? 44. While the Project sought to promote self-care and resilience among communities, it did not intend for CODESA members to fund CODESA activities from their own pockets. Unexpectedly, the final report indicates that many CODESAs required financial contributions from their members – approximately USD 20 every 6 months, for example. Members of CODESAs and HZ CODESAs were encouraged to make contributions for projects after they were elected. Qualitative interviews indicated that this was both a matter of principle (responsibility and commitment) and practicality. Members of CODESAs in Kongo Central likened these financial contributions to pre-existing practices in Bantu societies such as community savings groups or likemba, which represent widespread forms of pooling for community mobilization. Intermediate Result 3 (Component 3): Integration of the poorest and most vulnerable fringes of the population in the health facility committee and decisions on healthcare services. 45. Through component three, the Project sought to integrate the poorest and most vulnerable fringes of the population into the CODESA and decisions on health services by empowering CODESAs and to improve access and quality of services for vulnerable groups (VGs). Key elements included training CODESAs to identify and promote inclusion of VGs (including creating and updating VG lists); implementation of a quota system for participation and election of VGs in CODESAs; support and training to representatives of VGs; and pilot social and financial interventions to promote access and inclusion of VGs. As highlighted by this report’s theory of change, outcomes of IR-3 are linked to outcomes of components one and two, as new structures and processes from preceding components (e.g., matching grant mechanisms, contracting mechanism, initial training on roles and responsibilities, effective application of social accountability tools) either promoted effectiveness for achieving or were required for achieving IR-3. As such, IR-3 may be considered an extension of IR-2, with greater focus of activities and outcomes on VGs. Through a combination of access, inclusion, and collective planning activities, IR-3 was expected to directly increase social and financial risk protection in the community and improve responsiveness of health services to VG needs, ultimately for improvements in access and quality of health services. Page 20 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) Intermediate Result 4 (Component 4): Knowledge and learning – health facility committees are fully integrated into the health system and beyond. 46. The GPSA/CODESA approach was designed to be and perceived by many stakeholders as being appropriate to increase the resilience of communities. Given the socioeconomic and political situation in the DRC during Project implementation, the Project was frequently perceived as an appropriate response for mitigating social and financial risks, ensuring support to communities, and encouraging ownership of local solutions and self- care. 47. Cordaid reported that structured and organized CODESAs had become trusted bodies for the population, providers, and managers of the local health system. The final evaluation qualitatively identified key differences in perceptions of the centrality of the CODESA in the community, as indicated by perceptions of its role and importance among the community. 48. In Kongo Central, community members, chief nurses, and even CODESA members interviewed were undecided about the place of the CODESA in the local community. For example, some gave it a relatively central place while others treated it as a peripheral actor. Many community members interviewed in Kongo Central demonstrated limited awareness and understanding of the CODESA (e.g., “people we go to when we have a health problemâ€?), and a vague understanding of who RECOs were (known as papa or mama bonsiga in Lingala). These interviewees did not relate the role of CODESAs to HF management. A key factor in rural areas across contexts was that many CODESA members were already members of, or affiliated with, other key organisations. While this provided the CODESA with a strong local network, actions of individual CODESA members were not always associated with the CODESA by the community. Chief nurses also noted that, for instance, local mayors (bourgmestres) are CODESA members – but they were seen as mayors, not as a CODESA member. 49. In Kinshasa, it is unclear whether CODESAs were perceived by the community as central actors in lieu of RECOs and local leaders. This is largely because the two CODESAs effectively utilized local relationships of RECOs and with local leaders to address local issues and support their activities (local leaders helped to identify vulnerable people, for example). Despite the short project duration of only three months, focus groups and interviews suggested that the CODESA (and RECO) model would be valued and effective in Kinshasa, possibly due to the number and concentration of motivated, educated, multisectoral, or socially active people, as suggested by a CODESA president in an urban area of Uvira, South Kivu. “Since initiating community participation, we are really seeing a big change. It's dynamic because we do not only work with health workers, it is multi-sectoral: there are teachers and people from different backgrounds involved. We are really very close to these people, more than before. It all started with the RECO and CODESA training.â€? – Chief nurse, Kinshasa 50. The affiliation and support of Cordaid and health authorities in favour of CODESAs promoted their legitimacy and confidence among their members, as well as opportunities for inclusion in community strategies of other aid actors. A chief-nurse in Katana health zone explained how CODESAs were previously viewed as burdensome, Page 21 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) but that they have gradually become more independent with financial and non-financial support and have their own micro-projects. The project team also stressed that the Project had brought the provincial and health zone authorities closer to the CODESA (first annual report). Reinforced CODESAs may also be utilized in community strategies of other aid actors, such as in the case of the large hygiene promotion campaign “Villages et Ecoles Assainisâ€? (healthy villages and schools) led by UNICEF. Accordingly, chief doctors in South Kivu described the CODESA as key for achieving inter-sectoral initiatives. 51. A key impact for the Project was whether a meaningful and impactful coalition of actors had been constituted at the provincial level in support of sustaining the model. While national dynamics described earlier in Section II are important, health policy is implemented at the provincial level where there is opportunity to influence behavior more directly and outcomes at the community level. In South Kivu, Cordaid benefitted from pre- existing and strong relationships with the provincial health authority from CSF/CODESA and CODESA II projects and had harnessed and promoted lessons from those projects. In Kinshasa and in Kongo Central, the Project was less noticed by key policymakers and authorities. 52. In Kongo Central and Kinshasa, there was less evidence of a provincial coalition of actors in favour of the CODESA model. The Project engaged with regulators and interviews suggested it was well understood and supported, but there was little evidence that a wider coalition of provincial-level actors was set in motion. Actors not directly involved in the Project were typically not aware of it. Relatedly, AAP Kongo Central (which usually handles PBF) was contracted to run the Project in Kongo Central and core Project activities took place in just 1.5 years. As such, Kongo Central was unable to directly benefit from Cordaid expertise, resources, communications and relationships in the way South Kivu was able to, and the case for CODESAs in Kongo Central was not expanded upon as it was in South Kivu by Cordaid 53. There was evidence that a positive dynamic, in the form of a virtuous circle, was emerging in South Kivu. In South Kivu, the Provincial MSP actively engaged with different partners on opportunities presented by CODESAs and focused on disseminating information and implementing new rules for community participation (established with Cordaid). The case of South Kivu was framed by evaluators as a successful Project collaboration with local authorities. Guerzovich, Mukorombindo, and Eyakuze (2017b) stressed the importance of providing local authorities with respect, resources, but also training, mentoring and technical assistance for social accountability programs. From interviews in South Kivu, evaluators suggested the Project may have nurtured capacity enhancing processes for community participation in health (Guerzovich, Mukorombindo, and Eyakuze 2017a), built on long- standing relationships between Cordaid and the Ministry of Health of South Kivu. 54. With the support of the provincial health authority in South Kivu, key international aid actors (GIZ, UNICEF, USAID, International Rescue Committee (IRC)) reportedly adopted approaches from the Project, often having witnessed them in practice. In addition to the support of health authorities and longer-standing support for CODESAs in the DRC from Cordaid and the International Rescue Committee, CODESAs have received support from a wide range of actors since the Project started, including Cordaid (in North Kivu), GIZ, UNICEF, and USAID. In 2019, USAID began a new project in the peri-urban area of Kadutu, near Bukavu, with its local implementing partner, BDOM. Cordaid has a long-standing relationship with BDOM and has shared ideas and lessons learned from its projects with CODESAs. UNICEF recently adapted its approach to embrace principles of community accountability following field visits and meetings with Cordaid. The International Rescue Committee has Page 22 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) supported CODESAs in other areas of South Kivu and has adopted tools developed by the Project. Project evaluators described the long-term efforts and relationships of the provincial MSP, Cordaid and other actors as contributing to this coalition of support, where organizations were seeking to learn from each other rather than compete. At the health zone level, the Project received increasing demand for CODESA support, as other health zone officials had requested the Project be expanded to their areas (second annual report), and individual CODESAs had reported ad hoc collaborations with the IRC and World Vision. 55. In the Kivus, rising interest in the GPSA/CODESA approach was partly due to the number and concentration of aid organizations (aid hubs), both within and across sectors (variety of potential applications). Aid actors are known to cooperate given they typically work in different sectors or regions (non-competing). However, as evaluators highlighted, there is risk that with more actors working on and with CODESAs, imbalances in workload, resourcing, and funding may occur between CODESAs. Coordination will therefore continue to be essential to ensure optimal allocation and utilization of resources, for which both international aid clusters and health authorities are responsible. C. Efficiency 56. Given the small scale of the project, an efficiency analysis was not completed during the preparation stage, and was not conducted at the completion stage. D. Other Outcomes and Impacts Gender 57. The project had a strong focus on gender equity. CODESAs guidelines mandated that at least 30 percent of women were members, although it is unclear how inclusive CODESAs were of women in practice due to lack of data reporting on this metric. This mandate was particularly important as local customs and traditions of some communities did not encourage the presence of women in the same groups as men. However, the final evaluation characterized the requirement as insufficient to overcome barriers to women’s participation. Some CODESAs reportedly elected women as the Chair, a position that had once been reserved for men only, during the first year of the Project. 58. In the first two years, project activities focused on gender mainstreaming. According to the second annual report, women were better able to express themselves, some as decision makers, and through the community mapping process. Some CODESAs developed new awareness-raising strategies through songs at events in villages which highlighted the benefits of women’s participation in the development and improvement of health of all. The Project increasingly encouraged micro-projects initiated by women. For example, according to the third annual report, during the socio-economic crisis in 2017, women were viewed as hubs of the local economy in most households, particularly given their involvement in informal trading in cities and rural work in villages. As such, women were increasingly solicited for projects requiring community contributions. During this time, the Project increasingly encouraged micro-projects initiated by women, although the extent of this was not clear in the project Results Framework. For example, in the Kitona military Health Zone, a CODESA with a female Chair Page 23 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) reportedly initiated a small goat breeding project. Given the shortage of essential medicines for children in the health centre, some goats were sold to fund the health centre’s purchase of a small package of drugs. 59. A main lesson in the project’s final evaluation was that CODESAs could become more inclusive of women through intensified programmatic focus. It was recommended that organisations and authorities monitoring and supporting CODESAs push for the inclusion of more women in CODESAs and create fora where problems can be discussed from the perspective of women, and that national and provincial authorities formally amend CODESA regulations to ensure women were included in CODESAs and decision-making structures within their community. Table 3: Summary of key findings for project outcomes Page 24 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) Outcomes: summary of key findings Gender ï‚· Respect for gender was emphasized during training in all CODESAs including the need for CODESAs to have at least 30% women as members, although it is unclear how inclusive CODESAs were of women. According to the final evaluation, CODESAs could still become more inclusive of women through intensified programmatic focus on women. ï‚· According to Cordaid, Project activities focused on gender mainstreaming and promoted gender equality and voice of women, and the Project increasingly encouraged micro-project initiated or led by women. Perceptions of the Project and CODESAs ï‚· Qualitative interviews suggested the Project had instilled a multi-sector dynamic to community participation (motivating participatory community response to issues, and direct engagements for helping to provide services such as water). ï‚· The Project approach was perceived by many stakeholders as appropriate for increasing the resilience of communities in a tense socioeconomic and political context (mitigating financial risks, ensuring support to communities, and encouraging ownership of local solutions and self-care). ï‚· Affiliation and support of Cordaid and health authorities in favour of CODESAs had promoted the legitimacy of CODESAs and confidence of their members , as well as opportunities for inclusion in OVERALL Overall community strategies of other aid actors at HZ and CODESA levels. ï‚· Views of the primary advantages of CODESAs varied across levels of the health system . ï‚· According to Cordaid, community assessment processes were reported to have instilled a culture of accountability among stakeholders and beneficiaries. CODESA Operations ï‚· Overall, many CODESAs in the Project were able to leverage relationships with local leaders and other local organisations as opposed to competing with them. ï‚· CODESAs often required financial contributions from members as a matter of principle (responsibility and commitment) and practicality (resource requirements). ï‚· Organizationally, differences in how CODESA members were involved in and how information was utilized for decision-making contributed to existing power dynamics in health facilities and may have led to differences in effectiveness of CODESAs for meeting community needs and improving responsiveness of health services. In particular, information flows and decision rights were critical for ensuring CODESAs could provide an effective accountability interface. Two different HF-CODESA scenarios were identified in HF management: top-down and RECO-oriented, and collaborative in HF co- management. The need for information sharing and consequences may have been understood, accepted, and adopted differently by HFs and HZ teams in each of these scenarios. ï‚· Training on responsibilities, and effective and use of social accountability tools, such as the TOR and CSCs, were key for ensuring CODESA decision rights and productive information flows and for addressing Page 25 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) initial resistance from HF staff. When comparing contexts, evidence indicates that CODESA decision rights and information flows may improve over time to enable HF co-management (e.g., ‘old’ intervention areas). ï‚· Although the Project and CODESAs faced little opposition, the Project highlighted how initial challenges with HF staff may be overcome through training on refreshed CODESA roles, responsibilities, and tools. ï‚· High levels of CODESA engagement with HF decision making were self-reported in all contexts, particularly relating to infrastructure and buying drugs and equipment. Chief nurses significantly disagreed with CODESA presidents across contexts on the extent of CODESA involvement in decisions for HR management and general planning. ï‚· RECO activities were executed most frequently versus HF management activities, and there did not OVERALL appear to be a necessary trade-off in time spent between both: greater activity in both areas may have positioned CODESAs as more effective mechanisms for social accountability. Unplanned outcomes ï‚· All four Project Managers who had left the Project reported utilizing the CODESA approach and ideas of social accountability in other large-scale programs. ï‚· HZ CODESAs (unplanned) provided an additional mechanism for upward accountability, influence, and sharing of experiences. HZ CODESAs were also utilized to help monitor and support CODESAs in unreachable areas. There may be opportunity to strengthen accountability mechanisms associated with HZ CODESAs, to improve local health governance and commitment of actors to CODESAs across levels. ï‚· Cordaid and CODESAs benefitted from pre-existing and strong relationships with the provincial health authority and had harnessed and promoted lessons from previous CODESA projects. Relatedly, CODESAs in ‘old’ intervention areas engaged more frequently with higher -level leaders (above village level). ï‚· Evaluation research indicated greater decision rights among CODESA members and more productive use of information and data for collective decision making between CODESAs and HFs (versus Kongo Central). ï‚· To most Project stakeholders in South Kivu, the CODESA reflected a broader vehicle for collective action at the HF and beyond, and not simply a group of RECOs as in Kongo Central and Kinshasa. SOUTH KIVU ï‚· CODESAs were perceived by Project stakeholders as catalysts for collective action in communities (many CODESAs were viewed as having a more central role and greater legitimacy in the community). ï‚· The Project saw the emergence of a strong coalition of actors at the provincial level (strong commitment and engagement of provincial health authority in favour of the CODESA; integration of the CODESA in health planning; interest of NGOs and civil society in the CODESA ‘model’, including its application across sectors, and new CODESA-inspired initiatives). ï‚· Increasing interest in the GPSA/CODESA approach in the Kivus was partly due to the number and concentration of aid organizations (aid hubs), both within and across sectors. These organizations looked to learn from each other rather than compete. ï‚· With the support of the provincial health authority in South Kivu, key international aid actors (GIZ, UNICEF, USAID, IRC) reportedly adopted approaches from the Project, often having witnessed them in Page 26 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) the field. There is risk that with more actors working on and with CODESAs, imbalances in workload, resourcing, and funding may occur between CODESAs and will require coordinated approaches. ï‚· Many differences observed between South Kivu and Kongo Central may reflect differences in the maturity of CODESAs (organization, capabilities, community relationships, and activities), and in Cordaid relationships with health authorities. ï‚· Evaluation research indicated fewer decision rights among CODESA members for HF management (on average): in many cases, CODESA members were used to collect and transmit information to, and carry out instructions from, the chief nurse. Relatedly, phone surveys indicated that CODESAs appeared to be less involved with HF management than those in South Kivu. CODESAs also reported more frequent engagement with local leaders than CODESAs in South Kivu (leveraging local relationships for CODESA activities, for example), but less engagement with higher-level leaders (above village level). Insufficient CODESA decision rights and information flows may be addressed through improved use of social accountability tools such as TOR and CSCs. ï‚· CODESAs were largely perceived as a group of community health workers (RECOs), mostly preoccupied KONGO CENTRAL with health promotion (largely true according to survey data) but not necessarily with HF management. ï‚· Perceptions of primary advantages of the CODESA varied across levels . For most mid-level professionals interviewed (HZ officials), CODESAs were largely RECOs and the primary advantage of the CODESA was that it was closer to the HF relative to the existing RECO model (for health promotion and bringing patients to the HF). Provincial-level officials and CODESAs, however, believed the CODESA’s ability to investigate and communicate local problems and generate own local solutions was key. ï‚· Interviewed community members, chief nurses, and even CODESA members were undecided about the place of the CODESA in the local community (potentially due to lack of community awareness of CODESA roles and responsibilities; affiliation of CODESA activities with other community actors supporting CODESAs; and affiliation of CODESA members with other organizations, particularly in rural contexts). ï‚· There was less evidence of a provincial coalition of actors in favour of the CODESA model and the Project was less noticed by key policymakers and authorities (largely positive attitude of provincial health authority toward CODESAs; doubts about capacity of the CODESA to do meaningful planning; no specific engagement of civil society with the Project and CODESAs). Kongo Central was unable to directly benefit from Cordaid expertise, resources, communications, and relationships in the same way as South Kivu, and the case for CODESAs was not expanded upon by Cordaid as it was in South Kivu. Page 27 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) ï‚· In addition to lack of previous support from Cordaid (as in the case of Kongo Central), CODESA-related outcomes in Kinshasa were limited by the scope and duration (3 months) of implementation. ï‚· CODESAs were largely perceived as a group of community health workers (RECOs) , mostly preoccupied with health promotion but not necessarily with HF management. KINSHASA ï‚· CODESAs (and RECOs) effectively utilized relationships within communities, although it is unclear whether CODESAs were perceived by the community as central actors in lieu of RECOs and local leaders. ï‚· Perceptions of primary advantages of the CODESA varied across levels (similar to Kongo Central). ï‚· Focus groups and interviews suggested the CODESA (and RECO) model would be locally valued and effective in the urban context of Kinshasa, possibly due to higher concentrations of relevant local actors. Page 28 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) Outcomes: summary of key findings Gender ï‚· Respect for gender was emphasized during training in all CODESAs including the need for CODESAs to have at least 30% women as members, although it is unclear how inclusive CODESAs were of women . According to the final evaluation, CODESAs could still become more inclusive of women through intensified programmatic focus on women. ï‚· According to Cordaid, Project activities focused on gender mainstreaming and promoted gender equality and voice of women, and the Project increasingly encouraged micro-project initiated or led by women. Perceptions of the Project and CODESAs ï‚· Qualitative interviews suggested the Project had instilled a multi-sector dynamic to community participation (motivating participatory community response to issues, and direct engagements for helping to provide services such as water). ï‚· The Project approach was perceived by many stakeholders as appropriate for increasing the resilience of communities in a tense socioeconomic and political context (mitigating financial risks, ensuring support to communities, and encouraging ownership of local solutions and self-care). ï‚· Affiliation and support of Cordaid and health authorities in favour of CODESAs had promoted the legitimacy of CODESAs and confidence of their members , as well as opportunities for inclusion in OVERALL Overall community strategies of other aid actors at HZ and CODESA levels. ï‚· Views of the primary advantages of CODESAs varied across levels of the health system . ï‚· According to Cordaid, community assessment processes were reported to have instilled a culture of accountability among stakeholders and beneficiaries. CODESA Operations ï‚· Overall, many CODESAs in the Project were able to leverage relationships with local leaders and other local organisations as opposed to competing with them. ï‚· CODESAs often required financial contributions from members as a matter of principle (responsibility and commitment) and practicality (resource requirements). ï‚· Organizationally, differences in how CODESA members were involved in and how information was utilized for decision-making contributed to existing power dynamics in health facilities and may have led to differences in effectiveness of CODESAs for meeting community needs and improving responsiveness of health services. In particular, information flows and decision rights were critical for ensuring CODESAs could provide an effective accountability interface. Two different HF-CODESA scenarios were identified in HF management: top-down and RECO-oriented, and collaborative in HF co- management. The need for information sharing and consequences may have been understood, accepted, and adopted differently by HFs and HZ teams in each of these scenarios. ï‚· Training on responsibilities, and effective and use of social accountability tools, such as the TOR and CSCs, were key for ensuring CODESA decision rights and productive information flows and for addressing Page 29 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) initial resistance from HF staff. When comparing contexts, evidence indicates that CODESA decision rights and information flows may improve over time to enable HF co-management (e.g., ‘old’ intervention areas). ï‚· Although the Project and CODESAs faced little opposition, the Project highlighted how initial challenges with HF staff may be overcome through training on refreshed CODESA roles, responsibilities, and tools. ï‚· High levels of CODESA engagement with HF decision making were self-reported in all contexts, particularly relating to infrastructure and buying drugs and equipment. Chief nurses significantly disagreed with CODESA presidents across contexts on the extent of CODESA involvement in decisions for HR management and general planning. ï‚· RECO activities were executed most frequently versus HF management activities, and there did not OVERALL appear to be a necessary trade-off in time spent between both: greater activity in both areas may have positioned CODESAs as more effective mechanisms for social accountability. Unplanned outcomes ï‚· All four Project Managers who had left the Project reported utilizing the CODESA approach and ideas of social accountability in other large-scale programs. ï‚· HZ CODESAs (unplanned) provided an additional mechanism for upward accountability, influence, and sharing of experiences. HZ CODESAs were also utilized to help monitor and support CODESAs in unreachable areas. There may be opportunity to strengthen accountability mechanisms associated with HZ CODESAs, to improve local health governance and commitment of actors to CODESAs across levels. ï‚· Cordaid and CODESAs benefitted from pre-existing and strong relationships with the provincial health authority and had harnessed and promoted lessons from previous CODESA projects. Relatedly, CODESAs in ‘old’ intervention areas engaged more frequently with higher -level leaders (above village level). ï‚· Evaluation research indicated greater decision rights among CODESA members and more productive use of information and data for collective decision making between CODESAs and HFs (versus Kongo Central). ï‚· To most Project stakeholders in South Kivu, the CODESA reflected a broader vehicle for collective action at the HF and beyond, and not simply a group of RECOs as in Kongo Central and Kinshasa. SOUTH KIVU ï‚· CODESAs were perceived by Project stakeholders as catalysts for collective action in communities (many CODESAs were viewed as having a more central role and greater legitimacy in the community). ï‚· The Project saw the emergence of a strong coalition of actors at the provincial level (strong commitment and engagement of provincial health authority in favour of the CODESA; integration of the CODESA in health planning; interest of NGOs and civil society in the CODESA ‘model’, including its application across sectors, and new CODESA-inspired initiatives). ï‚· Increasing interest in the GPSA/CODESA approach in the Kivus was partly due to the number and concentration of aid organizations (aid hubs), both within and across sectors. These organizations looked to learn from each other rather than compete. ï‚· With the support of the provincial health authority in South Kivu, key international aid actors (GIZ, UNICEF, USAID, IRC) reportedly adopted approaches from the Project, often having witnessed them in Page 30 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) the field. There is risk that with more actors working on and with CODESAs, imbalances in workload, resourcing, and funding may occur between CODESAs and will require coordinated approaches. ï‚· Many differences observed between South Kivu and Kongo Central may reflect differences in the maturity of CODESAs (organization, capabilities, community relationships, and activities), and in Cordaid relationships with health authorities. ï‚· Evaluation research indicated fewer decision rights among CODESA members for HF management (on average): in many cases, CODESA members were used to collect and transmit information to, and carry out instructions from, the chief nurse. Relatedly, phone surveys indicated that CODESAs appeared to be less involved with HF management than those in South Kivu. CODESAs also reported more frequent engagement with local leaders than CODESAs in South Kivu (leveraging local relationships for CODESA activities, for example), but less engagement with higher-level leaders (above village level). Insufficient CODESA decision rights and information flows may be addressed through improved use of social accountability tools such as TOR and CSCs. ï‚· CODESAs were largely perceived as a group of community health workers (RECOs), mostly preoccupied KONGO CENTRAL with health promotion (largely true according to survey data) but not necessarily with HF management. ï‚· Perceptions of primary advantages of the CODESA varied across levels . For most mid-level professionals interviewed (HZ officials), CODESAs were largely RECOs and the primary advantage of the CODESA was that it was closer to the HF relative to the existing RECO model (for health promotion and bringing patients to the HF). Provincial-level officials and CODESAs, however, believed the CODESA’s ability to investigate and communicate local problems and generate own local solutions was key. ï‚· Interviewed community members, chief nurses, and even CODESA members were undecided about the place of the CODESA in the local community (potentially due to lack of community awareness of CODESA roles and responsibilities; affiliation of CODESA activities with other community actors supporting CODESAs; and affiliation of CODESA members with other organizations, particularly in rural contexts). ï‚· There was less evidence of a provincial coalition of actors in favour of the CODESA model and the Project was less noticed by key policymakers and authorities (largely positive attitude of provincial health authority toward CODESAs; doubts about capacity of the CODESA to do meaningful planning; no specific engagement of civil society with the Project and CODESAs). Kongo Central was unable to directly benefit from Cordaid expertise, resources, communications, and relationships in the same way as South Kivu, and the case for CODESAs was not expanded upon by Cordaid as it was in South Kivu. Page 31 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) ï‚· In addition to lack of previous support from Cordaid (as in the case of Kongo Central), CODESA-related outcomes in Kinshasa were limited by the scope and duration (3 months) of implementation. ï‚· CODESAs were largely perceived as a group of community health workers (RECOs) , mostly preoccupied with health promotion but not necessarily with HF management. KINSHASA ï‚· CODESAs (and RECOs) effectively utilized relationships within communities, although it is unclear whether CODESAs were perceived by the community as central actors in lieu of RECOs and local leaders. ï‚· Perceptions of primary advantages of the CODESA varied across levels (similar to Kongo Central). ï‚· Focus groups and interviews suggested the CODESA (and RECO) model would be locally valued and effective in the urban context of Kinshasa, possibly due to higher concentrations of relevant local actors. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. Key Factors during Preparation 60. Objectives for the Project were not clearly articulated and required examination of activities and the Results Framework to better understand expected outcomes and impacts. Project objectives were vaguely worded, including PDOs and intermediate outcomes. Most objectives were not specific or time-bound, and this impacted understanding and quality of reporting and results achieved. As a result, intended objectives were inferred from the expected measurable outcomes based on the operation’s key associated outcome targets and the description of intent in the legal agreement. Furthermore, reports and the evaluation indicated that the fourth component objective was not attainable within the project timeframe given capacity and levels of funding available, in addition to delays in funding for project implementation. 61. The Project followed a relatively complex yet flexible design. The design incorporated appropriately sequenced component activities, as well as a semi-structured, emergent approach to address third component objectives. However, a loosely formulated theory of change may have undermined understanding, monitoring and achievement of expected outcomes. The wording in component descriptions per the legal agreement provided flexibility for the Project to assess regional and community needs and to define activities more specifically as foreseen by the GPSA’s adaptive learning and management approach (see Box 1 below), such as those related to matching grants, subsidies, and social and financial interventions for VGs. Additionally, while the structure of components was relatively clear, the logic (including relationships and connections between component activities and intermediate outcomes and PDOs) and underlying assumptions were not clearly articulated in project documents, including the Results Framework, particularly given the absence of a clear theory of change. Relatedly, the project evaluation included analysis of the GPSA theory of change at the time, but this was not clearly linked to the Results Framework for the Project. As such, the project evaluation reframed core activities and intended outcomes in a way that obfuscated understanding of the original project structure and expected and realized outcomes as referred to in the Results Framework. Page 32 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) Box 1: GPSA’s adaptive learning and management approach The GPSA considers adaptive learning and management – conceived as an iterative process set within complex and dynamic socio-political contexts – as an important driver of collaborative social accountability. GPSA has found that an effective route toward supporting results is to broker multistakeholder coalitions composed of civil society, government, and private and international institutions that engage in collaborative social accountability processes using a problem-solving mindset. Integrating sectoral and governance interventions, as GPSA’s collaborative social accountability efforts do, opens new pathways to obtain scalability and sustain results. Social accountability is “collaborativeâ€? when citizens, civil society groups, and public sector institutions engage in joint, iterative problem solving to improve service delivery, sector governance, and accountability. This is opposed to confrontational, advocacy-based social accountability strategies that are based on the development of civil society’s countervailing power.12 GPSA’s adaptive programming approach cannot always be adequately measured through standardized criteria, as an attempt to formalize milestones and indicators at the start of the Project can risk derailing the iterative approach needed for success. Despite the above, this ICR sought to examine if adaptative approaches helped to solve the problems the Project targeted at the initial design phase, and whether the capacity of individuals, beneficiaries, and implementing partners improved under the Project. By carrying out interventions through a collective identification of tangible, locally relevant problems, collaborative social accountability mechanisms developed under the Project sought relevance, legitimacy and practicality. In line with the principles of adaptive programming, the Project favoured flexibility in its design, with a view towards scaling up support where the likelihood of sustainable reforms is greatest. GPSA projects adopt principles of adaptive programming as part of design, in line with the recognition that not every facet of the Project can be planned, and that implementing partners will inevitably face some level of uncertainty during implementation. Course-correction during project implementation is thus encouraged and rewarded based on effective supervision and implementation. For example, as information is gathered during implementation about what is happening and how it is measured, project indicators may subsequently need to be adapted to the reality on the ground. The measure of success is therefore based on the extent to which projects help implementing partners solve problems identified by local stakeholders using collaborative social accountability mechanisms.13 62. The Project selected appropriate stakeholders to engage and established an appropriate plan for collecting information and monitoring the achievement of outcomes. Specifically, the Project organized and aligned joint 12 Guerzovich, F. and Scommer, P. “Social Accountability and Open Government: Different Types of Collaborative Engagement.â€? Les Éditions de l'IMODEV. Vol. 7 (2018). Available at: . 13 Guerzovich, Maria F., Maria Poli, and Emilie Fokkelman. 2020. “The Learning Crisis and Its Solutions: Lessons from Social Accountability for Education.â€? Global Partnership for Social Accountability Note 13. World Bank, Washington, DC. Available at : . Page 33 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) monitoring missions between Cordaid, DPS, and health zone management teams, in addition to local CBOs when needed. As outlined in the first annual report, the involvement of different government health sector actors at different levels was key to increasing ownership of project activities among the community, but also in the health system. Data collection schedules and sources were, in large part, clearly outlined in the Results Framework. Further details on how monitoring was conducted and may have affected outcomes is provided in Key Factors during Implementation. 63. Retrospectively, the project evaluation and reports outlined three limitations related to project planning: underestimated budget allocations, underestimated resources required for more volatile areas of South Kivu, and underestimated (human, social, and financial) resources required for national-level engagement and advocacy. The second annual report14 explains how the project team had realized that some line items from the project budget had been underestimated, particularly for supporting interactions between HFCs and health authorities (e.g. HZ management teams). A proposal for a revised budget was submitted with the report that year. Further details are provided in Key Factors during Implementation. B. Key Factors during Implementation Factors subject to World Bank control 64. The initial delay in disbursement of funds significantly delayed program implementation and negatively impacted project partners and beneficiaries. The Project contract was signed by the World Bank and Cordaid on November 18, 2014, but Cordaid only received the first instalment of WB funding (USD 123,296.93) in June 2015. The Project adjusted the implementation timeline and some activities were shifted to the second year, including activities for community micro-projects. Critically, the delay in funding led to delays in contracts signed with health authorities (formalized terms of collaboration with government counterparts required for Project implementation), which in turn impacted the schedule for components and activities, including CODESA subsidies, matching grants, and supervision and support for HFCs. Detailed in the first annual report,15 the late disbursement meant that not all CODESAs (only 105 CODESAs in seven out of 11 health zones) had been activated within the first year. Late funding delayed the hiring of a project consultant and the subsequent development of baseline study questionnaires, which needed to be implemented at the same time as training. Consequently, for CODESAs was delayed. Only 26 of the 111 targeted health committees were reported as including representatives of the poorest groups (IRI-6) in the first year. In the first midterm report (2016), the Project noted poor continuity of activities in intervention areas and with project beneficiaries as a result of funding delays, which increased risk to development outcomes at the time. In multiple cases, issues of trust among CODESAs arose because delays in payment and resulting communication issues. For example, training teams started their work but was not able to promptly follow up on their promises of subsidies and matching grants. One health zone chief-doctor in Kongo Central stated: “We had problems with the CODESA. They called us jokers, and I began to avoid them. I tried to explain that the situation [the delay in funding and activities] did not depend on me and that the money would come later. This is hard to explain to people, and they did not trust me.â€? 14 GPSA Grantee Annual Progress Report. Reporting Period: November 2015 – November 2016. Date submitted: 31/01/2017 15 GPSA Grantee Annual Progress Report. Reporting Period: November 2014 – November 2015. Date submitted: 01/16/2016 Page 34 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) Furthermore, given elements of component 4 were contingent on implementation of other components, initial delays in funding meant that some activities, such as the recruitment of CSOs, were still in progress in Kongo Central when the extension for the Project was requested. 65. Despite the slow start, GPSA and the task team supervised and supported the Project team to adapt to the changing context and operational challenges, including funding issues and regional insecurity. In 2016, an accelerated implementation plan was developed for Kongo Central, where the Project experienced significant delays. Many year-1 targets were reached in year three, and activities that were planned for two to three years, such as subsidies and matching grants, were delivered in the last 1.5 years of the project. Similarly, piloting new social and financial interventions to support VGs began more than one year later than planned, with two direct consequences: i) the duration of the pilot was shortened, and ii) lessons from the pilot on integrating VGs were harder to integrate into the project. Some activities under the fourth component could not be carried out, largely due to insufficient funds and human and social resources for national-level advocacy, and the need to focus on volatile areas of South Kivu that required more time and resources than expected. Furthermore, flexibility was provided when the Project was eventually extended at no additional cost by three months to compensate for initial delays. 66. Cordaid reported strong relationships and regular contact between the World Bank task team and Cordaid Project Officer. In July 2015, the task team significantly contributed to the Cordaid team’s (in Bukavu) understanding of project tools. Experiences were shared with the World Bank team during the same visit, and more guidance and clarification was provided by the GPSA team virtually to support project implementation and help achieve expected objectives throughout the project. The GPSA blog also provided continuous access to information and other documents related to the Social Accountability approach used. Factors subject to control of implementing entities. 67. The Project was implemented in four different locations, each of which required different levels of engagement, resources, and expertise. In some locations, CODESAs had benefitted from Cordaid support prior to the project. These CODESAs ultimately had greater capacity, more participatory decision-making and greater operational effectiveness. Furthermore, community members were more aware of CODESAs and their role. The project evaluation found that Cordaid had greater resources and expertise in the original intervention areas of South Kivu. To help compensate for such structural differences, Cordaid’s Project Officer visited new project sites in to Kongo Central and Kinshasa to support activities and implementation, although this was during the last six months of the project. Project staff acknowledged that it was more difficult to expand into new provinces, as many positive outcomes in the original intervention areas were embedded into other community revitalization projects. For example, in South Kivu in particular, the PBF program that was implemented by Cordaid over decades and also supported by the World Bank contributed to increasing the support and resources of CODESAs. Page 35 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) 68. The Project had a high turnover of managers, which impacted project implementation. Over its implementation period, the project had four official managers and one interim manager.16 Some managers had limited experience with community health and were in charge of an already busy portfolio. These changes destabilized project operations and led to further delays due to time required to recruit and ramp up new leadership. Additional team changes included the relocation of a project advisor in Kongo Central to South Kivu. The project also contracted a local consultant to monitor and support activities in Kongo Central, regularly assisted by the project coordinator based in Kinshasa. 69. The project had significant delays in recruitment and operation of CSOs for cross-verification of CODESA achievements and in setting up a fully functioning mechanism for administering matching grants. Local associations (ASLOs, or CSOs in the context of the Project) were recruited for permanent cross-verification of CODESA achievements, including implementation of community micro-projects. However, the recruitment process for CSOs was still underway in early 2017. The operations of CSOs were further delayed, reportedly due to delays in funding in Q2 2017. It appears the delay in activation of CSOs also contributed to delays in implementation of matching grants, including establishing a fully functioning mechanism for administering grants. For example, although initial workshops launching social and financial interventions for VGs in the communities took place in June 2017, matching grants were issued in February 2018, which resulted in a shorter period for the impact study. Consequently, the delay in matching grants had negative implications for implementation of components and may have limited levels of achievement with respect to outcomes. 70. The financial support provided to the CODESAs (subsidies and matching grants), was implemented differently by health authorities in South Kivu and Kongo Central. This may have created a different set of incentives in each province. The frequency and ways of scoring (e.g., weighting) differed by province for both subsidies and matching grants. Specifically, in Kongo Central, matching grants were more easily obtained (multiple grant applications possible, and each CODESA in Kongo Central received one grant every semester during the Project – 81 in total) and more frequent, but less generous than those in South Kivu (pass/fail assessments for maximal amounts requested in many cases). In the similar case of subsidies, the model implemented in Kongo Central (based on more frequent, semesterly scoring) more closely reflects a form of PBF. Aside from budgetary constraints, it is not clear what led to these adaptations. 71. The Project introduced inclusive and participatory biannual and annual project reviews to collectively review, analyze, plan and adapt actions for improved results. The Project had multiple joint monitoring missions for Cordaid DRC, Cordaid The Hague, and DPS and ECZ representatives to improve the quality of project results. As such, different stakeholders across the health sector contributed to the achievement of project objectives. DPS maintained focus on regulation and development of approaches adapted to local contexts. HZ management teams and health facilities ensured close monitoring of CODESA activities. Local associations (CBOs) provided community-based verification of CODESA achievements and assisted with planning of community activities. CODESAs themselves were responsible for monitoring community needs. Local associations (ASLOs) were eventually recruited for permanent verification of CODESA achievements following delays. The Heads of South Kivu and Kongo Central DPS, heads of targeted health zones, selected ASLOs, and CODESAs were also trained in 16First project manager was November 2014 – December 2015; second project manager was December 2015 – August 2016; third (Interim) project manager was August 2016 – October 2016; fourth project manager was October 2016 – June 2018; fifth project manager was July 2018 – February 2019. Page 36 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) management of funds and reporting, planning and development of micro-projects (including use of community assessment maps), and multi-stakeholder decision-making processes. Cordaid was responsible for gathering project data and presenting on its experience at national and international levels (Ministry of Health, WHO Health Cluster, and donors) to encourage CODESA membership and scale up of the Project’s approach. 72. During 2017, performance contracts were signed at all levels and the Project introduced biannual and annual project reviews. The purpose of the reviews was to allow different stakeholders to review project strategies and results and make recommendations for each level of project implementation. According to the third annual report, this allowed stakeholders to learn from each other’s experiences to address their own challenges. For DPS, this provided opportunities to develop common understanding of indicators and how to correctly report on them, and to understand challenges and develop timely and appropriate approaches. These included community self-evaluation through ASLOs, the involvement of political and opinion leaders, and increased participation of the community in the process for decentralization. Additionally, during these reviews, the project team was able to understand and assess the need to adjust the community-share of matching grants for community microprojects when socio-economic conditions had worsened. 73. The CODESAs implemented at the health zone level were initially unplanned, but provided an additional mechanism for upward accountability, influence, and sharing of experiences. These higher level CODESAs served as useful intermediaries for monitoring those at the community level, and particularly in very difficult to reach areas. HZ CODESAs were comprised of representatives from each CODESA in a health zone, and were supported, and sometimes set up, by Cordaid. Relatively early in the Project, Cordaid decided to help set up such committees in South Kivu with the Ministry of Health, using different funding sources. This was inspired by encounters with the ComiteÌ? Territorial/Communal de DeÌ?veloppement (COCODEV: the Territorial/Communal Development Committee) in Kongo Central, which not only focuses on health issues but also unites interests of representatives from different catchment areas. In Kinshasa, similar entities exist and cover typical CODESA activities but also monitor CODESA activities in each health zone. According to the project evaluation, the Project benefitted from Health Zone CODESAs, as they provided opportunities for sharing experiences and approaches between CODESAs, which complemented the fourth component objective (knowledge and learning) and PDOs for service delivery improvements (access and quality). According to the third annual report, exchanges between CODESAs fostered competitiveness and incentive to improve performance among CODESAs. HZ CODESAs were also described as another upward mechanism for accountability and a key vehicle for lobbying health zone management teams and attracting new external partners to the health zone. In a tense security context, HZ CODESAs were also utilized by Cordaid to maintain support to CODESAs in unreachable areas. 74. The Project engaged less with national level stakeholders than expected. This was due to a combination of funding delays, insufficient funds and political changes. The CODESAs were not seen as a priority or a vehicle for change by some national-level actors. According to the project evaluation, officials at the national level were not aware of the project approach and had limited interest in CODESAs. This may be attributed to the Project not being able to sufficiently advocate and build coalitions at the national level, which required substantial human, social, and financial resources the Project could not commit. Consequently, national level advocacy was less formal than planned. While Cordaid was invited to contribute to the national level community health policy early into the GPSA/CODESA project, GPSA/CODESA could not organise meetings with national authorities in charge of community participation at the Ministry of Health (Particom, Division Participation Communautaire). The only other key forum in which GPSA/CODESA was presented was the national Health Cluster led by the Page 37 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) WHO. According to the evaluation, interviews with officials at the national level (e.g. PartiCom) suggested almost complete ignorance of the project at the central level of the Ministry of Health and a limited interest in the CODESAs which may be viewed as a past fad. However, the experience of GPSA/CODESA suggests that advocacy at the provincial-level is most beneficial. In Kinshasa and Kongo Central, the Project was not significantly recognized by key policymakers and the provincial health authority, whereas it had more recognition in South Kivu. Factors outside the control of implementing entities 75. The Project took place in a tense national socio-political environment. This may have contributed to implementation of delays, yet also may have increased interest in the GPSA/CODESA model and community self-care. The political and economic situation in the DRC deteriorated during 2017. The resulting crisis negatively impacted the country's development and social cohesion. Uncertainties and tensions surrounding the presidential election, which had been postponed multiple times, continued into 2018 and presented the project team with challenges to building strong relationships with higher-level officials. However, the situation reportedly highlighted the importance of democracy, accountability, and representativeness across communities, organizations and sectors and may, in fact, have strengthened local interest in GPSA/CODESA. The third annual report highlighted that citizens were increasingly trying to make their voices heard, network and join civil society groups to express their opinions, and to hold state officials accountable. 76. In 2017, state officials noticed more responsible management of the recent cholera epidemic in health zones where the Project was being implemented. The participation of the community in co-management of health facilities alongside state officials made it possible to quickly identify sources of the outbreak and to find suitable solutions to solve it. The decrease in cholera cases in communities where the Project was active was facilitated by citizens’ commitment to intensify sensitization sessions to systematically promote hygiene practices. Some communities had established drinking water sources which significantly reduced the incidence of cholera in their communities. As such, the epidemic ended relatively quickly, facilitated by high levels of community participation in efforts to control the disease. 77. CODESAs have potential to ensure community voices remain heard and to influence decisions for the good of the community. Health policy in the DRC is largely shaped at the provincial level. In considering ways to influence community-related decisions at the provincial level, Cordaid sought to capitalize on the electoral period after country commitments had been ratified, by encouraging discussions among CODESAs on local candidates and elected representatives. In this way, the Project identified that the CODESA may adopt a key socially responsible role during elections, which was to promote informed choices of candidates. 78. The project adapted to the declining socio-economic conditions across the DRC in 2017, including the provinces of South Kivu and Kongo Central. The Project took declining purchasing power into account by adapting implementation. Specifically, it changed the conditionality for accessing funding for community- oriented micro-projects: the required community share of contributions was lowered from 30-40 percent to 20-30 percent. In South Kivu this change occurred between the first and second round of the matching grants. Additionally, subject to the financial crisis, some banks and loans and savings cooperatives accessed by Page 38 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) CODESAs went bankrupt, which caused short delays in Cordaid’s disbursement of CODESA subsidies. The situation was resolved by Cordaid by establishing local accounts at larger, more credible banks. 79. The security situation in two health zones of South Kivu, Lemera and Uvira, degraded rapidly after November 2017 and soon prevented Cordaid’s team from being able to work in those areas. Different demonstrations from civil society, including La Lucha, and other political oppositions forces were reported regularly across the country, which affected activities of CODESAs in targeted zones of intervention. In particular, the security situation in the two health zones of Uvira and Lemera in South Kivu worsened considerably, and the Project could not transfer payments to local partners. Consequently, monitoring and support of CODESAs and communities was conducted through phone in these regions. In addition, the Project made use of local community organizations to avoid risk of losing contact with intervention zones. Accordingly, Cordaid was able to obtain and respond to necessary information on time through remote management and use of community- based organizations. 80. Decentralization had a positive impact on the Project and provided a clearer path toward achieving project objectives. The DRC advanced its decentralization process in the health sector in 2016. Accordingly, it promoted acceptability among government and partners of participatory institutions such as the CODESA. A bottom-up approach to operational planning had begun, from local to national level, and, consequently, CODESAs were solicited for their inputs into planning processes. As a result of this collaboration, formal implementation arrangements were signed between the Project and provincial health authorities to improve service delivery. By 2017, various contracts were signed at all levels; between DPS and Health Zones and between Health Zones and CODESAs and it was reported that the commitment of health authorities (Provincial Ministers of Health, Heads of Provincial Divisions of Health and Physicians Heads of Health Zones) had become increasingly evident. 81. Throughout project implementation, the decentralization process in the DRC was reported to have greatly contributed to the success of the Project. In 2017, Cordaid reported stronger engagement with provincial health authorities in strengthening and collaborating with HFCs during the previous year. DPS were increasingly aware of how functioning CODESAs could improve the performance of the health facilities and held more constructive dialogues with CODESAs. Understanding among the project team was that decentralization had impacted the attitude and motivations of the provincial health authorities to take the work of HFCs seriously. This was noted in Cordaid’s second annual report: “The role and place of the HFCs have been put at the center stage in this process. This impacted the attitude and motivations of the provincial health authorities to consider HFCs seriously. The whole decentralization dynamic is positive and an opportunity for the project objectives.â€? 82. Notably, the decentralization process may have improved the culture of accountability among stakeholders instilled by the Project. In addition to restoring dialogue between communities and provincial authorities, this project offers a common framework for promoting community response in the context of country commitment to decentralization of institutions. Importantly, however, South Kivu and Kongo Central were among the very few provinces which had not been divided into smaller provinces and therefore benefited from the renewed interest in decentralization without having to restructure radically. Page 39 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) Table 4: Summary of key findings that affected implementation and outcome Positive Negative Undetermined During Preparation ï‚· Appropriately sequenced and ï‚· Unclear wording of objectives reinforcing component activities and objectives ï‚· Lack of robust theory of change ï‚· Results Framework (detailed and realistic plan; IRIs aligned with ï‚· Underestimations in budget for IRs; programmatically important engagements between key actors, measures) and resource requirements for Factors achievement of all outcomes (e.g. subject to ï‚· Appropriate engagement and resources for insecure areas and control of plan for collecting information national-level advocacy) World Bank and monitoring the achievement and of outcomes implementer During Implementation ï‚· Adequate supervision and ï‚· Delayed disbursement of WB funds support from WB to project at the start of the Project negatively team given changing context impacted the Project schedule and operational challenges (significantly) and credibility of Factors (training, strong relationships, project partners subject to location, availability, World Bank programmatic flexibility, and control access and opportunities for K&L) Positive Negative Undetermined ï‚· Appropriate training, ï‚· Human resource challenges in ï‚· Project was coordination, and aligned roles project management destabilized implemented in four and objectives across project operations different contexts implementation levels for joint requiring different monitoring and support; ï‚· Significant delays in recruitment levels of introduction of participatory and operation of CSOs for cross- engagement, biannual and annual reviews for verification of CODESA resources, and quality improvement and achievements, and in establishing a expertise monitoring; emergent use of fully functioning mechanism for Page 40 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) CSOs and remote phone support administering matching grants ï‚· Financial support for in conflict-afflicted health zones (implications for implementation of HFCs (subsidies & Factors components one, two and three, matching grants) subject to impact study and integration of implemented control of ï‚· Development and use of (unplanned) Health Zone lessons, and level of achievement of differently by health implementing outcomes) authorities in each entities CODESAs for increased accountability, and influence, province (different K&L, monitoring and support ï‚· Limited national-level engagement incentives) and advocacy due to Impact of ï‚· Cordaid effort to extend delays, insufficient resources, ï‚· Organizational influence and relationships of political change, and insecurity. differences among CODESAs to the provincial level, CODESAs by having CODESAs promote (information flows, informed choices of electoral decision-making, candidates in a socially and power responsible way dynamics) ï‚· Strengthened local interest in ï‚· Tense national socio-political democracy, accountability, environment complicated work with representation, and authorities and implementation of GPSA/CODESA due to tense activities in communities national socio-political Factors environment ï‚· Marked decline in socio-economic outside conditions from 2017, and required control of ï‚· Decentralization provided a changes to conditionality for implementing clearer path to achieving project accessing funding for community entities objectives (stronger micro-projects engagement, awareness and interest in the value of ï‚· Escalated security situation South functioning CODESAs across Kivu prevented project team from levels) physically working in two health zones, and required use of CBOs and more resources for service delivery, remote monitoring and support IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. Quality of Monitoring & Evaluation 83. M&E has been assessed based on three key elements. First is quality of M&E design, or whether the operation’s theory of change is clear and whether adequate indicators were identified to monitor progress toward the PDOs using effective M&E arrangements. Second is quality of M&E implementation, or whether M&E data were collected and analyzed in a methodologically sound manner. Third is quality of M&E utilization, or whether M&E data on performance and results progress were used to inform project management and decision-making. Page 41 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) Quality of M&E design M&E According to the Results Framework 84. The Project theory of change was not formulated in the initial description of the project and was formulated for the ICR. Many underlying assumptions, mechanisms and relationships between key activities, outputs and outcomes were not clearly outlined in project documents. This may have undermined understanding, monitoring, achievement, and meaningful assessment of expected outcomes. 85. PDO-level objectives were not specific, and associated PDO-level indicators were not sufficiently representative of, nor specific to, the PDOs of improved access and improved quality of health services. PDO- level indicators did provide important monitoring information but they did not sufficiently describe expected PDO-level outcomes (PDOI-1 and PDOI-2 did not directly relate to either PDO, and although PDOI-3 was clearly related to PDO-1, it was not presented as a clear and valid measure of access in the RF). An indicator reflecting utilization of essential health services might have served as an appropriate composite proxy outcome indicator for improvements to both access and quality of care. 86. Intermediate results indicators were appropriately related to intermediate outcomes in the RF and, in many cases, adequately reflected project outputs and activities in each component. As such, the Results Framework was useful for monitoring progress toward intermediate and PDO-level outcomes, although it did not provide sufficient information for tracking of the PDO-level outcomes themselves. Baselines and targets or target levels for measures were provided in almost all cases. 87. Targets were not formally updated throughout project implementation to reflect on the ground realities. It was evident that some targets in project reports and ISRs had been adjusted to reflect a new schedule and goals, although these were not explained. It is unclear from project documents what the revised target should have been, particularly given discrepancies in reporting between the project evaluation and annual reports and ISRs for the number of HFCs actually reached. Formalized corrections to the Results Framework would have been extremely beneficial given the high turnover of project staff to promote clear and accurate understanding of progress and outcomes for stakeholders. 88. Findings indicate that many measures were programmatically important, but the naming of indicators and selected units of measure presented ambiguity, assumptions, and reduced validity of project indicators. In a couple of cases, important criteria for scoring were not specified and not all key component outcomes were necessarily accounted for (e.g. matching grants for VG-focused projects as part of component 3). Furthermore, indicators were not disaggregated (by region or context, for example), which limited visibility of trends at the project level and precision for monitoring and evaluation. However, all RF indicators included appropriate sources, schedules, and stakeholders responsible for data collection. 89. The Project selected appropriate stakeholders to engage and established an appropriate plan for collecting information and monitoring the achievement of outcomes. Specifically, the Project had organized and aligned roles and objectives for joint monitoring (and support) between Cordaid, DPS, and health zone management teams, in addition to local CBOs when needed. As outlined in the first annual report, the embeddedness of Page 42 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) different government health sector actors at different levels was key to increasing ownership of project activities among the community, but also in the health system. HFC and medical staff contracts had an RBF component that was expected to help evaluate the social contracts between citizens quarterly. Their performance was to be evaluated through a set of indicators by the HZ management teams, and then by provincial health authorities, and ASLOs were to lead cross-verification of CODESA achievements in the field for quality assurance and enhanced reliability. To ensure effectiveness, the Project trained stakeholders across levels in management of funds and reporting, planning and development of micro-projects (including use of community assessment maps), and multi-stakeholder decision-making processes. 90. Importantly, the Project recognized that not all outcomes are quantitatively measurable. To supplement RF monitoring, reporting templates were designed to capture a variety of key information relevant to the Project and to building collaborative social accountability. This included information on context; key outcome tracking explanations; direct engagement with government counterparts; coalition building with formal and informal allies to achieve Project objectives; engagement of state accountability institutions; capacity building of direct beneficiaries and key stakeholders; internal adaptive learning; data generation and use for social accountability; spill-over effects from the intervention; application and integration of gender equality; MEL system and milestones; and project management and institutional development. These reports also included information relevant to GPSA/World Bank capacity and knowledge support. This reporting information was key to monitoring and assessments of the Project. Project evaluations: Impact evaluation and final project evaluation 91. Two evaluations were officially planned for the Project: the first (impact evaluation) for component three to assess the degree of inclusion of vulnerable groups in HFCs and the second (independent evaluation) to evaluate the fourth component. Both evaluations were finalised in October 2019 given delays in Project implementation and were subject to limited timeframes and budgets. 92. The impact evaluation focused on the pilot social and financial interventions implemented for VGs and was originally planned for 2017. Accordingly, Cordaid selected social and financial interventions to pilot for an initial one-year period following preliminary research on barriers to access among VGs (Falisse & Mirindi, 2016). All HFs were to benefit from all interventions after a two-year pilot period. Two approaches were assessed in the evaluation: i) a financial intervention whereby matching grants specifically devoted to VG support, and ii) a social intervention whereby subsidized facilitation of VG-CODESA meetings where VGs could seek social support when trying to access health care. The two interventions were randomized across 80 health facilities of four health zones: Miti-Murhesa, Katana, Idjwi, and Walungu. Each of these health zones had received prior support for CODESAs (“oldâ€? intervention areas of South Kivu) and accounted for approximately 10,000 VGs. The pilot followed a 2x2 factorial design: 20 HFs were randomly selected for the social intervention, 20 for the financial intervention, 20 for both interventions, and 20 were kept as a control group. 93. Without access to impact evaluation documents detailing research questions, methods, measures, and analysis, it was not possible to adequately assess the validity of measures, data collection, analysis, and results. These were also not sufficiently detailed in the final project evaluation. While randomization helped to remove bias in allocation of interventions, it is not possible to verify whether the sample size used was sufficient to detect small differences without further information. Specifically, assuming two-way interaction effects were Page 43 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) accounted for, factorial design sample size calculations vary significantly depending on the size of standardized interaction effects expected or obtained – enough to determine whether the sample size was inadequate or adequate. 94. The second independent evaluation used mixed methods17 to address selected outcomes and provided key themes and information for assessing context, implementation and outcomes. In a 2-page concept note, it was acknowledged that the final evaluation would not be a ‘proper’ impact evaluation in the sense that identification of causal pathways would be limited, and mostly through qualitative research. The evaluation focused on selected intermediate-level outcomes, project contexts, and implementation, including the extent to which and how key stakeholders adhered to, perceived, and shaped the project. As such, the evaluation provided important qualitative information and observations of behavioural outcomes, in addition to key parts of the Project such as subsidies and matching grants. However, the evaluation included analysis based on the GPSA theory of change at the time, but this was not clearly linked to the Results Framework for the Project. The evaluation also reframed core activities and intended outcomes in a way that obfuscated understanding of the original project components and expected and realized outcomes as referred to in the Results Framework. Notably, the project evaluation did not explicitly analyse indicators in the Results Framework. Access to the health information system was also not possible within the short time frame of the study and, more importantly, data on baseline situations of HFs were to be collected at the start of the project but were not available, which limited attribution of results. Quantitative analysis used in the evaluation was descriptive and the evaluation did not provide statistical testing for significance of associations. Quality of M&E implementation 95. Cordaid did not carry out a sufficient baseline study at the start of the Project, which limited assessments of outcomes and attribution of results. The Results Framework was not updated throughout the Project so weaknesses in M&E design remained throughout implementation. To improve the quality of M&E implementation, the Project introduced biannual and annual project reviews for quality improvement and monitoring. These assessments brought all stakeholders together, including the heads of the provincial ministries of health, Health Zone management teams, local associations (ASLOs), and CODESA representatives. However, limited or no data were available for multiple Results Framework indicators across years, which resulted in an incomplete data set, and which limited monitoring and evaluation of outcomes. Only four indicators were reported against in all three ISRs. Other key data and information were not accessible for 17The final evaluation used a mix of interviews, focus groups, phone surveys, and grey literature review (CBO and annual reports, and presentations). It primarily drew upon: i) 15-minute (13 questions) surveys via phone with 164 HF chief nurses and 157 CODESA presidents in HFs which had benefitted from the project, accordingly, findings were subject to self-reporting bias, although were supported by qualitative findings from interviews with other stakeholders; ii) 35 interviews and 16 focus groups (5-9 participants each) in HF catchment areas (chief-nurses, CODESAs, CBOs, community members) and health zones (chief- doctor, HZ CODESA; iii) 14 Semi-structured key informant interviews (4 GPSA/CODESA project managers, 1 project officer, World Bank contacts, 3 DPS officials from South Kivu and Kongo Central, 2 representatives of faith-based organisations running health centres, and 2 representatives of NGOs working in community health. A Most Significant Change approach was used in both qualitative research and phone surveys to elicit key elements in relation to the four intended outcomes structured in the evaluation: 1) fully functional CODESAs as a mechanism of social accountability; 2) the community-led rehabilitation of health- care infrastructure; 3) CODESAs promoting social inclusion; and 4) a wide support/coalition of actors in favour of the GPSA/CODESA and social accountability practices in the DRC. Page 44 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) assessment of outcomes. For example, all CBO reports were made available for South Kivu in the final evaluation, but Cordaid did not provide this information for Kongo Central. 96. Data reported throughout project implementation were not consistent. For example, the project evaluation and project reports presented two different totals for the number of strengthened HFCs (182 and 185, respectively). Additionally, from ISRs and project reports in 2016, it is not clear why the first measure of IRI-1 (140 health committees know their roles and are equipped to ensure it) did not match and was, in fact, greater than that reported for PDOI-1 (105 strengthened HCCs). Data between 2016 mid-term (MR) and annual reports (AR) were also inconsistent, as IRI-1 was reported as 140 in MR 2016 and 181 in AR 2016. Likewise, 76 Matching grants were recorded for IRI-5 in MR 2016, but these did not begin until H2 2017. Moreover, coverage of health zones and health areas appears to have been misreported in annual reports, as 105 health areas were reached across 7 health zones in the first year of the Project, and 105 health areas were reached across 11 health areas in the second year. It was also not clear in project reports how beneficiaries were defined in counts of men, women, and children and youth. 97. The recruitment, training, and use of ASLOs (nine in South Kivu and 3 in Kongo Central) and the institutionally embedded system for joint monitoring, suggest that M&E functions and processes may be sustained after project closing. However, the likelihood of sustainability may differ by context due to a limited provincial-level coalition of actors in favour of the CODESA model in Kongo Central and Kinshasa; limited project engagement with Kongo Central DPS; limited engagement of local civil society with CODESAs and the Project in Kongo Central; and maturity of the intervention (only 3 months in Kinshasa). As Cordaid continues to scale in Kongo Central and expand activities to other provinces such as North Kivu, systems for M&E may be strengthened over time through subsequent related projects and support, particularly based on the findings from this project. Quality of M&E utilization 98. Information gathered through community assessments was used to inform development, implementation, and evaluation of community micro-projects and improvements to health facilities. According to Project reports, CODESAs were capacitated to identify critical issues in their respective communities using information and to solve them with local means. For example, during the cholera outbreak, stakeholders observed that communities were able to use, act, and share information from community assessments and meetings, which may have helped curb the impact of the outbreak in those communities. 99. The Project used monitoring information for quality improvement through inclusive and participatory annual and biannual joint reviews of the Project. Reportedly, these reviews allowed all stakeholders, beneficiaries and the project team to collectively review, analyze, plan and adapt project implementation at each level for improved results. The reviews complemented previous training on project planning and development, and decision-making. According to the third annual report, this allowed stakeholders to learn from each other’s experiences to address their own challenges. For DPS, this provided opportunities to develop common understanding of indicators and how to correctly report on them (for improvements to data quality and reliability), and to understand challenges and develop timely and appropriate approaches. These included community self-evaluation through ASLOs, the involvement of more political and opinion leaders, and increased participation of the community in the process for decentralization. Additionally, during these reviews, the Page 45 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) project team was able to understand and assess the need to adjust the community-share of matching grants for community microprojects when socio-economic conditions had worsened. 100. Cordaid shared its experiences at national and international levels, including to the Ministry of Health, WHO Health Cluster, and donors, to ultimately promote CODESA membership and scale up of the Project’s approach. The Project could not organise meetings with national-level authorities in charge of community participation at the Ministry of Health (the Particom, Division Participation Communautaire) until later into the Project. Cordaid also participated in two GPSA Fora and exchanged experiences with other beneficiaries of GPSA. While Cordaid expressed enthusiasm for sharing experiences with other actors throughout the Project, the Project had focused resources on service delivery given contextual challenges and did not necessarily have the capacity for effective advocacy at national levels. 101. The Project motivated Cordaid strategies and expansion of the GPSA approach to other contexts and sectors. According to Cordaid, experiences of the Project motivated expansion of the GPSA/CODESA model to other provinces (e.g. North Kivu), and the development of a similar program and practice across sectors (e.g. education). This is triangulated in Cordaid’s 2018-2020 triennial strategic plan, where the organization integrated social accountability and community engagement in all its community projects as a cross-cutting strategy to stimulate ownership and ensure sustainability of approaches (e.g. Global TB HIV / AIDS Program 2018-2020). Furthermore, during the Project, the national MSP and the World Bank requested that Cordaid develop two concept notes to scale up a community PBF program based on the GPSA/CODESA approach that centered on social accountability and community participation. This included the Community PBF In The City Of Kinshasa, and CODESA Redynamization in PDSS Health Zones. B. Environmental, Social and Fiduciary Compliance 102. Environmental Assessment for the project was not required (category C) as part of the Bank’s project safeguards. At time of appraisal, project screening identified minimal environmental and social risks related to project activities. No civil works were to be financed under this Grant. 103. As the Project was operating in an area where indigenous groups were present, it triggered OP 4.10 to ensure screening for inclusion of these indigenous groups into the project implementation. The Project used the Indigenous Peoples Plan developed for the DRC Prevention and Mitigation of Sexual and Gender Based Violence (SGBV) in North and South Kivu Project to address people and/or issues unique to the health project. The IPP for the SGBV Project was consulted on and publicly disclosed in the DRC and in the Bank InfoShop on July 9, 2014. A key aspect of this project is outreach to and consultation with vulnerable groups to ensure they are receiving adequate medical attention. In addition to special outreach for women and children, the project also sought to ensure other vulnerable groups in the project area, such as Indigenous Peoples, were included in consultations and studies regarding services. Page 46 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) 104. According to implementation status reports, all three monitored legal covenants (for TF-18164) were complied with.18 C. Bank Performance Quality at entry 105. Quality at entry refers to the extent to which the Bank identified, facilitated preparation of, and appraised the operation so that it was most likely to achieve planned development outcomes and was consistent with the Bank’s fiduciary role. 106. At the time of Bank appraisal, the strategic relevance and approach of the Project was clear, although project objectives were not specific. The Project and PDOs were consistent with the Bank CAS and the DRC national health sector development plan (PNDS 2011-2015) and complemented the launch of a series of larger programs and reforms supported by the Bank in the DRC, both at the time of approval in 2014 and thereafter.19 The issues of inadequate access and quality of health services were appropriately identified as key to improving health outcomes in targeted regions and components clearly addressed key requirements for ensuring HFCs can and do function effectively for improvements in access and quality. These included improved use of HFCs in the health system, sufficient non-financial and financial support to HFCs, sufficient inclusion and attention to vulnerable groups in HFCs, and increased integration of HFCs into the health system. 107. The Project was well positioned to build upon experience, evidence, strategies, and tools generated by earlier Cordaid and Bank pilots involving CODESAs. The approach toward strengthening HFCs across regions (strengthening local governance structures and processes; promoting community participation and collaborative social accountability; providing opportunities for HFCs to enable improvements to health services and in the community; and ensuring vulnerable groups were integrated into the health system) was particularly relevant in the context of the decentralization process, disengagement of the state, and conflict-afflicted areas in the DRC. Importantly, South Kivu and Kongo Central were among the very few provinces that had not been divided into smaller provinces and therefore benefited from the renewed interest in decentralization without 18 i) Finance Agreement (Legal Agreement 2.03) – Complied: Institutional and Other Arrangements. The Recipient has sole fiduciary responsibility under the Project, including for procurement and financial management and shall ensure, at all times during the implementation of the Project, that it maintains competent staff in adequate numbers, including a program manager, a financial manager, a monitoring and evaluation specialist and procurement specialists with experience in projects financed by international financial institutions; ii) Finance Agreement (Legal Agreement 2.05) – Complied: Project Monitoring, Reporting and Evaluation. (a) The Recipient shall monitor and evaluate the progress of the Project and prepare Project Reports in accordance with the provisions of Section 2.06 of the Standard Conditions and on the basis of indicators acceptable to the World Bank and set forth below in paragraph (b). Each Project Report shall cover the period of one calendar semester, and shall be furnished to the World Bank not later than forty five (45) days after the end of the period covered by such report; iii) Finance Agreement (Legal Agreement 2.06) – Complied: Financial Management. (a) The Recipient shall ensure that a financial management system is maintained in accordance with the provisions of Section 2.07 of the Standard Conditions. In the final ISR (May 2018), one overdue was reported (Financial Statement Audit) as being overdue for less than 4 months. 19 The World Bank financed Health Systems Strengthening for Better Maternal Child Health Project and the Multisectoral Health and Nutrition Project. Page 47 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) having to restructure radically. The Project also built upon experience, evidence, and tools generated by earlier Cordaid pilots and a Bank project involving CODESAs. 108. Capabilities and inputs of GPSA, the World Bank, and Cordaid were appropriately aligned to ensure program effectiveness. As the implementing agency, Cordaid brought significant local capacity, experience, and coordination for strengthening HFCs in the DRC (detailed in Section I). The Project was supported by the Washington DC-based World Bank GPSA team. The World Bank team in Kinshasa also provided capacity building and implementation advice. The technical capabilities of GPSA also strongly complemented the approach outlined for the Project. GPSA appropriately supported the project by providing technical assistance and advice on specific methodological challenges related to the implementation of social accountability tools. 109. The Project had a complex yet flexible design. It incorporated appropriately sequenced and reinforcing component activities and objectives, as well as a semi-structured, emergent approach to address third component objectives. However, more specific formal descriptions of activities in relation to components would have facilitated clearer understanding of the intended structure, activities, and outputs of components among different stakeholders for project sustainability. For example, part three of component one is vaguely described in the legal agreement. As such, the Project may have benefitted from more formal documentation from the outset. 110. The resources granted by the Bank were sufficient for achieving some project outcomes but not all, particularly given disruption to the Project in volatile areas of South Kivu. Excluding project management and implementation costs, most of the Project budget was allocated to three activities: the training and retraining of CODESAs, matching grants, and CODESA subsidies. However, it was reported that resource allocation for component four was inadequate to achieve objectives (human, social, and financial resources), particularly given the subsequent violence and insecurity in South Kivu, and costs for supporting interactions between HFCs and health authorities had been underestimated and required correcting during the Project. The Project may have benefited from measures to mitigate high turnover of project managers and disruptions to the Project, in addition to ensuring human resource capacity could be maintained by the implementer throughout the Project (relevant in the case of the project manager). 111. From a grant management perspective, the agreed timing of disbursements was not clearly defined by specific development outcomes (discussed in Section IV, Quality of M&E design). Timing of instalments formally centered on input-based measures (project financing and planning) for achievement of project outcomes. While it may be assumed this information was used in combination with other sources (e.g. project reports, engagement with stakeholders), for the Bank, this information alone does not directly inform understanding of progress, effectiveness, or efficiency with respect to outcomes. However, requirements for disbursements were consistent with the Bank’s fiduciary role and responsibilities. 112. The Project originally specified an operational plan for the achievement of most of project objectives in the agreed timeline and sought to directly engage government actors and important state accountability institutions such as health authorities, for M&E and implementation. Importantly, to promote collaborative social accountability, the Project was designed to establish and reinforce institutional structures and processes for effective functioning, monitoring, and use of HFCs in the health system and community, and to provide opportunities to address issues highlighted through social accountability mechanisms and tools. Page 48 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) Quality of supervision 113. Quality of supervision refers to the extent to which the Bank proactively identified and resolved threats to the achievement of relevant development outcomes. 114. Despite the slow start to the Project, GPSA and the TTL adequately supervised and supported the project team to adapt to the changing context and operational challenges (adequate training, relationships, location, availability, programmatic flexibility, and access and opportunities for K&L). Detailed further in Section III, a combination of factors affecting implementation, including insecurity and delayed funding, required significant adjustments to the project schedule and targets, and limited time available for achievement of key outputs and outcomes. Appropriate flexibility was provided when the Project was eventually extended at no additional cost by three months “to compensate for initial delays of between 7 to 9 months in project launches due to holdups in the signing of agreements with Regional Government entities; and to allow time for the completion of knowledge management productsâ€? (World Bank 2018b). Given the significant programmatic and contextual challenges for the Project in the DRC at the time, the focus of Bank support largely focused on achieving objectives for implementation within the Project timeframe. 115. Throughout the Project, Cordaid reported strong relationships and regular contact with the TTL and project officer, who answered questions quickly and sufficiently. As detailed in the previously, the Project was appropriately supported by the Washington DC-based World Bank GPSA team for technical assistance, and the World Bank team in Kinshasa provided capacity building and implementation advice. The GPSA blog also provided continuous access to information and other documents related to the Social Accountability approach. The project team was also able to participate in the third and fourth GPSA Global Partnership Fora organized by the World Bank in Washington DC in May 2016 and October 2017. These fora provided the project team with opportunities to learn about the MERL system and its importance in project management, to exchange experiences with other beneficiaries of GPSA funds, and to consider the future of GPSA. The project team consistently suggested continuous exchange of experiences with implementing partners in other GPSA projects for improved support. Accordingly, an exchange experience and visit to Morocco was originally scheduled for 2017 but was rescheduled and took place in H1 2018 following a scheduling conflict with the project team of CARE Morocco. 116. According to midterm reports, Cordaid consistently indicated satisfaction with the support provided by GPSA and Bank staff in 2016, 2017, and 2018. Survey results indicate that Cordaid received sufficient guidance from WB staff for engaging public sector institutions and accessing information; used or contributed to GPSA knowledge products or activities; applied knowledge gained from WB to project operations and analytical work; was able to learn from experiences of CSOs in similar contexts; and saw that collaboration with a GPSA Global Partner helped the Project. 117. Improvements to quality of performance reporting may have improved understanding and assessment of progress and achievement of project objectives. While performance reporting was candid, ISRs and Cordaid project reports lacked clarity and detail on specific activities, issues, and progress toward PDOs (e.g. exactly when contracts were signed, when matching grants were disbursed, issues of delays in signing contracts), and the quantity and quality of RF data collected through the project was not identified as problematic or Page 49 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) rectified. Data were largely unavailable due to the previously discussed limitations of M&E design and project reporting requirements, although not all indicators reported by the Project were included in ISRs. The Bank demonstrated appropriate attempts to elicit clarity and more detail in reports from the project team, but important programmatic information, such as specifically when matching grants were disbursed, was not made clear in project reports and ISRs. While this may be common knowledge among key Project stakeholders such as the WB team and Cordaid, it is not adequately clear to those outside or less close to the Project. D. Risk to Development Outcome 118. Under significantly declining socioeconomic conditions, it is critical that implementation of financial support be adapted to take declining purchasing power into account (e.g. lowering community share of community-oriented micro-projects). Similarly, financial partners (e.g. loans and savings cooperatives) should be appropriately assessed to avoid issues of bankruptcy and subsequent risks and delays to funding; relatedly, local bank accounts should be established at larger, more credible banks. In addition, differences in how financial support for CODESAs (e.g. subsidies and matching grants) are implemented across regions can distort incentives. Anticipating, monitoring, and understanding these differences is important for achievement and assessment of expected outcomes. Careful attention should be paid to frequency and scoring of subsidies and matching grants across target regions. Appropriate hypotheses may be tested to assess the optimal structure for financial support according to regional needs. V. LESSONS LEARNED AND RECOMMENDATIONS Project preparation 119. Significant human, social and financial resources are required for conflict-afflicted or insecure areas, CODESA and health authority interactions, and national-level engagement and advocacy activities (for CODESAs), and these should not be underestimated in program design. In conflict-afflicted areas, implementers may obtain and respond to necessary information on time through remote management and use of community-based organizations. Experience also suggests that advocacy at the provincial level may be most effective in areas where CODESAs have been previously supported (e.g. South Kivu). M&E 120. Projects that use social accountability to improve access to and quality of health services require relatively complex yet flexible designs. In this instance, the objectives were not clearly articulated, which made development of the adequate results framework, a concrete theory of change, and monitoring achievement difficult. A key lesson from this experience is that while the Project may have created many positive outcomes, the levels of achievement and their attribution to Project investments is not clear from the data collected. While this Project may be considered small-scale relative to other Bank projects, it is Page 50 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) critical that sufficient attention be paid to ensuring the Project functions well and can be evaluated successfully to maximize use of the Bank’s financial resources. The Bank is encouraged to determine whether its financial and non-financial support for smaller scale projects are likely to produce meaningful evidence and outcomes. Key lessons as it relates to evaluation of social accountability projects are the following: ï‚· In forming Project objectives, it is also critical that types of access (e.g. financial) and quality (e.g. clinical) be explicitly defined, and that multiple appropriate metrics and methods be incorporated for assessment (e.g. service utilization may represent an appropriate composite measure of improvements to service quality and access). ï‚· The logic, including relationships and connections between component activities and intermediate outcomes and PDOs, and underlying assumptions need to be clearly articulated in project documents. The theory of change must be clearly linked to a complete Results Framework containing appropriate SMART objectives and indicators. ï‚· Monitoring and evaluation efforts should be updated to ensure meaningful assessment of these initiatives as the project adapts. ï‚· Qualitative data to better understand mechanisms, drivers and degree of collaborative social accountability should be incorporated into reporting, as was the case, both for course correction and for more meaningful assessment of outcomes. ï‚· It was difficult to ascertain levels of progress, program effectiveness, or efficiency with respect to financial inputs. While requirements for disbursements may be consistent with the Bank’s fiduciary role and responsibilities, it is critical that disbursements be linked to results, and that reporting be clear in detailing specific activities, issues, and progress toward PDOs. ï‚· Involvement of different government health sector actors at different levels for joint monitoring, particularly at district and provincial levels, may help to increase ownership of project activities among the community, but also in the health system. ï‚· An institutionally embedded system for joint monitoring may promote sustainability of M&E functions and processes after project closing. ï‚· Inclusive and participatory biannual and annual project reviews may be used to collectively review, analyze, plan and adapt strategies and actions across levels for improved results (including for each level of project implementation). Implementation 121. Delays in the Bank’s disbursement of funds, particularly at the start of the Project, can significantly undermine credibility and trust of the implementer and local partners from the outset, both of which are critical for collaborative social accountability . If components of programs are staged and largely interdependent, the impact of such delays will be magnified and significantly undermine the effectiveness of the Project. The Bank must pre-empt technical issues and ensure prompt disbursements of funds. If delays ensue, the Bank should closely coordinate with the implementer and local actors to ensure timely and accurate information is provided, particularly for activities which have already begun (e.g. training), to preserve credibility and trust building in the community. Page 51 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) 122. During implementation, the tense socio-political environment did present challenges in project implementation, but it can appeared to help strengthen local interest in the HFC model . The communities targeted seemed to have greater interest in self-care, democracy, voice and accountability, and representativeness. The project’s approach for mitigating financial risks for the vulnerable, ensuring financial and non-financial support to communities, and encouraging ownership of local solutions and self-care can help to increase resilience of target communities. Consequently for future projects, there is opportunity to leverage this collective interest to help drive local support for the HFC model and other initiatives. For example, the HFC model may be used to help coordinate and promote disease control efforts during an epidemic (e.g. Cholera) through community assessments and meetings. In these cases, it is advised that monitoring and evaluation efforts be intensified to understand the specific contributions of the CODESA in controlling the spread of disease. 123. Decentralization can help promote acceptability among government and partners of participatory institutions such as the CODESA, and provide a clearer path toward achieving project objectives. For example, in the DRC, a bottom-up approach to operational planning had begun, from local to national level, and, consequently, CODESAs were solicited for their inputs into planning processes. Relatedly, the decentralization process can instill a culture of accountability among stakeholders and impact the attitude and motivations of the provincial health authorities to take the work of HFCs seriously, resulting in stronger awareness, interest, and engagement from health authorities across levels. Outcomes 124. The way that CODESA members are involved and how information is used for decision-making contributes to power dynamics within health facilities, and accounts for differences in the CODESAs ability to meet community needs and improve health service responsiveness. Information flows and decision rights may improve over time and are critical for ensuring CODESAs can provide an effective accountability interface. Effective training and retraining on roles, responsibilities, and effective and use of social accountability tools, such as the TOR and CSCs, can help establish CODESA decision rights and productive information flows (for co-management), as well as overcome initial resistance from HF staff. Implementation research may be used to better understand contributions of activities and tools for promoting decision rights and productive information flows. As CODESAs mature and their relationships between implementers and health authorities solidify, there will likely be an increase in productive information flows and decision rights among CODESAs. Pre-existing, strong relationships between implementers and provincial health authorities may help to promote a strong coalition of actors in favor of the CODESA co-management model. With sustained support and time, CODESAs may evolve to hold a central role and greater legitimacy in the community and reflect a broader vehicle for collective action at the HF and beyond (e.g. cross-sector initiatives). District-level HFCs (HZ CODESAs) can provide additional mechanisms for upward accountability. Moving forward, it will be critical that they are strengthened to allow for more sharing of experiences, monitoring and supporting HFCs in unreachable areas. HZ CODESAs may foster competitiveness among CODESAs for performance improvement and may assist in lobbying health zone management teams and attracting new external partners to the health zone. A higher concentration of aid organizations in target Page 52 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) regions may also increase opportunities for organizations to learn from each other rather than compete. In this scenario, coordinated approaches will be required to mitigate imbalances in workload, resourcing, and funding for CODESAs. . Page 53 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) SUPPORTING DOCUMENTS A. In-Depth Assessment of Results Framework 1. Each indicator of the Results Framework (RF) has been assessed according to properties of SMART indicators and in relation to respective expected outcomes. These include whether the indicator is (adapted from GAP, 2003): ï‚· Specific: indicator and unit of measure clearly relate to the outcome, described without ambiguities, and parties have common understanding of the indicator ï‚· Measurable: unit of measure specified can be counted, observed, analysed, tested, or challenged using available tools or methods ï‚· Achievable: performance targets were realistically achievable at the time; and the target accurately specifies the amount or level of what is to be measured to achieve outcomes ï‚· Relevant: is valid (an accurate measure of a result, behavior, practice or task) and describes the underlying issue ï‚· Time-bound: time-referenced and able to reflect changes, and provides measurement at time intervals relevant and appropriate to program goals and activities 2. Each indicator has been coloured red, yellow, and green, to indicate the degree to which an indicator adheres to each of the SMART properties. Green cells represent examples of indicators which adequately satisfy a property of SMART indicators, as justified in the Results Framework. In addition, comments are provided (where relevant) to inform improvements to indicator development, including whether the indicator is reliable (consistently measurable in the same way by different observers), precise (operationally defined in clear terms), and programmatically important (related to achieving objectives required for impact). As noted in this report, while all these properties are not reflected by indicator names (a limitation), indicators are assessed considering all additional indicator information provided in the Results Framework (indicator names and units of measure were often taken together to inform broader understanding and assessment of indicators). The phrase “formally updatedâ€? is used to refer to updates to the Results Framework. 3. A key limitation of the Results Framework was that targets were not formally updated after miscalculations for target HCCs and significant changes to implementation schedule (from review of the RF and other project documents, it was evident that some targets in project reports and ISRs had been adjusted to reflect a new schedule and goals, although these were not explained). Consequently, at least six indicators did not have appropriate targets through the project (PDOI-1, PDOI-2, IRI-1, IRI-2, IRI-6, and IRI-7). This may determine whether the indicator was, by definition, not achievable and, in some cases, specific (where uncorrected targets were included in indicator names – maintained throughout reporting). The following analysis excludes this common factor (but is noted here), to avoid repetition and to highlight other recommendations which may be adopted to improve quality at entry. As such, appropriateness of target levels was considered in place of actual targets. Additionally, indicators which accounted for an output at one point in time during the 4 years (IRI-3 and IRI-8) were loosely taken to be time-bound. Analysis of all Page 54 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) information provided on indicators in the Results Framework indicates that one indicator may be SMART (almost all indicator names were not time-bound). 4. The Results Framework was a key and valued source for understanding project plans. The following practices were identified for improvements to Results Framework indicators: ï‚· Greater alignment between outcome-oriented indicator names and units of measure would improve validity and reduce assumptions in the results chain. IRI-9, for example, uses the term “permanentâ€? and, although a behavior-related outcome is described by the unit of measure, it is not an outcome which sufficiently describes permanence. In the case of IRI-4, while the community scorecard process is an activity which may help to restore trust, it does not sufficiently imply that trust was restored as result of a community assessment. In many cases, the unit of measure and name did not fully correlate given framing of indicator names (the unit of measure was often a count, whereas indicator names were framed as binary outcomes (IRI-1, IRI-4, IRI-6, IRI-7, IRI-9, IRI-10). To reduce ambiguity, indicator names should be clear and adequately specific to reflect the outcome and unit of measure, and almost identically worded indicators across outcome levels should be avoided (e.g. PDOI-2 and IRI-3). For IRI-5, it was not necessarily clear from the indicator name (activities to improve health services) whether non-HF (community) micro-projects were to be included in this measure. ï‚· In some cases (e.g. PDOIs), the indicator did not accurately represent the desired level of outcomes. Relevance and specificity of indicators to outcomes may be improved by avoiding the use of process and output measures for measuring outcomes (if they are relevant, it is important to specify assumptions or to adjust indicator names to more accurately reflect the measure). ï‚· Improved naming of indicators to reflect all properties of SMART indicators (including time- bound) for greater clarity. ï‚· Avoid use of two measures for the same indicator where possible, else people may only report one, unless it is expressly highlighted that both measures are required for reporting on the indicator. ï‚· As a limitation, many indicators were not disaggregated (e.g. by context or province), for example, which may obscure visibility of trends and compromise precision for monitoring and evaluation. ï‚· Where possible, important metrics or criteria underlying a unit of measure may be described to improve validity. ï‚· Appropriately specify population where possible (e.g. IRI-5 is specific to 4 HZs of South Kivu (“oldâ€? intervention areas) ï‚· Additional indicators (or a note for development of future indicators) may be used to reduce assumptions in the results chain, and to more comprehensively represent all outcomes relevant to component activities (e.g. matching grants for VG-focused projects as part of component 3). Page 55 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) Page 56 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) Page 57 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) Page 58 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) B. Project Results Framework Annex 1: Results Framework and Monitoring REINFORCING SOCIAL ACCOUNTABILITY OF HEALTH SERVICES IN BAS CONGO AND SOUTH KIVU PROVINCES PROJECT Project Development Objective (PDO): “The development objective of this proposal is to improve access and quality of health care services in targeted regions (Bas Congo and South Kivu) through the strengthening of Health Facility Committees. â€? PDO Level Cumulative Target Values** Description Data Source/ Responsibility for Data Core Results Unit of Measure Baseline Frequency (indicator YR 1 YR 2 YR3 YR 4 Methodology Collection definition etc.) Indicators* Indicator One: Number of Health 79 (HCC from the 111 79 111 4 Training session Health zone management % of HCC Strengthened 190 Care Committee current pilot) reports teams (Under supervision trained/recycled Health Care Trained/recycled of project team) Committees (HCC). Indicator Two: Number of signed 79 (Previous HCC 120 190 190 190 4 (first Contracts Project team - % of contract 190 Social and evaluated of pilot) contract, then archived, signed and contracts between contracts of refreshing of Activities report evaluated. the citizens, the Healthcare contract each Healthcare Committees and year). Committees and medical staff medical staff signed. Medium Score of Medium Score of Score Score in Score Score Quarterly Compilation of Provincial health division - Evolution of performance of the first evaluation in progress in in performance Health Zone management performance Health care progres progre progres score team indicators of Committees and s ss s healthcare medical staff Contract signing CSOs Committees are concern and medical partners staff. involved in the carrying out of . the project: 36 CSOs, 190 Health care committees and medical staff of health facilities, 11 Health zone management teams, and 2 , provincial health divisions. The health care committees and medical staff contracts have an RBF component that will help to 10 Page 59 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) evaluate the social contract between citizens quarterly. Their performance is evaluated through a set of indicators by the health Zone management teams, the last are evaluated by the Provincial health division, and CSO are in charge of the cross- verification on the field Indicator Three: Number of identified Result of the Vulnera Rate in Rate Rate in Twice Impact study Project team routine Rate of Improved access to vulnerable people baseline impact ble progress in progres (Baseline and report collection of indicators identified healthcare facilities who access to study people progre s end impact functionality of the Health vulnerable for vulnerable healthcare facilities are ss study) Committees (Under the people who groups identifi lead of a consultant) access to ed healthcare drawn Quarterly Routine Health facilities staff facilities on collection of specific indicators of criteria Health facilities INTERMEDIATE RESULTS Intermediate Result, Component 1: Reinforcing the health facility committee system Intermediate Result Number of health Result of pre and 190 190 4 (Training & Training session Health Zone management Rate of health indicator One: committees members post test training health health recycling) reports and team committee Health committees who know their roles session, and the commit commit impact study members who know their roles baseline impact tees and tees Twice report Project team according with know their and are equipped to study their and (Baseline the impact study roles. assure it. member their and end s membe impact rs study) Number of The 79 health 7 (1 annual 6, Functionality Health Zone management Rate of health functioning health committees of the semester) assessment team (cross- verification by committees with committees pilot and their based on criteria CSOs survey) medium or high members functionality level. 11 Page 60 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) Intermediate Result Health committees The 79 health 190 190 190 190 4 (first Follow-up Cordaid team (For % of Health indicator Two: which contract with committees of the contract, then reports of provincial level committees Health committees the Health Zone pilot refreshing of provincial health verification) which has have capable management team contract each division reports contract with referees at the year). and contracts Provincial team (For Health the Health Zone upper-level (health Health committees archived. zone follow-up to health management zone) which are regularly facilities) team supervised , monitored and Quarterly Routine Health Health zone team (For % of health evaluated by the follow-up Zone and health HMIS routine reports) committees health Zone team facilities HMIS which are reports (Cross CSOs (Cross-verification) regularly verification by supervised , CSOs survey) monitored and evaluated by the health Zone team Intermediate Result, Component 2: Re-appropriation of the health facility through community actions and projects Intermediate Result Availability of the Previous and Reflecti The final 1 Draft or final Project team The new terms indicator One: A new guidelines and obsolete guidelines, on ToR is ToR of HFC. of reference of new social contract refreshed framework and the back ground about available committees is between the for health facility of the pilot project the new and Activities available citizens, the committees HFC dissemin reports committee and the ToR is ated. medical staff is conduct established. , a draft of new ToR is availabl e Intermediate Result Score-card process Execution of Score- 120 All HFC All All 4 Activities Health zone management % of HFC indicator Two achieved card process in 79 health have HFC HFC reports team (Cross verification by which have Trust is restored. Health areas of the commit conducte have have CSOs) done at least 3 pilot tees d at least conduct conduc score-card have one ed at ted at process. conduct score- least least ed at card their their least process second third one score- score- score- card card Page 61 of 66 card process proces 12 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) process. s Intermediate Result Number of activities 0 50 50 50 3 Minutes of Project team % of activities indicator Three: being funded to selection with to improve the Activities to improve health lists of laureates provision of improve health service delivery health services services. HF C( ≥ 2/3) Intermediate Result, Component 3: Integrating the poorest and most vulnerable fringes of the population in the health facility committee and decisions on health-care services. Intermediate Result HFCs which respect a Result of inclusion Selectio Scaling- All All 4 Activities and Health Zone management % of HFC indicator One: The quota-system for the approach in 79 n of the up in 90 HFC HFC survey reports teams which respect a committee participation of health committees 100 others quota-system represents vulnerable groups , of the pilot HFC to HFC CSOs for the everybody, with a experim participation of specific attention ent the vulnerable on the poorest inclusio gro ups.( ≥ 3/4) groups. n approac h Intermediate Result HFCs with social 0 100 190 190 3 Activities and Health Zone management % of HFC with indicator Two: commission working survey reports teams social Socio-economic within them commission barriers to access CSOs working within to care by them. ( 4 ≥ 3/) vulnerable people are lowered. Intermediate Result, Component 4: K & L: health facility committees are fully integrated into the health system and beyond. Intermediate Result A national framework 0 0 0 0 1 1 The framework Project team New law and indicator One: is organized report and guidelines for Health facility production health facility committees are a committees is well-known available strategy for improving access to health-care. Intermediate Result The sessions of 0 0 0 4 4 2 Activities report Provincial health division Number of indicator Two: A “co mité of technical eligible sessions system for technique/commission commissions (At least 2 permanent s provincials de and health expected reflection and p i lotage d e a l sa ntéâ€? provincial sessions for evolution of the wich include the HFC running each province) committees exists. topics in their committees. discussions. These Page 62 of 66 commissions 13 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) and committees are established by the health ministry of DRC as the only framework of dialogue in the health sector at the provincial level. They meet half-yearly. Intermediate Result The sessions of 0 0 0 1 1 2 Activities report Secrétariat Général à la Number of indicator Three: “comité/commissions of technical santé (Cen tral eligible sessions National attention nationals de pilotage commissions administration of the Health (At least 2 is given to health de la santéâ€? wich and health ministry in DRC), with expected facility committees include the HFC provincial involvement of project sessions for and other social topics in discussions. running team each province) accountability committees. measures in primary health These care. commissions and committees are established by the health ministry of DRC as the only framework of dialogue at the national level of the health sector. They meet half- yearly. 14 Page 63 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) A. RESULTS INDICATORS A.1 PDO Indicators Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Strengthened 190 Health Number 79.00 0.00 111.00 105.00 Care Committees (HCC). 30-Sep-2014 30-Sep-2014 18-Nov-2018 16-Feb-2018 Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 190 Social contracts Number 79.00 0.00 190.00 185.00 between the citizens, the Healthcare Committees and 30-Sep-2014 30-Sep-2014 18-Nov-2018 16-Feb-2018 medical staff signed. Page 64 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Improved access to Number 4.00 0.00 8.00 6.00 healthcare facilities for vulnerable groups 30-Sep-2014 30-Sep-2014 18-Nov-2018 16-Feb-2018 Comments (achievements against targets): A.2 Intermediate Results Indicators Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Health committees know Number 79.00 0.00 190.00 185.00 their roles and are equipped to assure it. 30-Sep-2014 30-Sep-2014 18-Nov-2018 16-Feb-2018 Page 65 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) Comments (achievements against targets): Page 66 of 66 Confidential The World Bank Reinforcing SAcc of health services by supporting health committees and the community diagnosis in Bas Congo and South Kivu (P150874) .. Page 67 of 66 Confidential