September 2016 Monitoring and Evaluation (M&E) of Anti-Corruption Action Plans 1 1 This input note was prepared by Elena Georgieva-Andonovska. September 2016 Contents 1. Background and objectives .............................................................................................................. 3 2. What is M&E and why it matters..................................................................................................... 3 3. M&E of anti-corruption measures ................................................................................................... 4 4. M&E tools ......................................................................................................................................... 5 4.1. Logical framework ........................................................................................................................ 5 4.2. Results framework ....................................................................................................................... 7 4.3. Theory of change (ToC) ................................................................................................................ 9 5. Results indicators ........................................................................................................................... 12 6. Mitigating Risks in Sectors ............................................................................................................. 17 7. The role of civil society in M&E...................................................................................................... 19 Annex 1. Types of M&E, Differences between M&E and M&E applications ....................................... 21 Annex 2. Impact and Result Evaluation in Colombia ............................................................................ 25 Annex 3. List of references and additional resources ........................................................................... 26 Annex 4. Specific Skill Requirements and M&E Competencies ............................................................ 28 List of Figures Figure 1. Example of a results chain ............................................................................................................. 8 Figure 2. Five-step approach to building a theory of change in anti-corruption interventions ................. 10 Figure 3. Theory of Change Analysis for Anti-corruption Enforcement Work ............................................ 10 Figure 4. Example of a logic model with a theory of change ...................................................................... 11 Figure 5. Examples of indicators and data sources for targeting corruption ............................................. 12 Figure 6. Disaggregation of a sample indicator .......................................................................................... 14 Figure 7. Key tools to identify, track and measure corruption risks and corruption in the health sector . 18 List of tables Table 1. Example of a logical framework matrix and guiding questions ...................................................... 7 Table 2. Example of a results framework...................................................................................................... 8 Table 3: Overview of skills for monitoring and evaluation ......................................................................... 28 September 2016 1. Background and objectives There are constant and growing pressures on governments and organizations around the world to demonstrate the results of their programs and activities. Governments are increasingly asked to be more responsive to demands from internal and external stakeholders for good governance, accountability, transparency and greater efficiency and effectiveness. As demands for greater accountability and results have grown, there is an accompanying need for useful and useable results-based monitoring and evaluation systems to support the management of policies, programs, and projects. The same motivation is driving Italy’s public agencies, which need to monitor and evaluate the effectiveness of their anti- corruption action plans. The purpose of this note is to provide public agencies in Italy with guidance and tools on how to conduct monitoring and evaluation (M&E) of their anti-corruption action plans. The note is structured as follows: Section 2 presents what M&E is and why it matters; Section 3 discusses M&E of anti-corruption measures; Section 4 presents useful M&E tools; Section 5 discusses good practices in selecting results indicators. The note contains also additional information on the types and applications of M&E (Annex 1), a list of references and additional resources (Annex 2) and information on the skills and competences required for conducting M&E (Annex 2). 2. What is M&E and why it matters2 The overall objective of an M&E systems is to strengthen public sector efficiency, effectiveness, accountability and transparency, with a view to enhancing service-delivery and ultimately contributing to the achievement of national goals. Monitoring and Evaluation (M&E) is a powerful public management tool that can be used to improve the way governments and organizations achieve results. Just as governments need financial, human resource, and accountability systems, they also need good performance feedback systems. Governments don’t build M&E systems because they have intrinsic merit. Governments build M&E systems because (1) those systems directly support core government activities, such as the budget process; national planning; development of policies and programs; and the management of ministries, agencies, programs, and activities, or (2) provide information in support of accountability relationships. Thus, M&E is not an isolated function of government, but rather part of the broader public sector performance system and is often linked to public sector reforms such as performance- based budgeting, evidence-based policy making, results-based management, and the like. The Organization for Economic Cooperation and Development (OECD) defines monitoring and evaluation as follows:  Monitoring is a continuous function that uses the systematic collection of data on specified indicators, to provide management and the main stakeholders of an ongoing development 2 Adapted from “Gorgens, Marelize; Zall Kusek, Jody. 2009. Making Monitoring and Evaluation Systems Work: A Capacity Development Toolkit. World Bank. © World Bank. https://openknowledge.worldbank.org/handle/10986/2702 License: CC BY 3.0 IGO.” September 2016 intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds.  Evaluation is the systematic and objective assessment of an ongoing or completed project, program, or policy, including its design, implementation, and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact, and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipients and donors.3 Thus, monitoring and evaluation are interlinked but independent exercises, with different processes and purposes. Monitoring gives information on where a policy, program, or project is at any given time (and over an extended period) relative to its targets and outcome goals. It is descriptive and looks at the “what” and whether “things are done right”. On the other hand, evaluation gives evidence about “why” targets and outcomes are, or are not, being achieved and “whether the right thing is done”. While monitoring examines progress vis-à-vis objectives and thus assumes causality, evaluation seeks to establish and use causality. Annex 1 summarizes some of the key issues differentiating between monitoring and evaluation. 3. M&E of anti-corruption measures To understand how effective anti-corruption measures are in reducing the risk of corruption, it is important to establish a regular monitoring mechanisms to measure the impact and changes, and to make adjustments to policies and institutional arrangements based on the findings. Such mechanisms should ideally be set up in connection with developing the anti-corruption strategy, program or action plan. This would require the government/agency to think about what type of a review mechanism it wants to introduce, which areas should be reviewed, what performance indicators should be used, which agencies should be responsible for the review(s), what resources are available or required, who should participate in the review, should external stakeholders also be included, how frequently should the review(s) be held and how should the results be reported.4 However, international experience shows that the impact of various anti-corruption measures is difficult to measure and ascertain. Governments often present outputs of anti-corruption policies and measures while outcomes remain ambiguous. Different countries use different approaches to the implementation and M&E of anti-corruption measures. In terms of the organization of monitoring, typically the responsibility for monitoring is shared between some central body, which gathers data on implementation, and all of the agents who are involved in the implementation and provide the data. Sometimes such central body also elaborates methodology for the monitoring. For example, Croatia has established a complex institutional structure for the monitoring of anti-corruption measures – the Committee for the Monitoring of the Implementation of Anti-corruption measures (presided by the Minister of Justice) and the National Council for Monitoring of the Implementation of the Anti-corruption Strategy (body of the parliament). Implementers of anti-corruption measures report regularly to the Minister of Justice. Another example of a mixed (centralized/ decentralized) approach is found in Lithuania where state and municipal institutions regularly measure the efficiency of the anti-corruption activities they conduct (for example, assess the 3 Source: OECD 2010. Glossary of Key Terms in Evaluation and Results Based Management, http://www.oecd.org/dac/evaluation/2754804.pdf 4 OSCE 2016. Handbook on Combating Corruption, http://www.osce.org/secretariat/232761. September 2016 quality and efficiency of measures provided for in the plans of measures for the implementation of the anti-corruption programs approved by their internal legal acts). On the central level, Lithuania’s Special Investigation Service (SIS) regularly assesses how state and municipal institutions implement anti- corruption measures and provides proposals concerning their improvement. The SIS analyses the efficiency of anti-corruption activities conducted by state and municipal institutions (for example, anticorruption reports made by the departments of these institutions, etc.) and regularly provides methodical assistance for the staff of state and municipal institutions performing the functions in the area of corruption prevention.5 Along with the organization aspect of measuring achievement, the choice of appropriate indicators is crucial. Typically anti-corruption policy planning documents and action plans are designed at least partly as tables with appropriate indicators. These may be indicators of outputs (direct actions as such) and outcomes (substantial impact of the actions)- see Section 5 for more details on how to select good indicators. For example, in Lithuania, the National Anti-Corruption Program (NACP) is the main inter- institutional action plan and anti-corruption programs adopted by other institutions must be oriented towards the objectives and goals specified in the NACP. The measures should attain tangible and measurable results and the NACP specifies criteria of the results (for example, increase of confidence in state institutions, quick provision of public services, simplification of the procedures for issuing licenses and other administrative requirements laid down in legal acts, public involvement in the law-making process, etc.). Box 1 details the method used by Georgia to monitor and evaluate its anti-corruption strategy and action plan. 4. M&E tools This section provides an overview of the most commonly used M&E tools, which can help public agencies think through the design of their M&E activities. Agencies can select the tool that they deem most helpful depending on their goals and the planning stage they are at (e.g. designing an intervention, choosing indicators, monitoring and evaluation, etc). A combination of different tools can also be used depending on the objective of the evaluation and the content of the program to evaluate. Further guidance on M&E tools and approaches can be accessed in the World Bank’s publication “Monitoring and Evaluation (M&E): Some Tools, Methods and Approaches”. 4.1. Logical framework A logical framework (or logframe for short) helps to clarify the objectives of any project, program, and policy or action plan. A logframe usually looks like a matrix (see Table 1) and aims to present information about the key components of an intervention in a clear, concise, logical and systematic way. It aids in the identification of the expected causal links—the “program logic”—along the results chain (see the next section). It might lead also to the identification of performance indicators at each stage in this chain, as well as risks which might impede the attainment of the objectives. The logframe is a vehicle for engaging partners in clarifying objectives and designing activities. During implementation of a program the logframe serves as a useful tool to review progress and take corrective action. 5 OECD 2015. Prevention of Corruption in the Public Sector in Eastern Europe and Central Asia. September 2016 Box 1. Georgia: Monitoring and evaluation methodology In 2015, the Anti-Corruption Council (ACC) of Georgia prepared a new Anti-Corruption Strategy and Action Plan (2015-2016) and a new Monitoring and Evaluation Methodology. The methodology enables permanent tracking of the status of implementation of measures, comprehensive assessment of the quality of implementation, evaluation of impact and identification of any existing gaps and challenges, as well as necessary budgetary, human or other resources to implement the envisaged measures. The Methodology was developed by the Secretariat of ACC as a result of a participatory process with the involvement of civil society, businesses and academia. The Methodology includes both monitoring and evaluation components and consists of three elements:  Tracking progress with a monitoring tool- the tool was prepared in partnership with 14 public agencies responsible for the anti-corruption Action Plan implementation. The tool shows progress on the actions taken by different agencies, the challenges faced during implementation and the quality of implementation. The tool is shared with members of civil society for their input. Based on the collected information and a discussion at a session of the working group on anti-corruption, a final assessment is prepared by the ACC Secretariat.  The tool enables tracking the status of implementation of a measure by looking at the following criteria: (a) implementation has not started; (b) is underway; (c) was suspended; (d) was terminated; (e) was completed; as well as the assessment of the level of implementation using the following ratings: (a) fully implemented; (b) largely implemented; (c) partially implemented and (d) not implemented.  Monitoring results through progress reports– these reports are compiled by the Secretariat, based on the narrative submissions of the responsible agencies and the monitoring tool. It describes the measures carried out, status of implementation and ratings through the same participatory process as described above. Annual reports are submitted for adoption to the ACC, presented to the Government of Georgia and made public.  Evaluating impact through evaluation report at the end of the planning cycle – an analytical document produced by the Secretariat through a participatory process (including civil society inputs, round table discussion and ratings) at the end of the planning cycle (every 2 years) containing a comprehensive assessment of the Action Plan’s implementation and the impact of the anti-corruption policy on the anti-corruption outlook of the country as well as critical analysis of existing challenges and needs for future action. Data sources use in the Evaluation Report include the assessments and ratings of international organizations, NGO reports, as well as field visits to implementing agencies and interviews with their staff. Source: Adapted from OECD 2015, Prevention of Corruption in the Public Sector in Eastern Europe and Central Asia September 2016 Table 1. Example of a logical framework matrix and guiding questions Intervention logic Objectively verifiable Sources and means Assumptions indicators of of verification achievement Overall What are the overall What are the key What are the objectives broader objectives to indicators related to sources of which the action will the overall objective, information for contribute i.e. at the impact level? these indicators? Specific What specific Which indicators Which factors and objectives objectives is the action clearly show that the conditions outside intended to achieve in specific objective of of the order to contribute to the action has been organization’s the overall objectives achieved, i.e. at the responsibility are outcome level? necessary to achieve the objective? Which risks should be taken into consideration? Expected The results are the What are the What are the What external results outputs envisaged to indicators to measure sources of conditions must be achieve the specific whether and to what information for met to obtain the objective extent the action these indicators? expected results on achieves the expected schedule? results? Activities What are the key What are the means What are the What preconditions activities to be carried required to implement sources of are required before out and in what these activities, e.g. information about the action starts? sequence in order to personnel, equipment, action progress? produce the expected training, studies, What are the results? supplies, operational action’s costs? facilities, etc.? Source: Author 4.2. Results framework A results framework represents the underlying logic that explains how the objective of an intervention is to be achieved. This is done by translating the results chain (see Figure 1) of an intervention into indicators that measure the degree to which inputs are being transformed into specific activities and outputs, and the degree to which a relevant target population is using those outputs as the anticipated outcomes of the intervention. A results framework builds on, and helps articulate, a project’s or program’s theory of change (see below)—the causal pathways from the planned interventions to the intended outcomes. September 2016 Figure 1. Example of a results chain Source: Adapted from World Bank 2013, “Results Framework and M&E Guidance Note”. Results frameworks typically have three main elements (see Table 2): (a) a statement of the intervention (project, program, plan, etc.) objectives; (b) a set of indicators to measure outcomes and impact that are linked to the objectives and a set of intermediate results to track progress toward achieving outcomes and promoting change; and (c) M&E arrangements specifying clear units of measurement for each indicator, baselines, annual and final targets for each indicator as well as the roles and responsibilities for collecting, reporting, and analyzing data on those indicators. Table 2. Example of a results framework Project/Program/Plan Objective(s): E.g. To reduce corruption in the construction sector Indicators Target Values Data collection and reporting Indicator Baseline YR1 YR2 YR3 YR4 End Frequency Data collection Responsibilit Name Target of reports instruments y for data collections Indicator 1,2, 3, etc Intermediate Indicators Target Values Data collection and reporting Indicator Baseline YR1 YR2 YR3 YR4 End Frequency Data collection Responsibilit Name Target and reports instruments y for data collections Indicator 1, 2,3, etc September 2016 Source: Adapted from World Bank 2004. “Monitoring and Evaluation (M&E): Some Tools, Methods and Approaches”. 4.3. Theory of change (ToC) A result framework must be based on a clear understanding and specification of how planned interventions are expected to lead to desired outcomes and change in behavior. A theory of change (ToC) is a way of teasing out the underlying logic(s) and assumptions of a policy, program or plan. Unlike the logical framework approach, a ToC is a conceptual map that focuses not only on the linkages between program components but also on the preconditions and assumptions that enable the intervention to work. ToCs are therefore useful in analyzing the complex and power dependent social transactions that anti-corruption interventions often seek to influence. Mapping the anatomy and internal logic of an intervention is necessary at the strategy, program, and project levels. It is also a useful tool for integrating or mainstreaming anti-corruption components into larger programs. Using a theory of change to approach understand a program’s logic and define results indicators can be particularly helpful in the case of anti-corruption measures. Countering corruption is about changing behaviors, attitudes, and the structures of accountability, transparency, and integrity. This kind of change is a contentious, political process. Linear models of change that rely on a stable set of outputs are rarely adequate to capture the multiple actors, relationships, and behaviors that characterize anti-corruption efforts. Anti-corruption efforts are not bound by location, time, or actor. Moreover, corruption has multiple causes (direct and indirect) and manifests differently according to context. Consideration of these complexities is required to identify a causal process and consistent theory of change. However, often anti- corruption interventions do not have a built-in theory to guide the results chain, making it difficult to meaningfully track the progress and measure the desired results. Theory of change can serve as a conceptual map of the change process from start to finish. It makes known the underlying assumptions about why and how a project will be successful, and maps out the intermediate steps that must be taken to reach a long-term result. An explicit theory of change outlines a robust framework for monitoring, and more importantly, leaves an evidence trail for evaluation purposes.6 The theory of change process for anti-corruption interventions includes five steps (see Figure 2). It starts with an analysis of the political and economic processes that prevail in the project context, including the incentives, relationships, distribution and contestation of power among stakeholders. The process then moves to identifying long-term goals, and mapping out the steps needed to reach those goals. The “reality check” is in considering the internal logic of the results chain, and the external factors that may influence outcomes. 6Source: UNDP 2015. A user’s guide to measuring corruption and anti-corruption, https://www.globalintegrity.org/wp- content/uploads/2015/11/Users-Guide-Measuring-Corruption-Anticorruption.pdf September 2016 Figure 2. Five-step approach to building a theory of change in anti-corruption interventions Source: Jesper Johnson, Theories of change in anti-corruption work: A tool for program design and evaluation, Chr. Michelsen Institute (CMI), 2012. Building the theory of change requires identifying where preconditions are necessary for plausible causal pathways to occur, as well as the challenges that may prevent goals from being achieved (shown as red boxes in Figure 3). Based on the outcomes identified, and given the prevailing political and economic constraints, indicators and targets are developed to monitor performance and assist with evaluations. The logic of the ToC should then be tested and validated by inviting relevant stakeholders to provide comments. Figure 3. Theory of Change Analysis for Anti-corruption Enforcement Work Source: Adapted from Jesper Johnson, Theories of change in anti-corruption work: A tool for program design and evaluation, Chr. Michelsen Institute (CMI), 2012. September 2016 Logic models are often incorporated into theories of change, allowing the development of indicators to track progress. The indicators developed through a theory of change process should meet certain SMART (specific, measurable, attributable, realistic, and time-bound – see also Section 5 for a more detailed discussion) conditions in order to serve as effective measurements of results (see next section). An example of linking a results chain to a ToC showing how anti-corruption activities can lead to improvements in transparency and accountability is presented in Figure 4 below. Figure 4. Example of a logic model with a theory of change Source: UNDP Global Anti-corruption Initiative (GAIN) (2014-2017). September 2016 5. Results indicators7 A key step8 in building a strong M&E system is to select the right indicators to measure the desired outcomes. According to the OECD, an indicator is a quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement, to reflect the changes connected to an intervention, or to help assess the performance of an actor. It is through the regular measurement of key performance indicators that we can determine if outcomes are being achieved. However, progress needs to be monitored at all levels of an intervention to provide feedback on areas of success and areas in which improvement may be required. Hence, indicators should be developed for all levels of the results chain in order to monitor progress with respect to inputs, activities, outputs, outcomes, and impacts. By measuring performance indicators on a regular, determined basis, managers and decision-makers can find out whether projects, programs, and policies are on track, off track, or even doing better than expected against the targets set for performance. Figure 5 provides examples of indicators and data sources for targeting corruption. Figure 5. Examples of indicators and data sources for targeting corruption Target Proposed Indicator Results Possible Data Sources Type of Data Global or chain National Enact legislation Increase the extent to Input UNCAC gap analysis and Administrative, National, designed to limit which national laws are Self-Assessment Report Assessment Some corruption compliant with UNCAC aspects can be globally compared Reduce incidence Number of individuals Outcome International Crime Perceptions, Global of bribery cases that report paying a Victim’s Survey Experiences Regional by X % by year Y bribe when interacting TI Global Bribery National with government Barometer officials Regional public opinion surveys Number of firms that Business Environment report paying bribes to and Enterprise Survey obtain services (BEEPS) GAC Surveys Countries ensure Increases the extent to Input UNCAC gap analysis and Administrative, National, adequate provision which national laws are Self-Assessment Report Assessment Some to detect and compliant aspects can prevent illicit flows with UNCAC be globally compared Halve illicit Volume of illicit Outcome Global Financial Integrity Administrative National, financial flows financial flows Global by year Y Source: UNDP Global Initiative on Anti-corruption (GAIN), 2014 7 Adapted from “Results Framework and M&E Guidance Note”, World Bank 2013, http://siteresources.worldbank.org/PROJECTS/Resources/40940-1365611011935/Guidance_Note_Results_and_M&E.pdf 8 For more information on the key steps to building a successful M&E system, see Kusek, J. Z., & Rist, R. C. (2004) Ten steps to a results based monitoring and evaluation system. A handbook for development practitioners. The World Bank, available at http://documents.worldbank.org/curated/en/638011468766181874/pdf/296720PAPER0100steps.pdf September 2016 When selecting indicators, best practice is to choose indicators which are SMART (specific, measurable, attributable, realistic, and time-bound):  Specific means that the indicator measures only the design element (input, activity, output or outcome), which is intended for measurement, and not any other elements in the project. For example, if the target output is to train 100 civil servants, the specific indicator to be measured will be the number of civil servants trained;  Measurable means that there are practical ways of measuring the indicator, being clear and unambiguous in terms of what is being measured (e.g., avoid words like ―successful‖ unless it is possible to define exactly what it would mean in the project context). For quantitative proportions or percentages, this means that both the numerator and the denominator must be clearly defined (Box 3). For quantitative whole numbers and qualitative data, it means defining each term within the indicator such that there can be no misunderstanding as to the meaning of that indicator. This is critical for ensuring that the data collected by different people at different times are consistent and comparable;  Attributable means that the indicator is a valid measure of the targeted issue and the project/program can be credited for the changes in that issue;  Realistic means that indicators selected must be realistic in terms of their ability to collect the data with the available resources. Some indicators present major problems for data collection owing to the cost or skills required (e.g., large-scale sample surveys). Being realistic in planning and identifying collectable information ensures that it will, in fact, be collected. This is an important factor to consider and may lead to compromises on other criteria.  Time-bound has several connotations. First, indicators must be time-bound in terms of the time spent in data collection. Second, indicators must reflect the timing of collection, being cognizant of seasonal differences. Third, the time-lag between activities and output and outcomes must also be reflected in the indicators that are chosen. The SMART principles are relevant for all indicators aimed at monitoring and evaluating performance. In addition, when the focus is on activities and programs developed and implemented by individual agencies, the following principles can help design good indicators:  Be unambiguous: The definition of the indicators should be operationally precise and there should be no ambiguity about what is being measured or how to interpret the results. Indicator protocols should adequately document definitions and how results are to be interpreted (contextualized). Subjective language can make indicators ambiguous and red flags include words such as: (i) operational, implemented, developed, approved, adequate, satisfactory; (ii) strategy, action plan, process, recommendations.  Less is better: Avoid too many indicators. If possible, limit the number of outcome and impact indicators to five or fewer and the overall number of project/program indicators to not more than 15.  Make it easy: Indicator data should be easy to collect. If possible, select indicators for which data collection mechanisms and systems already exist in the agency.  Use cost-effective indicators: A common challenge is to select indicators that are appropriate to the conditions on the ground and that can be collected with a reasonable amount of resources and within a reasonable period of time. When possible, avoid indicators that are too expensive to monitor, particularly if they involve the use of baselines that are not easily obtained. September 2016  Use existing data: In most situations, it is preferable to anchor the measurement of results on existing data sources. Basic monitoring information is usually available through a combination of administrative databases and sample or census-based surveys. Selecting the data sources is a fundamental decision that should be made together with the selection of indicators.  Select the units of measurement carefully: Units of measurement may be quantitative (e.g. kilometers, numbers of people, percentages) or qualitative (i.e., yes/no). In either case, both the indicator and the definition of the unit of measurement need to be clear. Once an indicator has been selected, it might be useful to try to “unpack” it and understand better its different aspects. An example of how an indicator can be disaggregated is presented in Figure 6 below. Figure 6. Disaggregation of a sample indicator Source: Johnson et al., How to monitor and evaluate anti-corruption agencies: Guidelines for agencies, donors, and evaluators, Chr. Michelsen Institute (CMI), 2011 September 2016 A common mistake, however, is to over-engineer an M&E system and select too many indicators. This has been the case with the national M&E systems of Colombia and Chile, which try to measure hundreds and even thousands of indicators (Boxes 2 and 3). This can lead to over- complication of the system, an overwhelming amount of information, in which the main messages get lost, and additional burden to the workload of program managers. The appropriate number of performance indicators depends on the scope and complexity of a program and on the type of performance indicator. Indicators can focus on the various parts of the results chain, beginning with inputs, moving through activities and outputs, and culminating in outcomes and impacts (long-term outcomes). Senior officials would tend to make use of high-level strategic indicators such outcomes and impacts, which as a rule of thumb should not be more than five per objective. Line managers and their staff, in contrast, would tend to focus on a larger number of operational indicators that target processes and services. September 2016 Box 2. Colombia’s M&E system The government of Colombia decided in 1991 to create an M&E system, which would be based initially on the World Bank’s approach to evaluation. A constitutional requirement for evaluation was introduced later that year, and the Bank and other donors quickly followed with a range of technical and financial support. The government also introduced a series of laws, decrees, and regulations to buttress the M&E system. There are two main components of the system, which is known by its Spanish acronym, SINERGIA (Sistema Nacional de Evaluación de Resultados de la Gestión Pública, or National System for Evaluation of Public Sector Performance): 1. SIGOB: Starting in 2002, Colombia developed the Government Goals Monitoring Information System (Sistema de Gestión y Seguimiento a las Metas del Gobierno—SIGOB), which uses a logical structure to consolidate and manage goals and indicators for all national government programs, including strategies for achieving the president’s priorities and the strategic objectives of the National Development Plan (NDP). SIGOB is an institutional arrangement and a technology platform that allows monitoring of performance indicators in real time, providing readings of government performance from different perspectives (for example, four-year NDP objectives, cross-cutting strategies such as poverty reduction, agency performance, presidential priorities, or long-term goals such as the Sustainable Development Goals). o To ensure the quality and timeliness of information, indicators and targets are: i) defined with the participation of the technical teams from the National Planning Department and line agencies responsible for overseeing and implementing sectoral and national programs; ii) validated with the sector ministers and agency directors; and iii) discussed with and approved by the president in the Council of Ministers. Once these internal processes are set, targets and indicators are made public through printed documents and through the SIGOB Web site, to which citizens have open access. Responsible program officials (“goal managers”) in ministries, whose names are publicly displayed in the system to foster information reporting accountability, report results electronically via the system. o SIGOB consolidates performance indicators and facilitates public consultation, widely disseminating the results of government programs. The system interoperates with the Integrated Financial Information System (Sistema Integrado de Información Financiera—SIIF), allowing the level of appropriation and budget execution associated with the objectives and goals to be tracked. The presidency and the National Planning Department regularly check the information and conduct managerial oversight meetings with the ministers and directors of institutions to identify progress and define courses of action to resolve implementation problems that may affect target achievements. SIGOB currently has 626 indicators: 104 on impacts (16 percent), 371 on outcomes (60 percent), and 151 on outputs (24 percent). Based on reported information, SIGOB facilitates the preparation of monthly, quarterly, and annual reports on government progress, including the Annual Report of the President to Congress. 2. SINERGIA’s second main component is the series of evaluations that are conducted in parallel to SIGOB . At the end of 2006, 15 evaluations were being conducted or had recently been completed, with another 22 planned for the following five years. The total cost of these evaluations is $11.1 million. The evaluations are contracted out to academia or consultants, with oversight by the planning department in close collaboration with both the evaluated agency and the donors funding the evaluation. Rapid evaluations are also being piloted, with a view to mainstreaming them in the budget and planning work of the finance and planning ministries. The three main types of evaluation in Colombia are rigorous impact evaluations, “institutional” evaluations, and “management” evaluations—the latter two focus on management and process issues. These evaluations are collaborative exercises involving the planning department, the sector ministry or agency responsible for the program being evaluated, and donors. Most of the funding for these evaluations is provided through donor loans. The impact evaluations are contracted out to academia or consultants, with oversight by the planning department in close collaboration with both the evaluated agency and the donors funding the evaluation. Sources: Mackay, Keith. 2007. How to build M&E systems to support better government. Washington, DC: World Bank; Castro, Manuel Fernando. 2011. Defining and using performance indicators and targets in government M and E systems. PREM notes. Washington, DC: World Bank. September 2016 Box 3. Chile’s M&E system The government of Chile has progressively developed its M&E system over a number of years, with most of the development having occurred since 1994. The system has been largely designed, implemented, and managed by the powerful Ministry of Finance (MoF), with the overall objective of improving the quality of public spending. The high utilization of M&E information is very impressive, and this alone makes Chile’s M&E system the strongest in Latin America and one of the strongest in the world. Chile has demonstrated that a whole-of-government M&E system can be built and operated at a relatively low cost. The six main components of the M&E system are as follows:  Ex ante cost-benefit analysis is required for all government projects (since 1974). This component is managed by the ministry of planning; it is the only component not managed by the MoF.  Performance indicators are collected for all government programs. They were first introduced on a pilot basis in 1994. The number of performance indicators has increased rapidly in recent years, from 275 in 2001 to about 1,550 currently. Of these, 25 percent relate to process issues, 57 percent to government outputs (that is, goods and services produced), and 18 percent to outcomes. Each ministry and agency provides the information to the MoF; there are about 11 indicators per entity. Entities are expected to have management information systems in place to produce this information. The MoF undertakes some data checking and data audits, and it includes the performance information in the budget bills it prepares each year.  Comprehensive management reports (1996). Each ministry and agency prepares one of these reports annually, based on MoF guidelines. The reports are intended to be the main public disclosure document. They report spending, use of funds, and performance; the reports thus draw heavily on the performance information that entities are required to produce and on the evaluations commissioned by the MoF. The reports also describe the progress made by the entity in achieving the formal institutional commitments it has agreed to with the MoF; these comprise specific actions the entity has promised to implement to improve its performance. The draft reports are reviewed by the MoF and the ministry of the presidency, and entities make any necessary revisions. The final versions of the reports are sent to the Congress.  Evaluations of government programs (1996). Some 185 of these rapid reviews have been conducted so far (that is, until the end of 2006). They entail the clarification and agreement (between the MoF and the ministry or agency whose program is being evaluated) of detailed program objectives. A logframe analysis and desk review of program performance is conducted, drawing on available performance information. Their average cost is about $11,000, and they usually take four to six months to complete.  Rigorous impact evaluations (2001). These evaluations involve primary data collection, sophisticated data analysis, and often the use of control groups. Eighteen have been completed so far, at an average cost of $88,000 and taking up to 18 months to finish. Excluding defense spending and income transfer payments, more than 60 percent of government spending has been evaluated so far.  Comprehensive spending reviews (2002). These reviews assess all programs within a particular functional area or ministry. They examine issues of inefficiency and duplication of programs. Eight of these desk reviews have been conducted so far, at an average cost of $48,000. Source: Mackay, Keith. 2007. How to build M&E systems to support better government. Washington, DC: World Bank 6. Mitigating Risks in Sectors M&E approaches for anti-corruption interventions in sectors are primarily concerned with risk assessments and the mechanisms for monitoring both risks and corrupt practices. Risk is composed of the likelihood of a corrupt practice occurring and the subsequent impact of that corrupt practice. September 2016 Assessments prioritize risks according to the characteristics and vulnerabilities of specific sectors such as education, health, agriculture, forestry, etc., and propose solutions to mitigate or eliminate them. Risk assessments focus on identifying the types of arrangements or practices that may lead to corrupt behavior, and may or may not include scores or ratings. They consist of evaluative data about organizational procedures, resource chains, and practices that can assist organizations in preventing corruption. Data is collected by experts through a variety of means, including interviews, surveys, observation, and then combined in the analysis stage with administrative data. The resulting data is not a measure of corruption, but rather, a measure of corruption risk. This data provides the basis for the corruption risk management action plan. The action plan should identify short-, medium-, and long-term priorities and indicators to manage the risks. Monitoring and evaluation mechanisms in sector approaches consist of a variety of tools, including anti- corruption instruments, data-collection methods, corruption/anti-corruption measurement methodologies, and financial tools prevalent within the sector. The selection of the most appropriate mechanisms depends on the purpose of the intervention and the available resources at hand. Figure 7 below presents some tools and data sources to identify, track and measure corruption risks and corruption in the health sector.9 Figure 7. Key tools to identify, track and measure corruption risks and corruption in the health sector Area Issue Tools to identify and track problems General Cross-cutting Political economy analysis in the health sector Vulnerability to corruption assessments Value chain analysis Sector accountability assessment Value for money audits Analysis of governance in health care systems Budget Budget processes Public Expenditure and Financial Accountability indicators (PEFA) Focus groups and interviews with public officials, recipient institutions, and civil society Payroll leakages Public Expenditure Tracking Surveys and Reviews (PETS, PERS) Household surveys Focus groups with public officials and health workers In-kind leakages Public Expenditure Tracking Surveys (PETS) Quantitative Service Delivery Surveys Facility surveys Focus groups with public officials, recipient institutions, and health workers Pharmaceuticals WHO Good Governance in Medicines program to assess transparency in drug supply and management International Drug Price Indicator Guide Internet based drug procurement databases Individual Job purchasing Official administrative records combined with facility surveys providers Interviews with public officials and former officials Governance and Anti-Corruption Country Diagnostic surveys 9 Adapted from OSCE 2016. “Handbook on Combating Corruption”. September 2016 Health worker Quantitative Service Delivery Surveys absenteeism Surprise visits Direct observation Facility records Focus groups or interviews with facility heads and patients Informal Informal payments Household surveys (e.g. World Bank Living Standards Measurement payments Surveys and Demographic and Health Surveys (DHS)) Facility exit surveys and score cards Focus groups/interviews with providers/patients and health staff Governance and Anti-Corruption Country Diagnostic surveys Corruption Perceptions of World Bank Governance Indicators (Control of Corruption), TI and Corruption Corruption Perception Index Perceptions Governance & Anti-Corruption Country Diagnostic surveys (WB) experience National level perception surveys by CSO and others Experiences of AfroBarometer, LatinBarometer, EuroBarometer, TI Global corruption Corruption Barometer National experience-based surveys Patient satisfaction surveys and report (score) cards Focus group surveys /studies Source: DFID, How To Note: Addressing Corruption in the Health Sector, 2010. Adapted from M. Lewis and G. Pettersson, “Governance in Health Care Delivery: Raising Performance,” World Bank Policy Research Paper, 2009. 7. The role of civil society in M&E Public agencies in Italy might also consider the beneficial role of civil society in M&E. In creating systems for monitoring and evaluating the implementation of AC plans, it can be tempting to focus only on data and information coming from government players, such as central ministries, sector ministries and agencies, subnational levels of government, and the Parliament or Congress. But this would ignore the important role civil society can play in the monitoring and evaluation of government performance. Civil society— nongovernment organizations (NGOs), universities, research institutes, think tanks, and the media—can play a role in M&E in several ways, including as both a user and producer of M&E information. For example, civil society can engage in third party monitoring and participatory monitoring activities and use various monitoring tools, which can provide the government with valuable additional information on their performance (see Box 4).  Third party monitoring (TPM) is defined as monitoring by parties that are external to an intervention’s direct beneficiary chain or management structure to assess whether intended outputs, outcomes, and impacts have been achieved by the project. TPM is mainly used to provide an independent perspective on project or government performance. It can be conducted by CSOs, think tanks, academic institutions, media, or private firms. These organizations generally have greater skills for monitoring than community representatives. However, there are large variations in skills, for example, between a firm that specializes in survey techniques and a grassroots CSO that specializes in social mobilization, advocacy, and facilitation. TPM usually involves project beneficiaries and at a minimum solicits their views in order to gather evidence and triangulate information. Yet, the findings and conclusions of third parties may not be fully aligned with the September 2016 views of project beneficiaries or communities, because they are ultimately meant to be the independent judgment of the organization conducting the monitoring.  Participatory monitoring refers to the active participation of project/program beneficiaries, affected people, communities, and other primary stakeholders in designing and implementing the monitoring. This definition goes beyond having consultations with primary stakeholders on predefined indicators, or asking them to provide information or feedback. It is built around agreeing on expected results, defining jointly with project beneficiaries about how to track progress, collecting required data, undertaking analysis, and developing practical action plans to resolve identified problems. Box 4. Tools of civil society monitoring  A community scorecard (CSC) or an Agency scorecard is a community-based monitoring tool that assesses services, projects, and government performance by analyzing qualitative data obtained through focus group discussions with the community. It usually includes interface meetings between service providers and users to formulate an action plan to address any identified problems and shortcomings.  Social Audit is a monitoring process through which project information is collected, analyzed and shared publicly in a participatory fashion. Social audits may go beyond the oversight of project finances and procurements to examine all aspects of the project, including level of access to information, accountability, public involvement, project outputs and outcomes. Social audits are typically carried out by community volunteers (social audit teams/committees) and findings are presented at a public forum/hearing.  A Citizen Report Card (CRC) is an assessment of public services by the users (citizens) through client feedback surveys. It goes beyond data collection to being an instrument for exacting public accountability through extensive media coverage and civil society advocacy that accompanies the process.  Citizen Satisfaction Surveys provide a quantitative assessment of government performance and service delivery based on citizen’s experience. Depending on the objective, the surveys can collect data on a variety of topics ranging from perceptions of performance of service delivery and elected officials to desires for new capital projects and services. Citizen satisfaction surveys have been used in many countries to monitor access to and quality of basic services. The surveys can be conducted by a government ministry or agency, private sector, or civil society organizations. Surveys generally contain relatively short questionnaires. Sources: World Bank, Social Accountability Guide, https://saeguide.worldbank.org/monitoring-non-state-actors; Van Wicklin III, Warren A.; Gurkan, Asli. 2013. How-to notes: participatory and third party monitoring in World Bank projects - what can non-state actors do? September 2016 Annex 1. Types of M&E, Differences between M&E and M&E applications Types of M&E10 Public agencies typically implement a combination of different monitoring activities as part of their overall management approach, with an increasing emphasis on monitoring results. Different types of monitoring include:  Compliance monitoring: Focuses on compliance with established standards and procedures.  Activity monitoring: Focuses on quantity, quality, and timeliness of inputs and activities.  Financial monitoring: Tracks cost of implementation according to budget and time frame.  Organizational monitoring: Tracks sustainability, institutional development, and capacity building in the project and potentially directs attention to problem areas.  Context monitoring: Focuses on the project context and environment, especially critical elements that can affect the implementation and progress of the project.  Beneficiary monitoring: Focuses on the perceptions of beneficiaries of the intervention.  Results monitoring: Focuses on achievement of planned results, especially outcomes and impacts. Evaluations can also be classified in several ways. For example, evaluations can be grouped according to the analytical paradigm (rationalist, pluralistic, participative), purpose (formative versus summative), content (goals, process, outcomes/impact), time perspective (ex-ante versus ex post), or the type of evaluator (internal versus external). Some of the most common types of evaluations are listed below.  Rapid assessment: This focuses mainly on potential impact identification and screening as a form of “impact pre-assessment.” It is useful in situations where time is short and there is need for a prompt response, such as establishment of refugee camps.  Process evaluation: This is a variety of the regular program evaluation (see next bullet) that focuses on the implementation and operations of a project or institution. It usually attempts to determine whether the project is aligned with its original design. The costs and time requirements for process evaluations are comparatively low.  Program evaluation: This is normally what is meant by the term “evaluation.” Evaluations are used to answer questions related to design, implementation, and results. The evaluation criteria recommended by the OECD are relevance, effectiveness, efficiency, impact, and sustainability (Box 5). The evaluation design, method, and cost vary considerably depending on the type of question one tries to answer. If cause-and-effect questions are being asked, the foundation for the evaluation is often a logic model or theory of change, which should be established at the beginning for each evaluation.  Impact evaluation: This is a specific kind of evaluation with a sophisticated methodology that measures changes in outcomes or impacts such as well-being that are attributable to the intervention. Rigorous methodology can establish various degrees of causality, using a counterfactual (what would have occurred in absence of the intervention) or a comparison group. Impacts and outcomes can materialize with a time lag. Therefore, impact evaluations must often start before an intervention to collect baseline data and continue afterwards to capture outcomes 10 Adapted from “How to monitor and evaluate anti-corruption agencies: Guidelines for agencies, donors, and evaluators”, Bergen: Chr. Michelsen Institute (U4 Issue 2011:8) 84 p. http://www.u4.no/publications/how-to-monitor-and-evaluate-anti- corruption-agencies-guidelines-for-agencies-donors-and-evaluators-2/ September 2016 and impacts. This approach can also prove useful for capturing unintended consequences linked to the intervention.  Organizational capacity assessment: So-called “performance assessments” in any field usually work by comparing the reality against predefined standards and criteria. As a result, these assessments are similar to audits, which assess the conformity of the intervention to procedures, norms, and criteria established in advance. They are not frameworks for evaluation. Organizational capacity “is the ability of an organization to use its resources to perform”. The assessment of such capacities includes an examination of systems and management processes, as well as of human, financial, and infrastructure resources. Box 5. OECD Evaluation criteria  Relevance: The extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor.  Effectiveness: A measure of the extent to which an aid activity attains its objectives. In evaluating the effectiveness of a program or a project, it is useful to consider the following questions: To what extent were the objectives achieved / are likely to be achieved? What were the major factors influencing the achievement or non-achievement of the objectives?  Efficiency: Efficiency measures the outputs -- qualitative and quantitative -- in relation to the inputs. It is an economic term which signifies that the aid uses the least costly resources possible in order to achieve the desired results. This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted.  Impact: The positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended. This involves the main impacts and effects resulting from the activity on the local social, economic, environmental and other development indicators. The examination should be concerned with both intended and unintended results and must also include the positive and negative impact of external factors, such as changes in terms of trade and financial conditions.  Sustainability: Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable. Source: The DAC Principles for the Evaluation of Development Assistance, OECD (1991), Glossary of Terms Used in Evaluation, in 'Methods and Procedures in Aid Evaluation', OECD (1986), and the Glossary of Evaluation and Results Based Management (RBM) Terms, OECD (2000). Differences between monitoring and evaluation Monitoring Evaluation Definition  Systematic and routine collection of  Systematic and objective assessment of an information from projects and ongoing or completed project, program, or program. policy, including its design, implementation, and results Timing  Monitoring is a periodically recurring  Evaluation assesses the entire project task already beginning in the cycle (a completed project or program or a September 2016 planning stage of a project or phase of an ongoing project or program program. that has been completed) Depth and  This is a regular part of project or  Evaluation reviews the achievements of purpose program management. It focuses on the project/program and considers the implementation of the project, whether the plan was the best one to comparing what was delivered with achieve the outcomes. (it involves what was planned. judgment)  Allows results, processes and  Evaluation measures achievements, as experiences to be documented and well as positive/negative and used as a basis to steer decision- intended/unintended effects making and learning processes  Evaluation looks for lessons to be learned  Key question: “What are we doing?” from both success and lack of success, and Focusing of efficiency also looks for best practices that can be applied elsewhere  Key question: What have we achieved? Focusing of effectiveness. Who  Monitoring is usually done by people  Evaluation is best conducted by an conducts it directly involved in implementing the independent outsider who can be project/ program. impartial in consulting with project/program staff Relationship  In general, monitoring is integral to evaluation. During an evaluation, information from between previous monitoring processes is used to understand the ways in which the project or M&E program developed and stimulated change M&E applications Project, program, and policy applications: M&E systems can be designed to monitor and evaluate at all levels of project, program, and policy. Information and data can be collected and analyzed at any stage and be used to better inform key decision-makers, the general public and other stakeholders. Therefore, M&E can and should be conducted throughout the life cycle of a project, program, or policy, including after completion. It thus becomes a critical part of a virtuous cycle of policy-making. Internal and external use of results-based M&E systems: M&E can be conducted at local, regional and national level. A functioning M&E system, at any level, provides a continuous flow of information that is useful internally and externally. Internal use of information from the M&E system is a crucial management tool that helps managers ensure that specific targets are met, by providing them with insights on progress, problems, and performance. M&E systems can help managers identify program weaknesses and take action to correct them. They can also help identify promising programs or practices. Likewise, the information from an M&E system is important to those outside the public sector who are expecting results and wanting to see demonstrable impacts. The information can build trust in a government or any other organization striving to better the life of its citizens or clients. Knowledge generation and learning: M&E systems are also a source of knowledge capital. They enable governments and organizations to develop a knowledge base of the types of projects, programs, and policies that are successful, and, more generally, what works, what does not, and why. M&E systems can also provide continuous feedback in the management process of monitoring and evaluating progress toward a given goal. In this context, they promote organizational learning. Broad public access to September 2016 information derived from results-based M&E systems is also important in aiding economic development both within and between countries. Transparency and accountability: M&E systems can promote transparency and accountability within organizations and governments. Beneficial spillover effects may also occur from shining a light on results. External and internal stakeholders will have a clear sense of the status of projects, programs, and policies. The ability to demonstrate positive results can increase popular and political support. September 2016 Annex 2. Impact and Result Evaluation in Colombia Colombia has a “results” evaluation (see figure below and description of the types of evaluations). Not sure if you want to include this in the note, but adding it just in case. 1) Executive evaluation. This type of evaluation studies the program from its design stage and analyzes the linkage between the design and the implementation process in order to assess the achievement of the goals and fulfillment of the outputs which are supposed to be delivered to the target population. The main source of information for this type of evaluation is administrative records and program documentation. 2) Operational evaluation. This type of evaluation does a deep analysis of the program’s macro and micro- process. The idea is to use the findings on the projects as inputs to guarantee that the outputs are being produced in the most efficient way. 3) Institutional evaluation. This type of evaluation studies the institutional framework behind the program and analyzes the structure of incentives and the organizational structure created for a program in order to operationalize it. 4) Results evaluation. This type of evaluation focuses on the analysis of the improvement in the program in terms of welfare of the population beneficiaries from the program. It also studies the effects that the delivery of outputs has on the population. 5) Impact evaluations. This is the most rigorous type of evaluation, because it is based primarily on the construction of experiments to determine the comparative effect of the program intervention on an individual within the program compared to an individual not receiving the program’s intervention. Source: UNICEF 2010. “From policies to results: Developing capacities for country monitoring and evaluation systems”. September 2016 Annex 3. List of references and additional resources Castro, Manuel Fernando. 2011. Defining and using performance indicators and targets in government M and E systems. PREM notes. Washington, DC: World Bank. Available athttp://siteresources.worldbank.org/INTPOVERTY/Resources/335642-1276521901256/ME12_v2.pdf Clark, Mari; Sartorius, Rolf; Bamberger, Michael. 2004. “Monitoring and evaluation: some tools, methods, and approaches.” Evaluation Capacity Development working paper series; ECD. Washington DC; World Bank. Available at http://documents.worldbank.org/curated/en/829171468180901329/Monitoring-and-evaluation-some- tools-methods-and-approaches Council of Europe 2010. Corruption Risk Assessment Methodology Guide. Available at http://www.coe.int/t/dghl/cooperation/economiccrime/corruption/projects/Albania/Technical%20Pape rs/PACA_TP%202%202011-Risk%20Assessment%20Methodology.pdf Gorgens, Marelize; Zall Kusek, Jody. 2009. “Making Monitoring and Evaluation Systems Work: A Capacity Development Toolkit”. World Bank. Available at https://openknowledge.worldbank.org/handle/10986/2702 DFID 2010. “How To Note: Addressing Corruption in the Health Sector”. Available at https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67659/How-to-Note- corruption-health.pdf Jesper Johnson, Theories of change in anti-corruption work: A tool for program design and evaluation, Chr. Michelsen Institute (CMI), 2012. Available at http://www.u4.no/publications/theories-of-change-in- anti-corruption-work-a-tool-for-programme-design-and-evaluation/ Kusek, J. Z., & Rist, R. C. (2004). “Ten steps to a results based monitoring and evaluation system. A handbook for development practitioners.” The World Bank, available at http://documents.worldbank.org/curated/en/638011468766181874/pdf/296720PAPER0100steps.pdf Mackay, Keith. 2007. “How to build M&E systems to support better government”. Washington, DC: World Bank. Available at http://documents.worldbank.org/curated/en/689011468763508573/How-to- build-M-E-systems-to-support-better-government OECD 2010. “Glossary of Key Terms in Evaluation and Results Based Management”. Available at http://www.oecd.org/dac/evaluation/2754804.pdf OSCE 2016. “Handbook on Combating Corruption”. Available at http://www.osce.org/secretariat/232761. OECD 2015. “Prevention of Corruption in the Public Sector in Eastern Europe and Central Asia”. Available at https://www.oecd.org/daf/anti-bribery/ACN-Prevention-Corruption-Report.pdf. September 2016 Regional Anti-Corruption Initiative (RAI) 2015. “Monitoring and evaluation of the implementation of national anti-corruption strategies and action plans: Methodology”. Available at http://rai- see.org/monitoring-of-ac-strategies/ U4 2011. “How to monitor and evaluate anti-corruption agencies: Guidelines for agencies, donors, and evaluators”. Available at http://www.u4.no/publications/how-to-monitor-and-evaluate-anti-corruption- agencies-guidelines-for-agencies-donors-and-evaluators-2 U4 2013. “Methods for learning what works and why in anti-corruption: An introduction to evaluation methods for practitioners”. Available at http://www.u4.no/publications/methods-for-learning-what- works-and-why-in-anti-corruption-an-introduction-to-evaluation-methods-for-practitioners/ USAID 2015. “Practitioner's Guide for Anticorruption Programming”. Available at http://pdf.usaid.gov/pdf_docs/PA00K7PG.pdf UNDP 2014. “Global Anti-corruption Initiative (GAIN) (2014-2017)”. Available at http://www.undp.org/content/dam/undp/library/Democratic%20Governance/Anti- corruption/globalanticorruption_final_web2.pdf UNDP 2015. “A user’s guide to measuring corruption and anti-corruption”. Available at https://www.globalintegrity.org/wp-content/uploads/2015/11/Users-Guide-Measuring-Corruption- Anticorruption.pdf U4 2012. “Theories of change in anti-corruption work: A tool for program design and evaluation”. Available at http://www.u4.no/publications/theories-of-change-in-anti-corruption-work-a-tool-for- programme-design-and-evaluation/ Van Wicklin III, Warren A.; Gurkan, Asli. 2013. How-to notes: participatory and third party monitoring in World Bank projects - what can non-state actors do? Available at http://documents.worldbank.org/curated/en/863281468337280255/How-to-notes-participatory-and- third-party-monitoring-in-World-Bank-projects-what-can-non-state-actors-do World Bank, Social Accountability Guide. Available at https://saeguide.worldbank.org/monitoring-non- state-actors; World Bank 2013, “Results Framework and M&E Guidance Note”. Available at http://siteresources.worldbank.org/PROJECTS/Resources/40940- 1365611011935/Guidance_Note_Results_and_M&E.pdf World Bank. “Monitoring and Evaluation (M&E): Some Tools, Methods and Approaches”. Available at http://www.worldbank.org/oed/ecd/tools/ September 2016 Annex 4. Specific Skill Requirements and M&E Competencies Carrying out the monitoring and evaluation requires a specific set of skills and competencies. Table 2 summarizes the skills mix for monitoring and evaluation, while Box 4 presents a set of technical and behavioral competencies needed for M&E, based on international experience. The list of technical and behavioral competencies could be used in job advertisements when hiring M&E specialists. Table 3: Overview of skills for monitoring and evaluation Monitoring skills Evaluation skills Skills for both M&E  Designing indicators,  Research study design  Analytical skills calculation methods, and  Statistical Methods  Calculating baselines definitions  Cost benefit analysis  Qualitative and quantitative  Calculating targets  Synthesizing evidence research skills  Identifying and reporting costs  Survey design and sample  Logical framework analysis  Organizing reporting channels calculations  Data interpretation  Drafting monitoring progress  Questionnaire design  Data storage and management reports/ writing skills  Data collection techniques  Quality assurance (data quality  Performance Management (focus groups, checks)  Translating results for decision interviewing, expert  Formulating recommendations making review) based on data  Developing standardized  Applying M&E information in approaches, drafting standards management decisions  Planning, communication and coordination between system stakeholders Source: World Bank Box 4: Technical and Behavioral Competencies for monitoring and evaluation Behavioral Competencies - Ability to identify and engage stakeholders at all levels- collaborates with partners on assessing progress and dealing with critical issues. - Written and oral communication - Time management, organizational and work planning - Teamwork and coordination - Problem Solving and reasoning Technical Competencies  M&E tools and concepts - Strong experience and familiarity with M&E concepts - Ability to develop/use M&E tools - Ability to develop and design frameworks and link the indicators with frameworks - Ability to identify, develop and evaluate indicators  Setting up and supporting M&E systems - Ability to develop, regularly update, harmonize, and communicate M&E plans that include identified data needs, standardized indicators, data collection procedures and tools, as well as September 2016 roles and responsibilities and budgets for implementing a functional M&E system. - Ability to finalize and operationalize Performance Monitoring Plans, determining M&E system set-up and data collection requirement  Data collection and data management competencies - Ability to identify the sources of data, collect, manage, analyze and interpret data - Familiarity with surveys and other data collection tools, including identification of data needs, data collection planning (including budgeting) and implementation, data analysis, report writing, dissemination, feedback, and data use. - Ability to manage the implementation of data quality assurance policies and procedures appropriate to the type of data and data source, including supportive supervision and data auditing - Ability to manage the implementation of data management systems and data sharing procedures.  Analysis - Ability to conduct and manage rigorous analyses of data - Perform and/or evaluate analysis of program activities to determine success in achieving targets and goals. Identify opportunities for improvement and take steps to make appropriate changes.  Evaluation - Ability to plan and design evaluations, draft and review scopes of work for hiring external firms and consultants, identify appropriate research questions, methods and plans, review and approve research instruments, and manage external consultants; - Ability to manage the evaluation process and use evaluation findings for program improvement.  Reporting - Ability to write reports, communicate & disseminate M&E information - Ability to use M& E data to support decision making - Ability to manage the dissemination of information in a targeted and timely manner. - Ability to identify, articulate, and support strategic use of data for program management and improvement Source: World Bank