34375 W O R L D B A N K W O R K I N G P A P E R N O . 4 1 Self-Assessment in Managing for Results Conducting Self-Assessment for Development Practitioners Rosalía Rodriguez-García Elizabeth M. White THE WORLD BANK W O R L D B A N K W O R K I N G P A P E R N O . 4 1 Self-Assessment in Managing for Results Conducting Self-Assessment for Development Practitioners Rosalía Rodriguez-García Elizabeth M. White THE WORLD BANK Washington, D.C. Copyright © 2005 The International Bank for Reconstruction and Development / The World Bank 1818 H Street, N.W. Washington, D.C. 20433, U.S.A. All rights reserved Manufactured in the United States of America First Printing: April 2005 printed on recycled paper 1 2 3 4 5 07 06 05 World Bank Working Papers are published to communicate the results of the Bank's work to the development community with the least possible delay. The manuscript of this paper therefore has not been prepared in accordance with the procedures appropriate to formally-edited texts. Some sources cited in this paper may be informal documents that are not readily available. The findings, interpretations, and conclusions expressed herein are those of the author(s) and do not necessarily reflect the views of the International Bank for Reconstruction and Development/ The World Bank and its affiliated organizations, or those of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply and judgment on the part of The World Bank of the legal status of any territory or the endorsement or acceptance of such boundaries. The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development/The World Bank encourages dissemination of its work and will normally grant permission promptly to reproduce portions of the work. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA, Tel: 978-750-8400, Fax: 978-750-4470, www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA, Fax: 202-522-2422, email: pubrights@worldbank.org. ISBN-10: 0-8213-6148-1 ISBN-13: 978-0-8213-6148-1 eISBN: 0-8213-6149-X ISSN: 1726-5878 DOI: 10.1596 / 978-0-8213-6148-1 Rosalía Rodriguez-García is Consultant to the Operations Policy and Country Services Results Secretariat of the World Bank. Elizabeth M. White is Senior Results Management Specialist in the same unit. Library of Congress Cataloging-in-Publication Data Rodriguez-García, Rosalía. Self-assessment in managing for results: conducting self-assessment for development practitioners / Rosalía Rodriguez-García, Elizabeth M. White. p. cm. -- (World Bank working paper; no. 41) Includes bibliographical references. ISBN 0-8213-6148-1 (pbk.) 1. Economic development projects--Management--Evaluation. 2. Economic assistance--Management--Evaluation. 3. Economic development--Planning--Evaluation. I. White, Elizabeth M. II. World Bank. III. Title. IV. Series. HD75.9.R63 2005 338.91'068--dc22 2005045769 Contents Foreword v Abstract vii Acknowledgments ix Abbreviations and Acronyms xi 1. Introduction 1 2. Self-Assessment Context and Managing for Results 3 Organizational Context 3 International Community Context 4 Results-Based Management Approach 5 Managing for Results 6 3. Conducting Self-Assessments 9 Self-Assessment Criteria 10 Results Orientation 10 Roles, Responsibilities, and Timing 11 Critical Phases and Steps in Conducting Self-Assessments 12 4. Challenges, Lessons, and Tips: A Focus on Process 23 Challenges 23 Lessons 24 Practical Tips 25 5. Conclusion 27 APPENDIXES A. Learning from Experience: Use of the Self-Assessment Process in the World Bank 29 B. Using a Self-Assessment Tool for Public Sector Management in Canada 31 C. Use of Self-Assessment in the Preparation of a Country Assistance Strategy and Achievement Report in the World Bank: The Experience of Tunisia 33 D. Process for Planning and Conducting Self-Assessments: Six Phases and 12 Critical Steps 35 Terminology 37 Bibliography 39 iii IV Contents LIST OF BOXES 1. Partnerships in Development 4 2. Twelve Critical Steps for Conducting Self-Assessments 13 LIST OF FIGURES 1. Managing for Results Framework 7 2. Necessary Phases for Successful Self-Assessment: From Direction to Utilization 12 LIST OF TABLES 1. Illustrative Assessment Questions and Their Use in Strategy Design 20 Foreword I n early 2003, the World Bank became the largest development agency to embrace "man- aging for results" as a guiding approach for improving the effectiveness of its assistance to countries. Managing for results deepens dialogue with clients and governments and re- flects ongoing efforts to sharpen the focus of business models on policy changes and demonstrable results on the ground. The Results Secretariat in the Operations Policy and Country Services Vice Presidency will support and assist this transformation by, among other things, making available proven tools and resource materials that staff--and our colleagues in client countries--can apply in their operational work. This Working Paper is the first in a series that we hope will facili- tate the dissemination of tools and techniques to help development practitioners enhance development effectiveness. Self-evaluation is well engrained within the World Bank. Together with independent evaluation and quality assurance, it helps maintain the Word Bank as a learning organization. Other development banks, bilateral and multilateral agencies, and the public and private sec- tors also incorporate self-evaluative exercises as a means to learn and improve performance. This Working Paper is directed to development practitioners. They may not see them- selves as evaluation experts, but they are, nevertheless, involved in planning, conducting, and managing self-assessment exercises and are direct users of evaluative information. The phases and steps outlined and described in this Working Paper can be utilized and adapted to plan, manage, and conduct different types of self-assessment, including project reviews, progress reports, and sector or multisector strategy completion reports. The paper is not meant to be a policy recommendation document or a complex ana- lytical exposition. Rather, it presents what we have learned about effective policies and pro- grams, and helps identify ways to achieve tangible and demonstrable results on the ground. Susan A. Stout Manager Results Secretariat Operations Policy and Country Services Vice-Presidency The World Bank v Abstract T he global development community is increasingly recognizing the need to manage better for results--that is, to steer development assistance to achieve results on the ground--and therefore, the strategic value of performance information at the operational level. The challenges are to provide high-quality and timely information for decision- making at critical points and to help development practitioners incorporate the use of outcome information in their business practices. Evaluative exercises help meet these challenges by providing information on outcomes achieved, examining the relevance of strategies to development impacts, underscoring good practices, and supporting and informing operations. This paper proposes a systematic and pragmatic approach to planning and conducting self-assessments of country and program strategies. The approach, which is grounded on inquiry principles and guided by a sound and logical conceptual model and process, is appli- cable to strategy development, progress reviews, and program management. The authors have used the approach in various settings--in the public and private sectors, in academic institutions, and in nongovernmental organizations (NGOs). This paper argues that the self-assessments it describes can make three strategic con- tributions. First, they use a participatory process, which engages those responsible for designing and implementing strategies and programs in the assessment process. Second, they improve strategy design by providing critical information that can be used for fine- tuning and further developing the strategy. Third, they focus on the outcomes that have been achieved during the program period, triangulating multiple data sources to assess progress, and deriving lessons that can be incorporated in the next program strategy. Thus, these self-assessment exercises become powerful tools in planning and managing for results at the operational level, and an important source of learning about achievements across sectors and countries. A well-done self-assessment is not a "quick and dirty" exercise; doing these self- assessments well requires time, effort, and resources. The fundamental purpose of this paper is to supply some of the tools, bringing inquiry concepts, process, and practice closer to development practitioner users, and to engage them in applying this informa- tion to the design and selection of strategies, the management of existing initiatives, and the assessment of progress to achieve better and faster results on the ground. vii Acknowledgments T he authors thank the technical experts in results-based approaches at the World Bank and other agencies who advise on the dissemination and use of materials such as this to support results-based approaches in operations. We are especially grateful to several colleagues in the World Bank who have provided valuable comments on previous versions of this paper, particularly to operations officers Cynthia Cook and Cecíle Fruman; to specialists in results-based monitoring and evaluation Elizabeth McAllister and Ray Rist; and to Operations Policy and Country Services (OPCS) Results Secretariat staff, par- ticularly Jack van Holst Pellekaan and Leslie Villegas. Editorial comments by Henry Chase, OPCS, are much appreciated. This paper also benefited from the valuable inputs and insights of reviewers outside the World Bank in bilateral and multilateral agencies, government, private sector, research and academic institutions, and NGOs. Our appreciation goes to Robert Bernstein, Management Sciences for Health; Sandii Lewin, the Global Fund to Fight AIDS, Tuberculosis and Malaria; James Macinko, New York University; Patricia Moser, Asian Development Bank; John Novak, United States Agency for International Development; Loretta Randolph, Organizational Development Systems; Jill Russell, International AIDS Alliance, United Kingdom; and Steve Williams, National Research Center, Duke University. ix Abbreviations and Acronyms ADB Asia Development Bank AfDB Africa Development Bank AIDS Acquired Immunodeficiency Syndrome AROE Annual Report on Operations Evaluation CAE Country Assistance Evaluation CAS Country Assistance Strategy CASCR Country Assistance Strategy Completion Report CDF Comprehensive Development Framework CODE Committee on Development Effectiveness EAP East Asia and Pacific Region ECA Europe and Central Asia Region ESW Economic and Sector Work FY Fiscal Year GTZ German International Assistance Agency HNP Health, Nutrition, and Population IBRD International Bank for Reconstruction and Development ICR Implementation Completion Report IDA International Development Association IFC International Finance Corporation IMF International Monetary Fund LCR Latin America and the Caribbean Region MDG Millennium Development Goal M&E Monitoring and Evaluation MNA Middle East and North Africa Region NGO Nongovernmental Organization OECD Organisation for Economic Co-operation and Development OED Operations Evaluation Department OPCS Operations Policy and Country Services PPAR Project Performance Assessment Report PRSP Poverty Reduction Strategy Paper QAE Quality-at-Entry (Review) QAG Quality Assurance Group RBCAS Results-Based Country Assistance Strategy RBM Results-Based Management SAR South Asia Region SSP Sector Strategy Paper SWAP Sectorwide Approach UNDP United Nations Development Program USAID United States Agency for International Development WDR World Development Report xi CHAPTER 1 Introduction T his paper describes a systematic approach and process for planning and conducting self-assessments1 that are applicable to strategy formulation, program management, and progress monitoring in the public sector, NGOs, and international agencies.2 The paper discusses critical factors and lessons drawn from the authors' experience in conduct- ing and supporting self-assessments for various purposes and in various settings, for exam- ple, to manage and assess progress of programs with governments and NGOs; to design program strategies and increase learning with international agencies; and to design and select country strategies and examine achievement of outcomes with development banks. Self-assessment is a key tool of managing for results that is linked with knowledge- building, learning, accountability, and informed decisionmaking.3 It is defined as any eval- uative exercise conducted by development practitioners to assess the progress of a program they are designing and implementing toward expected outcomes, for purposes of detect- ing errors or barriers and making timely adjustments to maximize the achievement of results on the ground. For development practitioners, self-assessment is not a mechanism for abstract inquiry, but first and foremost a management tool that: Promotesbothanendpointandastartingpointinmanagingforresults. Actively involves users in the process of stocktaking and examination of what has worked and what has not. 1. Self-assessment refers to any systematic evaluative exercise concerning strategies and programs conducted by those involved in their design and implementation. 2. This paper uses the term "program" in a generic sense. Institutions have different mechanisms to design and manage how they achieve results. 3. In Can Governments Learn? (F. L. Leeuw and others 1994), R. Rist asserts that learning can hap- pen in and for the benefit of any organization function, not only in and for decisionmaking (p. 192­193). 1 2 World Bank Working Paper Providesreadilyavailableandimmediatelyusefulinformationforstrategydesign,pro- gram management, prioritization, progress monitoring, learning, and accountability. Providesafocus,structure,andprocesstohelpensurethattheinformationgainedis timely, relevant, and reliable, and thus more likely to be used (White and Rodriguez- Garcia 2003). Turns what is often an informal feedback mechanism or a reporting exercise into a process that overcomes perceptions of subjectivity by improving the rigor and process of the self-assessment exercise. Helpsidentifyoperationallinkagesbetweenproductsandservicesaimedatcommon outcomes, thereby enhancing the probability of scaling up program results within the context of local realities. The basic elements of self-assessments apply whether the institution is a major multina- tional agency, a government department, or an NGO because inquiry is a universal process that embodies general principles and accepted standards of practice.4 The self-assessment process applies widely-accepted social science and management science inquiry methods and approaches.5 While it facilitates managerial decisionmaking and determines whether adequate implementation progress has been made to achieve expected outcomes,6 the self-assessment is also a key analytic tool that goes further than many desk-review exercises by deriving lessons learned that are explicitly integrated into business use. No matter the organizational setting, when a self-assessment is well and carefully done-- and recognizes appropriate caveats--it is a useful tool in managing for results that responds to a common goal: to provide timely, reliable, and useful feedback about ongoing and com- pleted actions and to derive relevant strategic implications that are immediately useful in strategy design and program management (IFAD 2004). A careful self-assessment reflects a pragmatic view of its purpose and intent: self-assessment should meet the needs of manag- ing specific programs or strategies and should yield useful information for decisionmaking in specific contextual situations.7 By conducting self-assessments and using the knowledge gained through them, practitioners assume the role of an agent of change working to improve a program in ways identified by the self-assessment. (Appendix A describes the use of self- assessments in the World Bank.) 4. R. Windsor and others (1994) in Evaluation of Health Promotion, Health Education, and Disease Prevention Programs discuss the agency decisionmaking model of inquiry, which is congruent with the approach to self-assessment presented in this paper. The authors note that the "primary work mode is the use of social and behavioral sciences methods within a client-specific problem-solving framework (rather than a hypothesis-testing framework). Measures of success would include the use of results in agency deci- sionmaking, more effective performance, and production of a respectable report" (p. 45). 5. Conceptually, this section adapts and builds on widely used social sciences inquiry methods. Ref- erences include Allen and others (2004), World Bank (2002a), Aga Khan Foundation (1993a and 1993b), Rosenthal (1984), and Rossi and Freeman (1993). 6. See World Bank (2003), "Annual Report on Operations Evaluation." The report defines monitor- ing as shown above, distinguishing it from evaluation, which uses attribution and causality to determine the extent to which outcomes have been achieved. 7. Lee Cronbach, a respected researcher and statistician, heralded a pragmatic viewpoint of evalua- tions as cited in Shadish and others (1991). CHAPTER 2 Self-Assessment Context and Managing for Results Organizational Context The growing emphasis in the last decade on improving management in the public and pri- vate sectors and international agencies has brought with it a threefold shift within the evalu- ative domain: (1) away from the traditional approach of external audits to the concept of internal evaluation, participatory evaluation, and self-study as indispensable tools for man- agement (Love 1991; Sonnichsen 2000); (2) away from the single project as the focus of analy- sis to programs, outcomes, and behaviors; and (3) toward a stronger link between knowledge and practice through learning, with evaluative information providing the bridge between the two (Rist 1995). Many governments and international agencies have adopted internal evaluation as one element of a broader results management system. For instance, the internal monitoring and evaluation (M&E) policy of the State of Oregon in the United States notes that "each agency shall monitor their . . . programs on a continual basis for the annual progress report . . . and support data shall be compiled and retained . . . to be made available [to decisionmakers]." The German International Assistance Agency (GTZ) defines the scope of internal evalua- tion as including all activities implemented within GTZ (impact analysis, progress reviews, final evaluation, cross-sector analysis) and encompassing self-evaluation and independent evaluation.8 Internal independent evaluation and self-assessment are processes that generate knowl- edge to manage for results: "Self-inspection is one effective tool that can be employed for organizations to discover and correct deficiencies, adjust activities and alter operational tasks to ensure effective and efficient goal attainment" (Leeuw and others 1994). However, 8. Both sources are found at www.oregon.gov and www.gtz.de/glossar. 3 4 World Bank Working Paper acquiring and applying knowledge are functions of the capacity and willingness to do so for learning. This means "rigorously examining on a recurring basis organizational actions and reasoning processes" (Leeuw and others 1994). International Community Context Given the complexity and extent of poverty, the international community recognizes that it must scale up those activities that make a difference and become more effective overall in providing financial and technical support. Improving the ability of countries, agencies, and institutions to manage for results is now central to the international community's commit- ment to harmonize efforts with partners to increase the effectiveness of development and to support better the achievement of country outcomes (see box 1). Managing to achieve results is not a new concept: the public sector and private companies have developed and practiced different schools of thought and approaches for many years.9 The common thread of these approaches is the value of maintaining a focus on strategic objectives (and measur- ing progress toward them), aligning organizational systems (both internal and external) to achieve objectives, and using timely information to create incentives for improved response to strategic, technical, or market-share challenges, cognizant of the implications for achiev- ing results. Thus, development agencies and development practitioners have increasingly empha- sized: (a) strategic planning and management that focuses on outcomes--rather than only on inputs and outputs--to improve organizational performance, and (b) measuring to cre- ate evidence for informed decisionmaking, learning, and knowledge building, and to report on results (Kusek and Rist 2004). The challenge is to incorporate sound management and evaluative principles, practices, and approaches into business practices so that the informa- tion produced is useful to and used by the development practitioner. This paper proposes that the systematic use of self-assessment by development practitioners and public sector Box 1. Partnerships in Development The effectiveness of development efforts should be judged on the basis of results. The traditional emphasis on aid disbursements and project inputs measured only how resources were allocated and used. But what really matters is the effect of public and donor actions on out- comes identified in a country strategy. This focus on country outcomes has changed the way the World Bank measures and monitors its development contributions and how it supports countries in tracking their own progress on results. Outcomes depend on effective partnerships at all levels in country-owned policies and strategies. Source: Partnerships in Development, 2004, p.21. The World Bank 9. See The Strategy-Focused Organization (Kaplan 1991) and Strategy Maps (Norton 2004) for a pre- sentation of cases studies about companies and government agencies that improved their outcomes and products by focusing on and aligning their business practices to the corporate strategy in a consistent manner. Self-Assessment in Managing for Results 5 decisionmakers address this challenge while complementing other elements of established institutional accountability systems (for example, quality-at-entry assessments, independent evaluations, product and processes quality control, ex post reviews, and mid-term progress assessments). Results-Based Management Approach Self-assessment has become an increasingly important tool in the troika of quality assur- ance, performance improvement, and corporate accountability. Many governments are now engaged in public sector reform to improve their policies, business practices, and insti- tutions with the intention of increasing financial accountability, efficiency, effectiveness, and transparency in the delivery of goods and services (Shaw 1999; World Bank 2000; Allen and others 2004). In turn, a key component of international aid to support these efforts has been the strengthening of countries' capacity to manage10 the reform process and monitor its progress. In the private sector, to respond to ever-changing environments, companies have kept testing innovative management approaches and methods to improve products and perfor- mance and to increase market share and profitability. One such approach--the strategy- focused organization--has been tested successfully during the last decade to determine its success in achieving results by transforming marginal-performing departments and com- panies into ones with improved performance and sustained profitability. The elements of one such reportedly successful model (the balanced score card in strategy-focused organi- zations) are driven by strategies that stress the ways institutions create value. The elements of this model are leading with strategy, translating the corporate strategy into operational terms, aligning the organization to create synergies, transforming strategy into everyone's everyday job, making strategy monitoring a continual process, and mobilizing change through executive leadership. Building on these experiences in the private and public sectors, the international com- munity has adopted managing for results--sometimes referred to as results-based man- agement (RBM)11--to anchor efforts to improve aid effectiveness.12 A review13 of RBM management experiences of selected bilateral and multilateral agencies found that agen- cies principally adopt the approach to improve strategy and management decisions (a 10. Management is the technical and social process of coordinating human, physical, and informa- tion resources to achieve desired results. Capacity is a critical dimension of success. The Strategy Focused Organization (Kaplan 2001) cites a survey of 275 portfolio managers who reported that the ability to exe- cute strategy was more important than the quality of the strategy itself, whereas for many years, man- agement experts have heralded the need to devise good strategies to generate superior performance. 11. In the selected proceedings of the "World Bank Seminar on Public Sector Performance," (Mackay 1998), E. McAllister advocates demystifying, simplifying, and clarifying terms, particularly clarifying dis- tinctions among results-based management, performance measurement, and program evaluation. This paper does not attempt to define these concepts, but agrees with the need for clarity of meaning. 12. This paper primarily uses the term "managing for results" instead of results-based management, but it considers the two concepts to reflect a similar orientation--management that is focused on improving performance--and thus uses both terms interchangeably. 13. See Baastel-Universalia (2002). This report describes the RBM approaches of CIDA, USAID, UNDP, UNICEF, IDB, and the World Bank. 6 World Bank Working Paper management tool) or to measure and report on performance (an accountability tool). Accord- ing to the report, implementing a managing-for-results system requires the following: Aclearvisionofthechangeprocessandawell-articulatedstrategythatbalancesatop- down with bottom-up participatory approach; Agoodcommunicationplantotransformtheorganizationintoalearningorganiza- tion by applying lessons learned and best practices; Solid buy-in of front-line managers, together with supporting guidelines, tools, resources, and incentives, to encourage staff to share accountability; Resultsinformationthatisintegratedinmanagementdecisionmakingandguidedby a robust internal M&E system; and Supportingfinancial andprogrammatic information systems. In most instances, the managing-for-results paradigm cuts across three key domains: (1) at the institutional level, managing for results incorporates well-proven management approaches from the public and private sectors to lead and measure operations with a view toward continual improvement in performance; (2) at the country level, injecting businesslike approaches into public agencies is expected to increase results and account- ability;14 and (3) at the global level, managing for results strives to render partners more efficient in their joint efforts to increase the effectiveness of development aid and of their own partnerships. Managing for Results The literature on the successful use of managing for results by governments, bilateral and multilateral agencies, and the private sector, plus the authors' own experience, have revealed a consistent pattern that points to core building blocks that can be applied to man- aging for results.15 This is supported by the findings of a review of documents published between 1996 and 1999 that attempted to assess what had worked and what had not worked with respect to efforts at implementing RBM in the Organisation for Economic Co-operation and Development (OECD) and other countries (Canada Office of the Audi- tor General, 2000). The key lessons were synthesized into the following good practices: Linkperformancemeasurestothecorporatepolicyorstrategicframework. Taketimeandmaintainmomentum. Alignmanagementsystemstosupportimplementation. Provideadequatefinancialandhumanresources. Establishaperformancemanagementculture. Demonstratesenior-levelleadershipandinvolvement. Communicatepurposeofperformancemanagementsystem. 14. See Rist (1995), footnote 10. 15. See World Bank (2003), "Annual Report on Operations Evaluation" and www.oregon.gov and www.gtz.de/glossar. Self-Assessment in Managing for Results 7 Useamanageablenumberofindicators. Alignperformancemeasureswithaccountabilityanddecisionmakingauthority. Demonstrate use of credible performance information. Whether managing for results is used primarily as a public sector management tool or a public sector measurement tool to increase agencies' effectiveness, it is driven by enlightened leadership that leverages a particular cultural and normative environment. Managing for results largely follows a framework that incorporates four components: (1) developing and communicating a clear corporate strategy that can be translated into specific operations and actions; (2) focusing planning and management on achieving outcomes that support corporate policy and operational strategies; (3) aligning business practices, procedures, and processes within the institution and with partners to achieve outcomes; and (4) improving monitoring, evaluation, and information use as well as reporting on performance for corpo- rate learning and accountability. A basic tenet of this framework is that when all components work together within a supporting organizational context, the synergies among them lead over time to a continual improvement in performance. Coherent and consistent leadership helps bring congruence among these building blocks to steer the organization toward delivering results. All of this is embedded in an organiza- tional context that provides formal and informal incentives to focus on results. Figure 1 depicts these components: strategy, focus, alignment, and continual M&E. These elements may be familiar. Many are, after all, core elements in different management models. What differentiates this framework from traditional management models is the synergy among strategy, focus, alignment, and monitoring information. This synergy is driven by leadership that cultivates strategy-based teamwork, encourages innovation, and establishes the right formal and informal incentives. The focus is on getting results by Figure 1. Managing for Results Framework External Environment Leadership Continual Monitoring & Evaluation, Reporting & Information Flow Focus on Align Business Outcome-based Practices, Process External Planning & and Structures Environment Management to Outcomes External Environment Communicate Goals and Strategy Culture and Subcultures Norms and Behaviors 8 World Bank Working Paper encouraging action based on information and by ensuring that congruence exists through- out the organizational system.16 While the figure includes the more tangible elements of managing for results, it is the less tangible ones (organizational culture, behavior, norms, incentives, and power relationships) that are critical to creating and sustaining the synergy and maintaining the focus. Self-assessment is a crucial element of the fourth component: M&E. By identifying early signals of programs that are on track, or going off track,17 M&E provides a feedback mecha- nism on the outcomes and consequences of actions.18 Whereas M&E provides the broader framework that guides the measurement and analysis of institutional or program perfor- mance, a self-assessment uses M&E data to provide information that can be put to immedi- ate practical use to fine-tune strategies, operations, and programs. This makes self-assessment relevant for timely decisionmaking, strategy design, and program steering. Moreover, self- assessment is conducted through a process that allows users to experience rather than infer the value of finding evidence on performance, creating a common point of reference for all involved in the exercise. 16. A systems model encourages consideration of all major elements that affect organizational effec- tiveness. These may include external conditions, organizational structure, the tangible building blocks mentioned above, individual staff and managers within the organization, organizational cultures and subcultures, and the interrelationships among all of the elements. Systems thinking has been widely stud- ied and accepted in management and organizational development circles. There has also been much lit- erature on institutional change processes and constraints to change. Presenting the elements and complexities of institutional change and organizational system design is beyond the scope of this paper. See "Diagnosing Organizations" Harrison (1994). 17. See "Overview and Conclusions" by Mackay (1998). 18. M&E adds a "fourth leg" to the governance chair traditionally built around budget, human resources, and auditing systems." See Kusek, Rist, and White (2004), p. 3. CHAPTER 3 Conducting Self-Assessments T he process for conducting self-assessment presented here is descriptive, not pre- scriptive, indicating the steps and actions that would help make the self-assessment a systematic and rigorous exercise that adds value to managing for results. The sys- tematic description of the procedural steps also aims at facilitating the management of the self-assessment process.19 This section supplies some of the tools of self-assessment, bringing inquiry concepts, processes, and practice closer to users--development practitioners20--and engaging them in the process. Inquiry is a universal process that embodies general principles and accepted standards of practice; hence the conduct of self-assessment is essentially the same whether the institution is a major multinational agency, a government department, or an NGO. A systematic, pragmatic approach to planning and conducting self-assessments is valu- able on two fronts. The process is close to the users of information and engages them to think outside their one specific sector or area, framing inquiry in terms of what the out- comes have been for the entire program and how performance during implementation has influenced achievement to date. The product (the report) should provide critical informa- tion that is timely, focuses on results, and addresses the relevance of the program (and ongoing activities) to the development goals and current conditions of the country or region. This information can be used to strengthen selectivity, alignment, and integration 19. R. Rist distinguishes between self-evaluation within management and the "management" of self- evaluation as a key distinction regarding the use of evaluation findings (in Sonnichsen 2000). 20. This paper uses "development practitioner" in its broader sense. It also uses the terms "program staff" or "program team" to mean those involved in implementation of a program and in the design of strategies. Country nationals and stakeholders who participate in the program in any significant way are considered part of "program staff" or "program team." 9 10 World Bank Working Paper of various institutional products and services, toward the goal of an integrated, results- driven program. Self-Assessment Criteria Reliable and useful self-assessments must meet the following criteria: Credible.Self-assessmentsshouldbeasobjectiveaspossible,andshouldbeseenas such. Systematic. Self-assessments should be planned on the basis of sound inquiry methods and should follow a logical procedural model. To the extent possible and applicable, self-assessments should be undertaken with the rigor associated with more formal studies. Valuable. Self-assessments should add significant value to management needs, such as strategy design, selectivity, resource allocation, and program management. This requires a focus on outcomes that are directly influenced by the program under review and that are likely to be attained and measured during implementa- tion of the subsequent program. User-owned.Ifself-assessmentfindingsaretobeused,theymustbeseenasuseful for better management for results by the development practitioner or program staff and by the client. The program staff and the client must be invested in the process. Results Orientation In addition, as a key tool in managing for results, the self-assessment should have a results orientation. A results-based self-assessment addresses: (a) the relevance of the program to the longer-term development goals of the country21 (often referred to as higher-order out- comes); (b) the achievement of outcomes directly influenced by the program during the period; and (c) how lending and nonlending activities, intervention mix, policy dialogue, and analytic work directly influence delivery of those outcomes, given the relative perfor- mance of the client, government, donors, stakeholders, and other factors. The self-assessment should illuminate how ongoing interventions22 can be integrated in the subsequent pro- gram or strategy on the basis of the timing of implementation and identification of syner- gies among individual operations and the overall desired outcomes.23 21. The approach distinguishes between outcomes that can be influenced directly by a program and those that are not as direct. The latter would reflect longer-term development goals and include such out- comes as poverty reduction, economic growth, or the Millennium Development Goals. 22. Intervention refers to the part of a strategy or program that actually reaches the population or the targets. 23. Higher-order outcome indicators change slowly, and data are usually infrequent and late because of the time required for data collection and processing. Therefore, the program or strategy needs to focus on outcomes that can be influenced (and data that can be collected) over the program implementation period--usually three to five years. Self-Assessment in Managing for Results 11 Therefore, a results orientation for the self-assessment acts as a catalyst for under- standing how the current program mix can help achieve outcomes; it also triggers program staff to reflect on ways to design the subsequent strategy or program, and it provides an opportunity for broader examination of other issues such as program strengths; program barriers; government assets and strengths vis-à-vis the program; partnerships with other donors, NGOs and civil society groups; and the agency's effectiveness. Throughout the ana- lytic process, lessons, synergies among individual operations and activities, and good prac- tices are identified and discussed, contributing to cross-fertilization, staff learning, and effective management. Roles, Responsibilities, and Timing Self-assessment can be conducted at many points in the program or strategy cycle: (a) as soon as a new program is being considered; (b) close to the completion of a program or broader strategy period; (c) as required by the implementation plan; (d) in response to changes in the environment (for example, significant program staff changes or political changes in coun- tries); (e) anytime program performance is of concern (if, for example, progress slows down significantly and suddenly, or intermediate outcomes are not being achieved even though implementation is progressing); and (f) in response to a board of directors or government request for information. It is particularly useful when starting the planning of a new program because it can jump-start the design process and reveal opportunities for increased synergy among operations. The self-assessment process should have a leader, most likely the program director or manager. It may be appropriate to designate another person as the self-assessment manager to oversee the process and ensure its successful completion. The self-assessment, like the program itself, occurs within contextual settings. Understanding the characteristics of such a context necessitates the engagement of key actors and observers. Therefore, the self- assessment process should involve all of those who are responsible for the implementation and monitoring of the program or strategy under examination: development practitioners, country nationals, and program stakeholders, as appropriate. The program staff who over- see implementation of the program are essential to the process, as they contribute data, expert knowledge, first-hand experience, and program memory. The level of involvement of country nationals--central and line ministries, project staff, and beneficiaries--should follow standard practices. Partners, civil society, and other stock- holders also add a valuable dimension to the exercise. Self-assessment is an opportunity for self-inspection; it is not about blaming, but about improving program quality so that out- comes can be achieved on the ground. Depending on the complexity of a particular self-assessment exercise and time consid- erations--as the timeliness of findings is important in the business process--an internal spe- cialist or an external consultant can be brought in to advise or take on the bulk of the work to support the self-assessment manager. The specialist would follow--and adapt as needed-- the conceptual and procedural approach described herein, with a view to deriving value for the corporate business process. 12 World Bank Working Paper The self-assessment process may take between 30 and 45 workdays--depending on the availability of program documentation, depth of knowledge of the specialist, and availability of program staff--and can be scheduled across several months (two to three months in the authors' experience). Critical Phases and Steps in Conducting Self-Assessments Our examination of self-assessment exercises has revealed a consistent set of required phases for successful self-assessment. Six factors are critical to enabling the systematic and logical examination of program achievement in a manner that engages the development practitioner in the results appraisal process and facilitates business use: (1) direction, (2) participation, (3) examination, (4) consultation, (5) verification, and (6) utilization. These factors align themselves in a cycle; representing the six key phases of the analytic pro- cess (see figure 2). Such a systematic approach to self-assessment provides opportunities for corporate learning, innovation, and improvement that are based on experience--a key facet of a learning organization (Senge 1990). The participatory nature of the approach adds value: research has shown that those who are expected to use evaluative findings for improv- ing program performance should participate in the fact-finding process and in analyz- ing the implications of the findings (IFAD 2004). Thus, the transformation of each self- assessment into a systematic participatory exercise that engages program staff, encour- ages teamwork, and enables managing for results renders the process both valuable and user-owned. Figure 2. Necessary Phases for Successful Self-Assessment: From Direction to Utilization Participation Participation Direction Examination Utilization Consultation Verification Self-Assessment in Managing for Results 13 This section describes the procedural framework and sets out the 12 key steps along the 6-phase continuum as a roadmap that can be used to guide the self-assessment process.24 (Appendix D depicts this 6-phase, 12-step roadmap). Development practitioners can expand on these steps, or they can combine some steps to fit the complexity of the program and the contextual situation. Box 2. Twelve Critical Steps for Conducting Self-Assessments Phase I. Direction The early engagement of the program director (or self-assessment manager) in the process and the direction he or she provides send the message that a focus on results is important. It also sets the stage for the use of findings by linking them to the design of the next program or strategy. Leadership from the program di- rector regarding the importance of the exer- Step 1 cise and how the findings will be used will add Review guidelines weight to the undertaking and help foster the & requirements for active participation of program staff. Finally, self-assessments the continual involvement of the program di- rector is needed to provide the bridge between ex post assessment and ex ante design of the next strategy. Step 1. Review requirements and guidelines. Consult the institution's guidelines and require- ments (content, format) about the expected de- liverable(s) and review the leading corporate frameworks, as they reflect corporate expecta- tions, to ensure that the product will meet insti- tutional needs. Step 2 Clarify parameters Step 2. Agree with the program director on self- & report assessment parameters and expectations. It is expectations with important to clarify the purpose and parame- manager ters of the self-assessment and agree on the expectations for the process and the report. It is also helpful for managers and staff to delin- eate the roles and responsibilities of each mem- ber of the program staff involved in the self- assessment process. (continued ) 24. This section incorporates a briefing note prepared by Rosalía Rodriguez-García and Cécile Fru- man on the experience of conducting a results-based country program achievement self-assessment of the World Bank's program in Tunisia in September 2003. 14 World Bank Working Paper Box 2. Twelve Critical Steps for Conducting Self-Assessments (Continued) Phase II. Participation Lack of program staff involvement in the pro- cess can severely hinder the self-assessment, as can misunderstandings about its purpose. Not only is it critical to involve program staff in the self-assessment, it is also important to in- clude the participation of program stakehold- ers (donors and country nationals) in mapping the type and sources of strategic information that need to be examined. As appropriate, pro- Step 3 gram beneficiaries can be included in the self- Clarify approach assessment process. and process with staff Step 3. Clarify the logic model and program staff understanding of process and report ex- pectations. This step helps establish an envi- ronment in which the self-assessment is seen as valuable for taking stock of individual opera- tions and for developing subsequent programs. It can be achieved through early meetings, led by the program director, to give program staff the opportunity to provide input on the logic model underpinning the program under exam- ination (for example, what were the objectives Step 4 and what should have been accomplished?). Map information Step 4. Map information needs and sources. needs & sources Identify: (a) resources available, (b) type of information needed (financial, portfolio re- views, analytic papers), (c) sources of informa- tion (web, printed, in the country, partners), (d) mechanisms to gather the needed informa- tion (interviews, group meetings, conference calls, documents, videoconferences, observa- tion), and (e) informants who can provide data or observations unavailable in print. Esti- mate the time needed for desk analysis, interviews, and observations (for example, is a country visit envisioned as part of the self-assessment?), and of potential barriers to the self-assessment (for example, key program staff may be unavailable for most of the self-assessment period). Although program staff may not need to participate in this exercise as a group, they should be con- sulted to ensure that no key source of information or critical element is overlooked. (continued ) Self-Assessment in Managing for Results 15 Box 2. Twelve Critical Steps for Conducting Self-Assessments (Continued) Phase III. Examination Thorough desk analysis helps ensure that the self-assessment incorporates to the extent pos- sible all the program products and outcomes, including quality reviews, independent evalu- ations, and other stock-taking exercises. The review of documents, discussion with govern- ment officials and key informants, and field observations should confirm the existence of program outputs and outcomes and the pro- Step 5 gress being made toward achieving them. Conduct Step 5. Undertake systematic desk review of systematic desk documents, correspondence, reports, and re- review lated publications. This step is the most time- consuming, but it is necessary for a sound diagnosis of the situation, appraisal of the facts, identification of cross-sector outcomes, and discovery of hidden synergies that can help elaborate good practices and lessons learned and contribute to the subsequent program strategy. It is often useful to start by examining the program strategy document, followed by progress reports, program appraisals, indepen- dent evaluations, and outcomes studies made Step 6 during the life of the program. This will help Interview establish a storyline of performance, which can program staff then be followed by the review of other types of documentation. The examination may include the following documents: National development plans, poverty re- duction strategies, or other national goal- setting documents, including baselines and other statistics for the national goals. MillenniumDevelopmentGoals(MDGs)documentsorotherglobalgoal-settingdocumentsrel- evant to the country, region, sector, themes, or programs. Program materials--specifically, progress reports, project status reports, trip reports, lending portfolio reviews, institution-building reports, aides-mémoire, or other documents that report on how the projects, programs, or sectors are delivering outcomes. Analytic, advisory, technical support, and policy dialogue products. Quality assurance reviews and independent evaluation reports. Lending portfolio and financial management updates and project status reports. Board of Directors special reports as appropriate. Statistical data from reputable national and international institutions and documents (for example, World Bank MDGs page for the country and poverty analysis or investments type reports; United Nations Development Program (UNDP) country demographic profiles; Demo- graphic Health Surveys; national census, surveys, and other national statistics). (continued ) 16 World Bank Working Paper Box 2. Twelve Critical Steps for Conducting Self-Assessments (Continued) Dataonprogressmadeonachievingnationaldevelopmentgoalsbysector,bygeography,and by population, as well as regional strategic papers as appropriate. Relevant partners' reports and other relevant reports, memoranda, and studies and related publications. Step 6. Interview all relevant program staff--operations, sectoral, financial, theme specialists, and others. Semi-structured interviews should be conducted both to obtain information about the pro- gram in a systematic way and to engage program staff in the self-assessment process. Interviews seek ex post information about program goals and priorities; how these were set; the extent to which expectations have been met; the barriers to and facilitators of implementation; unexpected positive and negative results; risks and opportunities; and good practices or lessons that have emerged. The interviews also seek to catalyze staff assessments of program outcomes, bringing their insights into the self-assessment to inform the next strategy or program. If program staff have participated broadly in group workshops, it may be necessary to interview only those members who were unable to participate or who are more communicative in a one-to-one situation. (continued ) Self-Assessment in Managing for Results 17 Box 2. Twelve Critical Steps for Conducting Self-Assessments (Continued) Phase IV. Consultation While the participation phase makes the pro- gram staff (those who are intimately involved in the design and implementation of the pro- gram) the focus of fact finding and data collec- tion, the consultation phase aims at gathering information from outside informants: other specialists, managers, or policymakers who are involved in the program or who are related to the program in a more independent manner. Step 7 Consultation with key informants, including Interview key those in the country, will add important ele- informants (e.g. ments to the self-assessment that may other- legal, operations, wise not surface. Consultation may include, as peers) appropriate: Sector, theme, and operations specialists and advisers, Independent evaluators and quality assur- ance staff, Funding and implementation partners both in headquarters and in the country, Step 8 Client representatives (including elected Consult officials, government staff, NGOs, civil soci- with relevant ety, and other national stakeholders), and donors & partners Staff of units responsible for operations policies. Step 7. Interview key informants staff from sec- tors, operations, themes, legal, policy, and man- agement. Use semi-structured interviews to gauge critical information from these sources, looking for confirmation or recalibration of find- ings, commonalities in interpretation of facts, and perceived synergies by those with a poten- Step 9 tially broader view of the program's risks, oppor- May conduct tunities, and performance. country visit Step 8. Consult with donors and partners. This step acknowledges the need to learn from other institutions and partners about their involve- ment in implementing the program or strategy, and their experience in working with the pro- gram team as well as their views of the collabo- ration among the participating agencies.25 (continued ) 25. Harmonization of efforts by donor and implementing agencies has been identified as a key ele- ment in improving development aid effectiveness. 18 World Bank Working Paper Box 2. Twelve Critical Steps for Conducting Self-Assessments (Continued) Step 9. Consult with the client and conduct a site visit if feasible. This consultation can take sev- eral forms: videoconference, teleconference, electronic virtual meetings, and so forth. While engaging the client early on is important, clients may differ as to the degree of involvement they prefer. In the case of Mozambique, the borrower and in-country partners were involved in coun- try program stock taking as part of the process of debating the poverty reduction strategy and the national development plan. In Tunisia, the client preferred long-distance consultation and to review and comment on the strategy achievement report once a draft was available (see Appen- dix C). Regardless of the degree of involvement and whether a site visit is conducted, it is impor- tant to incorporate the views of the client and other national key stakeholders in the self- assessment, to the extent possible. Consultation with elected officials, NGOs, academic institutions, the private sector, and civil society may be advisable and appropriate. Phase V. Verification The verification of information (financial, statis- tics, qualitative, evaluative, quality assurance) is a crucial phase in a rigorous self-assessment, helping to shift findings from information to evidence. The verification phase includes the preparation and review of draft reports to verify the accuracy of the information and the rele- vance of the analysis and conclusions. Step 10 Step 10. Review findings and draft report with program director or self-assessment manager Review findings and program staff. This step may involve double- with program checking information and identifying data gaps director, task manager, and and quality issues. A first "raw" draft report may program staff be used to guide the discussion and verification of findings with the self-assessment manager and program staff. The review of findings and conclusions of the self-assessment is an ite- rative process; several drafts will be needed before a complete draft report is available. Step 11. Prepare draft reports for staff and client review. This essential step needs to be planned and managed as part of the self-assessment Step 11 process. Prepare draft reports for staff and client review (continued ) Self-Assessment in Managing for Results 19 Box 2. Twelve Critical Steps for Conducting Self-Assessments (Continued) Phase VI. Utilization Self-assessment is useful in managing for results only if the quality, timely, and pertinent infor- mation gathered through a systematic and par- ticipatory process is used for decisionmaking, knowledge building, and learning, thus ena- bling improvements in performance and in the achievement of development outcomes. Step 12. Use the findings of the self-assessment Step 12 report for strategic program design decisions, Use findings for selectivity, monitoring, and staff learning. strategic program The self-assessment report should be useful, design, decision- credible, and valuable to program staff, man- making, and staff agement, the client, and stakeholders. By iden- learning tifying achievements, lessons learned, and good practices, and by bringing forward in a timely manner what did not work and why, the self-assessment report informs decision- making and becomes a vital tool in managing for results. It also provides a review of the ongoing program progress toward outcomes that will help the program staff set realistic out- come goals in the subsequent strategy period. Table 1 illustrates how the self-assessment can be used to guide strategy design by deriving answers to core strategic questions about the program.26 The self-assessment is a transparent process. Once the self-assessment process is con- cluded and a report is available, lessons can and should be shared with stakeholders. In the World Bank, the program staff for Tunisia and Cameroon organized several seminars to disseminate the lessons learned from both the programs examined and the assessment processes. In bilateral agencies such as the United States Agency for International Devel- opment (USAID), informal brown-bag luncheon seminars are often used to share lessons learned across specialties and institutions, including NGOs, for-profit entities, and other interested parties. Other groups conduct "exit meetings" to discuss findings and their uses. 26. This table brings together principles from strategy design and assessment questions. It is derived from the authors' experience in supporting results-based country strategies and offers illustrative ques- tions that would be applicable to different settings. 20 World Bank Working Paper Table 1. Illustrative Assessment Questions and Their Use in Strategy Design Managing for results Uses in program or framework Self-assessment questions strategy design I. Align program Relevance (to longer-term goals Alignment. Given new informa- interventions or higher-order outcomes) tion, does the program need to to broader Have the needs of government change its focus (e.g., from health strategic goals or population changed systems to infrastructure) to because of progress or slip- contribute to national goals (rele- page on obtaining key devel- vance)? Is the government request- opment goals? ing a different approach or focus because of the contributions of Did the country or regional other donors or new policy context change significantly decisions? during the period of imple- mentation (related to devel- Selectivity. Have policies of other opment goals and priorities)? donors shifted, changing the value added by the program? Is the cur- Was the program relevant and rent mix of instruments and inter- responsive to specified ventions so fragmented that national development goals? contribution to national outcomes will be marginal at best? Should the program be reoriented? Knowledge. What analytic work needs to be undertaken to under- stand better the constraints to achieving national goals? Risks. Have risks materialized that require a fundamental shift in the focus of the strategy as national goals are threatened? II. Focus on Program/strategy outcomes Selectivity. Is there an ongoing outcomes Were the expected outcomes financing commitment that could (for program achieve greater development influenced by the particular period) impact if the subsequent strategy strategy or program achieved? focused on the same area? Or is Why/why not? the contribution of the ongoing How did achievement of out- program sufficiently strong that comes "contribute" to new strategy can shift to other progress toward specified development priorities? Should national development goals? strategy selection be based on Was the mix of lending, non- what is working well or on where lending, interventions, policy there are more problems? dialogue, knowledge transfer, Realisminoutcomes.Givenimple- and analytic work appropriate mentation experience, progress to deliver the outcomes, and toward outcomes and country and how/how not? regional context, should the new program/strategy be more conserv- ative in what is realistically achiev- able? How is the current program positioned to deliver outcomes in the subsequent program period? (continued ) Self-Assessment in Managing for Results 21 Table 1. Illustrative Assessment Questions and Their Use in Strategy Design (Continued) Managing for results Uses in program or framework Self-assessment questions strategy design Mix. Is there a need to restructure operations, leverage components toward outcomes, or scale up promising interventions? III. Monitoring Contribution to outcomes Improvingperformance.Howcould and evaluating What was the performance of the staff improve support for the performance program staff during the client in the subsequent program? implementation in the area of What performance indicators can the staff set for itself to contribute Quality of products and to improvement in subsequent services? program implementation? Portfolio assessments? How could the process of strategy Country dialogue and aid design and consultations con- coordination? tribute to improved implementa- tion in the subsequent program? Technical support and analytical studies? What business processes and outcome-based reviews can the Overall knowledge staff put in place to help achieve transfer? outcomes? What was the quality of the strategy? Were the right Whatincentivesareavailableto foster the use of M&E information? means chosen to deliver the outcomes? Was the strategy Does capacity exist for data collec- right? tion, processing, and statistical analysis? IV. Focus on Did the previous program Emphasis on country-level capac- strengthening include a system for monitor- ity. Are expected outcomes from capacity to ing and evaluation? If not, the new program realistic if imple- manage for what was the basis for assess- mentation weaknesses have been results to ing whether or not progress addressed? Will the assumptions support has been made against underpinning the timing for strategy outcomes? achieving the outcomes hold? How did counterpart perfor- How will the new program estab- mance support or undermine lish a useful M&E system if the achievement of the outcomes country capacity is weak? What are in relation to program imple- the implications going forward? mentation issues, not neces- Should the new program consider sarily policy oriented? strengthening country capacity in this area as part of its new approach? Harmonization. What process should be in place for working with other partners in selectivity of support, monitoring mecha- nisms, and strengthening country capacity? CHAPTER 4 Challenges, Lessons, and Tips: A Focus on Process E xperience reinforces the principles discussed in this paper. From these experiences, challenges, valuable lessons, and practical tips have emerged. Challenges Program Outcomes. Recent experience found that most programs identify the longer-term or higher-order outcomes (for example, economic growth) and then outputs and activities. What is missing is explicit identification of expected program outcomes--those that could reasonably be achieved in the time period of the program (about four years) and that are directly related to the program of support. Typically, indicators largely reflected a combina- tion of longer-term or higher-order outcomes (for example, decreased infant mortality) and process milestones (for example, staff trained with new skills), and they lacked clarity on the outcomes that could be influenced by the program during the implementation period (for example, services expanded to additional geographic areas; retention rates for secondary schools students increased in target schools; improved household economy from better access of women to informal markets using new roads). The lack of these program out- comes is sometimes referred to as the missing middle. Therefore, in many instances, the self-assessment faced the challenge of "reconstructing" the missing middle of the program under examination, and then determining how far on the causal chain progress had been made. This required in-depth examination of the contributions of individual operations to the overall program and of how the different interventions, analytic work, policy dialogue, and technical support contributed to longer-term development goals or higher-order out- comes as well as the program-bounded outcomes. Identification of these synergies was 23 24 World Bank Working Paper important for a good self-assessment, but even more so for designing a subsequent strategy to be built on the ongoing operations. Only by reconstructing this missing middle were the causal and temporal relationships of ongoing operations to past, present, and future out- comes clear. This first challenge can be mitigated by comparing program objectives as included in the program strategy with those included in individual projects documents, operations, lending portfolio reviews, and analytic or technical documents. The causal and temporal relation- ships can also be validated with sector and other specialists and by double-checking with pro- gram managers and country officials regarding their understanding of the outcomes expected from the program and those achieved during the program period. Cross-Sector Orientation toward Outcomes. The integrated results approach as applied to self-assessment is often new to development practitioners. Consequently, the first incli- nation of program staff was to provide information on individual projects or policy studies before looking for ways to tackle cross-cutting issues or to appraise how the old and new lend- ing or intervention mix could combine with analytic and support activities to achieve pro- gram outcomes. This broader, cross-sectoral view of program results was reached through the participatory process described in this paper. Thus, the self-assessment must consider and incorporate for analysis different types of information to ascertain how different interven- tions are contributing or have contributed to outcomes on the ground. Data on Outcomes. Another challenge was that previous program strategies did not always consider how collection of data on outcomes would take place (in part because pro- gram outcomes--the missing middle--were not explicitly identified). Therefore, data col- lection relied heavily on national statistics for macro-level or output indicators, the result being that most time-bound program outcomes remained largely unaccounted for. Thus, subsequent strategy design should include outcome indicators and address sources of data for these indicators, and describe how monitoring, evaluation, and reporting will take place. Lessons The participatory process is important, but it should not be unduly long or cumbersome. Allow time for consultation, verification, and peer review, as such stocktaking contributes vital infor- mation to strategic program design. The participatory and iterative process presented in this paper takes time and effort, but that is necessary to ensure the thoroughness and usefulness of the assessment. It helps to use consultants with a good understanding of results-based approaches and corporate operations to do the bulk of the work. Accounting for the time needed to achieve, measure, and report outcome data is important. In assessing achievement of quantitative targets at the end of a program, it is important to remember that longer-term goals and higher-order outcome indicators change slowly and that it takes time to collect, process, and report data. Outcome data may not be available in time for the self-assessment, especially for the last year of the program. Programs need to sup- port the collection of outcome data systematically throughout the implementation period, and as intervals, as needed, through surveys and other appropriate means. Programs should also support the use of data and information in program management, policy reviews, and staff learning. A self-assessment brings added value to the next program or strategy design. The systematic process of conducting the assessment of past and current achievements brings to the fore- Self-Assessment in Managing for Results 25 front political, strategic, and technical issues related to the program that otherwise may have been left unattended, and points to possible ways to address them. First, experience suggests that making explicit the link between the ongoing interventions or services and analytic, advi- sory, or technical support activities, and between them and the new interventions, will enhance the program strategy. Often, past program strategies have focused largely on the pro- posed interventions and did not articulate the link between the ongoing and the new ones. Second, program strategies will be more robust when all activities--old and new--are ori- ented toward outcomes and explicitly discussed. Third, a focus on results and the interme- diate outcomes the program can influence underscores the M&E and reporting needs of the program for effective management of its implementation. Finally, activities and projects initiated in previous programs that are expected to be completed during the next program cycle deserve particular attention, as they are more likely to deliver the outcomes during the time period of the program being designed. The process of self-assessment described in this paper helps identify opportunities to fine-tune ongoing activities; improve synergies with the new interventions; deepen collaboration with partners; and enhance corporate dialogue with clients, partners, and other stakeholders in terms of contribution to the achievement of development results on the ground. Practical Tips A data repository on program outcomes will facilitate the examination of achievements for progress reports and self-assessments. Collecting data for the self-assessment may necessitate considerable consultation with a wide range of corporate sources and staff (including oper- ations managers and sectors specialists), the client, and outside sources. Having one major repository of data links (including on outcomes, outputs, process, and coverage) within the program management team will make the identification of data and the data review process more efficient and will make data more accessible for management decisions. Having a central repository for all advisory and analytic products of the program or country strategy will facilitate the review of products. Collecting analytic products (policy studies, brief- ing notes, market analysis) may prove to be an unexpected challenge, as products are some- times not available in the expected corporate site (online documents' database or other such source). In a few instances, this will require increased personal contacts and verification to ensure that all products are examined and included in the self-assessment. This challenge can be mitigated by enforcing regular filing of data in the electronic systems established in most agencies. Program staff should be expected to have met corporate requirements for report- ing data. For instance, at the World Bank, it is an institutional requirement to file analytic products in an electronic database called "Imagebank." However, experience shows that often this is not done. Progress reports are a useful tool to measure progress toward outcomes. In cases where there were no progress reports, the parameters of the self-assessment were broad and fluid, with a great deal of information to be captured, assessed, and synthesized, sometimes long after the fact. This underscored the fact that often there is limited recorded information on overall country strategy or program achievements and limited outcome data (there usually are more project-specific data). Progress reports that are also results-oriented will help monitor progress toward achieving program outcomes, and will be an important facilitator of end- point self-assessments and independent evaluations. CHAPTER 5 Conclusion T he self-assessment of programs or countrywide strategies is an important planning and management tool and an important source of learning about achievements across sectors and countries. Because of its critical role in documenting achievements and experiences in managing for results, this paper suggests that the process of self-assessment should be systematic, grounded on core inquiry principles, and guided by a sound and logi- cal conceptual model and process. It does not, however, need to be overcomplicated. The process can be adapted to the contextual situation by adding on or collapsing steps as needed. In addition to learning by inquiry, learning by doing is paramount. Therefore, the experience of institutions conducting self-assessments and designing results-based programs and coun- try strategies were examined to identify good procedural practices. This connection of method and experience assures the relevance of the self-assessment process to business prac- tices and corporate expectations. The programmatic achievement findings in the self-assessment report provide timely inputs into the learning process of the agency's staff charged with planning and implement- ing programs and countrywide strategies, to the client, and to other stakeholders. In this way, the self-assessment adds value to corporate operations by encouraging innovation, cultivat- ing teamwork, and contributing knowledge, good practices, and lessons learned from pro- gram implementation to the informed decisionmaking processes that underscore managing for results. 27 APPENDIX A Learning from Experience: Use of the Self-Assessment Process in the World Bank The self-assessment process was used by country teams to appraise the progress made by coun- trywide strategies in achieving program outcomes in order to trigger and inform the subsequent strategy design process.* The participatory self-assessment provided the approach and process to derive the information necessary to apply a results-based approach to strategy design and an out- come-oriented management view for more focused implementation. This necessitated (White and Rodriguez-García 2003): Assessing together the ongoing and planned countrywide lending portfolio, analytic prod- ucts, and policy dialogue activities to derive the subsequent strategy to achieve, and possibly scale up, development outcomes. Determining how the ongoing lending portfolio could help selectively focus the subsequent strat- egy on key development goals (and thus scale up) or whether or not the same development issues were still relevant (and thus perhaps scale back). It is not enough to compile lessons learned on what worked well, what worked less well, or what objectives had been achieved. Instead, this approach fostered a critical examination of the intermediate outcomes being delivered by the ongoing countrywide strategy and what the implications were for selectivity and focus in the subsequent strategy. Focusing on outcomes directly influenced by the World Bank's program. The self-assessment helped country teams identify those outcomes more likely to be directly influenced during the subsequent strategy (as the strategies are in series), not only higher-order, longer-term coun- try outcomes (such as poverty reduction or reductions in prevalence rates), and determine how the lending contributed through outputs or disbursements. This allowed for better planning of a full range of products and services to influence outcomes (for example, ongoing and planned lending, analytic work, policy dialogue, and partnerships). The self-assessment process added value by: Enablingasystematicapproachtotheexaminationofcountrywidestrategyoutcomesandiden- tification of strengths and gaps to achieving results. This encouraged more problem solving and improved planning. 29 30 World Bank Working Paper Providing direction for the preparation of the next strategy by enabling information-based selectivity and an outcomes focus for improved strategy articulation and presentation, bring- ing in independent evaluation and other studies to inform strategy design. Fosteringdialoguewiththeclientaroundissuesofachievingresultsandmutualaccountability. Informingfutureadvisoryandcapacity-buildingactivitiesaroundresults-basedmonitoringand evaluation, including statistics. *This information is based primarily on the experiences of Armenia, Cameroon, Tunisia, and, to lesser degrees, Mozambique and Zambia. When the methodology and process proposed herein were followed, the self-assessments were valuable for design, learning, and dialogue with stake- holders. They also facilitated discussions on trade-offs in terms of time and effort, composition and degree of staff involvement, and areas of engagement for enhanced client dialogue. APPENDIX B Using a Self-Assessment Tool for Public Sector Management in Canada Today's public sector managers have greater flexibility and discretion in deciding how best to use resources and deliver services. Good results information enables them to manage well in this envi- ronment. Better information on what is working and what is not helps managers provide more effective services to Canadians. With the tabling of Results for Canadians in March 2000, the government committed itself to man- agement excellence in three key areas: (1) a citizen focus in designing and delivering the applica- tion of sound professional and ethical values to guide public service management; (2) a focus on results as an integrating principle of management in all departments; and (3) a continuous exam- ination of its expenditures to assure responsible spending. Managing for results is seen as a cata- lyst for learning, innovation, and improvement. With this aim in mind, the Treasury Board of Canada Secretariat and the Office of the Auditor Gen- eral of Canada worked with senior managers across the public service to develop the Managing for Results Self-Assessment Tool. The tool brought together more than 10 years' experience in implementing results-based management to share practical guidance on key elements of man- aging for results. The tool may be used by an organization, a directorate, a branch, or even a unit within a government department or agency to take stock of its ability to manage for results. The tool clearly communicates the interconnected nature of the key elements of managing for results by emphasizing the need to view them in an integrated fashion. In 1997, the Office of the Auditor General assessed the existing state of results-based management in the federal government and in June 2003, the Treasury Board Secretariat introduced the Management Accountability Framework. This framework provides deputy heads with a clear statement of man- agement expectations in support of the vision of a modern public service set out in Results for Canadians. The Self-Assessment Tool supports the implementation of the framework by helping orga- nizations apply a more results-focused approach when assessing their management practices and reporting to Parliament on how well their services and programs respond to the needs of Canadians. The self-assessment tool helps departments and agencies assess their strengths and weakness in using results to support decisionmaking and improvement. The logic of the tool revolves around five pivotal supporting elements: commitment to results, results-based strategic planning, operational/ business planning, measuring results, and reporting on results. Source: The Managing for Results Self-Assessment Tool. Office of the Auditor General and Treasury Board Secretariat of Canada, January 31, 2003. 31 APPENDIX C Use of Self-Assessment in the Preparation of a Country Assistance Strategy and Achievement Report in the World Bank: The Experience of Tunisia Early on, the program team accepted the idea of designing a result-based country assistance strat- egy (RBCAS), beginning with the self-assessment of the achievements of the previous CAS. The team understood that the focus on "results" is not about attribution, but about management, and that it helps the World Bank to support the client better and to be a better partner. Tunisia was not a pilot country for this approach; rather, it adapted to the situation of a middle-income country some of the early lessons learned in making the CAS more useful as a tool to manage for achieving results on the ground. Internal Process Early on, several participatory retreats and meetings were held about the results agenda to develop common understanding and language. Theresultsapproachwasbeneficialinthatitforcedthecountryteamtothinkcross-sectorally by strategic objectives and outcomes and provided a platform for substantive discussions. The achievement report (self-assessment) helped the staff understand the concept and appli- cation of outcomes and was found to be of great value. The country team relied on expert facilitation. Process with the Client The client initially resisted the participatory process for joint identification of outcomes. The client voiced concerns as to the benefits of a results-based country strategy. The client reached agreement with the World Bank on the results chain and outcomes but remained uncomfortable with committing to indicators and targets. The client welcomed the self-assessment report and found it very valuable. 33 34 World Bank Working Paper Main Lessons Learned The causal logic from inputs to outcomes is better articulated in the strategy and provides a solid platform for implementation and for monitoring and evaluation. The process and product of the self-assessment (achievement report) were valuable for the country team and the client. The new approach to strategy design neither cost more, nor took longer than traditional country strategies. Suggestions for Country Teams Confirm institutional expectations. Engage the client early on and identify champions. Encourage the client to engage in partici- patory dialogue so that the client has a large part in designing the results framework and indi- cators. Clarify the message: "the results approach is a tool for better management." Invest in solid analytic work before designing the strategy. Think ahead--what are the implications for future monitoring and evaluation of outcome indicators? How can outcome data be systematically collected and used? Challenges ahead for the Tunisia Program Team Systematically monitor the process and act on data and information. Carry cross-sectoral collaboration into implementation. Involve the client in monitoring and use of information for decisionmaking. Use the results-based strategy as a tool for budgeting and management. Implement flexibly. Prepared by Cécile Fruman, Senior Country Officer, Maghreb Department. APPENDIX D Process for Planning and Conducting Self-Assessments: Six Phases and 12 Critical Steps 35 Step 10 I.I. Direction Review findings Direction with program director, task manager, and program staff III. Examination Step 1 Review guidelines & II. Participation requirements IV. Consultation for self- assessments Step 3 Step 11 Clarify approach Step 5 Prepare draft and process with Conduct Step 7 reports for staff systematic desk Interview key and client review staff review informants (e.g. legal, operations, Step 2 peers) Clarify parameters & report expectations with manager Step 6 Step 4 Interview Map information program staff needs & sources VI. Utilization Step 8 Consult with relevant Step 12 donors & Use findings for strategic partners program design, decisionmaking, and staff learning Step 9: May conduct country visit © Terminology Development Intervention An instrument for support aimed at promoting devel- opment (for example, policy advice, projects interventions) that reaches the population. Managing for Results The process of steering development assistance to achieve results on the ground, affecting synergies for improved decisionmaking, knowledge building, and learn- ing for enhanced accountability. Performance The degree to which a development intervention or devel- opment actor operates according to specific standards or criteria. Result The output, outcome, or impact of a development intervention. Results Chain The causal sequence of a development intervention, mov- ing from inputs and processes to outputs, outcomes, and impact. Self-Evaluation An evaluation by those who are entrusted with the design and delivery of a development intervention. Self-Assessment Any systematic inquiry into the progress made by a pro- gram or countrywide strategy in achieving outcomes, conducted by those involved in the program 37 38 World Bank Working Paper Goal The higher-order or longer-term national objective to which a development intervention is intended to contribute. Impact The long-term effects--positive and negative, primary and secondary--(for example, changes in socioeconomic or health status) produced by a development intervention, directly or indirectly, intended or unintended. Outcome The medium-term or shorter-term effects or behavior changes resulting from a development intervention. Also, specific program period-bounded outcomes. Effect The intended or unintended change resulting directly or indirectly from a development intervention. Output The products, capital goods, and services that result from a development intervention. It would include changes or products resulting from the interventions that are relevant to the achievements of outcomes and impact throughout the results chain. Process The systems, processes, and procedures set in place to achieve outputs, outcomes, and impacts. Input Financial, human, and material resources used for a devel- opment intervention. Indicator Quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement, reflect the changes connected to a development intervention, or help assess the performance of a development actor. Based on: OECD Glossary 2002 and OPCS. See www.oecd.org/dac/evaluation. Bibliography Aga Khan Foundation. 1993a. "Assessing Information Needs." Primary Health Care Man- agement Series. ------. 1993b. "Assessing the Quality of Management." Primary Health Care Manage- ment Series. Allen, R., S. Schiavo-Campo, and T. C. Garrity. 2004. "Assessing and Reforming Public Financial Management: A New Approach." Washington, D.C.: World Bank. Baastel-Universalia. 2002. "Results-based Management Study Tour Report." Unpublished working paper. November. Canada Office of the Auditor General. 2000. "Implementing Results-based Management: Lessons from the Literature." Ottawa. Grafstein, R. 1992. Institutional Realism. New Haven: Yale University Press. Harrison, M. I. 1994. "Diagnosing Organizations." In Applied Social Research Methods. Vol. 8. Newbury Park, CA: Sage Publications. International Fund for Agricultural Development (IFAD). 2004. "New Evaluation Process and Methodology." IFAD Website. Jarawan, E., S. Stout, and others. 2004. "Rapid-Results: Enhancing Capacity while Achiev- ing Results." Operations Policy Country Services Results Secretariat, January 7 Event, "Results: Everybody's Business." Washington, D.C.: World Bank. Kaplan, R. 1991. The Strategy-Focused Organization. New Haven: Harvard Business School Press. Kusek, J. Z., and R. C. Rist. 2004. Ten Steps to a Results-Based Monitoring and Evaluation System: A Handbook for Development Practitioners. Washington, D.C.: World Bank. Kusek, J. Z., R. C. Rist, and E. M. White. 2004. "How Will We Know Millennium Devel- opment Results When We See Them?" World Bank African Region Working Paper Series No. 66. Washington, D.C. 39 40 World Bank Working Paper Leeuw, F. L., R. C. Rist, and R. C. Sonnichsen (eds.) 1994. Can Governments Learn? Compar- ative Perspectives on Evaluation and Organization Leadership. New Brunswick/London. Love, A. J. 1991. Internal Evaluation: Building Organization from Within. Newbury Park, CA: Sage Publications. Mackay, K., ed. 1998. Public Sector Performance: The Critical Role of Evaluation, Selected Proceedings. Operations Evaluation Department. Washington, D.C.: World Bank. Norton, D. P. 2004. Strategy Maps. New Haven: Harvard Business School Press. Rodriguez-Garcia, R. 2003. "Framework for the Assessment of Principal Recipients' Capacity for Monitoring, Evaluating, and Reporting." Geneva: The Global Fund to Fight AIDS, Tuberculosis and Malaria (corporate authorship). Rossi, P. H., and H. E. Freeman.1993. Evaluation: A Systematic Approach. Newbury Park, CA: Sage Publications. Rosenthal, R. 1984. "Meta-Analytic Procedures for Socials Research." Newbury Park, CA: Sage Publications. Rist, R. 1995. "Development Questions and Evaluation Answers." In Robert Picciotto and Ray C. Rist, eds. Evaluating Country Development Policies and Programs: New Approaches for a New Agenda. San Francisco: Jossey-Bass. Senge, P. M. 1990. The Fifth Discipline: The Art and Practice of the Learning Organization. Doubleday. Shadish, W. R., Jr., T. D. Cook, and L. C. Leviton. 1991. Foundations of Program Evaluation: Theories of Practice. Newbury Park: Sage. Shaw, R. P. 1999. "New Trends in Public Sector Management in Health." Working Paper, World Bank Institute, Washington, D.C. Sonnichsen, R. C. 2000. High Impact Internal Evaluation. Newbury Park, CA: Sage Publi- cations. Valadez, J., and M. Bamberger. 1994. "Monitoring and Evaluating Social Programs in Developing Countries." Washington DC: Economic Development Institute of the World Bank. Von Schirnding, Y. 2002. "Health in Sustainable Development Planning: The Role of Indi- cators." Geneva: World Health Organization. White, E. M., and R. Rodriguez-Garcia. 2003. "Country Assistance Strategies Become `Results-Based.' " Prepared for Operations Policy and Country Services event "Results: Everybody's Business," January 7, 2003 at the World Bank, Washington, D.C. Windsor, R., and others. 1994. Evaluation of Health Promotion, Health Education, and Dis- ease Prevention Programs. 2nd ed. Mountain View, CA: Mayfield. World Bank. 1993. Getting Results: The World Bank's Agenda for Improving Development Effectiveness. Washington, D.C. ------. 2000. "World Bank Strategy: Reforming Public Institutions and Strengthening Governance." Poverty Reduction and Economic Management (PREM), Public Sector Group. Washington, D.C. ------. 2002a. Annual Report on Evaluation Capacity Development. Operations Evaluation Department. Washington, D.C. ------. 2002b. "CAS Results Framework and Matrices, and CAS Completion Report: Guideline Framework for Reports." Operations Policy and Country Services. Wash- ington, D.C. ------. 2003. Annual Report on Evaluation Capacity Development. Operations Evaluation Department. Washington, D.C.: World Bank. Self-Assessment in Managing for Results is part of the World Bank Working Paper series. These papers are published to communicate the results of the Bank's ongoing research and to stimulate public discussion. This paper argues that the self-assessment it describes can make three strategic contributions. First, it uses a participato- ry process, which engages those responsible for designing and implementing strategies and programs in the assessment process. Second, it improves strategy design by providing crit- ical information that can be used for fine tuning and further development of the strategy. Third, it focuses on the outcomes that have been achieved during the program period--triangu- lating multiple data sources to assess progress, and deriving lessons that can be incorporated into the next program strat- egy. Thus, these self-assessment exercies become powerful tools in planning and managing for results at the operational level, and an important source of learning about achievements across sectors and countries. A well-done self-assessment is not a "quick and dirty" exer- cise; doing these self-evaluations well requires time, effort, and resources. The fundamental purpose of this paper is to supply some of the tools, bringing inquiry concepts, process, and practice closer to development practioners and to engage them in applying this information to the design and selection of strategies, the management of existing intiatives, and the assessment of progress to achieve better and faster results on the ground. World Bank Working Papers are available individually or by subscription, both in print and online. ISBN 0-8213-6148-1 TMxHSKIMBy361481zv,:<:":%:) THE WORLD BANK 1818 H Street, NW Washington, DC 20433 USA Telephone: 202 473-1000 Internet: www.worldbank.org E-mail: feedback@worldbank.org