* Lessons & Practices O P E R A T I O N S E V A L U A T I O N D E P A R T M E N T BUILDING EVALUATION CAPACITY A key feature of successful organizations is decision makers faced with tough political their ability to learn from experience and to choices; it shows them what is and is not react to market or client responses to their working, and why. actions. In an environment determined more * Improving resource allocation and budgetary and more by markets rather than by states, process. How efficiently are government political institutions, the press, and the public revenues being spent? Will shifts in bud- are demanding more efficiency and respon- getary allocations increase efficiency? To siveness. The capacity to absorb information, answer such questions, governments need assess performance, and respond flexibly is to base their resource allocation and bud- becoming as vital for governments as for getary processes solidly in the evaluation private firms. of public expenditures. * Improving investment programs and projects. Evaluation agencies exist at least nominally Evaluation of investment projects can help in many developing countries, and national engender a performance-oriented culture audit offices exist in almost all. Generally, within government agencies. The exami- though, the evaluation capacities do not play nation of completed projects is an exercise an important part in decision making. They in public accountability, which is impor- could-and should. This issue of Lessons & tant in its own right. Beyond accountabili- Practices looks at the steps needed to build ty, good ex post evaluation provides feed- and benefit from evaluation capacity in the back to management-about weak points public sector in developing countries. in budgetary processes, about the perfor- mance of public agencies, about manage- Evaluation can strengthen public ment quality, and incentives in the public sector management sector. There are also important links back to policy evaluation, for example with Used effectively, evaluation capacity can play respect to criteria for and levels of external a critical role in four areas of a nation's public sector management: * Influencing policy analysis and formulation. C O N T E N T S Careful analysis of the costs and benefits of existing policies is key to informed, Evaluation can strengthen pubLic tough-minded, policy analysis and formu- sector management 1 lation. In today's more open economies, Ev aluation in developmg countries 3 policy consequences are more complex Lessons for planners 5 and sequencing issues more important. Developing a countrv- Objective evaluation also gives comfort to specific strategy 5 NOVEMBER 1994 NUMBER 4 Box 1: In Korea, an emphasis on operational performance Korea has evaluated government performance since 19b2. Evaluation is the responsibility of the Performance Evaluation Bureau (PEB) within the Economic Planning Board. Headed by a director general. PEB has five directors, 20 deputy directors. and 22 staff members. It evaluates the performance of government and of public enterprises, with the aim of increasing managerial efficiency. It establishes and refines the standard guidelines for evaluating government performance that are to be applied by all govern- ment ministries for their self-evaluations. It collects and a nalvzes the results of these evaluations, and reports on them twice a year to the cabinet council and the president. PEB also handles problems that may arise during the implementation of important government programs. The performance evaluation system focuses on the operational performance of the govern- ment's major policies and programs. It covers three kinds of evaluation: * Intensive analysis of government programs that have special economic and social importance. * Ministerial self-evaluation. * Evaluation of major government implementation programs. These include policies and projects, presidential directives, and presidential campaign commitments. Evaluation results are used to locate performance problems and to determine future resource allocations. PEB formulates recommendations to correct identified problems, and the Office of the Budget and ministries concerned are consulted to guarantee adequate feedback, before the recommendations are implemented. Once PEB has submitted its report to the cabinet council and the president, a press release on the report is issued. borrowing for public projects. All these ernment itself: the assessment of the factors make investment-level evaluation a relevance, performance, and cost of gov- potentially powerful instrument for learn- ernment agencies and ministries, and of ing from both success and failure, and for alternative ways of doing public business. gaining insights into how public institu- Performance-based auditing is one instru- tions need to change. ment for such reappraisals. What is critical * Examininig fundamental missions. A vital is a capacity for such evaluation indepen- dimension of public sector management is dent of the mainstream bureaucracy but the evaluation of institutions or the gov- with access to it. Om 2 LESSONS & PRACTICES Evaluation in developing countries uating externally financed programs and projects. In developing countries, demand for evalua- * Limited attention to the quality and timeli- tion began to emerge in the mid-1980s. ness of information and to the need for Several countries have evaluation units objectivity and reasonable independence attached to planning, finance, or prime minis- in conducting evaluation. ters' offices. The best examples of strong insti- * The high cost of evaluation research tutional mechanisms for evaluation are in and lack of access to rapid, low-cost East Asia (see Box 1). A movement to strength- research methods. en evaluation is underway in Latin America, * Shortage of trained staff. led by Brazil, Chile, Colombia, and Mexico, inspired by severe resource constraints and The Bank's experience public demands for greater accountability in To help borrower countries to strengthen the public sector. their evaluation capacity, the Bank started with efforts to ensure borrowers monitored But in most developing countries an and evaluated the projects it financed. evaluation culture-or the consistent use of Then it introduced a requirement for borrow- feedback in formulating policies and allocat- ers to contribute to project completion reports ing resources-is only incipient. Sensitivity to on the projects it supports. Since 1987, it has public criticism and the fear of political fallout had a program to support the development from evaluation findings are inhibiting factors. of broadly based evaluation capacity (see Many countries still lack the essential require- Box 2). Countries assisted by this program ments of effective evaluation: the quality of include Bangladesh, Brazil, Chile, China, information and access to it is often poor, Colombia, Indonesia, Mexico, Morocco, mechanisms for feedback into the decision Pakistan, Philippines, Sri Lanka, Thailand, making process are weak, and a culture of Venezuela, and Zimbabwe. The Evaluation accountability is not firmly in place. Capacity Development Program (ECDP) has also supported development agencies Current problems including the West African Development Common constraints and problems are: Bank, the Andean Development Corporation, * Lack of interest and commitment to the the Caribbean Development Bank, and evaluation function at the political level. the Islamic Development Bank. This attitude is often then manifested at the bureaucratic level. The experience, which has been mixed, * Lack of feedback mechanisms for applying emphasizes the following prerequisites for evaluation findings. This results in both effective evaluation: lack of demand for evaluation results and m Evaluation will not thrive unless there lack of institutional links between those is a demand for it. Demand for evaluation who carry out evaluation and those who results, and a performance-oriented cul- need to use its findings. ture in the public sector, are essential if * More attention to preparing and apprais- evaluation is genuinely to affect public ing programs and projects than to evaluat- policy and resource allocation. ing their performance on completion. * Conditions must favor the development of reli- * Little involvement of national staff in eval- able and accessible informntion. There is NOVEMBER 1994 Box 2: Evaluation capacity development program In 1987, the Bank established an evaluation capacitv development program (ECDP) in response to specific requests from borrowers. Managed bv operations evaluation staff, the program has the following elements: * Designiing action programs: After high-level countrv authorities have appointed a senior official to head the evaluation function, the task manager and staff visit Bank headquarters to discuss goals and constraints with evaluation, operations, and EDI staff. These discussions help identify the relevant experience within the Bank and internationally . Action plans are designed for instituting or strengthening evaluation, and a framework is laid out for cooperation with OED and others. To help focus action plans and system development, organization and process-diagnostic studies mav be recommended and planned. Follow-up meetings take place periodicaHyv in Wiashington or in the borrowing country. * Building awareness: Seminars and workshops on evaluation methods and tindings are held for government officials. Presenting the findings from the Bank's ow n ev alua- tions helps develop both the awareness of potential users of evaluation results and a consensus for support of evaluation within the country. * Documentation: Evaluation material from OED and other sources, including interna- tional development agencies, is made available. a Training: Seminars and workshops are held on the role and conduct of evaluation, with OED staff primarily contributing course material, case studies, and lectures. On-the-job training is provided through engaging borrowing country staff as observers or participants in OED's evaluation work. OED takes staff on secondment, where this constitutes a priority within the framework of cooperation arrangements. OED staff increasingly spend time during evaluation missions to discuss evaluation issues with the host institution. * Parallel evaluations: OED considers topics for parallel evaluation with borrowing coun- tries that have made a commitment to develop their evaluation function. Parallel evalu- ation is undertaken if these joint efforts coincide with OED priorities, as reflected in its work program, and do not affect OED's independence. (ED retains full responsibility for its reports and encourages the host institutions to prod uce their own. scope here for institutional development ty, ranging from firm enforcement of assistance, by the Bank and other agencies, covenants on financial reporting to much to strengthen the basic institutional and broader support, for example through professional infrastructure of accountabili- accountancy development projects, 4 LESSONS & PRACTICES financial sector adjustment loans, or pub- Plan in the context of public lic sector management operations. sector management * Evaluation results must be fed back into a Capacity building for evaluation needs to be receptive and responsive decision making planned and undertaken within the context of process. Government leaders, parliaments, improvements in public sector management. and publics need a basis for identifying The performance of the evaluation system will both specific shortcomings and systemic depend on the clarity of the institutional rela- problems in government performance. A tions within the country, the responsibilities of free flow of information helps managers at the function, and the means available. Any all levels. At the technical level, managers country considering establishing or strength- need feedback to identify what is working ening an evaluation function in the public sec- and what is not-and why. Senior deci- tor will need to address the factors that are sion makers must be able to monitor not listed in Box 3. only major policies and programs but the performance of the institutions running Start modestly them, so they can specify reform mea- It may be prudent to start modestly and put in sures, where needed. place simple mechanisms that can be imple- * Evaluation implies accountability. Evaluation mented immediately, rather than starting with findings must have-and be seen to the design and development of a comprehen- have-consequences for the institutions sive or very sophisticated set-up. and the individuals who are accountable for the conduct of public business. In principle, the evaluation of country policies and programs promises the highest payoff. Lessons for planners But such evaluation is politically and bureau- cratically more sensitive. It is also more Serve decision makers' needs demanding methodologically and may take Evaluation can be best developed if it is seen longer to establish itself. Meeting demand for by all concerned-within both the country and sectoral and project-level evaluation within the development community-as a way to line ministries may often provide a more effec- learn and to improve the performance of the tive starting point. public sector. Efforts to build evaluation capacity must be designed to serve the real Developing a country-specific strategy needs of domestic decision makers. Demand can arise from a prominent client or key stake- In most developing countries, building up an holder who sees the benefit of using evalua- effective evaluation capacity will require sus- tion results to improve public performance. tained, complementary efforts in three areas: to develop and nurture genuine demand and Key officials should take full advantage of political support for evaluation; to improve the demand that already exists in some parts supply capacity by building relevant skills and of government. Donors and international institutions; and to build financial information financial institutions can, through their loan and dissemination infrastructure. requirements and influence, help nurture demand by domestic agencies, but should On the demand side, in countries such as take care not to substitute for it. China and Thailand, auditors general played NOVEMBER 1994 5 Box 3: Factors to consider in institutionalizing evaluation How does a country go about building a public sector evaluation function' This checklist, in approximately sequential order, showxs items that will need rev iew and action. Relation to other public sector functions and institutions * Where evaluation is set up at the executive level of government, its relation to the management of the national budget, to public sector expenditLre decisions, and to the development of the strategic plan. * The relation between executive-level evaluation and the ftinctions of the a ud itors or controllers general, or other independent oversight institutions of the legislature. * Legal requirements for setting up an evaluation function. * Relation of evaluation to the audit or internal control functions. Centralization or decentralization * The appropriate division of responsibility between the central government, line ministries, provincial and local authorities. * Rules and regulations governing the evaluation function. Location * Independence of the function. an initial important role in raising awareness both these countries have moved to institu- about the need to establish a national evalua- tionalize the function and train staff. tion function. These agencies have also pur- sued the organization of evaluation training On the supply side, the Bank has supported programs for their own staff. In some coun- Brazil in several workshops and seminars tries the executive branch has followed the on evaluation in government and to discuss lead and taken steps to institutionalize the evaluation experience (for example on function. In Thailand, for example, a task force the environment, public sector management). under the prime minister, with the participa- In Colombia, the Bank has provided support tion of the auditor general, is studying how to for public policy evaluation through a public establish an evaluation function that will cover financial management project. In Chile, government agencies and public enterprises. where human resources are not a constraint, In Brazil and Colombia, constitutional require- the Bank has been asked for advice about ments have generated demand for evaluation how to establish an evaluation function to from the legislative and executive branches; complement the highly developed resource 6 LESSONS & PRACTICES (Box 3 cofltinnfed) * Links to the presidency, prime minister's office, and the planning, finance, and budget svstems. * Legislative branch requirenients. * Requirements of indiv idual agencies' line management, and the role of self-evaluation. * Relation of evaluation to research and academic institutions and private sector entities. Scope, coverage, focus * Policies, programs, or projects. * All public sector resource uses, or only public expenditures financed through public debt. * All sectors, including the parastatals. * Priority of externally funded programs and projects. * Impacts and the human development incidence of public resource use. * Accountability aspects of public sector decisions, and learning from experience and feedback to the decision making processes. * Ev aluation reporting channels and the follow-up to evaluation work. Financial resources, staffing, information requirements n lI& E systems. * Financial resources. * Professional skills and training. * Methods, guidelines, and norms. allocation system in place. In China and side the country could be disseminated. On Indonesia, the Bank's Institutional Develop- the supply side, it may be prudent to start ment Fund is supporting programs that focus with some simple steps. For example, the on demand as well as supply constraints. audit function may be strengthened and asked to undertake selective evaluations especially The choice of interventions, however, useful to senior policy makers. To alleviate needs to be closely tailored to a country's capacity constraints and to develop local particular circumstances. The following styl- skills, multilateral and bilateral agencies may ized framework may help assess country jointly evaluate their projects with country needs and opportunities. personnel. If training in basic disciplines rele- vant to evaluation is needed, donors and inter- Countries where demand, supply, and infrastruc- national financial institutions could fund such ture are weak. To nurture demand for, and training at local universities. Development of show the practical utility of, evaluation, best modern budgeting, auditing, and accounting practices in evaluation from within and out- systems may be useful. Interventions to NOVEMBER 1994 7 Box 4: Colombia legislates an evaluation function In 1990 the minister of finance and the head of the National Planning Department (NPD) called for a study of how to design a national evaluation system. The studv led the authori- ties to conclude that a public sector evaluation function wvould become a realitv only if there was a fundamental law requiring it. In 1991 the Colombian constitution was being rewritten. Authorities took the opportunity to draft an article making public sector evaluation mandatorv. The article conferred on the NPD the responsibility of establishing a national evaluation system. Late in 1991 the NPD organized a high-level meeting of present and past public sector offi- cials and private sector representatives to: * review alternative organizational approaches; * discuss evaluation experience arising from large, controversial public sector projects; * build consensus on the direction of the regulations regarding the evaluation function. Participants included members of the constitutional assembly, the minister of finance, the controller general, the head of NPD, the attorney general, and other sector ministers. Also participating were the deputy controller general of the United States, the WVorld Bank's director-general, operations evaluation, and other international officials. The NPD has established an evaluation function, and the responsible officers are develop- ing evaluation plans. Yet to be determined are the roles of the other key agents/ sLich as the controller general. In September 1993 the Bank approved a public financial-management project that incorporates a component for public policy evaluation. This comnponent aims at, among other things, verifying that public resources have been well programmed and honestly and efficiently spent. develop evaluation capacity should be simple, This approach can be used, strengthened cost-effective, and focused on activities that through technical assistance, even if local demonstrate the usefulness of evaluation. skills and institutional mechanisms in govern- ment are weak. Where skills in government Countries wlhere demand exists, capacity is weak. are limited, there may be scope for usingnon- There may be receptivity for training and governmental organizations in evaluation information on best practices and also for the and to listen to public feedback. Introducing use of government commissions/task forces to transparent performance standards, budgetary evaluate important programs and policies. classifications, and modern financial and 8 LESSONS & PRACTICES banking codes is also likely to have a high Instruments for the medium term payoff. The activities described in the preced- n Assess the legal framework affecting eval- ing paragraph may be useful in this case, too. uation. Changes may be needed to institu- tionalize evaluation and ensure its conti- Countries where demand is weak, capacity is nuity (see Box 4). promising. To build up demand, it may be * Exploit opportunities at the national and useful to link evaluation activities to public sector levels to support stakeholders who expenditure reviews and other public sector are interested in using evaluation results management activities, while seeking opportu- to improve the performance of policies nities in the course of the country dialogue for and programs. raising awareness among decision makers and * Incorporate evaluation findings in analy- legislators of the practical utility of effective sis and dialogue on public sector manage- evaluation. Local personnel may need more ment issues. practice-oriented-rather than academic- * Incorporate evaluation and monitoring training. This need has emerged in China, arrangements into project design and for professionals who are unfamiliar with eval- implementation plans, and include uation practices and methods. Where good specific indicators and evaluation work research institutes exist, encouraging local per- in terms of reference for mid-term sonnel to work on evaluation may be a good program/project reviews. beginning even when government is not well * Use project initiation workshops, mid- organized to fully use the evaluation results. term reviews, and public expenditure reviews and/or other regular activities to Demand and supply are relatively well developed. encourage and apply lessons of formal Here, it may be most useful to strengthen the and informal evaluations of policies, link between evaluation and resource alloca- programs, and projects. tion decisions and policy reform, and to pur- * Promote evaluation by users and sue improvements in the financial and infor- communities affected by programs mational infrastructure. and projects. Encourage agencies to evaluate their own performance and In most countries, different parts of the evalu- to establish feedback mechanisms to ation function are at different stages of devel- ensure results are applied. opment. This suggests the need to build on * Promote joint evaluations with external opportunities where they exist, and to accept assistance agencies and international that progress will not follow a predetermined financial institutions. Coordination with path. The design of country strategies for the development community can help developing evaluation capacity should be strengthen evaluation. based on a full appreciation of governments' * To be effective, evaluation work must be evaluation needs. It will normally be neces- rigorous and credible. Ensure that appro- sary to assess the demand within government priate skills, training, methodologies, and for different types of evaluation and deter- standards are provided. Here, private mine where the use of evaluation capacity sector and academic institutions can will be most productive. be valuable allies. NOVEMBER 1994 9 Observations on developing evaluation capacity Ro bert Klihgaard, Prvtfessior of Econoionic. Us, iwerziitti of Natal South Ahitca Why evaluate? abotit the success of particular projects or managers, a policy ot purposive Evaluation has several purposes: evaluations may engender a climate * To test or find clues for predictive rela- less subject to suspicion and abuse, and tionships that will help infornm subse- more open to the inputs of clients. quent decision making. * To create incentiv es for agents. How much evaluative capacity? Measuring performance carefuly and providing intense incentives are com- How- nmuch evaluation capacitv to establish plements. Better evaluations, other depends on howl helpful the ex'aluation things equal, enhance incentives and findings are expected to be, and on the reduce the risks for agents. uses to wvhich the findings and the evalua- * To enable agents to make mid-course tion process are to be put. These dimen- corrections. Evaluation data are the sions vxarxv considerably among tasks, more useful, the greater are the local agencies, prolects.and gov'ernments. and temporal variability in key factors affecting a program's success, and the Broadly speaking. the value of ex'aluation more susceptible the program is to depends on its ability to aftect managerial adaptations by its managers. decisions. This abilit) depends not only * To foster accountability and transparen- on the quality ot the ev aluation but also cy . Quite apart from what it shows on the incentives for politicians and "Mainstreaming" the development of that the Bank take up a more proactive evaluation capacity approach to supporting evaluation capacity development. In particular, it recommended A Task Force on Evaluation Capacity that assistance for developing evaluation Development reported to the Joint Audit capacity should no longer be regarded Committee of the Bank's Board of Executive as a distinct initiative but should form an Directors in July 1994. It stressed the close integral part of the Bank's country portfolio links between domestic evaluation capacity management work and of its support for in borrowing countries and the Bank's broader public sector management reform. The agenda on institutional development and report of the Task Force proposes a strategy portfolio management, and recommended for this purpose. 10 LESSONS & PRACTICES (Box 5 coztinuied) Establishing a vigorous ev%aluation func- bureaucrats, and perhaps for aid officials tion may be particularly important when and organizations. The economics of orga- governments particularly need account- nization suggests that one should spend ability and transparency-for example, more on evaluation when agents' incen- when a general lack of confidence must tives are affected bv evaluations and *%hen be oxvercome. A strong, visible, indeed what agents do greatly affects the 'prof- expensive, commitment to ev aluation itabilit " of the project or organization. may send a signal that has a v alue far One iniplication is that if evaluation find- beyond that measured in decision-analytic ings cannot affect agents' incentiv es much, or managerial terms. one should not ev aluate much. For evalua- tions to make a difference, incenti ves need These ideas may help explain why evalua- to be linked to evaluations. tion has not dev-eloped very far in some contexts. For example, a government Some of the managerial literature suggests will probably undervalue evaluation that evaluations are particularly useful when officials have weak links between when the environment or the technologp pay and performance, little authority is subject to uncertainty,where it changes to do mid-course correction, and little over time or locality or both, and where input or effect on project qualih', and managers are permitted to make mid- when evaluation has high opportunity course corrections. Thus evaluative capaci- costs. By improving incentives and ty is less useful in well-known, standard- decision processes, the v alue of ized technologies under stable conditions evaluation rises-which, other things where managers in any case have little equal, generates demand for better freedom of action. evaluation capacitv. NOVEMBER 1994 1 Suggested Reading Grav, Andrew, Bill Jenkins, and Bon Segsworth Evaliuation Leaniiig tr"n1 Interiiahticnal Expcne-uices Budgeting, Auditing, and Evaluation. Funfilons and Amsterdam, London, New York Tok%o North- Inlegralton in Sne-v CG,vernmenti. New Brunswick and Holland, 1992. London Transacbon Publishers. 1993. R1st. Rav C., ed. Progra,rn Evaluation and the Leeuw, F.L., Rav C. Rist. R.C. Sonnichsen, eds Can A1anawonet,nt l Gov GLiernlmenrt Palternus aLnd ProiSctS Go'evinrnent,l Learn? Comparative Per4pecluve.s on acrot-; Eilitl Nalion.s Nes%% Brunswick and London Or,rnizaltnal Learning New Brunswick and London Transaction Publishert, 194(0 Trancachon Publishers. 1993. World Bank. 'Report oi the Evaluahon Capacit% Mayne, J., NI.L. Bemelmans-Videc. 1 Hudson. Dev'elopment Task Force " Washington D C and R Conner. eds. AJdvaming Public Policy World Bank, June 30 1'4 Lessons & Practices is produced by the Operations Evaluation Department to help disseminate the World Bank's develop- ment experience. The views here are those of the Operations Evaluation Department staff and should not be attributed to the World Bank or its affiliated organizations. Please address comments and enquiries to the managing editor, Rachel Weaving, G-7137, telephone 473-1719. Internet: RWEAVING@worldbank.org 12 LESSONS & PRACTICES