U)P,p - 2 92 7 POLICY RE SEARCH WORKING PAPER 2427 Public Officials and Their A framework for understanding both bad Institutional Environment public sector performance and good through surveys of An Analytical Model for Assessing public officials-and for presenting the results to the Impact of Institutional Change policymakers in a format that on Public Sector Performance leads to more informed choices about public sector Nick Manning reform. Ranjana Mukherjee Omer Gokcekus The World Bank Poverty Reduction and Economic Management Network August 2000 POLICY RESEARCH WORKING PAPER 2427 Summary findings To perform well, public officials must be confident In building on the premise that public officials' enough about the future to be able to see a relationship actions-and hence their organizations' performance- between their efforts and an eventual outcome. Their depend on the institutional environment in which they expectations are shaped by their institutional find themselves, this framework avoids simplistic environment. If the rules are not credible or are unlikely antigovernment positions but doesn't defend poor to be enforced, or if they expect policies to be performance. Some public officials perform poorly and contradicted or resources to flow unpredictably, results engage in rent seeking, but some selfless and determined will be uncertain, so there is little point in working public officials work hard under extremely difficult purposefully. conditions. This framework offers an approach for Manning, Mukherjee, and Gokcekus present an understanding both bad performance and good and for analytical framework used to design a series of surveys of presenting the results to policymakers in a format that public officials' views of their institutional environment leads to more informed choices about public sector and to analyze the information generated in 15 reform. countries. They describe how survey results help map a Types of reforms discussed include strengthening the public sector's strengths and weaknesses and offer an credibility of rules for evaluation, for record approach to identifying potential payoffs from reforms. management, for training, and for recruitment; ensuring The framework emphasizes how heterogeneous that staff support government policy; preventing political incentives and institutional arrangements are within the interference or micromanagement; assuring staff that public sector. It emphasizes how important it is for they will be treated fairly; and making government policymakers to base decisions on information (not policies consistent. generalizations) that suggests what is most likely to work, and where. This paper-a product of the Poverty Reduction and Economic Management Network-is part of a larger effort in the network to develop practical strategies for the reform of public sector institutions. The surveys were funded by the World Bank-Netherlands Partnership Program. Copies of the paper are available free from the World Bank, 1818 H Street NW, Washington, DC 20433. Please contact Claudia Nolan, room MC4-562, telephone 202-473-0030, fax 202-522-7132, email addresscnolan@worldbank.org. Policy ResearchWorkingPapers are also posted on theWeb atwww.worldbank.org/ research/workingpapers. The authors may be contacted at nmanning@worldbank.org, rmukherjee@worldbank.org, or ogokcekus@worldbank.org. August 2000. (65 pages) The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the view of the World Bank, its Executive Directors, or the countries they represeni. Produced by the Policy Research Dissemination Center Public officials and their institutional environment: An analytical model for assessing the impact of institutional change on public sector performance Nick Manning, Ranjana Mukherjee and Omer Gokcekus Nick Manning is Senior Public Sector Reform Specialist in the Public Sector group of PREM network. Ranjana Mukherjee is a consultant in the Public Sector group of PREM network. Omer Gokcekus is Assistant Professor of Economics at North Carolina Central University and Scholar-in-Residence at Economics Department of Duke University. The authors wish to specially thank Gary Reid (ECSPE) and Yasuhiko Matsuda (LCSPS) for their review of this paper and their very helpful suggestions. The report benefited from discussions with Meta de Coquereaumont (DECRA), Steve Knack (PRMPS), Jeff Rinne (PRMPS) and Adeel Malik (PRMPS). Yasuhiko Matsuda (LCSPS) contributed to the conceptualization of the institutional environment in Section 2. Abbreviations and Acronyms BNPP Bank Netherlands Partnership Program CAPAM Commonwealth Association for Public Administration and Management DECRA Research Advisory, Development Economics and Chief Economist ECSPE Europe and Central Asia, Poverty Reduction and Economic Management HRMIS Human Resource Management Information System INE Instituto Nacional de Estadistica IRR Institutional Rate of Return LCSPS Latin America and Caribbean Region, Public Sector OED Operations Evaluation Departnent OLS Ordinary Least Squares OECD Organization for Economic Cooperation in Development NGO Non Governmental Organization PREM. Poverty Reduction and Economic Management SAFCO Sistema de Administraci6n Financiera y Control Gubernamental Bolivian Law 1178 on Financial Management and Control SURE Seemingly Unrelated Regression Equations ii Table of Contents 1. Introduction ...........................................................1 2. Conceptual Framework ..........................................................3 2.1 What is meant by institutional environment? ....................................................... 3 2.2 How can institutional environment be measured? ................................................ 7 2.3 How can performance be measured? ......................................... ................. 8 2.4 Whose performance? - different units of analysis ...............................1............... I 1 3. Survey methodology .......................................................... 14 3.1 Questionnaire design ......................................................... 14 3.2 Sampling methodology ......................................................... 15 4. Construction of Indicators for Institutional Environment ..................... .................. 17 5. Construction of Indicators for Performance ......................................................... 20 6. The Connection between Performance and Institutional Environment .................... 22 7. Presenting Opportunities for Reforms ......................................... ................ 24 7.1 Step 1 - it's the institutions not the people ......................................................... 24 7.2 Step 2 - encourage contestation in interpretation ................................... ............ 25 7.3 Step 3 - challenge prior assumptions .......................................................... 25 7.4 Step 4 - encourage informed speculation on performance pay-offs ........... ......... 26 8. Summary .......................................................... 34 Selected Bibliography .......................................................... 36 Appendices ......................................................... 41 Appendix 1: Illustrative questions to probe for dimensions of the institutional environment .......................................................... 41 Appendix 2: Institutional environment indicators in the East Caribbean States ....... 46 Appendix 3: Illustrative questions to probe for dimensions of performance ............ 48 Appendix 4: Performance indicators in the East Caribbean States ........................... 54 Appendix 5: Connection between performance and institutional environment ........ 56 Appendix 6: Econometric and analytical issues ...................................................... 59 Index of Figures Figure 1. Analytical model for BNPP-funded surveys ............................................. ....... 3 Figure 2. Credible rules, credible policy and predictable resourcing constitute the institutional environment within any perspective ...................... . 8 Figure 3 The main components of government employment .................... .................... 11 Figure 4. Officials say that the whole government is much more corrupt than their own organization ............................................................ 15 Figure 5. Construction of indicators for institutional environment in East Caribbean states ........................................................... 17 Figure 6. Construction of indicators for performance in East Caribbean states .............. 20 Figure 7. Performance and Institutional Environment in sampled Bolivian public sector organizations ........................................................... 23 Figure 8. Rule complexity affects performance in Albania ....................... .................... 25 Figure 9. Expected impact of reforms on components of performance in Bolivia ......... 32 Figure 10. Effect of institutional environment on results focus in Bolivia ....................... 56 Figure 11. Effect of institutional environment on accountability in Bolivia .......... .......... 57 Figure 12. Effect of institutional environment on employee morale in Bolivia ......... ...... 58 Figure 13. Expected impact of reforms on different components of performance in Bolivia Central Administrations .................................................... 65 Index of Tables Table 1 Illustrative prior assertions tested against survey evidence ............. ................ 26 Table 2 Reform Interventions and anticipated performance pay offs in the Bolivian public sector ............................................................. 31 Table 3 Reform Interventions and anticipated performance pay offs in Bolivia Central Administrations ................................................... 64 Statistical Table I Rule credibility indicator and components (East Caribbean) ........... 46 Statistical Table 2 Policy credibility indicator and components (East Caribbean) ........ 46 Statistical Table 3 Indicator of resource adequacy and predictability and components (East Caribbean) ............................................ 46 Statistical Table 4 Indicator of overall institutional environment and components (East Caribbean) ........................................................... 47 Statistical Table 5 Results focus indicator and components (East Caribbean) .............. 54 Statistical Table 6 Accountability indicator and components (East Caribbean) ......... ... 54 Statistical Table 7 Employee morale indicator and components (East Caribbean) ........ 54 Statistical Table 8 Overall performance indicator and components (East Caribbean) ... 55 Statistical Table 9 Calculating the IRRs (Bolivia) ........................................................ 63 1. Introduction This paper lays down the analytical framework that was used in designing a series of surveys of the views of public officials concerning their institutional environment and in analyzing the information that was generated in 15 countries. It describes how the survey results help to challenge preconceptions, map the strengths and weaknesses of a public sector, and identify potential pay-offs from reforrn interventions. The need for an analytical framework emerged from the lessons of past experience in the World Bank. The Operations Evaluation Department reported that during 1980-97, only one-third of the Bank's closed civil service reforrn interventions had successful outcomes.' Other reviews of the Bank's public sector reform efforts have identified shortcomings of the Bank's approach in this area, pointing out the risks of a narrow and 'technocratic' view of what is needed for public sector reform, and of a reliance on 'best practice' models that have not been feasible in the particular country setting. 2 The Bank's most recent strategy for reforning public institutions has identified that for the approach to be effective: "... we need to work with our partners to understand and address the broad range of incentives and pressures - both inside and outside of government - that affect public sector performance."'3 The strategy paper also points out that for the analytic work to be useful: "We need to start with a thorough understanding of what exists on the ground and emphasize goodfit rather than any one-size-fits-all notion of best practice. And we need to work with our clients and other partners to develop and apply analytic tools effectively. " The framework laid out in this paper addresses both these needs. Public officials are not inherently rapacious rent-seekers; they respond to the incentive structure they face. There is ample evidence, both theoretical and empirical, to suggest that the performance of public officials is greatly determined by the institutional environment that they find themselves in. The framework recognizes that incentive systems are different in different countries - and vary across types of organizations and types of officials in the same country. Recognizing this, it offers a method for designing surveys that uncover the sanctions and rewards that drive behavior from those who may be subjected to them - the public officials themselves. Surveys were designed to cover several areas. They elicit a contextual description of the public sector, including characteristics of respondents, their reasons for joining the public sector and the length of time worked in government. Second, they offer the possibility of I World Bank (1999). 2 World Bank (2000). http://www-wbweb.worldbank.orgfprem/prmps/ 3 Ibid, p.4-5. 1 presenting policy-makers with robust confirmation of the theory-based assertion that it is the institutional environment in general, and rule credibility, policy credibility, and resource adequacy and predictability in particular, which drive performance. Third, they provide an opportunity to inform in-country reform discussions with current data. Fourth, the surveys also allow country-specific hypotheses to be tested. Many widely held views on public officials are often repeated but without substantive evidence - more akin to "urban myths" than to empirical observations. The surveys allow such assertions to be tested and supported or refuted. Finally, the surveys allow some assessment of which aspects of institutional environment are impacting on performance. It suggests those reform actions that seem most promising for higher performance, given the country's existing public sector conditions. The surveys and analyses were financed under the Bank-Netherlands Partnership Program (BNPP). Surveys of public officials funded by the BNPP have been completed in Albania, Argentina, Bangladesh, Bolivia, East Caribbean States, Guyana and Indonesia. Surveys are in progress or are being initiated in Bulgaria, Cameroon, India, Kenya, Macedonia, and Moldova. This program has provided funds for data analysis of a separate survey of public officials in Armenia, and another data analysis exercise is also planned for an ongoing survey in Benin. The survey instruments were based on a model designed in collaboration with Professor Bert Rockman of the University of Pittsburgh. World Bank staff working on a particular country tailored this basic questionnaire according to country-specific background and issues in public sector reform. This paper summarizes the premises on which the surveys were designed and analyzed, and the analytical framework. In explaining that analytical framework, examples have been used whenever it was considered their use would help make ideas clear. The examples have been drawn from survey experience during 1999 and 2000, and country data in Albania, Armenia, Bolivia, Guyana, Macedonia, and five countries in the East Caribbean region. The selection of examples was guided by their usefulness in illustrating the steps in the analytical framework, and not to point out any strengths or weaknesses of any country's public sector. Section 2 lays out the conceptual framework. Section 3 summarizes the survey methodology. Section 4 describes the approach for measuring the institutional environment using the East Caribbean example as an illustration. The approach for measuring aspects of performance is described in section 5, again using the same example of East Caribbean states. The relationship between performance and institutional environment is described in section 6, using the illustration of the Bolivian public sector. The Bolivian example was also used in Section 7 to show how survey results can be used to identify promising prospects for reform. Section 8 concludes with an overview of how the analytical framework is being used in practice. 2 2. Conceptual Framework Public officials in developing countries are often viewed as unskilled, poorly educated and poorly motivated to perform their official tasks. The assumption is frequently made that public officials are primarily motivated to exploit any official privileges that their positions give them to engage in opportunistic behaviors. Given the poor performance of public bureaucracies in many developing countries, and the undeniable reality of the low public sector salaries, such an assumption may not be wholly unrealistic. Yet, there are many public officials who, despite such a poor incentive environment, may perform their tasks without shirking, stealing or subverting in spite of overwhelming incentives for opportunism. For every rent-seeking or poorly motivated public official, there may be another who is providing a valuable service under extremely adverse circumstances. There is perhaps more to be learnt from an examination of why some public officials perform against the odds than from reciting the standard litany of failing public services. This paper starts from the premise that public officials' actions - and hence the performance of their organizations - depend upon the institutional environment in which they find themselves. In this way the analytical framework avoids the anti-government bias that pervades much of the literature on the public sector in developing countries, without attempting a defensive justification for poor performance.4 The analytical model allows verification of the assumption that it is the institutional environment that impacts on the performance of organizations in the public sector rather than the waywardness or malfeasance of officials. Accordingly, this model suggests that reform interventions that improve any or all aspects of the institutional environment will result in higher perfornance of some or all of the areas in which performance is measured. Figure 1. Analytical model for BNPP-funded surveys Institutional environment: Performance: * policy credibility i * results focus * rule credibility * accountability . resource adequacy and * employee morale predictability 2.1 What is meant by institutional environment?5 Institutions are the humanly devised constraints, or set of relational contracts that guide public officials' activities. They are made up of formal constraints (e.g. rules, laws, constitutions), informal constraints (e.g. norms of behavior, conventions, codes of conduct), and their enforcement characteristics. For public officials, formal rules are laid 4 See Tendler (1997) for a summary of these perspectives. This section was developed in collaboration with Yasuhiko Matsuda of LCSPS. 3 down in their code of conduct and operation manuals, in the budget documents, and in the many decrees, directives and instructions through which policy is conveyed. The informal rules are what the officials collectively understand as appropriate behavior, 'how we do things around here'. For example, not vigorously implementing the minister's newly announced scheme might result in a transfer to a position in a remote and inaccessible area. Institutions provide the incentives that provoke or prohibit certain actions. Rules and regulations, formal and informal, together define the incentive structure of public officials within their organization, or within their peer group, or across the public sector as a whole.6 Fundamentally then, this institutional environment shapes the expectations of public officials. If there is a rule about the management of records in the organization, or about methods of performance appraisal, then behavior will vary according to whether the official believes that breaches of these rules really will be punished. Similarly, willingness to gear actions to support Ministerial policies is somewhat greater if officials believe that policies will remain in force for a period of time, and will not be undermined by other policies of equal force. Expectations that policies are likely to be soon reversed lead, at best, to second-guessing of what the next ones might look like. At worst they lead to cynical disregard for any announced policy. Theprincipal-agent perspective assumes incompatibility of interests between the principal and the agent.! The consequent pursuit of private interests can take various forms including shirking (i.e., under-production of outputs desired by the principal), stealing (i.e., embezzlement and other forms of bureaucratic corruption), and sabotage/subversion (i.e., pursuit of objectives that are blatantly against the organization's goals). The relationship between the principal and the agent thus conceptualized is a contractual one, whereby the agent agrees to perform certain tasks, which he/she would not have done without the "contract," in exchange for a set of positive incentives to be provided by the principal. The principal enforces the contract through the monitoring mechanisms. In this perspective, an organization is a chain of voluntary contracts in a hierarchical structure.8 Suggested performance improvements emerging from this perspective tend to refer to increases in official salaries, greater contestability in public service provision, and stiff penalties for rule breaking. 6 See Horn (1995) 7 The principal-agent theory is not the only application of economic theories to the study of organizations. For a more thorough and nuanced treatment of the 'new economics of organization,' see Moe (1984). For a systematic application of the transactions cost approach to explaining structures of public bureaucracies, see Horn (1995). More generally, for an application of principal-agent theory to the subject of state reform, see Przeworski (1999). 8 The logic of principal-agent relations can be extended beyond a particular organizational boundary to include an analysis of the relationship between a bureaucratic organization (as represented by its head) and government (i.e., minister) or politicians (i.e., president, legislature). Examples include Weingast and Moran (1983), McCubbins and Schwartz (1984), and McCubbins, Noll, and Weingast (1987, 1989). 9 See, for example, Klitgaard (1997). 4 While persuasive, there are some limitations to this model. First, the economic theories tend to focus on formal institutions (e.g., official wages, as opposed to career possibilities, as an incentive; formal auditing, as opposed to social/peer pressure, as a monitoring mechanism).'" Second, crude economic theories may draw our attention disproportionately to the "minimal state" agenda with its accompanying mantra of "reduce bureaucratic discretion". Third, the whole rationale of the principal-agent theory breaks down if we accept that part of the bureaucracy's functions is to protect long-term public interests by not being fully responsive to particular short-term concerns of the government of the day. 1 Finally, unlike profit-seeking private firms that presumably operate with a clear ownership pattern and an unambiguous line of accountability, public bureaucracies usually operate under supervision from multiple principals.12 In sum, the principal-agent perspective emphasizes: 1. Monetary incentives; 2. Mechanisms for monitoring contracts; and 3. Clarity of agency mission. Critics complain that such economic theories offer relatively little in the way of explanations for non-opportunistic behavior in the public bureaucracy. 3 Such seemingly selfless actions are perplexing within the principal-agent perspective. However, the perspective is also limited by its focus on performance that can be observed and measured. Some other tasks require such a high level of discretion with virtually no possibility of effective monitoring that the principal cannot rely on economic incentives. Other non-monetary incentives, such as identification with the agency's mission or with the agency itself as well as esprit de corps, must be considered. The principled-agent perspective purports to offer more empirically accurate account of organizational dynamics and bureaucratic behavior, but at the same time its analytical power is somewhat more limited than that of the principal-agent theories. Leadership is seen to be key'4, as is "a strong sense of mission, effective managerial practice, and high expectations about employee performance".,5 The perspective builds on the sociological traditions that have tended to see organization as an organic entity and have emphasized '° See Heyman (1988) and Miller (1994). 3' A large-scale survey-based study of senior bureaucrats and politicians in five western countries found that "policymaking by bureaucrats is characterized by continuity, stability, and predictability" in clear distinction to the views of politicians who see themselves as "advocates, partisans, and tribunes" (Aberbach, Putnam, and Rockman, 1981, p. 25 6-7). 12 See Moe (1984), Moe (1990), Simon (1991), and Fukuyama (1995). 1 3 They are "far better at explaining why bureaucrats shirk, subvert, or steal than they are at explaining why bureaucrats behave as 'principled agents' - workers who do not shirk, subvert, or steal on the job even when the pecuniary and other tangible incentives to refrain from these behaviors are weak or nonexistent" (Dilulio, 1994, p. 277). Herbert Simon also provides a critique of the principal-agent variant labeled transactions cost analysis (Simon 1991). 14 Dilulio (1994). Other influential organization theorists including Selznick (1957) and Wilson (1989) have also underscored critical roles of leadership in accounting for organizational performance. IS See Grindle (1997, p.491). 5 culture, leadership, and styles of authority and power'6, and that emphasize the limitations of formal contracts. 17 There is a sizeable literature that sees officials' identification with organizational objectives as key to their "principled" behaviors. Another source of motivation is public officials' long-term career paths. When bureaucratic career paths are fairly well defined and predictable, they induce a particular range of behaviors from public officials who are interested in improving their career prospects. Recruitment and promotion processes work as socialization mechanisms that selectively preserve particular types of individuals.'8 One key to performance within the principled-agent perspective is that career paths are long term, allowing reputational rewards to be developed.'9 Thus the principled-agent perspective emphasizes: 1. Clarity of agency mission; 2. Organizational culture and trust; and 3. Long-term career paths. Some political scientists see public sector organizations, not exclusively in developing countries, as webs of patron-client networks.20 This perspective observes the patron- client relationship as based more on trust and loyalty than short-term, transaction-specific utility-maximizing calculation and bargaining by both parties. Unlike a typical principal-agent relationship marked by incentive incompatibility between the two actors, a patron-client relationship is based on the recognition of mutual (albeit unequal) benefits that the two actors derive from the relationship. Furthermore, to the extent that incentive incompatibility threatens the possibility of mutual gains, the personal and long-term nature of the relationship between the patron and the client reduces the monitoring costs of the client's behavior, and thus mnitigates the agency problems. The client is bound by the sense of loyalty and obligation as well as motivated by the expectation of present and future benefits from the continued participation in the relationship. 16 See Weber (1946), Selznick (1957), and Simon (1974). 17 An interesting point is that Williamson's (1975) seminal study of the economics of organization finds the superiority of internal organizations (hierarchies) over markets because of the former's greater ability to control opportunism. 18 Aberbach, Putnam, and Rockman (1981) provide an example of this impact on a range of attitudes and behaviors observed among bureaucrats at senior levels, and Schneider (1993) provides an example of the reverse. 19 OECD (1 994a) also notes that in some countries "the official doctrine is that public servants receive a means of subsistence to enable them to serve the state". The strongest expression of this particular incentive logic is to be found in the German pay policy for established civil servants, Beamte. For this group, traditionally, pay "is not regarded as remuneration for work performed or for performance of function, but as a means of livelihood linked to office. (The remuneration) is intended to enable Beamte, to give full commitment to duty of service to state and to live at appropriate standard for service rank and (to) be independent". 20 See Dresang (1974), and Schneider (1991). 6 This perspective assumes that it is more than likely that the loyalty to the patron will overwhelm the client's identification with the organizational objectives. This militates against effective organizational performance if the patron utilizes employment opportunities in the public sector as a means to reward political supporters. This perspective also requires that career paths are long term, although in this case the reason is that the currency of jobs in which the patron rewards the client holds little value if jobs can be withdrawn at any moment. Patron-client relations require extensive informality. Comprehensive, formal rules with clear policy objectives and restraints on the ability of the principal to micro-manage would remove the ability of the patron to reap the rewards from having placed their clients in key positions. The patron-client perspective emphasizes: 1. Long term career paths; and 2. Informality. 2.2 How can institutional environment be measured? These perspectives emphasize different sets of arrangements and consequently they direct attention to different potential levers for improving performance. Whatever perspective is adopted on the motivation of public officials, three particular sets of concerns stand out as fundamnental - the nature of the rules that constrain the behavior of public officials, the nature of the policies that they are asked to implement, and the manner in which they are provided with resources. As shown in Figure 2, these are always the ingredients in the incentive mix, whatever the assumed recipe.21 Within these dimensions, the institutional environment shapes expectations of future constraints and incentives. An official, who has come to expect that rules will not be enforced, works in an environment of low rule credibility. His or her behavior is shaped by the expectation that the rules for personnel and budget management will not be enforced in the future. 22 The three dimensions of the institutional environment - rule credibility; policy credibility; and resource adequacy and predictability - represent measures of the expectations of public officials concerning the future. 21 See Manning and James (2000) for a discussion of credible regulation within the public sector. See Evans and Manning (2000) for a discussion of policy credibility. 22 This point that institutional arrangements impact individuals' actions in the present by shaping their expectations about the future is made extensively in the institutional literature. See, for example, Bendor and Mooklierjee (1987) and Mnookin and Kornhauser (1989). It is consistent with an assertion that the problem facing both public and private sector managers is one of maintaining their collective reputation as a relevant actor among their staff (See Seabriglt 1993). More generally, the literature on cooperation in the absence of third-party enforcement emphasizes that "the shadow of the future" (i.e., the degree to which actors expect to interact again under similar circumstances) significantly determines behavior in the present (See Axelrod and Keohane, 1985). 7 Figure 2. Credible rules, credible policy and predictable resourcing constitute the institutional environment within any perspective expectations of expectations of policy expectations of rule enforcement future resource flows principal-~~ ~ moetr emh-epa agency~ ~4 siedsiedsie 44e5 o 04'~ ~~~~ -~~~~~ C6~~~~~~~~~~~~~~~~~~~~~~~~C principal- monetary empha- empha- agent incentives sized sized perspective contract empha- empha- enforcemenst sized sized claot tof empha- empha- empha- tyagen sized sized sized principled- mission ___ agent culture and empha- empha- empha- empha- emnpha- empha- perspective MUtual trUSt sized sized sized sized sized sized long term empha- empha- exnpha- patron- arrsm sized sizedpri client informality opposed opposed opposed perspectivec _ _ _ _ _ _ __ _ _ _ _ _ _ _ _ Against this context, the study has adopted a framnework for measuing the institutional environment built around measurement of these three factors that constitute the institutional environment. Accordingly, indicators are constructed to measure how much rule credibility, policy credibility and resource adequacy and predictability the institutional arrangements are providing. 2.3 How can perfiormance be measured? In addition to the well-recognized difficulties in assessing organizational performance, quantifying performnance of public organizations runs into two main problems. 23 First, unlike private organizations, public organizations have no single performance indicator that can compare across different types of organization and products, such as profits earned or market share. In the public sector, only a few organizations work for profit. 23 Acs and Gerlowski (1996) list the classic difficulties including measurement errors, random effects, and factors outside the control of the organization. Carter, Klein, and Day (1995) lists ownership, trading/non-trading, competition, accountability, heterogeneity, complexity, and uncertainty as the organizational dimensions and argue that these dimensions show huge variation among public organizations. They also point out the difficulty in measuring the quality of service delivery. 8 Outputs of many organizations such as the audit body or the planning division of the Ministry of Finance are used only by other organizations within the public sector. Second, public sector organizations are often dealing in goods of both low contestability and measurability.24 In such circumstances it is generally impossible to find performance measures of the public sector that satisfy the ideal qualities of consistency, comparability, clarity, controllability, comprehensiveness, boundedness, relevance, and feasibility.25 For instance, when public agencies' perfornance is measured, the metrics are distinctly organization or service-specific. The waiting time for a patient to see a doctor is not readily compared to performance in primary school enrolment. The current extensive debate on performance in the public sector provides illustrations of benchmarks that can be applied over time, but has done little to solve the problem of comparability between diverse agencies and sectors.26 Nevertheless, performance needs to be measured. The analytical framework described in the following pages introduces a new approach to measure performance of public organizations. In particular, it suggests that there are some common assertions that lie behind currently used diverse measures of performance. In particular it identifies three dimensions that are key to any conceptualization of performance: how focused officials have been on results, their accountability for adherence to formal rules, and their morale.27 Results focus is prima facie evidence that public officials are striving to achieve organizational goals and clearly lies behind consideration of organizational efficiency and effectiveness. It is a reasonable, although empirically testable, proposition that rewarding good performance and punishing bad performance encourages a results focus. Accountability is perforrnance in the distinctive sense of having adhered to the formal rules and so enabling actual behavior to be tested against mandated standards. For public sector organizations, which work with public funds, accountability is an important element of performance. Corruption is a symptom of poor accountability performance - but the notion of accountability adopted for this study is wider than corruption.28 Accountability in this sense rests on the past enforcement of regulations. This measure is of course different from an assessment of the existence of credible and appropriate rules and regulations, which is an aspect of the institutional environment. This notion of accountability includes transparency in decision-making and allocation of government resources. This refers, for example, not only to the audit of public accounts and making 24 See Girishankar (1999). 25 See Shand (1997). 26 See for example OECD (1994b), Public Service and Merit Protection Commission, Australia (1998), Gore (1993) and Osborne and Plastrik (1997). 27 A study by the authors of this paper is in progress to empirically examine the link between these performance measures and conventional output and outcome indicators, and its findings will be available in 2000. 2S See World Bank (1998). 9 the audit report available to legislature, but also to a history of regular consultations with the private sector, citizens' groups and NGOs. Finally, employee morale is also identified as a component of organizational performance. This is both a performance goal in its own right, and a contributor to organizational effectiveness and efficiency as identified in Australian and Canadian studies of public employees of the 1980s.29 Employee morale is raised by job satisfaction and reduced by the orientational and attitudinal disconnect that managers have with their staff. 30 This latter concept of a disconnect between the managers of organizations and other officials working in them has been usefuilly termed 'vertical solitude'. 31 It is the orientational and attitudinal disconnect that managers have with officials below them. As managers rise in the hierarchy, two factors cause them to become "impersonal." First, they become increasingly aware that they are no longer managing front-line staff, but other people who think like them. Therefore, their thought patterns and worldview increasingly diverge from that of their staff Second, as they rise, the units they manage become larger until it becomes impossible to personally know everyone who works for them.32 As a result, communication between the managers and general officials in an organization deteriorates. The general officials are in more regular contact with the public and client groups, but do not clearly understand the expectations of their ministers and manager who set the tone and direction in their departments. 33 It should be emphasized that these are intermediate measures of performance. Given the heterogeneity of organizational outputs it is only at this intermediate level that measures can be found that contribute to how effectively those heterogeneous outputs are produced in different organizations, but which are homogenous enough to permit measurement and comparability across disparate public agencies. The key feature of these measures of performance is that they refer to perceptions of past events. While the institutional environment shapes expectations of the future, performance refers to past behavior. The conceptual framework described above proposes that performance is a function of the institutional environment. To test this, indicators were constructed that allow measurement of the degree of results focus, accountability and employee morale. It thus becomes a testable proposition that performance improves as the institutional environment improves." 29 See Jans and Frazer-Jans (1989) and Zussman and Jabes (1989). 30 See also Schein (1996). 31 See Zussman and Jabes (1989). 32 Schein (1996). 33 Zussman, and Jabes (1989). 34 Leadership as a determinant of organizational performance has been considered in a 1999 study of US Federal Executives (PricewaterhouseCoopers 2000) and a 1987 study of the Canadian federal public service (Zussman and Jabes 1989). 10 2.4 Whose performance? - different units of analysis In addressing the key question about whose performance we are interested in, this analysis slices the public sector in three related ways. First, it examines some performance questions at the whole of government level. Second, it examines what needs to be done to get a particular group of officials to perform more effectively. Third, it asks what can be done to improve the performance of some particular agencies or organizations within the public sector. Disaggregating the public sector by type of official and by type of agency is a pragmatic judgment, and ultimately in this and the country reports the determination was made largely on the basis of custom and practice - how comrnentators refer to units or groups within government in this setting. However, it is worth exploring briefly the challenges that lie behind this simple classification exercise. Groups of officials The terms 'public sector employees' and 'civil service' are often used to describe anyone who is paid by government. However, this is a very heterogeneous group and in reality comprises: an inner core of administrators; members of the armed forces; officials employed by productive enterprises that are majority-owned, directly or indirectly, by the state; health workers of any level of government; primary, secondary, or university teachers paid by government; employees of sub-national governments, etc. Figure 3 The main components of government employment35 TOTAL PUBLIC S E CITO R rtat.okownd ote GEERm ALee ete eoltet te|rts Fo uther, detil se Cross-N1,AtinlDt nGLrnn mlomn n ae,o h ol Bank Administrativoe an Civilto Se|rvc Reforom websitel at !tolI htot-/wl b--rltban_{ rglpb elicetrtstcvlevbcosh Se als Stvn (1994).| ~117Y'111111 VERNM1N em ploy-es ~ ~ ~ ~ ~ t P. .- r _. -- p -Ml 35 For ffirther details see Cross-National Data on Governument Employment and Wages, on the World Bank Administrative and Civil Service Reformi web site at http://wwwl.worldbank.org/publicsector/test/civilservice/cross.htm. See also Stevens (1994). 11 The surveys seek a country-specific approach to distinguishing between groups of officials. In the case of Bolivia, for example, officials from central administrations, decentralized institutions, regional administrations were surveyed and the analysis distinguished between consultants and other public officials. In Bangladesh, the analysis distinguished between officials of ministries, autonomous corporations and district-level officials. In the East Caribbean states, perceptions of managers, middle-level officials and general officials were differentiated. Types of agency The characteristics of the goods that the agency produces, or the features of its tasks, are likely to have important effects on the way public officials behave. Some tasks naturally involve a greater level of discretion by front-line officials, while others are more amenable to strict standard operating procedures. Wilson (1989) has developed a typology of agencies according to varying characteristics along two dimensions: whether agency outputs are observable or not, and whether agency outcomes are observable or not. Agency outputs may not be observable simply because the agency's bureaucratic operators conduct their duties out of view of the manager (e.g., police officers on the beat, teachers in classrooms). They may also be "unobservable" because what the operators do is esoteric (e.g., a professor developing a hypothesis in a shower). In more formal terms, four broad categories of agency can be identified. Ministries are in a legal or constitutional sense indistinguishable from the state as they often have no specific primary legislative basis and their assets are the general property of the state. Their functions and objectives are multi-purpose complex tasks and traditionally defined by legislation (Continental European traditions) or determined incrementally by Cabinet (UK and other Anglophone traditions). Their source of funding is almost entirely the state budget and they usually have nil or very minor revenue earnings. As they have no corporate or legal identity separate from the state, they have generally no legal competence to enter into corporate contractual relationships with suppliers. The term autonomous agencies can be quite confusing because it may refer to two very different things: "executing agencies" and "statutory commissions and independent regulators". Executing agencies reporting to Ministries (or reporting to Ministers but under day to day supervision of a Ministry) are often in the same legal or constitutional sense indistinguishable from the state. However, the relevant Minister generally defines their objectives, perhaps in framework agreement, and their source of funding can include some revenue earnings. They rarely have legal competence to enter into contractual relationships with suppliers. 12 Some agencies are statutory commissions in that they do have a separate legislative existence. They can be non-asset owning (legally distinct but not able to own assets in any sense) or asset owning (legally distinct and the owner of its assets). Their objectives are often enshrined in a charter and their functions are defined by the legislation, including the Minister's powers of direction. They can often be budget dependent (subvented) with significant revenue earnings. They can have legal competence to enter into contractual relationships. Such bodies tend to be established for regulatory purposes, although in some presidential systems they can also be service providing. Trading bodies also have a separate legislative existence, and can be established under commercial law in which case their charter and their articles of association define functions. They can be budget dependent, subsidized, but with significant revenue earnings. They generally have full legal competence to enter into contractual relationships - they can assume corporate liability and can be sued. In the present series of BNPP surveys, those in Bangladesh, Bolivia, East Caribbean and Guyana covered officials from the first three types of organizations. 13 3. Survey methodology The surveys were designed to elicit the perceptions of public sector officials. All employees employed in the organizations being surveyed were eligible and no control group was included. As no 'before and after' information was sought, a cross sectional design was selected. 3.1 Questionnaire design36 In drafting and pre-testing the questionnaires, care was taken to avoid: ambiguous words and phrases; questions asking two or more opinions simultaneously; words or phrases that could be expected to trigger off emotional responses; and information that could manipulate the respondent.37 Most pre-testing exercises pointed to excessive length of the questionnaires. The rule of thumb applied was that it would be unlikely that a public official could afford to spend longer than 45 minutes with an interviewer even assuming that the surveyor could hold the official's attention for longer. Although the responses had to be converted to numbers for analysis, verbal response scales were selected over numeric scales in most questionnaires. This happened because during pre-testing, responses on a numerical scale tended to show responses clustered close to the middle of the scale.38 The advantage of using descriptive words for each category of response was that each category could be relatively well defined. The number of categories was kept low, to avoid loss of meaning in translation. The middle category was excluded to force respondents to express preferences even when they are indifferent. Pre-testing showed that including a middle category biased responses towards the middle category even when respondents had a slight preference one way or other. 'Don't know' as a response category did not appear in any survey questionnaire except in Indonesia. Thus the number of respondents expressing an opinion was biased upwards. Filters and branches were used. Although filter questions added to the length of the questionnaire, they were used to ask whether or not something was a problem before asking the degree of the problem. Most questionnaires began with 'easy' questions like how long the respondent had been working with government or in the same position, or details about the personnel management system in place in the respondent's organization. Questions on the same theme were grouped together. Listing all categories in the same order minimized any existing order effect within questions. In most questions categories were listed from low to high, and from bad to good. Most surveys asked about corruption and other sensitive questions such as what an official did when he or she did not agree with a supervisor's decision. Recognizing that respondents could be unwilling to admit incriminating behavior in face-to-face 36 The survey instruments are available on http://wwwl .worldbank.org/publicsector/test/civilservice/survey.htm. 3' / See Rea and Parker (1997). 38 See Recantini, Wallsten, and Xu (2000). 14 interviews, the first few surveys introduced a self-administered component (containing sensitive questions) besides the main in-person interview. However, this method ran into logistical problems at the data entry stage. In the later surveys, sensitive questions were asked towards the end of the interview by which time greater rapport would have developed between the surveyor and respondent. Even if this made the respondent suspicious in any way or unwilling to answer further questions, it did not bias the whole interview. Some less direct questions were asked, such as "How serious is corruption in your organization?" instead of the more direct "What percent of officials in your organization take bribes?" Interestingly, in all surveys analyzed so far, officials admitted that their. own organization was corrupt, but less corrupt than the rest of government. Figure 4. Officials say that the whole government is much more corrupt than their own organization. 100'/ 80V. 60%! 94% 98% 20%' 40 444% 0°/.- ___ _____ my organiz?.ion government my or anization government Armenia Bolivia 3.2 Sampling methodology Probability sampling was the method employed in Bangladesh, Bolivia and Guyana. Stratified random sampling was utilized with the strata being the types of organizations described in Section 2.4.3 In East Caribbean countries, quota sampling was the method chosen because the number of employees at each level was known accurately from the 1999 budget estimates available for these countries.40 Focus groups were mostly utilized to pretest draft questionnaires. 39 However, in Bolivia, the strata employed were not the types of organizations but the hierarchical levels in each organization. The number of different types of organizations chosen was "purposive" to maintain some representativeness, and to ensure that certain agencies were included. 40 Fink (1995). 15 Sample sizes in different countries had to be determined within the constraints of cost, time and the number of officials willing and available for interviews. Samples varied in size from 62 in St. Kitts and Nevis (population 41,000; size of public sector 3,000) to 823 in Bangladesh (population 126 million; size of public sector more than 100,000). In determining sample size, the number of subgroups to be included, and a 0.05 level of significance of the null hypothesis for the mean of the population, were both kept in mind. The sampling unit was the individual official, while the unit of analysis was an organization. In the case of East Caribbean countries, the unit of analysis was the whole public sector. The response rate in the data sets received thus far was between 80% and 100%. There was some item non-response in several countries. For example, item non-responses were 50% in Bangladesh when officials were asked what they did when they disagreed with the opinions of their supervisors. 16 4. Construction of Indicators for Institutional Environment The survey findings provide an opportunity to enrich the factual basis of national debates on public sector reform, and to support or refute prior assertions and hypotheses in the light of the available data. Some other uses of the data require that indicators of the institutional environment and performance be constructed. As explained in Section 2 outlining the conceptual framework, the institutional environment was measured by considering officials' expectations of the incentives and constraints that would apply in the future in terms of rule credibility, policy credibility and resource adequacy and predictability. Indicators were constructed with rule credibility, policy credibility and resource adequacy and predictability measured in terms of these indicators on scales that ranged from 0 to 10, with 0 being the worst and 10 being the best. Questions addressed both personnel and expenditure management. The figure below shows how responses to questions were grouped in constructing the institutional environment indicators in East Caribbean countries.4" Figure 5. Construction of indicators for institutional environment in East Caribbean states institutional environment rule credibility policy credibility resource adequacy & predictability in record policy seasonal - management - consistency _ absences in internal policy necessary _ audit - coordination l skills in performance political institutional _ appraisal (of micro- _ capacity individual officials) management in project _ evaluation 41 In the East Caribbean states, public officials were surveyed in Antigua & Barbuda, Grenadines, St. Kitts & Nevis, St. Lucia and St. Vincent & Grenadines. The same questionnaire was administered in all these countries. In all settings, the rules for allocating questions were: Does this question capture expectations of current or future constraints and incentives? If not, then it is not suitable for inclusion in an indicator of the institutional enviromnent. If it does, then does the question capture expectations of: (i) rule enforcement in relation to budget or personnel management; (ii) binding policy direction; or (iii) adequacy and predictability of financial and personnel resources? 17 Relevant and related questions were grouped under the same category as shown in the above figure. Since the responses were all on a verbal scale, they were converted to the same numerical scale. For uniformity, qualitative responses from all questions were converted into numbers on the scale of 0 to 10. For example, yes/no questions were converted into "O" and "10". For questions with four qualitative responses such as always (1), generally (2), occasionally (3), almost never or never (4), the following formula was used for the conversion:42 Scaled response = 40/3 - (Un-scaled response)* 10/3 After the conversion, the following steps were taken to calculate rule credibility, policy credibility, resource adequacy and predictability, and eventually the institutional 43 environment indicators. First, the simple arithmetic average of all responses regarding the same specific aspect of the institutional environment (i.e. average of all questions in the same box in Figure 5 above) were calculated. Second, after converting responses into numbers on a 0-10 scale measures of policy consistency, policy coordination, and political micro-management were derived by taking the simple average of the aspects of the institutional environment under the same component. Finally, the institutional environment indicator for the whole public sector was derived as a simple arithmetic average of the three components of the institutional environment. In more detail, in the East Caribbean example shown in Figure 5, rule credibility was measured by creating an indicator covering the existence and perceived quality of formal rules in four areas: record management, internal audit, performance appraisal, and project evaluation.' Six questions from the survey were utilized. Questions used in that and other surveys to probe for the dimensions of the institutional environment are set out in Appendix 1. Appendix 2 shows how the environment indicators were constructed for the East Caribbean states. 42 For consistency, always the positive, e.g., useful, helpful, working, responses were converted to "10," and negative, e.g., not useful, not helpful, not-working, responses were converted into "O". For example, in the question "Are job openings advertised?" "Yes" was converted into "10," and "No" was converted into "O." Similarly, in the question "How consistent are the various policies your organization?' "Very consistent" was converted into "10"; "more consistent than consistent" into "6.7"; "more inconsistent than consistent" into "3.3"; and "very inconsistent" into "O". 43 Aggregating data for indicators entails a delicate tradeoff. All data sets on public official perceptions are unreliable individually to some degree - and by aggregating several subjective measures from various sources, we can reduce this measurement error somewhat. However, the gain in reliability from aggregation comes at the expense of a loss in conceptual precision. In other words, the tradeoff is between emphasizing individual data sets that may have inaccuracies, and aggregating them to gain accuracy but with the risk that they become more blunt. This paper seeks to balance the risks involved by aggregating the data within indicators at this point, but subsequently to use the disaggregated data, albeit cautiously, in considering potential reform payoffs (see section 7.4) The risks of aggregation are in any case less severe in this approach as it entails using data from different questions but within the same survey. 44 Later surveys are emphasizing the degree to which the rules are structured to require contestability. 18 Policy credibility was measured by creating an indicator covering three dimensions: whether policies were consistent; whether they are coordinated between the units of government e.g. ministries; and whether political interference/micro- management was felt likely. Responses from six questions were used to calculate the policy credibility indicator as described in Appendix 2. Resource adequacy and predictability was measured by creating an indicator compounded from six questions about: unpredictable seasonal absences of personnel due to severe weather conditions e.g. hurricane; anticipated supply of necessary skills; and about more general anticipated capacity. Appendix 2 provides details. Using a similar method, an overall institutional environment indicator was also constructed on a scale of 0 to 10 (O = worst, 10 = best) by taking the simple arithmetic mean of the rule credibility, policy credibility and resource adequacy and predictability indicators. In this way, as Appendix 2 illustrates, an overall institutional environment indicator was calculated for the entire public sector of each of the East Caribbean states. Using this approach, it is possible to measure rule credibility, policy credibility or resource adequacy and predictability of a particular organization (e.g. Finance Ministry) or by types of organizations (e.g. regional administrations) when sufficient data is available for each type of organization. This was possible in the Argentina, Bolivia, Guyana and Indonesia data sets, and such calculations were made. 19 5. Construction of Indicators for Performance Performance was measured in terms of results focus, accountability and employee morale by asking public officials questions that probed their perceptions of these dimensions of performance. To measure how much results focus, accountability and employee morale, relevant indicators were constructed with scales which ranged from 0 to 10 - with 0 being the worst and 10 being the best.45 As with the institutional environment indicators, the performance indicators drew on considerations of both personnel management and expenditure management. Although combined within the indicator, they permit disaggregated analyses to be undertaken subsequently. Figure 6. Construction of indicators for performance in East Caribbean states |performance| |results-focus aocouritability epoe geared to enforceability satisfaction organization's a i regulations t objectives up-to-date training informiation w cseffetie in record an pnsmtct management r evaluation .n_indictor of eformance in internal in performaneo audit __ appraisal i aernt-based o o delega ation |reward and|ll |punishment|ll .|to tho public In the East Caribbean, the degree of results-focus was measured by creating an indicator cpvering: whether an organizanjob'satisatio.es were geared to its objectives; whether the organization was considered efficient; and whether a merit-based reward and punishment system was in place. Seven questions were utilized. Questions used in that and other 45 In all settings, the rules for allocating questions were: Does this question capture perceptions of past performance? If not, then it is not suitable for inclusion in an indicator of performance. If it does, then does the question capture perceptions of: (i) the degree to which a concern for results or outputs shaped behavior; (ii) rule-based behavior in relation to budget or personnel management; (iii) morale and job satisfaction. In addition, a measure of vertical solitude was constructed from the differences in perceptions concerning the environment between managers and their staff? 20 surveys to probe for the dimensions of performance are set out in Appendix 3. Appendix 4 shows how the performance indicators were constructed for the East Caribbean states. Accountability was measured by using 15 questions to create an indicator covering, inter alia, enforcement of regulations; demonstrated accountability to the public at large, and to civil society and parliament. In assessing the institutional environment, the existence of credible rules was tested in four areas: record management, project evaluation, internal audit and performance appraisal. In measuring performance, the past enforcement of these rules was tested in the same four areas. Appendix 4 shows how the performance indicators, including that for accountability,. were constructed for the East Caribbean states. Employee morale was measured by creating an indicator covering employee satisfaction and vertical solitude. Four questions were used to calculate satisfaction indicator as shown in Appendix 4. Vertical solitude was defined in Section 2.3 above. The indicator for vertical solitude was calculated by first scoring managers' perception of each aspect of the institutional environment and then the general officials' perceptions of the same. Then, the (absolute value) difference of these scores in each aspect was averaged to yield an indicator of vertical solitude.46 The overall performance indicator was calculated by taking the simple arithmetic mean of the results-focus, accountability, and employee morale. 46 This formula was adopted from suggestions by Jeff Rinne of PRMPS. 21 6. The Connection between Performance and Institutional Environment To describe and quantify the relationship between performance and institutional environment, a simple model was used. Aggregate performance and its three components, i.e., results focus, accountability for formal rules, and employee morale were regressed on institutional environment. Accordingly, performance, Yi, was modeled as a function of institutional environment (EO), and an error term ei47: Y= a + b Ei + ei On current assumptions, for the Bolivian public sector there is a statistically significant positive relationship between institutional environment and performance indicators. The estimation result for Bolivia was the following:48 Aggregate Performance = 0.47 + 0.74 (Institutional Environment) t-value = 5.54 R2 = 0.70 Number of observations = 15 Figure 7 is the result of cross-tabulation of institutional environment and aggregate performance indicators for 15 different public organizations in Bolivia. It shows the relationship between performance and institutional environment. Similarly, the effect of institutional environment on each performance area was tested. Strong relations existed between the institutional environment and each area of performance. The associated charts are provided in Appendix 5. 47 An error term was attached to capture measurement errors, random variations, as well as other determinants that have been omitted from the equation. 48 It is important to note that the aggregation of questions to construct the indicators is a question of judgment, although simple and explicit rules are used to minimize variations. These results may change as the country papers are prepared if the grouping of questions is altered. 22 Thus there is empirical evidence that: * Institutions do indeed matter;49 and * Some performance areas are more susceptible to the institutional environment than others ." Figure 7. Performance and Institutional Environment in sampled Bolivian public sector organizations PERFORMANCE: INSTITUTIONAL ENVIRONMENT 8 ----- - ----__ __ __ __ __ y 0.7429x + 0.4709 R= 0.7026 * audit banking-regulati n 7 / + central bank 06 A 4 tax + ine t evteleom-regulation agrarian refotennstiution Santa Cru2 * components5ofperformance-whic agrepresentednAppendiculture Te e t customs labor road 4 . 5 ~~~~~6 7 8 9 institutional environment 49 This assertion was based on differences in the estimated coefficients, i.e., slope, and in goodness- of-fit measures, i. e. R2, for the relationships between institutional enviromment and three different components of performance-which are presented in Appendix 5. 50 This same relationship is being tested using data sets from the public sectors of Armenia, Bolivia, East Caribbean states, and Guyana. Results of these tests will provide more empirical evidence. 23 7. Presenting Opportunities for Reforms To offer some options for consideration by policy-makers, four steps were taken as described below. The first step emphasizes the importance of presenting policy-makers with robust confirmation of the theory-based assertion that it is the institutional environment in general, and rule credibility, policy credibility, and resource adequacy and predictability in particular, which drive performance. Having made this point and having attracted attention to the question of how and when that environment can be changed, the second step is to maximize access to the data to encourage policy-makers and advisers to contest possible interpretations and to suggest locally appropriate reform interventions. The contextual description of the public sector, including characteristics of respondents, their reasons for joining the public sector and the length of time worked in govemment, enrich the factual basis of these discussions. The third step is to examine any prior assertions and topical reforms in light of the available data. In testing country-specific hypotheses, myths that have been consolidated through repetition rather than observation can be supported with evidence or refuted. Finally, where feasible, as an encouragement to further debate, a particular heuristic approach is employed to move beyond theory into an empirical investigation of which institutions matter in particular and where might the largest "performance pay offs" be found. 7.1 Step 1 - it's the institutions not the people In all settings where the surveys have been undertaken, a summary country report sets out the evidence that supports the theory-based assertion that it is the institutional environment in general, and rule credibility, policy credibility, and resource adequacy and predictability in particular, which drive performance. In making this point, the intention is to deter simplistic denigration of government or public officials. Although there is every reason to be concerned about poor performance and rent-seeking by some public officials, there are as many reasons to be impressed by the selflessness and determination of others working under conditions of extreme difficulty. Setting out the evidence offers an approach for understanding both good and bad performance and for presenting the results to policy makers in a format that lead to more informed choices about reform interventions. 24 Figure 8. Rule complexity affects performance in Albania 10 9 Wocal-budget con rl hr * central inst-budget 8 personnel cZ7 ginlbdet ional- * local-personnel 4; 6 *-persucntral inst- i 4 -judca- 3 - personnel 2- 1 0 0 1 2 3 4 5 complexity of rules In Figure 8 the horizontal axis shows the perception of officials of the complexity of budget management and personnel management rules in their respective administration (local governments, central institutions, regional administrations, judicial bodies and the control branch). The vertical axis shows the perception of these same officials on the enforceability of these rules. Both complexity and enforceability of rules have been ranked on a scale 0 to 10 with 0 being the worst possible, and 10 being the best possible.5 7.2 Step 2 - encourage contestation in interpretation The country reports also provide a summary of the descriptive data, presented in a format that facilitates comparisons between countries. In parallel, appropriately anonymized survey data is being placed on Internet sites. Open discussion fora are being facilitated to encourage policy makers, academics and other commentators to test alternative interpretations of the findings, to contest possible interpretations and to suggest locally appropriate reform interventions. 7.3 Step 3 - challenge prior assumptions The country reports then examine any prior assumptions or topical reforms already under consideration in the light of the available data. These prior assumptions have been derived from country dialogue with government officials and business and civil society representatives. They tend to fall into two categories. Some are assertions of facts (e.g. most civil servants would rather work in the private sector if there were jobs available) 51 Acknowledgments are due to Monica Das Gupta of the World Development Report 2000 who contributed to this figure. 25 and some are assertions of causal relationships (e.g. the way in which they were recruited affects the degree to which officials support government policy). Table 1 offers some examples of the assertions tested. Table 1 Illustrative prior assertions tested against survey evidence country factual assertion assertion of causal relationship Albania there is wide variation in performance is particularly highly understanding of the proposed civil correlated with officials' support service law for government policy Eastern managers feel that they have unpredictable availability of Caribbean inadequate autonomy in budgetary personnel as a result of natural matters disasters is damaging to performance records management systems are performance is undermined by the unreliable lack of incentives linked to the attainment of agency objectives there is intense resistance to donor funds for technical regional initiatives assistance lead to lack of incentives and hence to poor performance Public Accounts Committees are automatic renewal of contracts not considered strong results in poor accountability for performance Bolivia the SAFCO law is not considered political patronage and interference by officials to have improved in day-to-day activities undermines accountability performance The factual assertions are supported or refuted by simple reference to the data. Some assertions of a causal relationship are sufficiently precise to allow simple testing by reference to the data. Other such assertions, particularly those that suggest that performance is in some general sense undermined by some aspect of the environment, require that some indicators of performance be drawn from the conceptual model set out in section 2. 7.4 Step 4 - encourage informed speculation on perfornance pay-offs Where feasible, as an encouragement to further debate, a particular heuristic approach is employed to move beyond theory into an empirical investigation of which institutions matter particularly and where might the largest "performance pay offs" be found. The intention of this final step is to move beyond the general finding that institutions matter for performance, and to open up consideration of which institutions matter particularly. In making this step, three key points should be borne in mind. First, given potential problems of data quality, there is a risk of drawing over-elaborate conclusions that have a veneer of statistical robustness but are unlikely to be replicable. Second, at the general 26 level, interpretation of the survey data is strongly guided by theory. As section 2 indicates, theory gives us every reason to believe that institutions affect performance and that rule credibility, policy credibility, and resource adequacy and predictability are strongly implicated. However, theory provides no guidance on which of these institutional arrangements matter most and the findings at this disaggregated level are purely empirical. Finally, the model that has been developed is plausible but only one from very many that could have been constructed. In the country studies that offer detailed results, the model is only used to develop one set of interpretations where dialogue with government and World Bank operational staff suggest that this is useful. The key stages in making an informed speculation on performance pay-offs are as follows: Stage 1 - Develop an appropriate model Model I Performance = f (Institutional Environment). This model has already been estimated in the previous section. It showed that predicted correlations are high when simple averages of the elements of institutional environment is regressed against aggregate performance as well as its three main components. However, this information by itself is not enough for decision-makers in governments. They already know that performance is sometimes low because of weaknesses in the institutional environment; and if they could improve the whole institutional environment, the overall performance would improve. They need more precision. The logical next step is to move from Model I towards one in which the institutional environment is disaggregated into its components. Model II Performance = g (Rule Credibility, Policy Credibility, Resource Adequacy and Predictability). Estimating Model II will also not give enough detailed information. Decision-makers in government will agree that improving policy credibility can improve performnance, but they will still ask what specifically they need to do to improve performance. Accordingly, a third model is required to capture the relationship between performance and particular elements of the institutional environment. This third model needs to be disaggregated enough to provide input for reform interventions. Improving the institutional environment as a method for improving performance is a practical strategy only if we can assess which particular elements of the institutional environment really matter. In other words, decision-makers are looking for reliable answers on the following questions: 27 * Which reform efforts are most likely to succeed? * Will the same intervention be equally effective across the whole public sector or even within the same sorts of organizations? To illustrate by returning to the analysis of the Bolivian public sector, in that context 11 components of the institutional environment were identified: six components captured rule credibility; four referred to policy credibility; and one referred to resource adequacy and predictability. Rule credibility in the Bolivian public sector's institutional environment comprises: existence of rules in recruitment, existence of rules in evaluation, existence of rules in training, existence of rules in recording, fair treatrnent, and predictable career path. Policy credibility comprises four specific aspects of the institutional environment, i.e., policy consistency, policies communicated clearly to employees, policies supported by officials, political interference/micro-management. Just one question examined the predictability of resources.52 Accordingly, for the Bolivian public sector, a third model was written where performance is a function of eleven different components of the institutional environment. Model III Performance = h (Existence rules in recruitment, Existence of rules in evaluation, Existence of rules in training, Existence of rules in recording, Fair treatment, Predictable career path, Policy consistency, Policy communication, Policy support, Political interference/micro- management, Resource predictability) To unpack the relationship between institutional environment and performance, or in other words to quantify the relationship between performance and different potential reform efforts, an econometric model was utilized to estimate model I .3 Its estimated coefficients were used to derive appropriate indicators. For example, to find how much perfornance changes upon changing the institutional environment via making policies consistent, the following formula was utilized: Rate of performance improvement A performancelApolicy consistency polkyComillmy performance, 52 In each country where BNPP surveys were administered, survey managers customized the survey instrument to country-specific concerns, stressing some areas over others. As a result, not all surveys provided questions to capture all these aspects of the institutional environment or of performance. 53 In a linear model, the estimated coefficient for a specific aspect of the institutional environment variable is the partial derivative of the performance with respect to that aspect of the institutional environment. 28 where, AperformancelApolicy consistency = P foa , and apolicy consistency performanceo = initial performance level. Stage 2 - Examine institutional rates of return The illustrative equation below shows the percentage change in performance through making policies consistent. In that sense, it is the rate of return on reform effort to make policies consistent. Therefore, it is called the institutional rate of return of policy consistency (or policy consistency IRR): IRpolicycDnsistency A performance /A policy consistency performanceo In general, a comparatively high-value IRR for any reform effort means that the intervention is more likely to have a bigger impact on performance than another with low-value. A comparatively low value would indicate that the effort is less likely to have impact and should not be taken up if considerable results are desired. IRRs can be calculated for each of the eleven reform interventions listed above in the Bolivian examnple. A detailed econometric model specification, the estimation method, regression results, and how the estimated coefficients were used in deriving IRRs for the Bolivian public s.ector (based on this formula which was described above) are all presented in Appendix 6. Since all the input variables are capturing institutional environment, it is theoretically possible that the input variables are correlated. For example, rule credibility could be associated with resource predictability. However, two factors ensure that multicollinearity is not a serious problem. First, as the model is offered as a heuristic device for attracting policy-makers attention to particular points of entry to reform, this is not a particular drawback provided that any correlation is positive. If policy credibility were to deteriorate as resource predictability improved, the resulting combined effect would be hard to interpret. However, in all the survey results analyzed to date the correlation where it exists has been positive. Second, multicollinearity is a sample phenomenon as well as a theoretical one. A good rule of thumb is to find variables that are theoretically relevant (for meaningful interpretation) and that are also statistically non multi-collinear (for meaningful inference). As the correlation matrix in Appendix 6 shows, the correlation coefficients between the explanatory variables in the Bolivia data are low.54 The term "Institutional Rate of Return" has been coined deliberately to echo the more common Internal Rate of Return. Its purpose is to draw attention to the heterogeneous 54 In addition to presenting this matrix, Appendix 6 further explains how survey responses were converted to a binary mode thus lowering correlation among the explanatory variables. 29 results that are likely to be obtained from varying different elements of the institutional environment. It should not however be taken to imply greater accuracy than is actually possible at the present stage of development of this methodology. Given the imprecision of the measure, IRRs are used to rank order expected payoffs by type of intervention or change in institutional arrangements or practices, rather than to predict exact performance improvements. Stage 3 - Identify anticipated performance pay offs As noted, IRRs are intended to draw attention to the varying results that are anticipated from different interventions, but given their probable imprecision it is the general magnitude of the pay off rather than the exact value that is relevant for policy makers. The following table presents the potential reform interventions to improve institutional performance in the public sector of Bolivia. Only those reform interventions where IRRs were statistically significant for one-tailed test are identified. (For details, see Appendix 6.) Reform interventions with statistically significant IRRs are classified according to whether they are likely to impact on one, two or three areas of performance. 30 Table 2 Reform interventions and anticipated performance pay offs in the Bolivian public sector' Reform Intervention Improvement in Performance Number Results Accoun Employee of areas focus tability Morale affected Strengthening the credibility of rules for evaluation 3 V / V Strengthening the credibility of rules for record management 3 / _/ Ensuring that staff support government policy 3 Strengthening the credibility of rules for training 2 / Preventing political interference/micro- management 2 / / Assuring staff that they are treated fairly 2 / V/ Strengthening the credibility of rules for recruitment 1 / Making government policies consistent 1 ~~~~/ Figure 9 demonstrates how these anticipated payoffs accumulate to distinguish between the degrees to which each reform intervention is expected to impact on performance. 55 See footnote 48. 31 Figure 9. Expected impact of reforms on components of performance in Boliviae Reform Interventions affectine three nerformance areas. I M results focus °accountability *employee morale Reform Intervention ensuring staff support for government policies strengthening credibility of niles for record managemnent strengthening credibility of rules for evaluation 0% 10% 20% 30% 40% Percentage increase in present level of performance Reform Interventions affectins two performance areas. assuring staff of fair preventing political I Reform Intervention intervention/nmcro- 15% 12% moanagement stengLhening credibility of rules for uaining 0% 10% 20% 30% Percentage increase in present level of performance Reform Interventions affectine one Derformance area. Reform I Intervention strengthening credibility of rules for recruitment 0% 3% 5% Percentage increase in present level of performance 56 The unit of measurement for the improvement in the performance due to a reform intervention is the percentage change in the present level of performance. For details, see Appendix A6.4. 32 Decision-makers in government having limited resources to invest in public sector reform will want to be in a position to choose interventions that will probably result in better performance. The information presented in Table 2 and Figure 9 allows them to prioritize among possible interventions. Available resources can be concentrated on those interventions that affect all three or at least two areas of performance. Bolivian decision- makers would probably choose to strengthen rules in performance evaluation and record keeping and work to ensure that officials are committed to supporting government policies. This same analysis can be performed for a type of organization (e.g. centralized administrations in Bolivia), a particular organization (e.g. the Ministry of Finance or a particular group of officials such as consultants). Appendix 6 presents the findings for central administrations in Bolivia. 5' For IRRs of all types of organi7ations and groups of officials, see forthcoming Bolivia country report by Yasuhiko Matsuda and the authors of this paper. 33 8. Summary The public sector is a diverse environment in which to work. Although there are some general conclusions that can be drawn, staff perform better or worse in different agencies for highly diverse reasons. The Anna Karenina principle applies: "all well-performing agencies are alike; there are so many preconditions for effective performance that every dysfunctional agency is dysfunctional in its own way"." To perform well, public officials need to be confident about the future - not to the point of smugness, but certainly to the point that they can see the relationship between their efforts and any eventual outcome. The institutional environment within which they are working shapes these expectations. If the rules are not credible, with little prospect of enforcement, if they expect policies to be contradicted or resources to flow unpredictably, then they cannot envision any relationship between their effort and public sector performance. Rationally, there is little point in working purposefully as the results are so uncertain. This paper sets out an analytical framework that was used in designing a series of surveys of the views of public officials concerning their institutional environment and in analyzing the information that was generated. It describes how the survey results help to map the strengths and weaknesses of a public sector, and offers an approach for identifying potential pay-offs from reform interventions. Above all, the framework emphasizes the heterogeneity of incentives and institutional arrangements within the public sector - and the need for policy makers to have information that moves beyond generalities and indicates what is most likely to work and where. In building on the premise that public officials' actions - and hence the performance of their organizations - depend upon the institutional environment in which they find themselves, this framework avoids any simplistic anti-government positions, without attempting a defensive justification for poor performance. Although there is every reason to be concerned about poor performance and rent-seeking by some public officials, there are as many reasons to be impressed by the selflessness and determination of others working under conditions of extreme difficulty. This framework offers an approach for understanding both good and bad performance and for presenting the results to policy makers in a format that lead to more informed choices about reform interventions. The methodology depended on use of indicators to measure both institutional environment and performance, but the measures need to be interpreted with caution. As in the design of any performance indicators, it is important to ask: who wants the indicators, who uses them and why. The reliability and the usefulness of indicators depend on the credibility of the information on which they were built, on the accuracy and comprehensiveness of the data. Finally, indicators are not the ends, they are a means for decision makers to raise questions and highlight issues for further discussion and investigation in the light of local, country-specific knowledge. Indicators are not precise 59 Tolstoy's original words were that "Happy families are all alike; every unhappy family is unhappy in its own way". Gary Reid pointed out the relevance ofthe observation for public sector performance. 34 measures, they do not provide comprehensive coverage of all areas of an individual's or agency's work, and they can not tell the whole story. 35 Selected Bibliography Aberbach, Joel D., Robert D. Putnam, and Bert A. Rockman. 1981. Bureaucrats and Politicians in Western Democracies, Boston, MA.: Harvard University Press. Acs Z. J., and D. A. Gerlowski. 1996. Managerial Economics. New Jersey, Prentice Hall. Australian Public Service and Merit Protection Commission, 1998. State ofthe Service Report 1997-9, Canberra: Public Service Commissioner. Axelrod, Robert, and Keohane, Robert. 1985. "Achieving Cooperation Under Anarchy: Strategies and Institutions". World Politics. 38-1: 226-254. Bendor, Jonathan, and Mookherjee, Dilip. 1987. "Institutional Structure and the Logic of Ongoing Collective Action". The American Political Science Review. 81-1: 129-154. Carter, N, R. Klein, and P. Day (1995) How Organizations Measure Success: The Use of Performance Indicators in Government. London: Routledge. Dilulio, Jr. John D. 1994. "Principled Agents: The Cultural Bases of Behavior in a Federal Government Bureaucracy." Journal of Public Administration Research and Theory, 3: 277-318. Evans Gord and Manning Nick. 2000. "A practical approach to assessing central government policy-making institutions in cabinet government: learning lessons from recent World Bank institutional analyses". Unpublished World Bank paper, (http://wwwl.worldbank.org/publicsector/toolkits.htm) Fink, Arlene. 1995. How to Sample Surveys. The Survey Kit, California: Sage Publications. Girishankar, Navin. 1999. Reforming Institutions for Service Delivery: A frameworkfor Development Assistance with an Application to the Health, Nutrition and Population Portfolio, Policy Research Working Paper No. 2039, Washington D.C., The World Bank. Gore, Al. 1993. Creating a Government that Works better and Costs Less: The Report of the National Performance Review. New York: Plume. Greene, William H. 2000. Econometric Analysis, Fourth Edition, New Jersey: Prentice Hall. 36 Grindle, Merilee S. 1997. (Editor) Getting Good Government: Capacity Building in the Public Sectors of Developing Countries. Harvard Studies in International Development. Boston: Harvard Institute for International Development. Heyman, David B. 1988. "Input Controls in the Public Sector: What Does Economic Theory Offer?" IMF Working Paper, Washington D.C.: International Monetary Fund. Horn, Murray J. 1995. The Political Economy of Public Administration: Institutional Choice in the Public Sector. New York: Cambridge University Press. Jans, Nick and Frazer-Jans Judy. 1989. Careers and Culture in the SES, Canberra: Australian Government Publishing Service. Kaufmann, Daniel, Sanjay Pradhan and Randi S. Ryterman. 1998. New Frontiers in Diagnosing and Combating Corruption. World Bank PREM Note No. 7: Washington, DC. Kaufmann, Daniel, Aart Kraay and Pablo Zoido-Lobaton. 1999. Aggregating Governance Indicators, and, Governance Matters. World Bank Policy Research Working Papers Nos. 2195 & 2196: Washington D.C. www.worldbank.org/wbi/gac. Klitgaard, Robert E. 1997. "Cleaning Up and Invigorating the Civil Service." Public Administration and Development (U.K.) 17: 487 - 509. Knack, Steve and Nick Manning. 2000. Policy attributes. Unpublished World Bank document. McCubbins, Matthew D., and Thomas Schwartz. 1984. "Congressional Oversight Overlooked: Police Patrols versus Fire Alarms." American Journal of Political Science, 28: 165-79. McCubbins, Matthew D., Roger G. Noll, and Barry R. Weingast. 1987. "Administrative Procedures as Instruments of Political Control." Journal of Law, Economics, and Organization, 3: 243-77. . 1989. "Structure and Process, Politics and Policy: Administrative Arrangements and the Political Control of Agencies." Virginia Law Review, 75: 431-482. Manning, Nick and Oliver James. 2000. Regulation within the public sector- draft background paper. Unpublished World Bank document. 37 Manning, Nick, Ranjana Mukherjee, and Omer Gokcekus. March 2000. The experience ofpublic officials in Bolivia: What works in a weak institutional environment? Unpublished World Bank document. Milgrom, Paul, and J. Roberts. 1992. Economics, Organization, Management, New Jersey: Prentice Hall. Miller, Gary J. 1992. Managerial Dilemmas: The Political Economy of Hierarchy. New York: Cambridge University Press. Mnookin and Komhauser. 1988. "Bargaining in the Shadow of the Law: The Case of Divorce," Yale Law Journal 88: 950-972 Moe, Terry M. 1984. "New Economics of Organization." American Journal of Political Science, Vol. 28, No. 4. pp. 739-777. . 1990. "The Politics of Structural Choice: Toward a Theory of Public Bureaucracy." In Oliver E. Williamson (ed.), Organization Theory: From Chester Barnard to the Present and Beyond. New York: Oxford University Press. Nunberg, Barbara, and John Nellis. 1995. Civil Service Reform and the World Bank. World Bank Discussion Papers; No. 161. OECD, 1994a. Public Service Pay Determination and Pay Systems in OECD Countries, Paris: Organization for Economic and Cooperative Development. . 1994b. Performance Management in Government: Performance Measurement and Results- Oriented Government, Public Management Occasional Papers 1994 No. 3. . 1999. Elements of Model Civil Service Laws for Developing Nations at http://www.oecd.org/puma/sigmaweb/acts/civilservice/cslaws.htm; Civil Service Legislation: Checklist on Secondary Legislation (and other Regulatory Instruments) at http: //www. oecd. org/puma/sigmaweb/pubs/ENGPDF/sp 14. pdf, both from OECD's SIGMA program. Osbome, David and Peter Plastrik. 1997. Banishing Bureaucracy: The Five strategies for Reinventing Government. New York: Plume. PricewaterhouseCoopers Endowment for the Business of Government. 2000. Results of the Government Leadership Survey: A 1999 Survey of Federal Executives at http://www.pwcglobal.com/gx/eng/ins-sol/main/index.html Przeworski, Adam. 1999. "On the Design of the State: A Principal-Agent Perspective " in Luiz Carlos Bresser Pereira and Peter Spink (eds.), Reforming the State: Managerial Public Administration in Latin America. Boulder. Colo.: Lynne Rienner. 38 Rea, Louis and Richard Parker. 1997. Designing and Conducting Survey Research: A Comprehensive Guide, San Francisco: Jossey-Bass Publishers. Recantini, Francesca, Scott Wallsten and Lixin Xu. 2000. "Surveying Surveys and Questioning questions: Learning from the World Bank Experience". World Bank Policy Research Working Paper No. 2307, Washington D.C.: The World Bank. Seabright, Paul. 1993. "Managing Local Commons: theoretical issues in incentive design". The Journal of Economic Perspectives. 7-4: 113-134. Schein, Edgar H. 1996. "Three Cultures of Management: The key to organizational Learning" in Sloan Management Review, Fall 1996. Selznick, Philip. 1957. Leadership in Administration: A Sociological Interpretation, University of California Press (1984, reprint version). Simon, Herbert A., "Organizations and Markets," Journal of Economic Perspectives, Vol. 5, No. 2 (Spring 1991): 25-44 Stevens, Mike 1994. Preparingfor Civil Service pay and Employment: a Primer in Rehabilitating Government: Pay and Employment Reform in Africa ed. David L. Lindauer and Barbara Nunberg, Washington D.C.: The World Bank. Shand, D. 1997. The Role ofperformance Indicators in Public Expenditure Management, Unpublished IMF document. Simon, Herbert A. 1974. Administrative Behavior: A study of decision-making processes in administrative organization, New York: Free Press. . 1991. Organizations and Markets. Journal of Economic Perspectives, Vol. 5, No. 2 (Spring 1991): 25-44. Tendler, Judith. 1997. Good government in the tropics. Baltimore, Md.: Johns Hopkins University Press. Weingast, Barry R. and Mark J. Moran. 1983. "Bureaucratic Discretion or Congressional Control? Regulatory Policymaking by the Federal Trade Commission" in Journal of Political Economy (U. S.); 91:765-800. Williamson, Oliver E. 1975. Markets and Hierarchies, analysis and antitrust implications: a study in the economics of internal organization, New York: Free Press. Wilson, Jamnes Q., 1989. Bureaucracy: what government agencies do and why they do it, New York: Basic Books. 39 World Bank, 1994. Governance: the World Bank's Experience. Washington D.C.: The World Bank. . 1998. New Frontiers in Diagnosing and Combating Corruption, PREM Notes Number 7, October 1998. Washington D.C., The World Bank. . 1999. Report No. 19211 Civil Service Reform A Review of World Bank Assistance April 27, 1999 Operations Evaluation Department. . 2000. Reforming Public Institutions and Strengthening Governance: A. World Bank Strategy. http://www-wbweb.worldbank.org/prem/prmps Zussman, David and Jabes, Jak. 1989. The Vertical Solitude: Managing in the Public Sector, Halifax: The Institute for Research on Public Policy 40 Appendices Appendix 1: Illustrative questions to probefor dimensions of the institutional environment9 The test applied in grouping these questions was that these questions capture expectations of current or future constraints and incentives - and thus the degree to which officials are discounting the future and management is maintaining its reputation in relation to: 1. rule enforcement in relation to budget or personnel management; 2. binding policy direction; 3. adequacy and predictability of financial and personnel resources. Rule Credibility a. personnel management rules Within an organization, different people are hired for different reasons. This survey is interested in learning about general hiring trends. In your organization would you say that employees were hired more often for: 1. good performance at entry 2. test qualifications 3. met job requirements 4. personal connections 5. political connections 6. family status in community 7. payment of gift to some 8. public authorities 9. knew a senior official (Guyana) Does your organization have a system where annual objectives/ targets that are measurable are agreed to between emnployees and supervisors at the beginning of every year? (yes/no) (Guyana) What is your assessment of the existing formal, written guidelines related to personnel mnanagement? Please tell us the extent to which you agree with the following statements. Answer on a scale of 1 to 4, where 1 corresponds to 'strongly disagree' and 4 corresponds to 'strongly agree' 1. relatively simple to understand and implement 2. relatively flexible, providing our organization with a lot of discretion in their implementation 3. helpful for effective personnel management (Macedonia) Now, we will ask you a series of questions about all types of payments in addition to basic salary. Please answer for each of the following types of payments. This type of payment is distributed on the basis of 1. Clear rules 2. Rules that require/allow for some discretion 3. Full discretion, not on the basis of any rules (i) public transportation allowance (ii) food allowance - (iii) holiday payment (K-15), (iv) payment for overtime work, - (v) bonuses for innovation and rationalization, (vi) membership in Boards of Managers (vii) other bonuses or premia (Macedonia) 59 The questions themselves were developed by individual survey task managers and they may have drawn from several different sources in preparing them; The sources are not all known and thus exact citations can not be provided for each question. 41 Are positions like yours generally advertised to the public? (yes/no) (Bolivia) In your organization, would you say that the employees are hired more often for: 1. Performance on an examination 2. Personal connections 3. Political connections 4. Famnily relations 5. Payment or gift to some public authorities 6. Good CV (Bolivia) Does your organization have Individual Annual Operations Plans (POAI) for its employees? (yes/no) (Bolivia) Can an employee appeal a dismissal? (yes/no) (Bolivia) Does your organization have a Human Resource Management Infornation System (HMIS) containing records of employees? (yes/no) (East Caribbean States) As far as you know, does your organization have a stated performance appraisal system? (yes/no) (East Caribbean States) Are the criteria for appraisal known to you? (yes/no) (East Caribbean States) b. budget management rules For each of the following aspects of budget mnanagement, does your organization have formal, written guidelines describing how these functions are to be canied out. If so, are they carried out in full, in part, or not at all? Please answer on a scale of I to 4, where 1 conresponds to "not at all" and 4 corresponds to "yes, in full" 0 budget preparation 0 procurement 0 budget execution O audit 0 production of fnal end-of-year accounts (Macedonia) What is your assessment of the existing formal, written guidelines related to budget managernent? Please tell us the extent to which you agree with the following statements. Answer on a scale of I to 4, where I corresponds to 'strongly disagree' and 4 corresponds to 'strongly agree' * relatively simple to understand * relatively flexible * helpful for effective and efficient planning of programs/activities (Macedonia) Are there rules in place, which govem security, movernent, and storage of financial records? (Macedonia) Does your organization have an intemal audit function? (yes/no) (East Caribbean States) c. general rules Are there rules in place, which govem security, movement, and storage of records? (yes/no) (Macedonia) Is there a fornal system of evaluating programs/ projects being implemented by your organization? (yes/no) (East Caribbean States) Policy Credibility Agencies receive instructions and policies from different parts of the government, some comnplement each other, others conflict. Overall, how do you see the relationship between these various instructions/ policies? I. very consistent 2. more consistent than inconsistent 42 3. more inconsistent than consistent 4. very inconsistent (Guyana) How are changes in policy usually conmmunicated to you? I. Written correspondence 2. Orally 3. staff meetings 4. management 5. not at all (Guyana) How often do you disagree with the policies that your organization is asked to implement? 1. Often 2. Occasionally 3. Rarely 4. Not at all (Guyana) In your opinion, how would you rate the incidents of political interference that exists in your organization? 1. very frequently 2. frequently 3. infrequently 4. Almost never (Guyana) Do you believe your budget institution has a clear set of strategic objectives? (yes/no) (Note to interviewer examnple of a clear strategic objective might be: (a) for an agency regulating drug safety, to facilitate the introduction of safe medicines within the domestic market; (b) for a ministry of education, to build a labor force with the skills and adaptability required for a modern economy (Macedonia) What are the reasons for imperfect implementation of formal, written policies, guidelines and regulations related to personnel management? And among these potential reasons, what are the two most important reasons? 1. the guidelines are overruled by other directives within our organization or from other government units 2. contradictory among themselves 3. guidelines are not known or disseminated (Macedonia) What are the reasons you believe account for imperfect implementation of formal, written guidelines related to budget management? And arong these potential reasons, what are the two most imnportant reasons? 1. the guidelines are overruled by other directives within our organization or from other government units 2. contradictory among themselves 3. Guidelines are not known or disseminated (Macedonia) Agencies receive instructions and policies from different parts (of government), some of which are incompatible with each other. How compatible (consistent) are the instructions and policies which your agency receives with each other 1. Very Consistent 2. Consistent 3. Inconsistent 4. Very Inconsistent (Bolivia) In your view, which of these measures would contribute to improving the performance of your organization? inter alia... Protection from political interference (Bolivia) Which of these measures do you believe can be successfully implemented successfully in your organization? inter alia Protection from political interference (Bolivia) 43 How strongly do you agree with the following statement which is about the public administration in general not about specifically your organization. I =strongly disagree 4=strongly agree [for Ministers/Deputy-Ministers only] The process of formulating the national budget involves close consultation among members of the Cabinet. (1) (2) (3) (4) The process of formulating the national budget involves close consultation between the Ministry of Finance and the line ministries. (1) (2) (3) (4) The process of formulating municipal budgets involving close consultation with the Ministry of Finance (1) (2) (3) (4) The process of formulating the organizational budgets involves close consultation between the budget managers and department/division managers . (1) (2) (3) (4) (Bolivia) If decrees, executive orders, or policies interfere with the ability of your organization to do its-job, are methods available to have them changed? (Bolivia) Political rnicro-management or interference by political leaders and ministers in day to day working of public sector entities can sometimes hamper the efficiency of these organizations. In your organization, is the problem of political micro-management 1. Serious 2. Present, but not serious 3. Not present 4. Helping your organization perform better (East Caribbean States) Before an election, does each party clearly outline in its election manifesto the national priorities it would pursue if voted to power! (yes/no) (East Caribbean States) After a govenmment is formed, to what extent are policies formulated within the previously announced priorities I. Most ofthe time 2. Some time 3. A few time 4. Very rarely (East Caribbean States) Resource Adequacy and Predictability a. budget resources Do you think that actual funds approved for next year's budget will exceed or fall short of budgeted requirements? 1. exceed 2. fall short 3. other [specify] (Guyana) In what way do you expect that actual funds will differ from budgeted Funds? I. a large amount 2. a moderate amount 3. a small amount 4. other (Guyana) Do you expect that actual funds for your organization will differ from budgeted funds this year? (yes/no) (Macedonia) Do you expect that actual funds will differ from budgeted funds by a large amount or a small amount? (1) (2) (3) (4) 1=a small amount; 4=a large amount (Macedonia) How typical, in your experience, is it for actual funds to differ from budgeted funds? Would you say that it is rare, not typical, fairly typical very typical? (1) (2) (3) (4) 1 =rare; 4=very typical (Macedonia) 44 To what extent does the public administration and your organization have adequate financial resources to provide an appropriate level of service? Please answer on a scale of 1 to 5, where 1 corresponds to 'completely inadequate' and 5 corresponds to 'completely adequate'. Are the following resources in short supply for your organization? Please also identify the two resources in most short supply. Number of staff, skills of staff, computers, office supplies (excluding computers), other goods and services, mnanagerial support, other (specify) (Macedonia) Here is a list of attributes which people mention for an organization to work efficiently. If your organization lacks any of them, write it down in your list, please. inter alia... Disbursement of budget resources without delay (Bolivia) b. personnel resources How many weeks per year are lost from regular duties because you have to work on emergencies like natural disaster? 0 1-2 3-4 More than 4 (East Caribbean States) Some people consider that emigration of educated and trained persons from the Caribbean to US, lUK and Canada drains the public sector of skilled persons. How severe is this problem in your organization? 1. Very severe 2. Severe 3. Moderate 4. Mild (East Caribbean States) 45 Appendix 2: Institutional environment indicators in the East Caribbean States60 Statistical Table 1 Rule credibility indicator and components (East Caribbean) Record Internal Performance Project Rule management audit appraisal evaluation Credibility Antigua and Barbuda 8.4 3.5 4.3 3.7 5.0 Grenada 8.8 3.6 9.3 6.1 6.9 St. Kitts and Nevis 7.1 6.5 5.6 5.0 6.1 St. Lucia 8.6 5.5 8.1 4.1 6.6 St. Vincent And Grenadines 8.9 6.9 4.2 5.1 6.3 East Caribbean 8.6 5.2 6.6 4.7 6.3 Statistical Table 2 Policy credibility indicator and components (East Caribbean) Consistency Policy Micro- Policy coordination management Credibility Antigua and Barbuda 4.1 5.3 4.2 4.5 Grenada 3.7 5.4 3.8 4.3 St.Kitts andNevis 7.3 6.9 3.3 5.8 St. Lucia 6.0 6.4 3.5 5.3 St. Vincent And Grenadines 4.2 5.1 3.8 4.4 East Caribbean 4.8 5.7 3.7 4.8 Statistical Table 3 Indicator of resource adequacy and predictability and components (East Caribbean) Lost man-days skills Institutional Resource from natural available capacity adequacy and disasters predictability Antigua and Barbuda 4.1 5.5 7.7 5.8 Grenada 3.9 7.4 7.9 6.4 St. Kitts and Nevis 5.3 5.9 7.4 6.2 St. Lucia 5.3 6.5 6.2 6.0 St. Vincent And Grenadines 5.3 6.1 6.3 5.9 East Caribbean 4.5 6.3 7.1 6.0 60 See footnote 48. 46 Statistical Table 4 Indicator of overall institutional environment and components (East Caribbean) Rule Policy Resource Overall Credibility Credibility predictability Institutional from Statistical from Statistical from Statistical Environment Table I Table 2 Table 3 Antigua and Barbuda 5.0 4.5 5.8 5.1 Grenada 6.9 4.3 6.4 5.9 St. Kitts and Nevis 6.1 5.8 6.2 6.0 St. Lucia 6.6 5.3 6.0 5.9 St. Vincent And Grenadimes 6.3 4.4 5.9 5.5 East Caribbean 6.3 4.8 6.0 5.7 47 Appendix 3. Illustrative questions to probefor dimensions ofperformance6' Test applied: These question capture perceptions of past performance and of 1. the degree to which a concern for results or outputs shaped behavior, 2. rule-based behavior in relation to budget or personnel management; 3. morale and job satisfaction. Results Focus Are there documented work / performance standards for each function in your organization? (yes/no) Are they generally achieved? (Guyana) In your opinion, to what extent would you say that your organization has realized its mnission? 0 to 25% 26% to 50% 51% to 75% 76% to 100% (Guyana) In the past year have staff in your organization been rewarded for any of the following reasons? 1. Taking prompt action on customers / Clients/ end users' requests or complaints 2. Providing outstanding service 3. Improving qualifications 4. Excellent attendance record 5. Consistent courtesy Overall, do you think these rewards were justified? 1. always 2. generally 3. occasionally 4. almost never 5. never (Guyana) Across individuals within your budget institution, all types of bonuses and similar payments are I. very unequally distributed 2. unequally distributed 3. equally distributed 4. distributed on the basis of performance (Macedonia) In the past year, has anyone in your organization been the subject of the following types of sanctions for poor performance or unprofessional conduct? 1. Dismissal 2. Demotion 3. Salary decrease for a specified length of time 4. Suspension of duties with withdrawal of salary 5. Postponement of promotion or pay increases 6. Public censure and reprimand 7. Warning 8. Initiating criminal proceedings (Macedonia) 61 The questions themselves were developed by individual survey task managers and they may have drawn from several different sources in preparing them. The sources are not all known and thus exact citations can not be provided for each question. 48 What were the causes of the disciplinary actions? 1. Poor work performance 2. Unauthorized absence 3. Working on a second job during official work hours 4. Soliciting or accepting bribe 5. Embezzlement 6. Conflict with supervisor (Macedonia) During the past two years, has your organization received any sanctions or measures for not performing well? If yes, which of the following mneasures? 1. Our budget was reduced 2. Our budget was increased 3. Responsible staff was disciplined 4. We received a warning 5. other (specify) (Macedonia) In the past year, have staff in your organization been recognized/rewarded for any of the following reasons? 1. Taking prornpt action on consurer requests or complaints 2. Providing outstanding service 3. Other 4. No recognition/reward (Bolivia) How many employees of your organization with excellent performance have not received recognition/reward? 1. Many 2. Some 3. A few 4. No one has received recognition (Bolivia) Overall, do you think that these rewards were justified? 1. Always 2. Generally 3. Occasionally 4. Almost never or never (Bolivia) At the end of the year, are physical outputs of your organization measured and evaluated against the expenditure incurred? 1. Most of the time 2. Frequently 3. A few tirnes 4. Very rarely (East Caribbean States) In the past year, how mnany ermployees in your organization received? Reward or recognition for good performance? * 0% * about 10% * about 25% * about 50% Some kind of penalty for non-performance? * 0% * about 10% * about 25% * about 50% (East Caribbean States) In your performance evaluation, how much importance is laid on your achievemnent of your targets I. a lot 2. some 3. a little 4. not at all (East Caribbean States) 49 Accountability When you first joined the government, were you assured that you would get the job? (yes/no) (Guyana) Before you applied for your current job were you assured that you would get the job? (yes/no) (Guyana) When was your performance last appraised by your supervisor? I . last quarter 2. mid year 3. end of first quarter 4. one year ago 5. two years ago 6. three years ago 7. more than three years ago 8. performance not appraised (Guyana) How difficult is it to obtain adequate and timely information from the records management system in your organization? [ very difficult D difficult D very easy D easy (Guyana) In your opinion, to what extent is corruption a significant problem in the public sector as a whole? 1. very significant 2. significant 3. somewhat significant 4. not significant at all (Guyana) How often are cases of corruption, generally, reported to the proper authorities? 1. Almost always 2. Frequently 3. Occasionally 4. Almost never (Guyana) In the past year, has anyone in your organization been the subject of the following types of sanctions for poor performance or unprofessional conduct? 1. Dismissal 2. Demotion 3. Salary decrease for a specified length of time 4. Suspension of duties with withdrawal of salary 5. Postponement of promotion or pay increases 6. Public censure and reprimand 7. Warning 8. Initiating criminal proceedings (Macedonia) What were the causes of the disciplinary actions? I . Poor work performance 2. Unauthorized absence 3. Working on a second job during official work hours 4. Soliciting or accepting bribe 5. Embezzlement 6. Conflict with supervisor (Macedonia) For each of the following aspects of budget management, does your organization have formal, written guidelines describing how these functions are to be carried out. If so, are they carried out in full, in part, or not at all? Please answer on a scale of 1 to 4, where 1 corresponds to "not at all" and 4 corresponds to "yes, in full". (Macedonia) Is there a problem of finding important financial records in your organization? (1) (2) (3) (4) (5) 1 =extremely problematic 5==no problem at all (Macedonia) 50 Please evaluate the following opposing statements about practice in monitoring commitments and mnanaging aggregate expenditure. In your organization, a system for monitoring and controlling the total stock of expenditure commrritments incurred is * not present * present * present and moderately effective * present and highly effective In your organization the total stock of expenditure commitments is monitored frequently throughout the year and controls are in place to ensure that total commitments do not exceed total budget authorization (1) (2) (3) (4) (1) totally disagree (4) totally agree (Macedgnia) How easy is it to find necessary records in your department? Please answer on a scale from 1 to 4, where 1 corresponds to "extremely difficulf' and 4 corresponds to "very easy". (Macedonia) In general, cases of cormption are always, frequently, occasionally or never reported to proper authorities? 1. Always 2. Frequently 3. Occasionally 4. Almost never or never (Bolivia) Would you say that, in your organization, corruption is a 1. very serious problem, 2. a serious problem, 3. a somewhat serious problem, or 4. not a serious problem at all? (Bolivia) How difficult is to obtain information from those records? 1. Very difficult 2. Difficult 3. Easy 4. Very easy (Bolivia) How reliable is the information contained in your organization's HRMIS? 1. Very reliable 2. Reliable 3. Not reliable 4. Completely unreliable (East Caribbean States) Does your organization's HRMIS contain the following? 1 . Updated records of all training you attended 2. The dates when you got increment or moved from one scale to another 3. Perfornance evaluation reports 4. The date you will retire from service (East Caribbean States) When were your own records last updated on HRMIS? 1 . In 2000 2. In 1999 3. In 1998 4. Do not know when the records were updated (East Caribbean States) How easy or difficult is to obtain for official purposes records of employees in your organization? I . Very easy 2. Easy 3. Difficult 4. Very difficult (East Caribbean States) 51 Have you ever met with associations / groups of citizens who are the users, consumers of the services produced by your organization? (yes/no) (East Caribbean States) Are you obliged to share official infonnation, especially that related to performance of your organization, with them? (yes/no) (East Caribbean States) Is your organization's audit report required to be presented to parliament? (yes/no) (East Caribbean States) If yes, how effective is the internal audit in ensuring compliance with internal control systems? 1. Very effective. 2. Effective. 3. Partially effective. 4. Ineffective (East Caribbean States) How long ago was your last appraisal conducted? 1. Less than 6 months ago 2. Less than a year ago 3. More than a year ago 4. I have never been appmised (East Caribbean States) Employee Morale62 Is your salary sufficient to meet your living expenses? (yes/no) (Guyana) Would you describe the quality of life that you have maintained over the past year as: I. Unacceptable 2. Acceptable 3. Good (Guyana) During periods of economic difficulty and austerity, some people talk about decreasing the size of the government as a means of saving money. Decreasing the size of the government would, in sorne areas, entail organization streamlining and staff cutbacks. Given that, how worried are you about losing your job sometime in the near future? 1. Very Worried 2. Somewhat Worried 3. Not At All Worried (Guyana) How often do you receive salary or wages late? 1. very often 2. a few times 3. sornetimes 4. no time at all (Guyana) How worried are you about losing yourjob some time in the near future? 1. very worried 2. somewhat worried 3. not very worried 4. not at all worried (Macedonia) In what year did you receive your last promotion? When do you expect to be considered for your next promotion? * Don't know * Don't expect to be considered for promotion (Macedonia) Would you describe the quality of life that you have raintained over the past year as unacceptable, acceptable, good or excellent? (Macedonia) 62 To further assess performance, a measure of vertical solitude was also constructed from the differences in perceptions conceming the environment between managers and their staff. 52 Are your earnings from your job in the public administration (including base salary and other payments) sufficient to pay your living expenses? (yes/no) (Bolivia) How probable is it that in the near future you will become unemployed? 1. Very probable 2. Somewhat probable 3. Not very probable 4. Not at all probable (Bolivia) Do you feel that you are contributing to your organization's performance? (yes/no) (East Caribbean States) Do you enjoy your work? (yes/no) (East Caribbean States) How probable is it that in the near future you will lose your job? 1. Very likely 2. Likely 3. Unlikely 4. Very unlikely (East Caribbean States) 53 Appendix 4: Performance indicators in the East Caribbean States63 Statistical Table 5 Results focus indicator and components (East Caribbean) Geared to Efficiency merit based reward Results focus organization's & punishment objectives Antigua and Barbuda 6.0 3.6 0.7 3.6 Grenada 6.7 4.9 1.7 5.1 St. Kitts and Nevis 7.4 5.4 2.0 5.2 St. Lucia 6.9 4.1 1.3 5.0 St. Vincent And Grenadines 6.9 4.5 1.2 5.2 .~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ East Caribbean 6.8 4.4 1.3 4.9 Statistical Table 6 Accountability indicator and components (East Caribbean) Enforcement of regulations in_ _ A ^ Accountability to Accounta CD 0 ~~~~~~~~~~~~bility 18- ~ ~ ~ O 8 w e~~~ from g" previous '~~~ ~~ ~~8 columns Antigua and 5.4 7.7 4.6 5.6 5.8 5.0 4.4 6.2 5.4 Barbuda Grenada 4.7 8.1 6.0 7.4 6.5 4.5 4.6 7.3 5.7 St. Kitts and 6.1 9.2 7.4 5.7 7.1 5.9 4.6 7.3 6.2 Nevis St. Lucia 5.1 7.9 5.5 6.5 6.3 5.9 4.4 7.2 5.9 St. Vincent 4.8 7.6 5.4 4.5 5.6 5.5 4.1 7.4 5.7 And Grenadines East Caribbean 5.1 8.0 5.7 6.1 6.2 5.3 4.4 6.2 5.5 Statistical Table 7 Employee morale indicator and components (East Caribbean) Country Employee morale Antigua and Barbuda 8.0 Grenada 7.8 St. Kitts and Nevis 8.5 St. Lucia 7.6 St. Vincent And Grenadines 8.2 East Caribbean region 8.0 63 See footnote 48. 54 Statistical Table 8 Overall performance indicator and components (East Caribbean) Results-focus Accountability Employee Morale from Statistical from Statistical from Statistical Performance Table 5 Table 6 Table 7 Antigua and Barbuda 3.6 5.4 8.0 5.7 Grenada 5.1 5.7 7.8 6.2 St. Kitts and Nevis 5.2 6.2 8.5 6.6 St. Lucia 5.0 5.9 7.6 6.2 St. Vincent And Grenadines 5.2 5.7 8.2 6.4 East Caribbean 4.9 5.5 8.0 6.1 No measure of vertical solitude was calculated in the East Caribbean. The very considerable differences in the perceptions of managers prevented the establishment of any single reference point against which other staff s perceptions could be tested. However, in other country reports where indicators could be calculated at organizational level, an indicator of vertical solitude was constructed. 55 Appendix 5: Connection between performance and institutional environment' Figure 10. Effect of institutional environment on results focus in Bolivia 7 result focus =0.54 19 IE + 0.7997 R2= 0.2853 6 ____________________ bW king-regula ion tax 5 ;c ] + ~~~~~~~~Santa Cruz central bank ;l jzglaria a~~~~~cu reform 4 __iinance __tele rn-regulation * road * agriculture 3 __Chuquisac i 5 6 7 8 9 institutional environment 64 See footnote 48. 56 Figure 11. Effect of institutional environment on accountability in Bolivia 9 accou itability = 1.043 IE - 1.0788 R2 = 0.741 *audit t central bank 7 . ;~~~~~~~~~~~~~~~ tele DM-regu atien 6 5 +7f~~~~~saci t labor * customs 5 6 7 8 9 institutitonal environment 57 Figure 12. Effect of institutional environment on employee morale in Bolivia 8 mnwale = 0.6631 IE + 1.5892 R2 = 0.7606 biking-regula on 7 . ine / t + c~~~~~~~~~~~~~~rentral bank 4 tax */e telec _m-regulation 5 6 7 8i a 9orm institutiomsl envion / + Santa Cnuz Chuguisa L 4 .... .... 5 6 7 8 9 institutional environment 58 Appendix 6: Econometric and analytical issues This appendix has five sections. Section A 6.1 presents the econometric models. Section A 6.2 discusses the data and estimation problems and describes how these problems were tackled. Section A 6.3 includes the estimation results for the entire Bolivian public sector. Section A 6.4 briefly explains the construction of the IRRs for Bolivia. Section A6.5 has the IRR tables for centralized organizations. A 6.1. Econometric models Econometric Model I: Yij= a + b E +ei, where Y = performance indicator, with j = aggregate performance, accountability, employee morale, and results focus; IE - institutional environment; and e = a well-behaved stochastic error term. A linear functional form was chosen to keep the analysis simple. This model was estimated by using data points from 16 Bolivian organizations. The estimation results are presented in Section 6 and in Appendix 3. Econometric Model III: 11 where, Yj = performance-j, with j = aggregate performance, accountability (ACC), employee morale (MOR), and results-focus (RFO); X k= component-k of the institutional environment, with k=l,2, ..., 11; and uj= a classical error term. In particular, xl = Existence of rules in recruitment x2 = Existence of rules in evaluation x3 = Existence of rules in training x4 = Existence of rules in recording x5 = Fair treatment 59 x6 = Predictable career path x7 = Policy consistency x8 = Policy communication x9 = Policy support xlO = Political interference/micro-management xl 1 = Resource predictability A 6.2. Dealing with data and estimation problems: the example of Bolivia Number ofobservations Having 1 1 independent variables (and a constant term), and only 15 organizations, i.e., data points was a problem. In order to calculate IRRs for different organizations and different groups of public officials, responses from each public official were utilized in running regressions. That basically gave (after some data cleaning) 690 observation points for the entire Bolivia; and 240 for central administration, and so on. Collinearity among independent variables Rather than doing nothing or changing the functional form, explanatory variables were transformed into binary responses: an indicator which is less than 5 out of 10 points were converted into zero, otherwise into one. This transformation also allowed the interpretation of the estimated coefficients in a more sensible way. Instead of referring to increasing policy credibility by one or by a certain amount, it was possible to refer to making policies credible. Similarly, instead of referring to two public officials who believe that rules in place 72.4% or 39.8%, it is possible to refer to distinguish between public officials who believe and don't believe respectively that rules are in place. Correlation coefficients for the (converted) institutional environment indicators (Bolivia) xl x2 x3 x4 x5 x6 x7 x8 x9 xIO xlI xl 1.00 x2 0.13 1.00 x3 0.05 0.03 1.00 x4 0.09 0.05 0.05 1.00 x5 0.04 0.10 0.07 0.05 1.00 x6 0.21 0.08 0.09 0.12 0.20 1.00 x7 0.12 0.06 0.03 0.07 0.03 0.14 1.00 x8 -0.09 0.00 0.00 -0.03 0.01 -0.03 -0.02 1.00 x9 0.11 0.05 0.13 0.09 0.06 0.19 0.13 -0.04 1.00 xlO 0.31 0.10 0.08 0.12 0.14 0.47 0.09 -0.07 0.23 1.00 xl1 0.14 -0.01 0.07 0.03 0.03 0.10 0.13 -0.04 0.03 0.09 1.00 60 Correlation among the error terms of different performanice regression equations In the left hand side of Model III, there are three specific performnance indicators, namely Results-focus (RF) Accountability (ACC), and Morale (MOR) as a function of eleven different aspects of the institutional environment (iE). In other words, we are saying that much of the systematic component of these three different performances will consist of the same type of variation-variation due to the aspects of the institutional environment. But, it is also likely because of the matching across public officials that random components of the performances will be related. Thus, some knowledge of the random component of RF variations would give us some predictive insight into the random component of ACC variations. Consequently, it is reasonable to expect that the errors of these three regression equations--RF= a(IE), ACC= v(IE), MOR= c(IE)--may be correlated across equations. However, when the errors are correlated across the equations, OLS results in biased variance estimates. Therefore, to improve efficiency, Seemingly Unrelated Regression Equations (SURE) approach was used in the estimations. If the errors of the equations were not correlated, then SURE results and OLS results became the same. 61 A 6.3 SURE Estimation results for the Bolivian public sector Estimates for equation: Results-focus N=690 Variable Coefficient Standard Error z=b/s.e. P[IZI_z] Mean of X Xi 0.1843148 0.23181 0.795 0.42654 0.4725 X2 0.6025640 0.27018 2.230 0.02573 0.2072 X3 -0.7502945E-02 0.27814 -0.027 0.97848 0.1913 X4 0.5985908 0.27651 2.165 0.03040 0.8058 X5 0.3981092 0.22865 1.741 0.08165 0.6232 X6 -0.2858045 0.26078 -1.096 0.27309 0.6420 X7 -0.5335056E-01 0.31301 -0.170 0.86466 0.8536 X8 -0.1540673 0.31894 -0.483 0.62905 0.8667 X9 0.7550640 0.24320 3.105 0.00191 0.6870 X1o 0.5296796E-02 0.25926 0.020 0.98370 0.4130 X1i -0.3912180E-01 0.26093 -0.150 0.88082 0.7681 Constant 3.411840 0.50980 6.692 0.00000 Model test: F[ 11, 678] = 2.52, Prob value = 0.00421 Estimates for equation: Accountability Variable Coefficient Standard Error z=b/s.e. P[IZI_z] Mean of X Xi 0.3022615E-01 0.12985 0.233 0.81593 0.4725 X2 0.6108728 0.15134 4.036 0.00005 0.2072 X3 0.4409731 0.15580 2.830 0.00465 0.1913 X4 0.7030247 0.15489 4.539 0.00001 0.8058 X5 0.1116405 0.12808 0.872 0.38338 0.6232 X6 0.5904624 0.14607 4.042 0.00005 0.6420 X7 0.1345138 0.17533 0.767 0.44296 0.8536 X8 -0.2312646 0.17865 -1.294 0.19550 0.8667 X9 0.3926405 0.13623 2.882 0.00395 0.6870 X10 0.9710046 0.14522 6.686 0.00000 0.4130 X1i 0.1574662 0.14616 1.077 0.28133 0.7681 Constant 4.051039 0.28556 14.186 0.00000 Model test: F[ 11, 678] = 22.58, Prob value = 0.00000 Estimates for equation: Employee Morale Variable Coefficient Standard Error z=b/s.e. P[|Z|_z] Mean of X Xi 0.2087146 0.70065E-01 2.979 0.00289 0.4725 X2 0.3327698 0.81663E-01 4.075 0.00005 0.2072 X3 -0.6659126E-01 0.84071E-01 -0.792 0.42831 0.1913 X4 0.4759933 0.83577E-01 5.695 0.00000 0.8058 X5 1.095540 0.69109E-01 15.852 0.00000 0.6232 X6 0.4592120 0.78821E-01 5.826 0.00000 0.6420 X7 0.1867915 0.94608E-01 1.974 0.04834 0.8536 X8 -0.8713628E-01 0.96401E-01 -0.904 0.36605 0.8667 X9 0.4057072 0.73509E-01 5.519 0.00000 0.6870 X1o 0.1470672 0.78362E-01 1.877 0.06055 0.4130 X1i 0.2771287E-02 0.78869E-01 0.035 0.97197 0.7681 Constant 4.136974 0.15409 26.848 0.00000 Model test: F[ 11, 678] 54.36, Prob value = 0.00000 62 A 6.4 Constructing the IRR indicators The following is a demonstration of how an IRR was constructed by using the estimated coefficients-which were reported in the previous section. Based on the following formula, which was explained in section 7, particularly for policy consistency, two pieces of informnation were required to construct a specific IRR. A performance / Apolicy consistency IRRpolicyconsistency performance For instance, to construct Results-focus IRRs (1) we need estimated partial derivatives of the results focus with respect to particular potential reform interventions-i.e., the coefficient of the related variable from Model III. Yet, for statistical significance of the constructed IRR, only the estimated coefficients, which were statistically significant for one-tailed test, were used in the construction of the indicators. (2) we need the initial level of the results-focus indicator. As the reported regression results show, four of the estimated coefficients were statistically significant for one-tailed test. The initial level of the result focus-indicator was 4.481. Thus, the IRRs were constructed as shown in the following table. Statistical Table 9 Calculating the IRRs (Bolivia) Institutional 8 (results focus) environmentk ( Resultsfocuso Results focus- a(institutional environment,,) IRRk Rules in evaluation 0.603 4.481 13.4% Rules inrecording 0.599 4.481 13.4% Fair treatment 0.398 4.481 8.9% Policy support 0.755 4.481 16.9% 63 A6.5 Reform interventions and improvements in performance: Bolivia's Central Administrations65 Table 3 Reform Interventions and anticipated performance pay offs in Bolivia Central Administrations Reform Intervention Improvement in Performance Number Results Accoun Employee of areas focus tability Morale affected Ensuring resource predictability 3 V V Ensuring that staff support government policy 3 / V/ V/ Strengthening the credibility of rules for record management 3 / / / Strengthening the credibility of rules for evaluation 3 / V Preventing political interference/micro- management 2 V/ V Strengthening the credibility of rules for training 1 V Figure 13 then demonstrates how these anticipated payoffs accumulate to distinguish between the degrees to which each reform intervention is expected to impact on performance. 65 See footnote 48. 64 Figure 13. Expected impact of reforms on different components of performance in Bolivia Central Administrations' Reform Interventions imyactin2 three performance areas. * results focus 3accountabilit employee morale ensuring resource predictability ensuring staff support for government 8% Reform policies Intervent'sVengthening credibility of rules for I14% T 8% record management strengthening credibility of rules for 10%i evaluation l . ~ ~ ~ ~ ~ ~~I III 0% 10% 20% 30% 40% 50% Percentage increase in present level of performance Reform Interventions imnactine one or two performance areas. Reform Intervention l strengtheiing credibility of rules for training 9 preventing political intervention/micro- fvJ 'ILI15%K LI 20% management 0% 10% 20% 30% 40% Percentage increase in present level of performance 66 As explained above, the unit of measurement for the improvement in the performance due to a reform intervention is the percentage change in the present level of performance. 65 Policy Research Working Paper Series Contact Title Author Date for paper WPS2403 Can the World Cut Poverty in Half? Paul Collier July 2000 E. Khine How Policy Reform and Effective David Dollar 37471 Aid Can Meet International Development Goals WPS2404 The Distribution of Mexico's Public Gladys Lopez-Acevedo July 2000 M. Geller Spending on Education Angel Salinas 85155 WPS2405 Marginal Willingness to Pay for Gladys Lopez-Acevedo July 2000 M. Geller Education and the Determinants of Angel Salinas 85155 Enrollment in Mexico WPS2406 How Mexico's Financial Crisis Gladys Lopez-Acevedo July 2000 M. Geller Affected Income Distribution Angel Salinas 85155 WPS2407 Utility Privatization and the Needs of Antonio Estache August 2000 G. Chenet-Smith the Poor in Latin America: Have We Andres Gomez-Lobo 36370 Learned Enough to Get It Right? Danny Leipziger WPS2408 What Makes Banks Special? A Study Biagio Bossone August 2000 E. Mekhova of Banking, Finance, and Economic 85984 Development WPS2409 How Did the World's Poorest Fare Shaohua Chen August 2000 P. Sader in the 1990s? Martin Ravallion MC3-556 WPS2410 Is Functional Literacy a Prerequisite Niels-Hugo Blunch August 2000 H. Vargas for Entering the Labor Market? Dorte Verner 37871 An Analysis of Determinants of Adult Literacy and Earnings in Ghana WPS2411 Natural Openness and Good Shang-Jin Wei August 2000 H. Sladovich Government 37698 WPS2412 Urbanization without Growth: Marianne Fay August 2000 A. Abuzeid A Not-So-Uncommon Phenomenon Charlotte Opal 33348 WPS2413 Foreign Direct Investment in Services James Markusen August 2000 L. Tabada and the Domestic Market for Expertise Thomas F. Rutherford 36896 David Tarr WPS2414 Pension Reform and Capital Market Dimitri Vittas August 2000 A. Yaptenco Development: "Feasibility" and 31823 'Impact" Preconditions WPS2415 Infrastructure Restructuring and Ian Alexander August 2000 M. Salehi Regulation: Building a Base for Antonio Estache 37157 Sustainable Growth Policy Research Working Paper Series Contact Title Author Date for paper WPS2416 The Swiss Multi-Pillar Pension Monika Queisser August 2000 A. Yaptenco System: Triumph of Common Sense? Dimitri Vittas 31823 WPS2417 The Indirect Approach David Ellerman August2000 B. Mekuria 82756 WPS2418 Polarization, Politics, and Property Philip Keefer August 2000 P. Sintim-Aboagye Rights: Links between Inequality Stephen Knack 37644 and Growth WPS2419 The Savings Collapse during the Cevdet Denizer August 2000 I. Partola Transition in Eastern Europe Holger C. Wolf 35759 WPS2420 Public versus Private Ownership: Mary Shirley August 2000 Z. Kranzer The Current State of the Debate Patrick Walsh 38526 WPS2421 Contractual Savings or Stock Market Mario Catalan August 2000 P. Braxton Development-Which Leads? Gregorio Impavido 32720 Alberto R. Musalem WPS2422 Private Provision of a Public Good: Sheoli Pargal August 2000 S. Pargal Social Capital and Solid Waste Daniel Gilligan 81951 Management in Dhaka, Bangladesh Mainul Huq WPS2423 Financial Structure and Economic Thorsten Beck August 2000 K. Labrie Development: Firm, Industry, and Ashl Demirguc,-Kunt 31001 Country Evidence Ross Levine Vojislav Maksimovic WPS2424 Global Transmission of Interest Jeffrey Frankel August 2000 E. Khine Rates: Monetary Independence and Sergio Schmukler 37471 the Currency Regime Luis Serven WPS2425 Are Returns to Investment Lower for Dominique van de Walle August 2000 H. Sladovich the Poor? Human and Physical 37698 Capital Interactions in Rural Vietnam WPS2426 Commodity Price Uncertainty in Jan Dehn August 2000 P. Varangis Developing Countries 33852