GET Note: Managing Grand Design Public Administration Reforms “Recently Asked Questions” Series August 2013 Managing & Monitoring Grand Design Public Administration Reforms 98809 Tony Verheijen I. Overview A grand design attempt at public administration reform can be thought of as any centrally designed, multiple agency reform program or process designed to modernize or improve the performance of administrative structures at the center of Government – usually with a focus on addressing persistent underlying inefficiencies. International practice shows that reforming selected central institutions (especially those that hold the purse strings) is a different matter altogether from addressing performance issues in large ministries with a service delivery mandate. Therefore, it is of critical importance to ‘unpack’ these particular reforms and uncover the persistent issues that arise in countries attempting to pursue such reforms. The four grand design cases highlighted here were selected for their comparability in terms of size and economy, and as examples of reforms from different regions. The cases presented here are Brazil, Nigeria, Russia and Tanzania. Each of these cases has specific characteristics, based on a unique country or reform context, but they share the features of a broad, across-the-board reform approach (in three of the four cases with a clear sub-national dimension that is distinct from the national one). This note focuses on the three critical design aspects of such reforms: a) reform coherence, b) effective anchorage and, c) blending technocratic solutions with substantive service delivery improvements. II. Reform coherence Critical to a comprehensive reform process is the successful linking of people, process and financing issues. This is even more important in comprehensive, across-the-board reforms, considering the cost implications and the need to secure buy-in from many stakeholders. For instance, if a country moves towards programmatic and results-based budgeting, but does not adjust either the way establishment numbers are set or the way salary envelopes are calculated, the reform’s impact will remain very limited and disappointing. At the same time, this has been the weak link in many across-the-board reform processes. In the case of Russia, during the early 2000s, budget management, and civil service and administrative reform were managed by separate (and rival) agencies, which negatively impacted progress. On the one hand, Russian reforms are a case of how reforms can progress regardless of the fragmentation that has hampered them. The separation of performance management from performance budget and civil service reform from GET Notes – Recently Asked Questions Series intends to capture the knowledge and advice from individual engagements of the World Bank’s Global Expert Team on Public Sector Performance (PSP GET). The views expressed in the notes are those of the author – Tony Verheijen- and do not necessarily reflect those of the World Bank. For more information about the PSP GET, contact the GET team leader Bill Dorotinsky (wdorotinsky@worldbank.org) and visit http://go.worldbank.org/ITQYT7A0K0 The World Bank 1 administrative reform has created artificial boundaries that have at times been costly. Regardless of this institutional problem, there has been some cross-program learning, as both budget reform and administrative reform programs have used similar incentive- based systems to move agency and regional level reform forward, and both have used general reform frameworks leaving implementation to agency and sub-national level institutions. Only civil service reform has used a more top-down and rigid approach. Administrative and budget reform have both used innovative approaches, drawing on OECD country best practices, while civil service reform has been more incremental and traditional. The consequences of the relative disconnect have been a tension between modernizing structures and institutional management practices and hierarchical top-down personnel management approaches. More recently steps have been taken to loosen some of the rigidities in personnel management, in particular through the implementation of a new pay system. It is interesting to note that while there is institutional rivalry at the central level, a number of regions have actually presented integrated reform programs that successfully blend the three main aspects of public sector reform. Tanzania presents a similar case of weak integration. Public service reform, decentralization and public financial management reform have been managed out of different Ministries, without adequate integration. This disconnect has been visible mostly through a lack of perceived results at the service delivery level (and local level), while central systems have moved forward on key aspects of reforms. The creation of a reform coordination unit under the Head of the President’s Administration aims to address the horizontal and vertical integration aspects of the reform process as the credibility of reforms has been increasingly questioned. Nigeria (in the 2000s) and Brazil (starting in 1998) had better integrated reform practices, with the National Public Service Reform strategy in Nigeria comprising public financial management (PFM) and public service management (PSM) reforms, including civil service management systems, and the constitutional and legal reform packages in Brazil are also linked. This facilitates progress in linking improvements in budget management with better public service delivery. III. The importance of legal and strategic frameworks Even in a context of reform fragmentation, having appropriate instruments is critical. Here the following issues are important: i. Linking the reform framework to deliverables and performance:  Tanzania: In the early 2000s, public service reforms were formally linked to the poverty reduction strategy and program, but the linkage lacked substance, hence it was ineffective (due to procedural issues).  Russia: The civil service reform framework introduced in 2001 was inward looking (with no clear linkage to overall reform strategies), while the Administrative Reform Concept (approved in 2005) was linked to business climate considerations (but not to human development). The World Bank 2  Nigeria: The national strategy for public service reform (mid-2000) took a holistic view of Human Resource Management (HRM), PFM and Institutional Governance Review (IGR), including aspects of state level reform. ii. Appropriateness of instruments (strategies vs. legislation)  Legislation is essential especially on human resource issues, in particular if pay reforms and staffing adaptations are needed. Legislation is less important to altering performance management, though this varies by administrative tradition.  Constitutional changes in Brazil facilitated overall reforms of HR systems.  The Russian Civil Service Law (CSL), to the contrary, imposed rigidity on performance management reforms. IV. Issues of sequencing The sequencing of such reforms is critical. i. Ideally a reflection on the role of the state needs to precede other reform steps (as divestiture and restructuring become much more difficult once acquired rights increase), both as a way to better frame strategic options (what kind of state and what should it provide, how should this be funded) and in order to identify some of the savings that may finance the cost of system improvements. Without such a reflection, fiscal risks will increase considerably. Brazil, Tanzania and Russia all went through a process of reflection on this and a related ‘right-sizing’ process at the onset. ii. Strategic frameworks and legislation follow, as well as elements of reform of structures and business process, which may or may not including pay enhancement packages. As far as implementation is concerned, at this stage strategic and selective reward mechanisms to drive reform processes can be useful, but preferably these should be limited to critical change management positions. Generally these reward mechanisms do not generate visible results on service quality. An example is the Tanzania Select Accelerated Salary Enhancement process (SASE) (even if it did not last). Some Russian regions also applied this practice. iii. Improving service delivery should follow the introduction of strategic frameworks and legislation. Approaches include linking performance appraisal to service delivery results, introducing complaints and feedback mechanisms, and re-engineering service delivery processes. It is at this stage where ideally performance-related-pay awards could be introduced across the civil service to induce performance improvements. Examples include, the Tanzania Medium Term Pay Reform strategy (July 2010), the second phase of pay reform in Russia (under the New Pay System initiated in 2010), and performance related pay in Brazilian states. However, the proportion of performance related pay The World Bank 3 relative to base pay is extremely important on this issue. This also needs a specific reflection on how to address local level systems. While it is possible to successfully manage reforms without going through this, missing step 1 is likely to drive up fiscal costs (and make reform difficult to sell in the end, as well as non-sustainable), and combining steps 2 and 3 is likely to lead to losing sight of service delivery improvements. Nonetheless, political constraints will always exert pressure to achieve rapid progress. V. The risk of technocratic bias The discussion on technocratic bias is the next critical element. Administrative reforms often fail when they are too ‘internal’ in their approach and do not sufficiently link to citizens. Technocratic bias risks are particularly high on civil service reform issues, as they often take a trade union focus rather than a citizen/client focus. This is particularly strong in areas where professional standards are critical and where performance measurement is controversial (in particular education) and has given rise to diverse and conflicting results. Technocratic bias is less of an issue with performance management reforms connected to investment climate and business environment considerations. However, even in those areas, functional review and business process reengineering often fall afoul of an introverted orientation. The importance of linking reforms as early as possible to sector and overall government results is critical. This will be further discussed under monitoring and evaluation (M&E), but the technocratic bias issue is an important enough risk in its own right to be flagged up front. VI. De-coupling central and sub-national grand design reforms In terms of risk of reform failure, international experience shows that reforms managed across levels of government in large and complex states need a diversified approach to succeed. A case study of seven countries (five OECD countries and two of the BRIC group), conducted in 2006 shows the complexities inherent in managing administrative and civil service reforms across levels of government and the critical lessons to be taken into account.1 International practice emphasizes the importance of addressing reform needs at the sub- national level through tailor made instruments. In some cases, the type of state organization (federal, semi-federal, decentralized unitary, unitary) makes this a requirement, since federal and semi-federal systems do not allow for imposed reforms. Even in unitary systems there is a strong argument for an approach that is based on competition, incentives and/or the application of indirect constraints, rather than direct financing or imposition. 1Tony Verheijen, Yelena Dobrolyubova, Yulia Shirokova, Maya Gusarova and Natalia Ivanova. 2006. Russian Federation Institutional Reform in Russia: Moving From Design to Implementation in a Multi- Level Governance Context. Report No. 35576-RU. World Bank, Washington, DC. The World Bank 4 Based on the above principles, two distinctive approaches can be looked into. One is the use of incentives and competition between regions, with the two most relevant cases from middle income countries in transformation being Russia and Nigeria. Both countries use a system of financing that is built on incentives and competition. Nigeria, in addition, uses fiscal responsibility legislation to impose ceilings on state level expenditures, akin to the approach used in Brazil (see Box 1). Box 1: Lessons from reforms across levels of Government 1. Ensure Integrated planning and sequencing of reform measures at the regional level Sub-national governments tend to have lower capacity to implement administrative, civil service and budget reforms. Thus, regions will be unable to service competing and conflicting demands from the center. Efforts should be made to achieve integrated action plans for reform implementation, including administrative, civil service and budget reform. 2. Coalition building and networking are the key factors in success Too little has been invested in consultations with the regions on reform implementation. Review of international practice highlights this as a key success factor in reform implementation in multi-level governance systems. Thus, policymakers could draw on the experience of countries such as Brazil, Australia, Canada and Spain to develop institutionalized consultation models. 3. Fiscal incentive systems need to take into account regional disparities in reform and fiscal capacity Competition and fiscal incentives are important potential drivers for reform, but in a context with a high level of disparity (fiscally and capacity-wise) a multi-faceted approach is required. This would include a continuation of merit-based competitions to draw out best practice demonstration examples of reform, along with a parallel facility which would target low capacity regions only, built on the following principles:  Allocation that is based on clear rules and is made only if the regions adhere to certain minimum rules and standards and take certain agreed basic initial steps based on an independent diagnostic  Criteria for the review of applications that are based on specific characteristics of disadvantaged and post-conflict regions. 4. Stepping up diffusion of best practices to enhance methodological support The establishment of best practice networks on civil service reform and performance management over the last year has demonstrated the huge potential that such relatively inexpensive reform tools have for disseminating practices and moving forward in reform implementation. Thus, it is proposed that these tools be expanded to address a broader set of issues. The Canadian experience in using best practice dissemination could provide useful inputs in this regard, as could the less formalized Austrian approach. 5. Capacity building programs: The need to be comprehensive and integrated Investment in capacity building is often limited largely to donor-funded programs. In view of capacity constraints and the complexity of the next stages of the reform program (performance-based budgeting and performance management as well as service standards improvement), the design of a budget funded capacity building program (integrating other donor support) is an urgent and essential step to take. The Brazilian FNAGE program could provide particularly useful input in this regard. 6. Monitoring: Building an outcome-oriented system The development of a well-tailored monitoring system is critical to the ability of the government to adapt implementation approaches along the way. Whereas this is a relatively new aspect of reform management, and thus less international practice is available, there are several tools available to build on (see discussion in section IV below). Based on ‘institutional reform in Russia: Moving from design to implementation in multi-level governance context.’ Source: World Bank, 2006 The World Bank 5 VII. Monitoring and evaluating reforms Measuring the impact of grand design reform programs has some specific features, as its purpose is establishing (positive) impact of reform, rather than traditional sector program implementation. The critical issue in this regard is to look at what kind of progress reforms can be associated with and at what level government wishes to measure this. For this, a link with other performance tracking systems operating in government needs to be made (key performance indicators, baselines, targets etc.) and to be economical with the development of indicator tracking systems. International practice on reform program monitoring generally distinguishes progress at four levels: 1. M&E on reform/process outputs (business processes adapted, rules rewriting, number of jobs evaluated, improved revenue levels); this is quite widely known and has a largely internal significance. While measuring progress at this level is important to feel the pulse of reform processes, it has much less significance when support building is concerned. One tool that has been specifically designed to track progress in reform application, and particularly on performance management is the Common Assessment Framework (CAF) piloted by some 2000 public sector organizations in Europe. The tool was originally for use in EU member states, but is currently applied well beyond its boundaries. CAF is built around the notion of critical reform tools for performance management, and is designed for application to individual institutions rather than across government. Following 10 years of piloting in over 30 countries, the CAF methodology has evolved with time. Initially focused on measuring the initiation of performance tools in different aspects of public management (strategic planning, HRM etc.), and applied mainly in smaller public sector organizations, it has evolved to a more comprehensive tool also applied in larger and more complex organizations. The current version of the process is based on eight criteria for assessing organizational excellence2: results orientation; citizen/customer focus; leadership and constancy of purpose; management by processes and facts; people development and involvement; continuous learning, innovation and improvement; partnership development; and corporate social responsibility. Annex 1 provides further information on the criteria and the subcriteria. Each of these criteria is subsequently assessment through a four step rating: (i) base, (ii) initiation, (iii) realization, (iv) maturity. Each of these steps in turn has a set of 3-4 assessment criteria based on which the organization can be ‘scored’. CAF is based on a combination of self-assessment and external validation processes (either by peers or by clients). The model (Figure 1) can be visualized as follows: 2 Based on www.eipa.nl/caf The World Bank 6 Figure 1: The CAF Model Source: European Institute of Public Administration (www.eipa.nl/caf) While requiring time and capacity, the application of CAF in lower capacity states in Europe, in Russian regions, in Central Asia as well as in African and Latin American contexts shows a strong degree of universality, and it is the only comprehensive internal assessment system specifically designed for the public sector. CAF, or a simplified version of it, is one potential instrument to assess the quality and implementation progress of grand design reform programs. 2. M&E on system outputs (kilometers of roads constructed against planned number, number of schools built, number of cases treated within deadline etc.). Output based reform monitoring tools are, in principle, a step up from the previous set of tools, which mainly measure the introduction/rationalization of processes, with CAF having some output dimensions, but remaining at the process level. Output measurement is widely applied in OECD country contexts as it is a critical element of moving towards any form of performance budgeting. This is a significant tool but it requires important resources. In addition, it is critical that it is seen as objective by those being assessed. Furthermore, the issue of external publication is important, as little is gained if this is retained internally in government. 3. M&E on relative performance: using “actionable indicators” to assess performance. This concerns indicators of change, which break down in two sub- categories, one is process related (such as PEFA indicators on budget management, AGI HRM indicators), a second one is result oriented (scorecards, etc.). See Box 2 for further details. Budget monitoring is commonly used to assess performance on results-oriented output indicators. Internal output monitoring can be supplemented by external output monitoring, under which government can contract NGO/CSOs to monitor output delivery. This is a widely The World Bank 7 used tool in South Box 2: Actionable Indicators Asia (scorecard There are several types of actionable indicators, but for this surveys, direct particular discussion there are two that have particular monitoring surveys), relevance: and increasingly used also in Africa and in  Actionable indicators that address process reform Commonwealth of issues, such as PEFA and HRM AGI indicators. Independent States  Actionable indicators that address reform results, such (CIS) countries. as doing business, global competitiveness index etc.. However, unlike the internal budget Frequently used actionable indicators include ‘doing business’, monitoring process, it as a way to assess the impact of public management is more difficult to modernization on business climate quality, the ‘global apply such tools in the competitiveness index’, again looking into improved performance on business and investment climate. The HDI targeted manner index, while having a high level of aggregation, is a further necessary to establish substantive indicator set that can benefit an assessment of performance patterns. progress in terms of state effectiveness. If indicators at this level are to be used, it is important to 4. M&E on development consider the following questions: outcomes: to assess a) How attributable is progress on the indicator/indicator long term impact and set to the reform (relatively straightforward on ‘doing relevance of reforms. business’ but less so on, for instance, HDI or GCI) or, Outcome-based M&E better, can aspects of the indicator set be linked to is particularly suited grand design reforms? for processes designed b) Can relevant progress indicators be identified that over a long time would cover most critical sectors? period and which can measure real impact. For this, grand design reforms would need to be linked with long term sector strategic indicators and an exercise to identify which indicators can be impacted through such reforms should be conducted. This could include Millenisum Development Goal sub-objectives for social sector ministries, revenue to GDP performance over time for financial sector ministries. The process of identifying these indicators requires extensive dialogue with sectors, through which the linkage between long term sector strategic goals and grand design reforms could be established. This could include the link between adequate and quality staffing of schools and enrollment levels (while enrollment levels are influenced by many other factors, the presence of qualified teachers is one), the link between maternal health indicators and availability of qualified medical staff and the availability of medicines in public health facilities, etc.. The World Bank 8 VIII. Case Studies Russia: competitive access to additional resources Russia has used incentives as a way to drive regional level reforms in two aspects of governance reforms, public financial management and administrative reform. In civil service reform, a more direct approach, based on mandatory framework legislation, was used. In relation to public financial management at a sub-national level, a competitive approach to support improved budget management at sub-national level was initiated in 2000 through a combined technical assistance and competition-based transfer window as a way to address weaknesses in regional level public financial management. Initially based on external financing, the competitive mechanisms was subsequently funded through the state budget and is credited with having brought about a significant improvement in budget and overall public financial management at the sub-national level. A similar approach was used in the second phase of the implementation of administrative reform, under the administrative reform concept adopted in 2005. In this case, an annual envelope of approximately US$ 20 million in annual additional financing for administrative reform proposals submitted by the regions was made available, starting from 2007. These amounts, while relatively small, have proven to be an important reform stimulus, as has the prestige of gaining these through competition. Reform proposals respond to annual priorities under the administrative reform program, which vary in terms of substantive priorities and in relation to national and sub-national sequencing. The logic behind this approach is that it is impossible to centrally manage the implementation of administrative reform across a large territory, but at the same time, it is important for central government to stimulate the implementation of such reforms and to push for quality in the implementation of the reform program. Each region is expected to draw up its own reform program (and funds were made available in the first round of financing to do that), following set priority areas, but additional financing is provided only to proposals which meet specific criteria. Attempts to add to this a facility for low capacity and post-conflict regions have not yet come to fruition, even though this had been deemed essential in terms of reducing disparities in development between higher and lower capacity regions (as lower capacity regions do not typically succeed in competitive approaches and could therefore be excluded from reform processes). More recently, the Russian Federation has started to focus on the regional disparities in capacity and performance, with a particular emphasis on fragile regions. The administrative reform process is further stimulated through the development of a federally defined performance target system (REAPAS) defined by the Ministry of Regional Development. The performance indicator targets set for regional executive authorities (adopted in April 2009) includes, among others, the status of institutional reforms, assessed through the implementation of performance related pay systems. The World Bank 9 Nigeria: competitive and benchmarking based access to resources Nigeria also uses access to reform financing as a way of stimulating reforms at the sub- national level, in this case using donor financing as a carrot. Only those regions that were ready to implement a set of conditions would have access to this financing and the criteria included: (i) A formal request for support submitted to the Federal Ministry of Finance; (ii) Evidence of willingness and readiness to undertake a state PFM performance assessment (PER and PEFA or PEMFAR) and adopt an agreed action plan for addressing weaknesses; data must be available and accessible for the exercise; (iii) Commitment to Public Procurement and Fiscal Responsibility reforms; at least draft bills should be ready in the state; (iv) Fiduciary performance of the state (procurement and financial management) on past/existing donor financed projects3; and (v) Balance in geo-political representation of states. Starting with three states in 2005, the program has subsequently grown to include 12 states in 2010, all of which receive a package of investment and technical assistance. An important further carrot is that access to financing under other World Bank sector programs is conditional on the state participating in the PFM and HRM improvement program, which includes support for payroll and HR management, support for budget system reforms and support to procurement reforms. In Nigeria, this access to finance model is supplemented by fiscal responsibility legislation. Brazil: fiscal constraints and liberalization of framework legislation (allowing for redundancies) A different approach was taken by Brazil, which set fiscal constraints through the fiscal responsibility law and reduced constitutional constraints on redundancies in public service employment system through a constitutional reform package adopted in 1998. The strong signal sent from the federal level induced and facilitated the design and implementation of reforms at state level, though speed and depth have varied significantly between states. Of further importance in the Brazilian case is the combination of top down pressure through fiscal constraints and bottom-up pressure through the introduction of participatory budgeting practices at state and local level, of which Brazil is one of the front runners. While progress at sub-national level in Brazil is uneven, with some states being more advanced in reforms than the federal government and others lagging, the enabling measures at the central level have strongly facilitated sub-national progress. The state of Minas Gerais has been particularly active in this regard. In 2003, they launched an ambitious reform program to introduce a results focus into the public administration focused on increasing efficiency and restoring the fiscal 3 The fiduciary performance of states on the Health Systems Development Project II (HSDP II) was used. The World Bank 10 balance. This entailed restructuring the state’s administration, setting quotas for new civil servant hires, and improving the procurement system. Starting in 2003, the government shifted focus toward strengthening the underlying performance monitoring processes centered around the “State for Results” agenda.4 More recently, the state of Pernambuco has undergone similar reforms, under the model “Todos por Pernambuco” introduced in 2008. The reform effort in Pernambuco focused on more closely aligning planning, budgeting , and the management of government programs. Contrast: what happens if one ignores the local level? The case of Tanzania provides a contrasting case, as long term efforts to modernize and rationalize the state administration have stalled as a result of the lack of attention for sub- national level reforms. The Tanzanian case shows how a lack of coordination between agencies responsible for public service reform (under the President’s Office) and decentralization (under the Prime Minister’s office) led to public sector reforms being largely limited to reforming central agencies (with resources to finance reforms remaining at central level), while little or no progress was made at the service delivery level. This became visible during the second phase of the PRAP program (2010). As citizens did not see the impact of administrative reform, and politicians felt similarly that such reforms ‘did not deliver’, support declined and external funding dried up. This is now being corrected through the inclusion of pilot municipalities in the administrative reform program, and a reduction of financing for reforms in central ministries. Potential lessons learned The above review of lessons learned highlights a few critical lessons: i. Ignoring service delivery levels will derail support for grand design reforms. Even programs initially successful such as the Tanzania programs (long seen as one of the most successful in the Africa region) will not continue to get political support if service delivery results do not follow. Thus a reflection on how to include sub- national government in reform programs is critical. ii. Designing reforms sub-nationally as if one were dealing with central government is unlikely to succeed, especially in large countries and even more so in decentralized governance contexts. Cases where larger countries have effectively stimulated reforms (without directly managing them) have shown better results. iii. A laissez-faire approach, such as applied in Brazil, primarily using enabling legislative measures (but also allowing for sub-national borrowing) has advantages of creating buy-in, but runs the risk of creating gaps between reforming and non-reforming regions. 4 Gita Busjeet. “The State Results-Based Management System of Minas Gerais, Brazil.” World Bank PREM Notes, Number 18, April 2012. The World Bank 11 IX. Conclusions What can be taken away from these lessons? The importance of ‘anchors’ The importance of legal/strategic frameworks as anchors: reforms can move to a certain level without an organizing framework, but eventually this is inevitable. The importance of a legal framework depends in part on administrative and political tradition, but in the end alignment with a new system needs to happen (i.e. a strategic framework is a condition sine qua non, but legal frameworks can follow reforms as their features get better defined). The importance of sequencing Sequencing is an issue that requires thorough analysis. The Tanzania case shows how under-analyzed sequencing issues can impact reform objectives. Similarly reforms in Russia had to go through course corrections as divergent approaches to reform hindered. A number of single agency reform cases (not discussed in detail here), such as Slovakia, Afghanistan, etc. show the risk of heavy investments in reforming one ministry that cannot be replicated because of political, financial or other constraints and the impact of the absence of a solid sequencing strategy. Some critical principles for sequencing:  The importance of recognizing the difference between central ministry reforms (usually heavily focused on internal process aspects) and service delivery level institutions (focused on better delivery).  When instruments for reform are centrally defined: the importance of adapting them to context (strategic planning, performance appraisal and performance based rewards have different features and impacts depending on the level at which they are rolled out; structured feedback mechanisms are most important where citizens interface directly with public services). Brazil and Russia are positive examples, while reforms in Uganda and Tanzania reflect the cost of ignoring these realities.  Ensuring that capacity and political support are built up in advance of roll out.  Allowing enough time - experience shows that reforms take 15-20 years to gestate and that forcing implementation into overly ambitious targets is likely to lead to early reform demise (Tanzania).  Central support capacity: without sufficiently developed central support capacity implementation is likely to fall short of target. The World Bank 12 References Brazil Busjeet, Gita. “The State Results-Based Management System of Minas Gerais, Brazil.” World Bank PREM Notes, Number 18, April 2012. OECD. 2010. OECD Reviews of Human Resource Management in Government Brazil 2010 FEDERAL GOVERNMENT. Paris, France: OECD. Pessoa, Mario. 2010. “Brazil – PFM reforms 1985/2015.” Presentation June, 2010. Velloso, Tarsila and Jeffrey Rinne. 2010. “Managing for Development Results: Improving Public services Delivery in Minas Gerais, Brazil.” Working Paper, World Bank, Washington, DC. World Bank. 2003. “Project Performance Assessment Report Brazil, 1986-2001: Public Sector Management Project (Loan 2721-BR), Private Sector Finance Project (Loan 3268-BR), Rio Grande Do Sul State Reform Loan (Loan 4139-BR), Rio de Janeiro State Reform Privatization Project (Loan 4211-BR), Mato Grosso State Privatization Project (Loan No. 4189-BR).” Report No. 25971. World Bank. 2008. “Project Appraisal Document (Vol I) on a Proposed Loan in the Amount of US$976 Million to the State of Minas Gerais, Brazil with the Guarantee of the Federative Republic of Brazil for a Second Minas Gerais Development Partnership Project.” Report No: 43232-BR. Project ID: P101324. World Bank. 2009. “Implementation Completion and Results Report on a Loan in the Amount of US$8.8 Million to the Federative Republic of Brazil for a Fiscal and Financial Management Technical Assistance Loan.” Report No: ICR00001109. Project ID: P073294. World Bank. 2012. The state results-based management system of Minas Gerais, Brazil, PREM Note Number 18, 2012 Nigeria Adamolekun, L. 2006. Africa: Rehabilitating Civil Service Institutions, Main Issues and Implementation Progress in Raadschelders, J. and Th. A.J. Toonen, The Civil Service in the 21st Century. London, Palgrave Olowu, Dele. 2010. “Civil Service Pay Reforms.” International Review of Administrative Services. 76(4) 632–652. Ugbebor, Gilbert O. 2006. “Improving Performance Appraisal System as a Basis for Promotion and Reward: A Nigerian Case Study.” World Bank. 2009. “Project Appraisal Document on a Proposed Credit in the Amount of SDR 75.8 Million (US$120 Million Equivalent) to the Federal Republic of Nigeria for a Second State Governance and Capacity Building Project.” Report No: 5 1483-NG. Project ID: P097026. Russia Bogetic, Zeljko, William Dorotinsky, Francesco Scaduto-Mendola, and Stepan Titov. 2010. “Russia’s Decade of Public Sector Reforms: Achievements, Challenges, and Opportunities.” Working Paper, World Bank, Washington, DC. The World Bank 13 Dobrolyubova, Yelena. 2004. Performance Indicators for Public Administration Reforms in Russia: Possible Approaches. Gusarova, Maya V. 2011. “Performance Management in Russia.” Presentation April 25, 2011. Gusarova, Maya V. 2013. Russia Knowledge Sharing Case Success. Manning, Nick and Neil Parison. 2004. International Public Administration Reform: Implications for the Russian Federation. Directions in Develoment Series, World Bank, Washington, DC. Tuncer, Baran. 2002. “Russia: Bank Assistance for Public Sector Management and Governance.” Operations Evaluation Department Working Paper 27999, World Bank, Washington, DC. Verheijen, Tony , Yelena Dobrolyubova, Yulia Shirokova, Maya Gusarova and Natalia Ivanova. 2006. Russian Federation Institutional Reform in Russia: Moving From Design to Implementation in a Multi-Level Governance Context. Report No. 35576-RU. World Bank, Washington, DC. Avaiable here. Verheijen, Tony, Yelena Dobrolyubova. 2007. Performance management in the Baltic States and Russia: success against the odds? International Review of Administrative Sciences, Vo. 73, No 2, 205-215 Tanzania Kithinji Kiragu, Rwekaza Mukandala, and Denyse Morin (2004), Reforming Pay Policy: Techniques, Sequencing, and Politics, in Kpundeh, S. and B. Levy, Building State Capacity in Africa, World bank Stevens, Mike and Stefanie Teggemann Comparative Experience with Public Service Reform in Ghana, Tanzania, and Zambia in Kpundeh, S. and B. Levy, Building State Capacity in Africa, World bank Schalkwyk, Andrew and Jennifer Widner. 2012. Creating an Affordable Public Service: Tanzania, 1995-1998. Innovations for Successful Societies Series, Princeton University. World Bank. 2010. “Performance Results and Accountability Project (PRAP) in Support of the Public Service Reform Program – Phase II: Aide Memoire.” P092898. Uganda Kithinji Kiragu, Rwekaza Mukandala, and Denyse Morin (2004), Reforming Pay Policy: Techniques, Sequencing, and Politics, in Kpundeh, S. and B. Levy, Building State Capacity in Africa, World bank Sharma, Natasha and Tim Williamson. 2011. “Performance-Based Budgeting and Budget Monitoring in Uganda.” World Bank PREM Notes 10 (June): 1-12. World Bank. 2010. Public Sector Reform and Capacity Building Unit, Country Department AFCE1, Equity in Public Services in Tanzania and Ugana, Policy Note. The World Bank 14