Report No: AUS0003357 . Estonia Evaluating the influence of local service quality monitoring on citizens and public officials Key findings from an empirical study . March 2024 DIME and Governance Global Practice | Bureaucracy Lab The work underlying this report was financed by the EU in collaboration with the EC's DG REGIO. \ © 2023 The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved This work is a product of the staff of The World Bank. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Attribution—Please cite the work as follows: “World Bank. 2024. Evaluating the Influence of Local Service Quality Monitoring on Citizens and Public Officials: Key Findings from an Empirical Study. © World Bank.” All queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Acronyms BL Bureaucracy Lab (World Bank) EC European Commission LG Local Government MoF Estonian Ministry of Finance NGO Non-Governmental Organization QVSR Quadratic Voting for Survey Research RCT Randomized Control Trial WB World Bank Contents Acknowledgements........................................................................................................................................ i Executive summary ....................................................................................................................................... 1 Chapter 1: Service delivery monitoring across local governments in Estonia .............................................. 6 1.1 Context ................................................................................................................................................ 6 1.2 Objectives of the report ...................................................................................................................... 7 Chapter 2: Conceptual framework and empirical approach ........................................................................ 9 2.1. Conceptual framework ...................................................................................................................... 9 2.2. Empirical approach .......................................................................................................................... 10 4.2 Citizens’ survey.............................................................................................................................. 10 4.3 Public officials’ survey ................................................................................................................... 12 Chapter 3: Main findings............................................................................................................................. 15 3.1 The demand side: citizen perceptions of their LG and engagement with the dashboard ............... 15 3.1.1 Citizens’ evaluation of their LGs................................................................................................. 15 3.1.2 Priorities of citizens .................................................................................................................... 18 3.1.3 Citizens’ uptake of the dashboard ............................................................................................. 24 3.2 The supply side: public officials' utilization of and engagement with the dashboard in decision- making..................................................................................................................................................... 25 3.2.1 Public officials’ evaluation of their LGs ...................................................................................... 26 3.2.2 Public officials’ uptake of the dashboard................................................................................... 31 3.2.3 Perceived usefulness of the dashboard by public officials ........................................................ 34 3.3 The state of the service: the environment in which public officials make decisions ....................... 36 3.2.1 Attitudes of public officials ........................................................................................................ 37 3.2.2 Management of LGs ................................................................................................................... 39 3.2.3 Capacity of public officials ......................................................................................................... 44 3.2.4 Work environment of public officials......................................................................................... 46 3.4 Was the dashboard acted upon? ...................................................................................................... 49 3.6 Bringing it together ........................................................................................................................... 51 3.7 Lessons from public officials ............................................................................................................. 52 3.7.1 Lessons from top-performing LGs.............................................................................................. 52 3.7.2 Lessons from consultations on the dashboard .......................................................................... 53 Chapter 4: Policy recommendation ............................................................................................................ 56 4.1 Enhancing citizen engagement with the dashboard................................................................... 56 4.2 Supporting the interpretation of the data on the dashboard .................................................... 56 4.3 Encouraging public officials to engage with citizens in the policy narrative .............................. 57 4.4 Long-term plan to enhance engagement with and improvement of the dashboard ................. 58 4.5 Continue expanding the scope of the dashboard ....................................................................... 59 Annexes ....................................................................................................................................................... 60 Annex 1: Timeline of the project ............................................................................................................ 60 Annex 2: Research Appendix .................................................................................................................. 61 Annex 3: Minuomavalitsus Dashboard ................................................................................................... 63 Annex 4: Methodological Notes ............................................................................................................. 64 4.1 Study samples ............................................................................................................................... 64 4.2 Quadratic Voting for Survey Research .......................................................................................... 64 List of figures, boxes and tables Figure 1. Exemplary view of the Minuomavalitsus dashboard ..................................................................... 7 Figure 2. The Dashboard ‘Production Function’ ........................................................................................... 9 Figure 3. Citizens’ Survey Response Rates Across LGs ................................................................................ 11 Figure 4. Demographics of the 2021 citizen survey .................................................................................... 11 Figure 5. Public Officials’ Survey Response Rates Across LGs..................................................................... 13 Figure 6. Demographics of the 2022-2023 public officials’ survey ............................................................. 14 Figure 3. Citizens’ satisfaction levels with their LG ..................................................................................... 16 Figure 4. Citizens' satisfaction with their LGs across regions ..................................................................... 17 Figure 5. Citizens' perception of the alignment between their priorities and those of local governments .................................................................................................................................................................... 18 Figure 6. Citizens' perception of the alignment between their priorities and those of local authorities across LGs.................................................................................................................................................... 19 Figure 7. Citizens' perception of the alignment between the priorities of local residents and those of local authorities ................................................................................................................................................... 20 Figure 8. Citizens' perception of the alignment between the priorities of local residents and those of local authorities across LGs ................................................................................................................................. 21 Figure 9. Distribution of "credits" allocated by citizens during the QVSR module ..................................... 22 Figure 10. Correlation between the QVSR scores allocated by citizens and service quality scores from the dashboard by sector ................................................................................................................................... 23 Figure 11. Views of the dashboard across time .......................................................................................... 25 Figure 12. Public officials' report on the prioritization of sectors and perceived preferences of citizens across sectors .............................................................................................................................................. 27 Figure 13. Public officials' report on the prioritization of sectors and citizens' own preferences across sectors ......................................................................................................................................................... 28 Figure 14. Performance of the sectors across LGs as reported by public officials ..................................... 29 Figure 15. Public officials’ report on the extent to which they possess the necessary information across LGs............................................................................................................................................................... 30 Figure 16. Public officials’ report on the extent to which a benchmarking tools would be useful across LGs .................................................................................................................................................................... 31 Figure 17. Information sources used by public officials on the performance of LGs ................................. 32 Figure 18. Frequency with which the dashboard was used........................................................................ 33 Figure 19. Variation in the extent to which public officials use the dashboard across LGs ....................... 34 Figure 20. Benefits of using the dashboard ................................................................................................ 35 Figure 21. Variation across LGs in the extent to which public officials report the dashboard has improved information flow ......................................................................................................................................... 36 Figure 22. Change in the motivation of public officials compared to when they joined the service (relative to 100) ......................................................................................................................................................... 38 Figure 23. Salary satisfaction levels across LGs .......................................................................................... 39 Figure 24. Management quality indices across LGs .................................................................................... 41 Figure 25. Overall management quality index across LGs .......................................................................... 41 Figure 26. Performance evaluations across LGs ......................................................................................... 43 Figure 27. Usage of performance evaluation results .................................................................................. 44 Figure 28.Percentage of public officials receiving training across LGs ....................................................... 45 Figure 29. Perceptions on trainings among LG public officials ................................................................... 46 Figure 30. Perceptions on communication and collaboration with colleagues.......................................... 47 Figure 31. Perceptions on trust among colleagues..................................................................................... 48 Figure 32. Perceptions on the support among colleagues in coming up with innovative ideas ................ 49 Figure 33. Changes in service quality in the worst-performing service areas pre- and post- dashboard .. 51 Box 1. Impact evaluations in policy making ................................................................................................ 50 Box 2. Quadratic Voting for Survey Research (QVSR) details ..................................................................... 65 Acknowledgements This report is the product of the Bureaucracy Lab, a partnership between the Governance Global Practice and the Development Impact Evaluation Department (DIME) of the Research Group. It is part of a multi- country collaboration with the European Commission (EC), funded via the Part II Europe 2020 Programmatic Single-Donor Trust Fund with the EC (TF073353), the objective of which is to empirically understand the personnel determinants of, and mechanisms influencing, productivity in public administration and service delivery units in EU Member States. The report was prepared by a WB team led by Daniel Rogger (Senior Economist) and Zahid Hasnain (Lead Governance Specialist), and consisting of Ayesha Khurshid (ET Consultant), Robert Lipinski (Consultant) and Katre Väärsi (Consultant). The team also benefited from valuable support on the survey design from Michael Best (Assistant Professor at Columbia University). Valuable guidance and advice were received from a group of researchers: Alessandra Fenizia (George Washington University), Gianmarco Leon Ciliotta (Universitat Pompeu Fabra), and Christian Schuster (University College London). The team would like to thank Roby Senderowitsch (Practice Manager, Governance Global Practice, Europe and Central Asia), Pedro Arizti (Senior Public Sector Specialist), and the Governance ECA Ops Team for their overall advice, guidance, and support throughout the development of this report. The team would like to express its gratitude to government officials of the Ministry of Finance (MoF) — Karl Annus, Ats Aasmaa, Andreas Aljas, Andrus Jõgi, Kaie Küngas, Mari Kalma, and Piret Zahkna. Special thanks go to officials from all the public institutions who supported the implementation of the survey and who took the time to respond to the questionnaire. From July 1, 2023, the relevant counterparts were part of the Ministry of Regional Affairs and Agriculture. Finally, we are grateful for the close collaboration and continuous support of officials from DG REGIO and DG REFORM of the EC, particularly Lewis Dijkstra, Philippe Monfort, and Mina Shoylekova. i Executive summary Main findings Through the Open Government Partnership, Estonia’s central government have worked with civil society to create a ‘policy dashboard’, known as the Minuomavalitsus dashboard, that compares levels of service delivery across 79 Estonian local governments (LGs) on an annual basis.1 It scores each LG along 20 dimensions of local public service delivery, ranging from broadly defined governance to more specific areas like libraries, using more than 300 sub-indicators and criteria (see Figure 1 of the main report for a view of the dashboard). Each of the 16 dimensions was evaluated on a 0-9 scale (with 9 being the highest level of quality), depending on how many of underlying quality criteria were met. To examine the influence of the LG dashboard on both citizens and public officials, the WB Bureaucracy Lab collaborated with the MoF to implement two large-scale surveys, at the citizen and LG public official levels. These surveys were designed in line with a conceptual framework, to better understand the demand and supply side channels through which the dashboard informs service delivery, and the overarching environment in which public officials establish policy. By comparing the service delivery data on the dashboard with the survey of citizens, we find that local governments with better services have overall more satisfied citizens. There exists a positive correlation between the satisfaction levels described above and the quality of local services provided by a LG , as measured by the dashboard. This implies there is a rationale for the dashboard to assess service levels across LGs and stimulate greater investment by LG leadership and officials in service areas in which they are relatively weak. Citizens have varied preferences as to what a local government should be prioritizing and how well they think their local government is doing in addressing these priorities. Similarly, public officials have varied beliefs as to what they think the citizens in their LG want the government to prioritize across local governments. The data reveals notable disparities in the self-reported performance of sectors within LGs by public officials. Together, these findings imply that there is not a general agreement in many localities as to what should be prioritized and how well government is working in those areas. Beyond variation across local governments, we find that exists significant variation in perceptions of performance and priorities within LGs themselves, indicating that even public officials within the same jurisdiction may not possess consistent, or even accurate, information regarding the relative importance or performance of their own sectors. This makes a strong rationale for the dashboard as not simply a presentation of information, but a means of encouraging a conversation amongst public officials and citizens as to how the local government is faring and what the government’s attention should turn to. Across many local governments, public officials state that they have adopted the dashboard as a key source of information. 57 percent reported using the dashboard multiple times a year to obtain information on the performance of their LG, though there is substantial variation across LGs. It has grown over the past three years to become an important source of information for staff of the local government 1 The dashboard is publicly available at https://minuomavalitsus.ee/. 1 and already overtaken some key sources of information such as formal field visits and formal citizen interactions as a source of information on LG performance. This is a real success of the dashboard, and we find evidence that it has become a part of local government staff's policy toolkit. However, citizens have engaged far less with the dashboard or its implications. The numbers of visits to the dashboard, even during peak times, is not particularly large. When directly asked if they knew of the dashboard webpage using the citizen survey, only 4 percent of citizens reported being aware of it. It is understandable that not everyone is interested in LG matters, and therefore are unlikely to hear (or if they do, remember) a dashboard of this type. However, such a low percentage of citizens being are aware of the dashboard indicates limited public awareness. There is a modest level of agreement on the impact of the dashboard on information exchange between citizens and LGs, with significant variation across LGs. On the other hand, a majority of public officials find the dashboard useful for their work, particularly for benchmarking the performance of LGs and identifying areas for improvement. The variation on the uptake and utility of the dashboard across LGs suggests that the administrative environment within LGs seems to be playing a significant role in shaping the perceived impact of the dashboard. Intrinsic and extrinsic factors, such as public service motivation and wages, significantly shape the behavior and effectiveness of public officials in shaping policy. Public officials across a majority of LGs report a decline in their motivation compared to when they joined the service. The survey identified extrinsic factors like wages, management quality, and service conditions as most commonly associated with lower motivation levels among public officials. To better understand the low levels of motivation, the study find that management quality varies significantly across LGs, with an average score of 3.5 on a scale of 1 to 5. Furthermore, specific management quality indicators show variation across LGs, with the highest composite score of 3.8 for effective organizational targeting, 3.18 for good performance management practices, and the lowest score of 3.0 for recruitment practices. This showcases the extent to which management is decentralized and the way in which management practices are inconsistently implemented across offices . To better understand the low scores on performance management practices across LGs, the survey finds that only 58 percent of public officials reported undergoing formal performance evaluations in the past two years, with substantial variation observed across LGs in this practice. Performance evaluations primarily serve to assess training needs (42 percent), while a smaller percentage use them for awarding bonuses (17 percent) or setting pay (15 percent). The lack of standardization and weak accountability systems may negatively impact public officials' motivation and performance. The utilization of performance evaluations to inform training needs can improve capacity building initiatives that support the adoption of tools like the dashboard. The study finds that a significant majority of public officials (86 percent) have received training within the last year. However, there is variation in training rates across LGs, indicating potential disparities in its accessibility. Leveraging this foundation, integrating components in trainings related to the adoption and utilization of the dashboard presents a significant opportunity for further enhancing public officials' skills and knowledge in their daily work. To support the integration of the dashboard in the day-to-day work of public officials, the survey finds that there exists a strong culture of collaboration and innovation within LGs, with 89 percent of public 2 officials reporting efficient communication with colleagues and 62 percent acknowledging clear delineation of responsibilities. Furthermore, the survey emphasizes a culture of innovation, with 76 percent of public officials reporting support for developing new approaches to work. The existing collaborative culture and openness to new ideas and technologies create a conducive environment for the adoption of the dashboard in the daily work of public officials. The dashboard's practical value as a policy tool requires evaluating on-the-ground changes in Estonian local governments, going beyond survey responses. Evaluating the dynamic improvement in service quality is complex, and greatly influenced by information availability. To determine the dashboard's impact, the WB's Bureaucracy Lab is conducting an impact evaluation study using a Randomized Control Trial (RCT). Some initial descriptive evidence does suggest the potential role of the dashboard as a policy tool. While the change is not consistent across all local governments, the trend indicates the dashboard's potential use in identifying and addressing urgent policy problems in certain areas. Recommendations Enhancing citizen engagement with the dashboard  Conduct a comprehensive awareness campaign to introduce the dashboard to citizens, media, and civil society organizations to emphasize its benefits in promoting transparency, accountability, and citizen participation in governance.  Foster strong partnerships with civil society organizations by involving them in the dashboard's development and governance. This will allow for the inclusion of their perspectives and needs through consultations and feedback mechanisms.  Showcase how the dashboard has influenced policy decisions and improved governance outcomes, integrating it into policy discussions, speeches, and official documents.  Establish mechanisms for feedback on the dashboard's utility, data accuracy, and relevance. Regularly update the dashboard based on user feedback and evolving needs to enhance engagement and usefulness. Supporting the interpretation of the data on the dashboard  To support management decisions, the dashboard can provide clear and concise information in an intuitive format for easy interpretation by public officials.  Provide recommendations or action plan templates alongside the displayed data, based on best practices and expert insights, to facilitate managerial actions and enables decision-makers to translate insights into policy actions.  Develop a community of practice with professionals, experts, and stakeholders helps bridge the gap between the interpretation of data on the dashboard and its utility for the work of decision- makers.  Enhance the dashboard's impact on policy decisions by creating an ecosystem that links dashboard utilization to existing policy evaluation systems and performance management processes for public officials. 3  Customize the dashboard to meet the specific needs and preferences of user groups such as the public, local governments, mayors, and staff, providing differentiated user access to align with their preferences and priorities.  Utilize the dashboard's ability to disaggregate data and provide granular insights into specific groups, enabling policymakers to address unique needs and challenges through targeted interventions and policies.  To support understanding and interpretation by time-constrained officials, provide cyclical performance reports derived from the dashboard that extract relevant data and insights into the strengths, weaknesses, and overall performance of local governments in accessible short and long formats.  Shift the focus of workshops on the dashboard towards active utilization, management, and fostering ongoing conversations on policy and performance to effectively leverage its potential in operations and decision-making.  Leverage individual consultations with local government officials to implement a monitoring framework that discusses and measures actions taken in response to survey findings, incentivizing improvements based on the dashboard's utilization. Encouraging public officials to engage with citizens in the policy narrative  Actively engage citizens in policy discussions based on the dashboard's findings enhances democracy and informed decision-making. The reports and workshops provided by the MoF to local officials can play a pivotal role in highlighting the importance of citizen engagement in the decision-making process.  Incorporate a citizen engagement module in these resources to create a structured approach for involving citizens, ensuring their perspectives are included in policy discussions.  Create special reports that highlight key dashboard findings to foster citizen engagement , capturing their attention and stimulating interest in exploring the data further. Long-term plan to enhance engagement with and improvement of the dashboard  Formulate a comprehensive long-term plan for the dashboard, involving stakeholder engagement, performance metrics, and continuous development to enhance its usage and capabilities.  Establish clear objectives, monitor impact, foster innovation, and develop a framework for tracking effectiveness and incorporating new features to maximize the dashboard's long-term value.  Develop a community of practice to encourage engagement , support officials in utilizing the dashboard, and foster continuous learning and improvement.  Strengthen tailored performance reports aligned with local government needs to enhance engagement and maximize the dashboard's value.  Leverage positive survey findings to highlight the dashboard's benefits for informed decision- making and efficient governance. 4  Understand non-government actors' perspectives, tailor features to their needs, and enhance their utilization of the dashboard through targeted outreach and engagement.  Integrate an evaluation toolkit to systematically gauge the dashboard's performance , usability, and impact.  Incorporate an annual reflection point to evaluate and align the dashboard with evolving organizational goals and priorities for enhanced effectiveness and relevance. Continue expanding the scope of the dashboard  Sustain the MOF's commitment to expanding and adapting the dashboard to changing needs, as exemplified by the inclusion of new indicators in the 2022 version.  Enhance the dashboard's flexibility to effectively respond to future major shocks and uncertainties, enabling stakeholders to make informed decisions during challenging times.  Incorporate a system for regular assessment of engagement, ensuring ongoing optimization and alignment with evolving stakeholder needs.  Implement an automated system to analyze policy priorities discussed in local council meetings , providing real-time insights and keeping stakeholders informed about the latest developments. 5 Chapter 1: Service delivery monitoring across local governments in Estonia 1.1 Context Administrative divisions are far more than the lines drawn on maps. People living in different administrative units, be it counties, local governments, or city districts, often lead very different lives. They also experience different quantities and qualities of public goods, be they green spaces, bicycle paths, public transport, waste management or educational facilities. The lives of most people begin to be affected by the quality of local governance from early on in their lives, as kindergartens, schools, and playgrounds are all at least partially managed by local authorities. This interaction continues later on as LGs takes responsibility for provision of household utilities, like water and heating, cultural and sport facilities and promotion of public health. And for many, care or elderly homes become one of the necessary locally provided services at an older age. In Estonia, the local government (LG) structure is composed of 79 LGs. These LGs wield considerable autonomy, deciding and organizing numerous of local issues independently. Despite the diversity in size and demographics across these municipalities, a noteworthy aspect is that all LGs share a common mandate. They are obligated to implement the same tasks and provide an identical range of services to their inhabitants. This uniformity, irrespective of the municipality's size or urban-rural distinction, underscores a commitment to ensuring equitable access to essential services for all residents. 2 At the same time, the European Commission has pointed out in its yearly country specific recommendation that the levels of the services provided by local governments in Estonia (LG) vary substantially.3 Furthermore, there exists little information available that would enable citizens and policymakers at local and central government levels to understand and fully grasp the differences in service levels. This information is essential for public administrators at both local and central government levels to address and effectively improve disparities in services across LGs. It is also necessary for the citizens to be able to consciously formulate their policy priorities and reform demands. Novel information on local government service levels can inform and shape the efforts of local public officials in delivering public goods directly. By observing new information on the relative quality of public services over which they have influence, public officials may shift their investments towards improving particular services. At the same time, novel information on service levels may generate indirect pressure on public officials from citizens. Newly available information on service levels can drive citizens to become more informed about and engaged with public policy. By leveraging local accountability mechanisms over public officials, informed citizens can shape the nature of policy making in their local constituency. In Estonia, through the Open Government Partnership, civil society and the central government of Estonia have created a policy dashboard, known as the Minuomavalitsus dashboard (hereafter ’dashboard’) comparing government service levels across 79 Estonian LGs on an annual basis since 2016. 4 Hundreds of government officials, academics and NGO representatives worked together to bring about a dashboard 2 Ministry of Regional Affairs and Agriculture, Republic of Estonia. 3 European Commission. (2022). 2022 European Semester: Country Specific Recommendations - Estonia. 4 The term ‘LG’ is used throughout the report to refer to one or all of the 79 Estonian tier-2 administrative units. Minuomavalitsus is Estonian for ‘My Municipality’ 6 scoring each municipality along 20 dimensions of local public services delivery, ranging from broadly defined governance to more specific areas like libraries, using more than 300 sub-indicators and criteria (see Figure 1 below for a view of the dashboard). Each of the 20 dimensions was evaluated on a 0-9 scale (with 9 being the highest level of quality), depending on how many of underlying quality criteria were met. Figure 1. Exemplary view of the Minuomavalitsus dashboard Governance quality across 79 Estonian LGs in 2021 Source: https://minuomavalitsus.ee In August 2020, the first version of the dataset, and the dashboard, became publicly available and has been updated regularly ever since at the dashboard’s website.5 This created ground for discussions on what are society’s expectations of the service levels, what are the differences in these levels that are acceptable, and what should be changed. In theory, this in turn is expected to increase LGs’ accountability and nudge them to provide better services and make their processes more effective. It also enables the central government to better target its intervention instruments and management by improving their knowledge of both the absolute and relative condition of the different public services in their locality. 1.2 Objectives of the report Despite its publication in 2020, the effectiveness of the dashboard as a policy instrument has yet to be comprehensively evaluated. This report takes a significant step towards assessing the effectiveness of the dashboard by examining its influence on citizens and local public officials. Through this, the report 5 For a complete list of indicators, and an example of results from the dashboard across sectors and LGs, see Annex 3. 7 seeks to understand the perceptions of citizens and public officials towards service delivery at the LG level. To that end, it sets out to answer several interrelated questions about the dashboard as a:  Measurement tool: What is the level of satisfaction of citizens and local public officials with the different sectors of their LG’s functioning? To what extent it is related to the quality of those sectors, as measured by the dashboard?  Diagnostic tool: What are the priorities of citizens and local public officials? How do they compare with each other and with the quality scores of different public services sectors as measured by the dashboard?  Policy tool: Are citizens and local public officials aware of the dashboard? If yes, are they using it to inform their activities when it comes to local governance? Is the LG environment conducive to the adoption of the dashboard as a policy instrument? The Ministry of Finance (MoF) in Estonia has invited the World Bank's (WB) Bureaucracy Lab to conduct an empirical assessment of the dashboard's effectiveness in LGs. This assessment utilizes two large-scale surveys conducted between 2021 and 2023, which gathered data from representative samples of Estonian citizens and local public officials across 79 LGs.6 It is our hope that by addressing these questions, the report can achieve a threefold benefit. One, by demonstrating the dashboard’s utility in the Estonian context, the report aims to provide an empirical basis to sustain the theoretical and practical efforts going into the maintaining, updating, and expanding the dashboard in Estonia. Secondly, by assessing the LG environment and the challenges faced by public officials, the report aims to identify areas for improvement within the administration that can enhance the adoption and impact of similar initiatives. Third, by carefully documenting the data gathering process, the theoretical mechanisms at work, as well as the effects, the report can help to promote the creation of new projects and inform the adaptation of the existing ones, which would seek to measure the quality of local public services delivery in other countries. 6 Three LGs are not included in the citizens’ survey (the first of the two surveys), both due to their small populations and small island status: Kihnu Vald, Ruhnu Vald, and Vormsi Vald. 8 Chapter 2: Conceptual framework and empirical approach 2.1. Conceptual framework The study design is based on a conceptual framework of a production function of local service delivery (see Figure 2). A production function is the process by which “inputs,” or the resources available to an organization, are ultimately converted to outcomes that citizens care about. In the context of the dashboard in Estonia, exposure to the dashboard, which provides citizens and public officials with information about the performance of LGs is the input that is ultimately converted into improved policy outcomes at the LG level. The production function for the LG dashboard in Estonia identifies two main channels through which exposure to the dashboard leads to improved policy outcomes: the demand side and the supply side. On the demand side, exposure to the dashboard leads citizens to shift their beliefs about the performance of their LGs. Specifically, citizens become more aware of the strengths and weaknesses of their LGs and become more engaged with local public officials. This increased engagement allows citizens to voice their concerns and priorities, which can influence local policy decisions. Additionally, as citizens become more informed about LG performance, they may be more likely to hold officials accountable for their actions, further incentivizing officials to prioritize policies that are in the public interest. On the supply side, exposure to the dashboard leads public officials to shift their beliefs about the performance of LGs. Specifically, officials become more aware of the strengths and weaknesses of their own LGs, as well as those of competing LGs. This increased awareness can influence the policy priorities of officials, as they seek to improve the performance of their own LG and compete with other LGs, be it for intrinsic or instrumental reasons. As a result, officials may prioritize policies that are more aligned with the needs of citizens and that can improve the overall service delivery within the LG. Figure 2. The Dashboard ‘Production Function’ Source: Authors’ own visualization 9 In addition to the supply and demand side channels, the environment and culture within LGs will play a crucial role in the effective utilization of the dashboard as a policy tool. It is essential to explore the internal dynamics and context within LGs in which public officials design and implement policies for effective service delivery. For the dashboard to be effectively utilized, the environment needs to be conducive to its adoption and integration into policy processes. This includes factors such as motivation, management, organization, collaboration, and the willingness to embrace data-driven decision-making. Understanding and addressing these environmental factors are vital to ensure the successful implementation and utilization of the dashboard for policy purposes. 2.2. Empirical approach To examine the influence of the LG dashboard on service delivery, the WB Bureaucracy Lab collaborated with the MoF to implement two large-scale surveys, at the citizen and LG public official levels. These surveys were designed in accordance with the conceptual framework established earlier (see Figure 2), to better understand the channels through which the dashboard informs service delivery, and the environment in which public officials establish policy. 4.2 Citizens’ survey The citizen’s survey involved a random sample of 100 citizens in 76 LGs, drawn from the Estonian population register, as implemented by the MoF.7 The citizen survey was divided into three modules:  The first module asked basic demographic questions.  The second contained questions about LG management and satisfaction, asked using Likert or binary response options to understand the extent to which citizens are satisfied with service delivery across sectors in their LGs.  The final module implemented the Quadratic Voting for Survey Research (QVSR) module to elicit citizens' preferences about different dimensions of LG performance displayed on the dashboard. For more details on QVSR, please see Annex 4. To complement direct survey evidence of the impact of the dashboard, we also used information from web site visit tracking, the dashboard indicator data, and local government budget data. The survey was implemented online, between June and October of 2021 by Kantar Emor, a survey firm based in Estonia. The target was to survey 100 respondents in each municipality , with the exception of Tallinn, where, due to a much larger population size, the target was 300 responses. In total, a sample of 7,219 people from across 76 LGs provided their responses. The number of respondents in a municipality varied between 50 in Narva-Jõesuu Linn and 397 in Tallinn. Figure 3 provides an overview of a full spatial variation in the response rates. 7 Three LGs are not included in the citizens’ survey, both due to their small populations and small island status: Kihnu Vald, Ruhnu Vald, and Vormsi Vald. 10 Figure 3. Citizens’ Survey Response Rates Across LGs Source: World Bank’s Citizens’ Survey in Estonia (2021) The survey sample predominantly consisted of Estonian respondents, accounting for 91 percent of the participants. Within the sample, female respondents were slightly more represented, comprising 59 percent, while male respondents accounted for 41 percent. Furthermore, individuals with below university education constituted 58 percent of the sample, compared to those with a higher education diploma, who constituted 42 percent of the participants (see Figure 4). These demographic characteristics provide a snapshot of the composition of the citizen survey sample to help contextualize the findings across specific population groups. Figure 4. Demographics of the 2021 citizen survey Source: World Bank’s Citizens’ Survey in Estonia (2021) 11 Finally, it should be noted that the survey was concluded shortly before recent LG elections, held in October 2021. As such, its results should not be viewed as an evaluation of the performance of the current local authorities, whose mandate started only after they survey had finished collecting citizens’ feedback. Rather, it should be seen as a potential input in deciding on reforms and new policy directions by the incoming local decision-makers. 4.3 Public officials’ survey To obtain insights into the supply-side effects of the LG dashboard, we implemented the second survey on a stratified random sample of public officials in each of the 79 LGs.8 The survey was divided into three modules: 1. The basic demographic characteristics of public officials across LGs 2. Public officials' use and perception of the dashboard. By gaining insights into their experiences of the dashboard, this module seeks to shed light on the effectiveness of dashboard in facilitating data-driven decision-making and enhancing transparency in local governance 3. The organizational and personnel management practices, skills, attitudes, and behaviors of public officials across LGs. These factors have been identified as crucial determinants of public administration productivity based on academic literature and WB engagements. We ensured that the survey methodology effectively captured these factors, providing valuable data for analyzing the effects of the dashboard on LG policy outcomes. The survey's sampling frame was created by the WB team using information available on the websites of each LG. The WB team scraped the names, roles, phone numbers, and email addresses of public officials working in the LGs. To categorize the officials, the WB team manually assigned them to sectors based on their roles and departments, and according to the sectors listed on the dashboard. Additionally, officials were classified into Manager, Specialist, Employee, or Support Staff roles. Support Staff were excluded from the sampling frame due to their limited involvement in decision-making within LGs, aligning with the study's objectives. Once the frame was finalized, the sample was drawn. For LGs with fewer than 20 officials, all officials were included in the survey. In LGs with more than 20 officials, a stratified random sampling approach was used to ensure adequate representation of LGs across sectors. The public officials’ survey was implemented by a local survey firm, Kantar Emor, and was implemented over the phone with prior appointments from public officials. The survey began in November 2022 and was completed in April 2023, with extensive support from counterparts in the MoF. The survey received a total of 1121 respondents across 79 LGs in Estonia. While there exists variation in response rates across LGs (see Figure 5), the overall response rate was 79 percent, indicating a strong willingness among LG officials to participate in the study.9 8 The survey of LG public officials was implemented across all 79 LGs. 9 For LGs where the response rate is higher than 100 percent, we were able to interview more respondents than targeted. The is coincidental since the survey firm conducted additional interviews with respondents who had their appointments cancelled or were unreachable, necessitating extra appointments, and interviews, for data collection. 12 Figure 5. Public Officials’ Survey Response Rates Across LGs Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) The majority of public officials that responded to the survey were highly educated (84 percent had a university education), with a small proportion of respondents not having a university education (16 percent). Female respondents were represented at a significantly higher frequency than men (77 percent vs. 23 percent). On average, the respondents were 49 years old, and their tenure of public officials in the civil service was 16 years (see Figure 6). Based on our assessment, the respondents represent a comprehensive demographic of LG public officials in Estonia, enhancing the robustness of the findings of this study and allowing a better contextualization of the findings. 13 Figure 6. Demographics of the 2022-2023 public officials’ survey Source: World Bank’s Public Officials’ Survey in Estonia (2022-2023) 14 Chapter 3: Main findings This chapter presents findings from the data and is structured around the conceptual framework described in Chapter 2, with the first section focused on the demand side of the production function and therefore analyzing citizens’ perceptions and engagement with the dashboard. The second section examines the supply side, documenting the degree to which public officials utilize the dashboard in their decision-making processes and their overall engagement with the tool. Finally, the third section allows us to gain an understanding of the administrative environment across LGs to inform the effective utilization of the dashboard as a policy tool among public officials. 3.1 The demand side: citizen perceptions of their LG and engagement with the dashboard The Minuomavalitsus dashboard indicates substantial variation across Estonia in the level and quality of public service and resulting service outcomes. However, different citizens may value the services they receive and the state of their local government’s public service differently (see a brief discussion in the appendix of such considerations in the research literature). As such, they may be differentially satisfied with different ‘mixes’ of public services. We therefore begin our assessment of the demand side of public service by looking at satisfaction with life in LGs overall, and then assess satisfaction with different services within that. 3.1.1 Citizens’ evaluation of their LGs One of the primary goals of the citizens’ survey was to evaluate their satisfaction levels with LGs. Figure 3 below shows the distribution of responses to several questions asking about this topic from different angles. Overall, a significant proportion of Estonians expressed satisfaction with their LG as a place to live, with 62 percent indicating they were somewhat or very satisfied with the LG as a place to live. Similarly, 54 percent of respondents reported being satisfied with the locally provided services. In terms of whether they are living in their favorite LG, 32 percent of the respondents stated that their current municipality is their preferred place to reside in Estonia. While this percentage may initially seem relatively low, it is worth noting that 53 percent of respondents selected the response option "Don't Know" for this question, suggesting that it may not be a dilemma they commonly consider. When excluding the undecided respondents, the preference for the current municipality increases to 69 percent (see Figure 7). Regarding LG council performance, there was a less positive picture. Approximately half of the respondents agreed that their council is doing a good job (50 percent), is efficient and well-run (43 percent), and provides good value for money (40 percent). Notable variations in responses based on demographic characteristics such as gender, age, or ethnic background were minimal. As such, there would seem to be room to improve the perception of local government. 15 Figure 7. Citizens’ satisfaction levels with their LG Source: World Bank’s Citizens’ Survey in Estonia (2021) 16 Local governments with better services have overall more satisfied citizens. There exists a positive correlation between the satisfaction levels described above and the quality of local services provided by a LG, as measured by the dashboard. Figure 8 shows that the average general satisfaction levels of respondents in a municipality tends to rise together with the average quality of local public services. From Figure 8 below it can also be gauged that the satisfaction levels tend to be lowest in the Northeastern Estonia – a pattern that is observed for most questions in the survey (even though for these LGs the correlation with service quality is positive). Figure 8. Citizens' satisfaction with their LGs across regions Source: World Bank’s Citizens’ Survey in Estonia (2021) Generally, Estonians tend to be moderately satisfied with their LG, but there remains scope for improvement. In particular, given some of the regional differences observed, there seems to be a geographic inequity in the extent to which Estonians are satisfied with their LG as a place to live and the services provided there. The dashboard therefore aims to measure and address an important aspect of citizens' satisfaction levels and provides clear guidance on the policy investments that are needed within a particular local government. Secondly, the results of the survey already indicate the utility of the dashboard as a measurement tool. As citizens in LGs with better quality of local services tend to be 17 more satisfied, the dashboard can continuously be used as a measurement tool to monitor the progress of interventions implemented to improve the quality of local services using a cross-LG panel dataset .10 3.1.2 Priorities of citizens Beyond overall satisfaction, it is important to understand to what extent citizens’ priorities, whatever they might be, are reflected in the decisions of the LG. The citizens survey we ran allows for a broad differentiation between two possible dimensions of question – individual and group-based. When citizens were asked whether they felt that the decisions of the local authorities reflected their priorities, the citizens provide a bleaker view than in the case of satisfaction questions. The distribution of responses to that question can be seen in Figure 9. Only 1 percent of respondents claim that their priorities are ‘completely’ reflected by local authorities’ activities. Even if the criteria are relaxed, only 1 in 4 respondents said that their priorities are reflected ‘completely’ or ‘to a large extent’. 40 percent answered that ‘only in some areas’ do they see a match between their priorities and the decisions made by the local authorities. On the other end of the spectrum, only 18% report ‘never’ or ‘almost never’, which might be seen as somewhat encouraging. Figure 9. Citizens' perception of the alignment between their priorities and those of local governments Source: World Bank’s Citizens’ Survey in Estonia (2021) 10 Panel refers to the fact that the same LGs are observed across multiple units of time (here years) which allows for temporal comparisons to be made. 18 There is minimal variation in respondents' views on this question across LGs (see Figure 10). In the majority of LGs, an average of 20 percent to 40 percent of respondents choose the top two responses (citizen priorities are "completely" or "to a large extent" reflected in the decisions of their LGs). Only in a few LGs does this average fall below 20 percent. Similarly, a similar share of LGs has response averages above 40 percent, with only Muhu Vald showing an average above 60 percent. These figures imply that there is substantial room for LGs across Estonia to engage with citizens on their decision making to either describe the rationale for their activities or to better align their work to their citizens’ preferences. Figure 10. Citizens' perception of the alignment between their priorities and those of local authorities across LGs Source: World Bank’s Citizens’ Survey in Estonia (2021) On the other hand, the survey also contained a group-based version of this question reading asking respondents about the extent to which they think their local council acts on the concerns of residents - thus clearly shifting the focus from the citizen’s own priorities to a broader reference frame of ‘local residents’. For this question the responses were more positive, as shown in Figure 11. Here almost a half (47 percent) of respondents said that the council acts on the concerns of local residents ‘to a great deal’ or a ‘fair amount’. This of course leaves a significant proportion of residents unclear on how their LG engages with citizen priorities. 19 Figure 11. Citizens' perception of the alignment between the priorities of local residents and those of local authorities Source: World Bank’s Citizens’ Survey in Estonia (2021) Perceptions of local government responsiveness to residents’ priorities seems to vary considerably across LGs, as seen in Figure 12. In some LGs, especially located along the coast of the country and in its southeastern parts, more than 60 percent of residents on average indicate that the council acts on residents’ concerns. In these localities, although it seems that an exact alignment between one’s own priorities and local council’s actions is rather difficult to achieve, the council on the whole is rather responsive to residents’ concerns. For some LGs this is an area to better engage with citizens on, with far fewer perceiving local governments as responsive to resident priorities. 20 Figure 12. Citizens' perception of the alignment between the priorities of local residents and those of local authorities across LGs Source: World Bank’s Citizens’ Survey in Estonia (2021) To gain insights into the specific priorities of citizens that LGs may or may not be reflecting, the QVSR module offers valuable data. Figure 13 presents the distribution of "credits" allocated by respondents across Estonia within the module, with more credits implying a preference of citizens for local governments to invest more in those areas.11 On average, the highest allocation of credits was observed in the areas of education, with Pre-school Education receiving a score of 1.33 and Basic Education receiving a score of 1.27. Conversely, the lowest credits were allocated to Governance (0.20) and Museum (0.22). This distribution sheds light on the relative importance that citizens attribute to different sectors, providing valuable insights into the perceptions of citizens on their perception of service delivery in their respective LGs. 11 The areas or service sectors included in the study were 16 given the presence of only 15 at the start of the project. 21 Figure 13. Distribution of "credits" allocated by citizens during the QVSR module Source: World Bank’s Citizens’ Survey in Estonia (2021) 22 To what extent are the sectors chosen by residents in the QVSR are also given higher priority by LGs, or at least are those that achieve a higher service quality as indicated by the dashboard. Figure 14 reveals that such correlation is only observed for certain sectors . Moreover, the correlation is generally moderate, ranging from 0.307 for Governance to 0.456 for Mobility. This means that the variation in local government activities does not closely track the priorities as expressed in our citizen survey. Figure 14. Correlation between the QVSR scores allocated by citizens and service quality scores from the dashboard by sector Source: World Bank’s Citizens’ Survey in Estonia (2021) At the time of the writing of this report, the dashboard’s service levels data does not clearly reflect on the extent to which they match the priorities of citizens. As such, it is hard for LG staff to identify where there is the biggest mismatch between survey evidence of citizen preferences and the LGs service delivery levels. The dashboard could play a stronger role in supporting citizens expression of their preferences (through surveys of citizens such as the one used in this evaluation that asks questions about priority sectors) and by engaging with citizens to explain the performance of their LG. It could be that local residents are not fully aware of the actual state of local services in all areas. For example, individuals without children may have limited knowledge about the quality of local educational services. It is also plausible that residents currently lack appropriate tools or do not seek out information to stay informed. 23 The question is the extent to which citizens are already engaging with the dashboard. We now turn to that question. 3.1.3 Citizens’ uptake of the dashboard We can use measures of how many individuals visited the dashboard to assess the extent to which there was uptake from any actor (and at higher levels of numbers only plausibly citizens). There are clear spikes in dashboard online visits in two periods – the launch of the dashboard and an experimental trial of a Facebook-based social media campaign (see Figure 15). The second one corresponds to the launch of the geo-targeted Facebook advertising campaign that was run by the World Bank’s research team working on this project as part of a randomized evaluation of the dashboard’s effectiveness. The numbers of visits, even during these peak times, are not particularly large. When directly asked if they knew of the dashboard webpage (minuomavalitsus.fin.ee), only 4 percent of respondents to the citizen survey reported being aware of it. It is understandable that not everyone is interested in LG matters, and therefore are unlikely to hear (or if they do, remember) a dashboard of this type. However, such a low percentage of respondents being are aware of the dashboard indicates limited public awareness. On the other hand, a clear message from Figure 15 is that citizens are responsive to conscious efforts aimed at promoting the dashboard. As such, there is ample scope for Estonia’s Open Government Partnership and relevant members of the central government to undertake initiatives that introduce the dashboard to citizens, media and civil society organizations, and embed it more strongly into the policy narrative of the Estonian polity. The role of the dashboard in highlighting the relative status of service delivery across local governments can be argued to be important for purely transparency reasons. Whether its existence translates into higher policy effectiveness, when it comes to demand-side effects (see the discussion on the conceptual framework and Figure 2 above), is however less clear. Certainly, low awareness almost necessarily entails that only a small group of citizens could possibly use the dashboard to advocate for reforms in LG. Yet since this group already knows about the dashboard, they might be expected to be above-average concerned about LG matters. In other words, this group of residents might be small but vocal. If the dashboard changes their voices, it may change policy and policy effectiveness. The discussion of the extent to which that might be true and might translate into on-the-ground changes in an LG, is left until after the supply-side of the conceptual framework, i.e., the role of local public officials as measured by our second survey, is considered. 24 Figure 15. Views of the dashboard across time One feature of Figure 11 should be elaborated on. It demarcates ‘treatment groups’ from an experiment that we ran and has lines for four distinct groups of local governments (and their corresponding visitors to the dashboard): control, citizens, bureaucrats, and both. In control LGs, we did not undertake a social media campaign. In ‘citizen’ LGs, we targeted advertising at citizens. In ‘bureaucrats’ local governments we undertook an analogous campaign to communicate the dashboard to public officials. And in both we undertook a campaign targeting both officials and citizens. The distinct lines showcase how visitors based in those local governments interacted with the dashboard over time, and thus show that citizen-focused social media campaigns have the biggest effects. 3.2 The supply side: public officials' utilization of and engagement with the dashboard in decision-making To gain a comprehensive understanding of the dashboard, it is crucial to understand how the dashboard has interacted with the work of public officials, and whether it has impacted their decision making. Given the emphasis on making results public, and the discussion above, it is important for us to assess the policy priorities of individual public officials, examine the extent to which they align with those of citizens in the LG they are serving, and assess the extent to which engagement of public officials with the dashboard might bridge this gap. 25 3.2.1 Public officials’ evaluation of their LGs Our survey of public officials asked a representative sample of public officials in each local government what their assessment of their LG’s strengths and weaknesses were in terms of service delivery and what areas should be prioritized. There exists a notable alignment between the sectors that public officials believe should be prioritized and the sectors they perceive citizens prefer to be prioritized. This alignment is particularly pronounced in sectors that have are dominantly citizen-facing, such as education and welfare (see Figure 16). This implies that public officials may be inclined to focus on sectors they perceive as crucial and important to the citizens they serve. This understanding and action towards citizens’ priorities suggests that public officials work on and prioritize sectors that are deemed significant for the satisfaction of citizens. Figure 16 presents the proportion of public officials that stated citizens in a local government would flag a sector (labelled underneath each panel) to be prioritized versus the officials’ own perceptions of what should be prioritized. As such, it showcases the extent to which officials believe they are following the preferences of citizens in their work. Overall, the panels all show a positive trend, implying that officials believe they are working in the areas citizens would like them to. 26 Figure 16. Public officials' report on the prioritization of sectors and perceived preferences of citizens across sectors Prioritization of sectors and perceived preferences of citizens across sectors Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) However, the alignment between the prioritization of sectors by public officials and the preferences reported directly by citizens themselves is not nearly as closely aligned as public officials would believe. By comparing the stated preferences of public officials with our citizen survey, we can see the relationship is not statistically significant, even for citizen-facing sectors (see Figure 17). The difference is particularly significant for social services for adults, children’s welfare and mobility. This suggests that the information on which public officials base their decisions and priorities may not accurately reflect the current state of service delivery in LGs, or the prioritization of citizens. It raises concerns about the sources and accuracy of the information that public officials rely upon to gauge citizen needs. 27 Figure 17. Public officials' report on the prioritization of sectors and citizens' own preferences across sectors Source: World Bank’s Public Officials’ Survey in Estonia (2022/23); World Bank’s Citizens’ Survey in Estonia (2021) Furthermore, it is crucial to emphasize the substantial variation in public officials' perceptions of sector performance both across and within LGs. Figure 18 illustrates the performance scores assigned by public officials to four sectors within their respective LGs, rated on a scale of 1 to 10.12 The red square represents the average score for each LG, while the blue squares represent individual public officials' scores. The data reveals notable disparities in the self-reported performance of sectors across different LGs. However, even more intriguingly, significant variations exist within LGs themselves, indicating that even public officials within the same jurisdiction may not possess consistent, or even accurate, information regarding the performance of their own sectors. 12 Only 4 sectors were chosen for their assessment given their more widespread representation in the survey across LGs. 28 Figure 18. Performance of the sectors across LGs as reported by public officials Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) The disparities in public officials' understanding of their own sectors highlight the need for a reliable source of information on the relative performance of service delivery. The dashboard of course aims to play that role. This information is crucial in bridging the gap between officials' perceptions and the actual state of service delivery to enable informed decision-making. It encourages informed decisions amongst public officials in the same local government, and across local governments. Given the disconnect between perceptions and realities, it is important to assess the extent to which public officials perceive themselves to have sufficient information. LG public officials generally believe that they have sufficient information on the current state of service delivery, stating that in their sectors they have sufficient information to effectively make decisions on approximately 81 percent of their projects. However, it is important to note the variation observed across LGs (see Figure 19). In some LGs, public officials report not possessing sufficient information to effectively work on a third of their projects. 29 Figure 19. Public officials’ report on the extent to which they possess the necessary information across LGs Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) What this means is that while the majority of public officials feel adequately informed about service delivery for a substantial portion of their work, the dashboard can provide a more robust discussion of the accuracy of that work, and seems to have played that role in a number of cases. In some local governments, there exists a greater disparity in access to information. In those local governments there remains a need for even more reliable sources of information on service delivery, implying a further role for an expanded dashboard. To what extent do officials believe a dashboard like Minuomavalitsus is useful? The survey asked public officials the extent to which they would find a benchmarking tool to compare the performance of sectors and LGs useful for their work. 75 percent of LG public officials reported that such a tool would be useful for obtaining feedback on the performance of their LG. However, there is noticeable variation across different LGs, with as low as 35% in some LGs (see Figure 20), implying the dashboard team may have to market the dashboard more in LGs where demand is currently low. 30 Figure 20. Public officials’ report on the extent to which a benchmarking tools would be useful across LGs Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) Overall, the findings in this sub-section reveal a notable misalignment between public officials' prioritization of sectors and the actual preferences of citizens, and a significant variation in the perceived performance of sectors within LGs, indicating a lack of consistent and reliable information available to public officials regarding service delivery, highlighting the importance of bridging this gap. To address these challenges, it is imperative to provide public officials with access to a centralized and comprehensive source of information — such as the dashboard— that empowers them to better understand the state of service delivery and make policy decisions accordingly. The high level of interest expressed by public officials in such a tool underscores its need and potential impact. Moving forward, the next section dives into the actual utilization of the dashboard to explore whether public officials have been actively leveraging it as an integral component of their decision-making processes. 3.2.2 Public officials’ uptake of the dashboard The public officials’ survey was directly used to illicit information from public officials about the extent to which they utilized the dashboard. The findings suggest that 25 percent of respondents reported using the dashboard as a primary resource for obtaining information on the performance of LGs (see Figure 21). Furthermore, the survey results also indicate that half of the public officials surveyed have utilized the dashboard at least once to gather information on the performance of their LG. This level of utilization indicates the dashboard has played a useful role as an ongoing source of performance-related insights, highlighting its value and utility in informing decision-making among public officials. It has already 31 overtaken some key sources of information such as formal field visits and formal citizen interactions as a source of information on LG performance. Figure 21. Information sources used by public officials on the performance of LGs Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) 32 Figure 22. Frequency with which the dashboard was used Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) More encouragingly, among public officials that indicated using the dashboard, 57 percent reported using the dashboard multiple times a year to obtain information on the performance of their LG (see Figure 22). This indicates that, to some extent, among the users of the dashboard, it serves as a consistent and dependable source of information. It further emphasizes the value of the dashboard as a tool that public officials could benefit from using in their decision-making processes. 33 Figure 23. Variation in the extent to which public officials use the dashboard across LGs Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) However, it is important to acknowledge the variation in the adoption rates across LGs (see Figure 23), ranging from 9 percent to 100 percent having used the dashboard in the past year. While the aggregate results are encouraging, particularly considering the relatively recent introduction of the dashboard as a tool for public officials to utilize for more reliable information on the state of service delivery in their LGs, there is a need to further convince officials in many local governments of its utility. In particular, the dashboard’s success will depend on the extent to which it can be further sold to public officials as useful for their work. This is discussed next. 3.2.3 Perceived usefulness of the dashboard by public officials This section of the report aims to evaluate the extent to which the dashboard has been beneficial for public officials to improve service delivery within LGs. Among public officials that use the dashboard, a majority (60 percent) reported that they find it useful for their work . Among these public officials, the most frequently identified benefit was the dashboard’s ability to benchmark LGs, overall and within sectors (see Figure 24). Approximately one-third of all respondents expressed that they found it very useful to compare their own LG's performance with that of other LGs. Similarly, benchmarking the performance of their sector within their LG against other similar LGs was equally useful. By providing public officials with a platform to assess their LG's performance in relation to other comparative LGs, 34 the dashboard proves to be a valuable a tool for identifying areas for improvement and a possible avenue to learn from the successes of other LGs. Figure 24. Benefits of using the dashboard Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) Furthermore, the survey focused on trying to understand the flow of information between citizens and LGs. This is particularly important to understand given the mismatch in prioritization between public officials as outlined in Section 3.2.1. Only 28 percent of public officials reported that the dashboard has improved this information exchange, indicating a relatively modest level of agreement among respondents. Interestingly, when analyzing the results at the individual LG level, in most local governments, fewer than a third of the respondents agreed that the dashboard has enhanced the exchange of information between citizens and their respective LGs (see Figure 25). This variation across LGs suggests that the environment within LGs seems to be playing a significant role in shaping the perceived impact of the dashboard. 35 Figure 25. Variation across LGs in the extent to which public officials report the dashboard has improved information flow Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) Given the disparity observed across LGs, to ensure the effective utilization of the dashboard as a key source of information for decision-making and policy formulation, it is essential to create a conducive environment within which public officials operate. This encompasses several factors, including effective management practices, collaboration both within and across teams, a motivated and skilled workforce, and the cultivation of conducive attitudes and behaviors among public officials. The state of these analog complements across LGs in Estonia are discussed next. 3.3 The state of the service: the environment in which public officials make decisions The scope of this section is on the human resource and organizational factors that complement interventions, such as the dashboard, to improve evidence-based service delivery. Individual-level and organizational-level drivers impact the performance of public officials across LGs and directly interact with their capacity to make informed policies. While different administrative levels have different functions, roles, responsibilities and interests, there are a set of common individual and organizational aspects that influence the actions and behaviors of the public officials:  The attitudes of public officials. A large economics, psychology and public administration literature has shown that it is the combination of intrinsic and pro-social motivation—the desire to work hard for the enjoyment of the task itself and the desire to exert effort for the benefit of others and not for oneself—that drives the performance of public officials. As such, positive 36 attitudes and behaviors—motivation, job satisfaction, engagement—are a good proxy measure of staff effort given the unique difficulties of monitoring the daily activities of public sector employees.  Management quality, such as goal setting, regular monitoring of the achievement of targets, and incentives, both monetary and non-monetary, for staff to be performance-oriented impact the productivity of a LGs.  The extent of capacity building activities to both upgrade knowledge and skills to ensure that public officials have the skills to use the dashboard  The work environment of LGs, recognizing that service delivery is a group output that requires collaboration within and across teams. It is well known that investments in individual training in the absence of a conducive work environment that incentivizes and enables staff to apply their knowledge to collective work are likely to be wasted. Therefore, teamwork, operating protocols, and trust are all important aspects of organizational culture that impact individual knowledge and attitudes, and service delivery. The public officials’ survey aimed to gain insights into the working environment of public officials in LGs, specifically focusing on the factors outlined above. The subsequent sections of this report dive into an in- depth analysis of the state of the LGs, taking into account these factors, in order to better place the variation in the adoption of the dashboard across LGs. By examining these factors, we can gain a better understanding of the factors influencing its adoption across LGs. 3.2.1 Attitudes of public officials In light of the challenges associated with monitoring output in public sector jobs, individual and organizational performance rely heavily on public service motivation as well as the commitment to continue serving in the public sector. In addition to compensation playing a key role, the day-to-day decisions and choices made by public officials, and consequently their performance, are also influenced by factors like professional pride and a sense of societal responsibility. These intrinsic and extrinsic sources of motivation significantly shape the behavior and effectiveness of public officials, extending beyond mere financial incentives. The survey findings reveal that the motivation levels of public officials have decreased over time . When asked about their current levels of motivation compared to when they initially joined, public officials, on average, reported a decrease of 9 points (relative to a starting point of 100). This indicates a slight decline in motivation over time. However, it is important to note that there is considerable variation in this change across LGs (see Figure 26). Some LGs exhibit a more substantial decrease in motivation, while others maintain relatively higher levels. These variations highlight the influence of local context and organizational factors on the motivation levels of public officials that may be resulting in public officials having varying experiences across difference LGs. This points towards the need to understand the reasons behind these varying levels of motivation among public officials. 37 Figure 26. Change in the motivation of public officials compared to when they joined the service (relative to 100) Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) The survey was able to shed light on the factors that contribute to motivation among public officials. When asked about the reasons behind their higher levels of motivation, public officials commonly attributed it to engaging and challenging work assignments (68 percent) , their belief in their own competence to succeed within the organization (72 percent), and the autonomy granted to them in carrying out their responsibilities (51 percent). These factors tend to serve as sources of intrinsic motivation that drive their commitment and dedication. On the other hand, when public officials reported lower levels of motivation, several key factors emerged as common reasons. One prominent factor is wages (60 percent), as officials expressed concerns about their compensation and its perceived inadequacy. Another factor identified is the lack of quality management (50 percent), which may include issues related to leadership, communication, and decision- making processes within their respective LGs. Additionally, public officials highlighted the condition of the service (28 percent), encompassing aspects such as holiday leave and other allowances, as factors that affect their motivation negatively. This suggests that most commonly, the decrease in motivation is closely linked to extrinsic factors , particularly compensation. While 94 percent of public officials report that they are motivated by working on tasks that have the potential to benefit others, underscoring that they, to some extent, are pro-socially motivated. However, the extrinsic factors, such as wages and management practices, are contributing to the decline in overall motivation levels. To gain a comprehensive understanding of this issue, the survey asked public officials about their extent of satisfaction with their salaries. Only 39 percent of public 38 officials reported being satisfied with their salaries. This indicates a notable dissatisfaction among a significant portion of the public sector workforce with their compensation. Furthermore, the data highlights substantial variation in salary satisfaction levels across LGs (see Figure 27). This suggests that disparities in salary structures exist, contributing to differing levels of satisfaction among public officials in different regions. Figure 27. Salary satisfaction levels across LGs Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) To some extent, benchmarking salaries against private sector employees and salary compression might explain the low levels of salary satisfaction among public officials. 75 percent of public officials report they would be able to earn higher wages in the private sector compared to the public sector. The extent of this perception among public officials could be explained by public officials benchmarking their salaries against a small segment of highly paid private sector employees, inflating the perception of bias. Additionally, public officials may be experiencing stagnant pay levels during their time in the service. Poor management emerged as another key source of demotivation among public officials. To further the assessment of low motivation levels, the measurement of management quality across LGs is outlined next. 3.2.2 Management of LGs A key element of the public sector work environment is the nature and qualities of management under which a public official works. Following reports of poor management contributing to lower motivation 39 among public officials, it is important to assess the quality of management within and across LGs. Management practices such as incentives, goal setting, and the regular monitoring of target are shown to be important for performance and motivation. Studies suggest that practices that support employee performance result in better motivation and higher trust.13 Similarly, management quality, and autonomy of public officials in particular, has been shown to be positively associated with productivity and task completion rates.14,15 Furthermore, better quality managers were found to raise productivity among Italian civil servants in the Social Security Agency by 10 percent.16 To measure management quality across LGs in Estonia, an index was constructed from relevant questions in the public officials’ survey that were directed towards managers in LGs to provide a standardized measure of management quality. This index was created using questions from the World Management Survey (WMS). The WMS is a survey-based measure produced by academics and used to measure the quality of management in many private and public sector organizations globally. The WMS module included modules on organizational goal and target setting, performance evaluations and incentives, autonomy in decisions and flexibility of procedures, degree of staff involvement and staffing practices. The index is constructed by first standardizing each individual response to the overall public administration sample mean by subtracting the mean from each individual response and dividing by the standard deviation. This allows responses of public officials to be compared to other public officials. A composite index is then constructed by summing over questions that measure management quality and is divided by that same number. A higher level of any index indicates a stronger belief or practice. For example, a higher management quality index indicates that the public official has better management skills than the average official. 13 Whitener, E. M. (2001). Do ‘High Commitment’ Human Resource Practices Affect Employee Commitment? A Cross- Level Analysis Using Hierarchical Linear Modeling. Journal of Management, 27(5), 515-535. DOI:10.1177/014920630102700502 14 Rasul, I., & Rogger, D. (2018). Management of Bureaucrats and Public Service Delivery: Evidence from the Nigerian Civil Service. The Economic Journal, 128(608), 413-446. 15 Rasul, I., Rogger, D., & Williams, M. J. (2021). Management, Organizational Performance, and Task Clarity: Evidence from Ghana’s Civil Service. Journal of Public Administration Research and Theory, 31(2), 259-277. 16 Fenizia, A. (2022). Managers and Productivity in the Public Sector. Econometrica, 90(3), 1063-1084. 40 Figure 28. Management quality indices across LGs Figure 29. Overall management quality index across LGs Source: World Bank’s Public Officials’ Survey in Estonia (2022/23)17 17 The number of LGs for which data is missing in this map is due to the lack of respondents identifying as managers. 41 Management quality varies significantly across LGs. Using the index created to measure it, the average management quality score across all LGs is 3.5 (on a scale of 1 to 5). However, the range of scores spans from 2 to 4.6, indicating considerable dispersion in management quality across LGs (see Figure 28 and Figure 29). This dispersion can be attributed to various contextual factors that may hinder managers from performing optimally, such as limited resources, unique challenges, or governance constraints. Additionally, lower capacity among some managers may also contribute to the variation in scores. It is crucial to recognize this dispersion as it underscores the differing experiences of public officials (non- managers) across LGs that shape their capacity for effective service delivery. Furthermore, specific management quality indicators show dispersion across LGs (see Figure 28). Questions which assess effective organizational targeting like setting of goals and targets, convergence of individual staff goals with those of the organization and communication of such goals report the highest composite score of 3.8. Questions that measure good performance management practices scored 3.18, while questions that gauge staffing, that focus on the recruitment of talented individuals and their promotion criteria, scored the lowest at 3.0. More interestingly, there exists a wide variation across these indicators within LGs, highlighting how different aspects of management differ within the same environment. This showcases the extent to which management is decentralized and the way in which management practices are inconsistently implemented across offices. It is important to acknowledge the existing challenges highlighted by low scores on the index on performance management practices. To gain a comprehensive understanding of the effectiveness of performance management of public officials across LGs, the survey specifically asked questions to public officials on the implementation of these practices. Only 58 percent of public officials reported undergoing formal performance evaluations within the past two years. Furthermore, it is important to note the substantial variation observed across LGs in this practice, with evaluations ranging from none in some LGs to 100 percent (see Figure 30). Since 2022, LGs are required to have development discussions with public officials. However, more than 61 LGs do not meet the requirement of at least 90 percent participation from public officials in these discussions.18 These limited figures and wide disparity underscores the extent to which performance is managed in a very decentralized manner not only across LGs, but within LGs as well. 18 https://minuomavalitsus.ee/en/kov?service=10-0-3-01&year=2022 42 Figure 30. Performance evaluations across LGs Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) Given the limited implementation of performance evaluations, it is important to understand the purpose and utilization of these evaluations among public officials who undergo them. Notably, 42 percent of public officials reported that performance evaluations are primarily used to assess the training needs of public officials, highlighting their significance in identifying areas for improvement (see Figure 31). Additionally, a relatively small percentage reported that evaluations were used for awarding bonuses (17 percent) or setting pay (15 percent). On the other hand, a notable proportion (23 percent) indicated that performance evaluations were not utilized at all, suggesting missed opportunities for leveraging these assessments to enhance performance and accountability. These findings highlight the widespread variation in how performance evaluation results are used across LGs in the country. 43 Figure 31. Usage of performance evaluation results Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) The absence of standardization, with weak accountability and rewards systems can have a negative impact on the performance and motivation of public officials across LGs. This lack of consistency in management practices may extend to other areas, including the adoption of new technologies like the dashboard. Given limited motivation levels among the workforce, the widespread implementation and utilization of tools, such as the dashboard, within and across LGs may face additional challenges. Beyond targeting better management practices, one area that can be capitalized upon to support the adoption of the dashboard is effective capacity building. The recognition of capacity building as a significant outcome derived from performance evaluation results underscores its importance among public officials and management across LGs in Estonia. The extent and usefulness of capacity building across LGs in Estonia are discussed next. 3.2.3 Capacity of public officials A significant majority of public officials (86 percent) report having received training within the last year. This finding is highly encouraging and aligns with the importance placed on identifying training and development opportunities as part of the performance evaluation process. To some degree, the data also reveals variation in the extent to which public officials have received training across LGs, with training rates ranging from 50 percent to 100 percent across LGs (see Figure 32). The observed disparity in training rates among public officials across LGs suggests that training opportunities may not necessarily be equally accessible to all. While the overall uptake of training is generally high, it appears that the availability and 44 uptake of trainings are primarily demand-led (from public officials themselves or through performance evaluations). Figure 32.Percentage of public officials receiving training across LGs Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) To gain insights into public officials' perceptions of the trainings they received, the survey examined key aspects on relevance, frequency, and alignment of trainings with work responsibilities. Among public officials that have received trainings, findings indicate a highly positive perception of trainings, with over 90 percent of public officials reporting that the trainings were directly relevant to their work (see Figure 33). Furthermore, a significant majority of public officials agreed that the trainings aligned well with their job requirements. Encouragingly, the survey also revealed that over 40 percent of public officials expressed a desire for more regular trainings, highlighting demand for continuous training among public officials. This underscores the significance of trainings in enhancing the skills and knowledge of public officials in carrying out their day-to-day tasks. 45 Figure 33. Perceptions on trainings among LG public officials Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) Widespread training initiatives among LGs in Estonia have proven to be highly useful and beneficial for public officials in their daily work. By equipping public officials with the necessary tools and knowledge, these training programs empower them to effectively perform their duties. Building upon the success of these existing training initiatives, there is a significant opportunity to capitalize on this foundation and integrate components related to the adoption and utilization of the dashboard in the daily work of public officials. In addition to capacity building, fostering a collaborative environment across LGs can also contribute to the effective and widespread uptake of the dashboard. The extent of collaboration and trust within LGs will be further discussed in the next section. 3.2.4 Work environment of public officials A collaborative and trusting environment may encourage the sharing of knowledge, best practices, and experiences among public officials within LGs. Given the variation in the uptake of the dashboard within LGs, the extent of this collaboration should be better understood. For this assessment, the survey asked respondents about their perceptions on collaboration with colleagues and about the extent of trust and support within teams. According to the survey findings, there is a strong culture of collaboration within LGs in Estonia. An overwhelming majority, 89 percent of public officials, reported that their communication with colleagues is efficient, and 62 percent of public officials acknowledged that their colleagues actively avoid duplicating work and make each other's responsibilities clear (see Figure 34). These findings suggest that the culture 46 of collaboration plays a significant role in promoting effective teamwork, highlighting a strong sense of coordination and cooperation within LGs. Figure 34. Perceptions on communication and collaboration with colleagues Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) 47 Figure 35. Perceptions on trust among colleagues Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) In addition to a culture of collaboration, the survey reveals that public officials in Estonia tend to trust and value their colleagues. An overwhelming majority, 89 percent, reported feeling valued by their colleagues for the work they do, indicating a positive and supportive working environment (see Figure 35). Furthermore, 85 percent of public officials reported relying on their colleagues for support during challenging situations and the same percentage expressed trust in their colleagues. This indicates a positive and supportive working environment across LG, with public officials reflecting a high level of confidence and trust in the abilities of their colleagues. Emphasizing a culture of innovation, 76 percent of public officials, reported that their colleagues support each other in developing new and improved approaches to their work. While there does exist some variation in across LGs (see Figure 36), public officials in over three-quarters of LGs report very high levels of such collaboration (over 64 percent). These findings indicate that trust and mutual respect among colleagues play a crucial role in fostering collaboration, support, and innovation within the LG context in Estonia. 48 Figure 36. Perceptions on the support among colleagues in coming up with innovative ideas Source: World Bank’s Public Officials’ Survey in Estonia (2022/23) These findings indicate the presence of a strong culture of collaboration and support within and across LGs in Estonia. Public officials reported a collaborative environment where ideas and technologies are actively discussed and adopted. This environment tends to foster innovation and provides a solid foundation for the adoption of the dashboard within LGs as a policy tool . While the dashboard is relatively new, the existing collaborative culture creates a conducive atmosphere for its widespread adoption. The openness to collaboration and the willingness to embrace new ideas and technologies bode well for the successful integration of the dashboard into the daily work of public officials. 3.4 Was the dashboard acted upon? The practical utility of the dashboard as a policy tool has to be evaluated by going beyond the survey responses and looking at the on-the-ground changes occurring across Estonian LGs. However low or high self-declared engagement of different policy actors with the dashboard might be, if it doesn’t translate into objectively measured improvements in the quality of local public services, then arguably it provides limited value added. Evaluating the dynamic improvement in public services quality is by its nature a multi- dimensional exercise, since it is driven not only by information availability and priorities of local political actors, but also by available resources, national and local development plans, exogenous economic shocks (e.g., Covid-19 pandemic, war in Ukraine), and the secular time trend. As a result, the WB’s Bureaucracy Lab is working on an impact evaluation study through a Randomized Control Trial (RCT) (see Box 1) 49 aiming to causally determine the relative importance of the dashboard in driving changes in local public services quality. Box 1. Impact evaluations in policy making RCT is an evaluation technique that draws from experimental design in order to measure the impact of a development project. Pilots and randomized experiments are useful when introducing policies aimed at bigger populations when no baseline data is available to measure the efficiency of the treatment or intervention. The strength of such designs lies in their simplicity. Experiments in which at least two groups are artificially created: one that receives treatment, and one that does not, allow for comparisons and enable to determine the effectiveness of the intervention. The frontier of government analytics includes the use of RCTs to investigate routes to strengthened public administration. In an OECD context, impact evaluation methods are now used to provide rigorous evidence on optimal reform trajectories. Linos (2018) experiments with recruitment strategies in the US to improve the diversity of police departments and Rogger and Sibieta (2016) experiments with the co-production of LG environmental services in the UK. Public officials require empirical evidence to determine if policies are working as intended and to identify improvements. Impact evaluations can support future policy reforms by mapping out the causal mechanism behind the impact of policies and help explain the process through which policies improve outputs such as service quality. Sources: Tourangeu, 200319, Webber & Prouse, 2018 20 However, some initial descriptive evidence can be provided to demonstrate the potential role of the dashboard as a policy tool. Figure 37 below shows the annual changes in local public service quality, as measured by the dashboard, averaged across 3 worst-performing service areas for each of 79 LGs. On average, before the dashboard became available, the 3 worst-performing service areas in each LG tended to improve by 0.42 points per year (the 2018-2019 average). After the dashboard was published, the average improvement in jumped to 0.70 (2020-2021 average).21 The change is not universal, and some LGs did see larger improvements pre-dashboard, but the trend is suggestive. This might indicate that the dashboard is in fact used to identify and address policy problems, at least in the most urgent areas. 19 Tourangeu, R. (2003). Recurring Surveys: Issues and Opportunities. A Report to the National Science Foundation Based on a Workshop Held on March 28-29, 2003. 20 Webber, S. & Prouse, C. (2018). The New Gold Standard: The Rise of Randomized Control Trials and Experimental Development. Economic Geography, Taylor and Francis Journals, 94(2), 166-187. DOI: 10.1080/00130095.2017.1392235 21 The dashboard was published in August 2020. Given lags in policy implementation it is assumed that it couldn’t lead to any measurable changes already in 2020. 50 Figure 37. Changes in service quality in the worst-performing service areas pre- and post- dashboard 3.6 Bringing it together Overall, these findings highlight important challenges on the road to the better adoption and integration of the dashboard by citizens and public officials. It provides key avenues to explore to support the process to better align the actions of public officials with citizen priorities, leveraging the dashboard's potential. The survey findings indicate that there is room for improvement in aligning LG actions with citizen priorities. On the demand side, targeted outreach efforts have led to an increase in the usage of the dashboard by citizens, showing its value as a tool for providing information on LG performance to citizens as a means for dialogue with LG officials. The availability of such information empowers citizens to hold their LGs accountable and voice their concerns or expectations more effectively. On the supply side, while public officials perceive that their priorities align well with what they perceive the priorities of citizens to be, in reality, there exists a notable misalignment between the two . There is a need to bridge the gap between officials' perceptions and citizens' priorities. Furthermore, there is significant variation in officials' perceptions of the performance of sectors within the same LG, further highlighting the need for the dashboard as a reliable source of performance information. Despite the relatively recent introduction of the dashboard, there has been uptake in its use, but it varies significantly across LGs, underscoring the need to better understand the environment in which public officials operate, adopt new tools, and implement policies. An assessment of this environment suggest that public officials suffer from declining motivation levels, resulting from compensation and management related issues. While management quality varies across LGs, performance management initiatives have been outlined as means to develop targeted capacity- building programs for public officials. Building on existing training initiatives, this is an opportunity to capitalize on for the integration of the dashboard into public officials' daily work. Furthermore, collaboration and support among colleagues is strong within LGs, providing a conducive environment for the adoption of the dashboard as an innovative solution to key challenges. 51 3.7 Lessons from public officials In addition to obtaining empirical evidence on the dashboard and complementary factors, the evaluation process included two additional components. Firstly, lessons were gathered through interviews with LG officials from three top-performing LGs (based on dashboard scores) in late 2020 and early 2021. These interviews aimed to capture their general experiences with LG management and their insights on successful dashboard implementation. Secondly, the dashboard was presented to local public officials across the country through a series of consultations conducted in 2021 and 2022. These qualitative approaches complemented the quantitative surveys and dashboard scores, enriching the evaluation process with valuable insights and perspectives from the LG officials themselves. 3.7.1 Lessons from top-performing LGs Learning about the experience of the LGs, which provide the highest level of local public services in specific sectors (as measured by the dashboard) was the goal behind conducting a series of interviews with relevant local public officials. Across interviews the following emerge as the overarching factors of success:  Cooperation: successful provision of local public services, be it sector-specific or overall, universally depends on cooperation of different policy actors. This has been either underscored as the key to the success in the top-performing LGs, a returning thread of conversation, or the key remaining challenge to be met. In the case of sector-specific cooperation, it might involve practices such as inter-school cooperation in the municipality, consisting of student and teacher exchanges, common activities, experience and resource sharing. Links between the sectors can also be fostered in this sphere as kindergartens and schools at different education levels can cooperate with each other. The private sector might also contribute. For instance, some LGs report working with architects to modernize the school infrastructure. In a broader view, cooperation between neighboring LGs proves essential for ensuring higher quality of different public services. In some instances, LGs rely on each other’s facilities, which is particularly beneficial given the high fixed costs of certain public services (e.g., a swimming pool, modern library, additional school places), making it impractical for every single municipality to invest in the same amenities. This is particularly observable for LGs benefiting from a well-developed public infrastructure of larger cities in the vicinity. In other cases, LGs might pool resources for a common investment, which will have cross-municipality benefits. Learning the lessons of best practice from other LGs also appears to play a role in the management of the most successful LGs.  Expertise: high quality of outputs can rarely be achieved with a correspondingly high quality of inputs. This is also the lesson drawn from the interviews with local public officials. In LGs with the highest quality of public services (output), the fundamental role of expertise of public officials (input) is repeatedly highlighted. Recruiting and retaining people with sufficient expertise, be it in the core public administration or in specific sectors, like schools, comes with a set of challenges, as their services tend to be highly competitive. This is why LGs benefit from offering attractive employment conditions, including higher compensation, balanced workload, performance recognition, or longer leaves. Advantages in this field stem also from conditions over which the local authorities have less short-term control, including the location (e.g., in relation to largest settlements or major transport networks) or ‘reputation’ of a municipality. When it comes to 52 management of the existing personnel it is highlighted that efficient work is facilitated rather than hindered by a flexible working environment. Micro-managing in general is found to be counter- productive. Instead, providing the public officials with latitude in achieving pre-specified goals appears to bring fruits. This positive association between staff autonomy and work outputs is in line with evidence found for other public administration institutions across the globe (Rasul, Rogger, and Williams 2023). Providing employees with a voice in longer-term development planning is also perceived as beneficial in top-performing Estonian LGs.  Modernization: forward-looking thinking and medium- to long-term planning are essential in developing high quality public services in a sustainable fashion. Even among the top-performers in specific sectors (or arguably most so among them), there is a recognition that there remain a lot of sub-sectors to be reformed. This might involve physical improvements (e.g., modernizing school infrastructure), legal changes (e.g., simplifying the bureaucratic procedures), organizational improvements (e.g., unified IT system; four-eyes decision-making). There also appears to be an important role for the ‘can-do’ attitude in the face of exogenous shocks. In cases in which the outstanding issues in local public services provision have been laid bare by external developments, the top-performing LGs appear to embrace rather than avoid the challenge. For instance, the need to modernize the school environment to ensure it meets the learning needs of the 21st century is widely acknowledged, and a variety of solutions are tried to address it. Even the arguably greatest external shock of recent decades – the Covid-19 pandemic – has been seen as an opportunity to address the remaining shortcomings, for example when it comes to the digital skills of public officials.  Inclusivity: in a democracy the definition of ‘high quality’ of public services needs to go beyond the material quality and also consider how universal the reach of public services is. Some of the interviewed local public officials acknowledge this dimension explicitly, some appear to do so only implicitly. Nevertheless, in all the interviews, the importance of providing inclusive public services can be discerned. In some instances, this involves ensuring the availability of necessary support for the children with special needs in schools; in others organizing activities bringing together the youth and elderly; still in others addressing the accessibility problems faced by people living in remote areas of a municipality. The most recent iteration of the dashboard from 2022 incorporates this perspective, by evaluating a new service area – accessibility. This indicator has a comprehensive coverage, ranging from the social transport service, through accessible schools, to promotion of the Estonian Library of the Blind. Thanks to highlighting accessibility as a separate category of public services quality, LGs obtained a (more) objective measurement and diagnostic tool to inform their work on providing inclusive public services. 3.7.2 Lessons from consultations on the dashboard In order to promote the dashboard and increase public officials’ competencies in using it, a series of consultations were run by external specialists, in close cooperation with the WB’s Bureaucracy Lab and the MoF. The consultations ran in three rounds between March 2021 and December 2021. Although they 53 had multiple goals, including awareness-raising, capacity-building, and feedback-gathering, they provide a series of overarching lessons about the dashboard:22  Measurement conundrum: constructing a dashboard that quantifies the quality of public services across specific service areas comes with a list of advantages, but also raises several concerns. First, it should be underscored that the public officials in general appreciate the opportunities provided by dashboard (see also section Chapter 3.2.3). Those include the ability to benchmark LG performance, both against the past years and against other LGs. It is also convenient that so much information is condensed in a single place and visualized in an interactive and diversified manner. However, some public officials express doubts about the ‘epistemological’ approach taken in the dashboard. For some, the quality of public services can hardly ever be quantified objectively, especially when measured across time, in a cross-municipality perspective, and, ultimately, using a simple 0-9 scale. A partial response to this critique also emerges from the consultations though. Already in the current form, with a long list of sub-indicators and quality criteria, many public officials find it difficult to explore all the dashboard in its full version. If the measurement was to be nuanced even further, in all likelihood the understanding of the dashboard would become even lower, which in turn could affect its usage.  Ensuring relevance: providing an annual, cross-municipality overview of several dimensions of public services invariably raises questions about comparability and relevance of (some) results. On one hand, providing a universal set of standardized, objective criteria can increase the utility of the dashboard as a comparison tool. This is validated by the high levels of appreciation of the dashboard, voiced both during the consultations and in the public officials' survey. On the other hand, local officials appear to be particularly concerned about imbalance in the assessment introduced by using a set of criteria common to all LGs. For one, the discrepancy in the capacities and needs of rural and small LGs as opposed to those of urban and large ones, appears to be insufficiently accounted for. Moreover, the within-LG differences are averaged out in the dashboard, which some public officials perceive as a problem to be solved. Several LGs face high internal variability, for instance covering both industrial and rural areas or spanning a large geographical area. This often leads to different levels of public services provision (e.g., schools, public transportation) and divergent needs of local citizens, which the dashboard cannot capture at the moment. Finally, some service areas or specific criteria, for instance, museums, libraries, public transportation, are not applicable in certain LGs. The common thread among these pieces of feedback is the demand for greater flexibility of the dashboard assessment, to achieve an optimal trade-off between comparability and relevance.  Fostering the dialogue: close and sustained cooperation between the policy actors involved in the dashboard creation, in particular the LGs and the MoF team responsible for its maintenance, emerges as essential in improving the dashboard’s relevance and usage. Various types of cooperation appear to be desirable. One of the most commonly discussed ones is the facilitation of data provision by LGs. The procedure of providing the data going into the dashboard is not always clear (e.g., as there are some many multiple criteria) and timely (e.g., there are limited possibilities to update LGs’ results in response to achievements made in-between the annual data 22 In addition, a series of specific suggestions has been raised, related to particular (sub-)indicators, data sources, and methodological decisions. 54 collection exercises). Secondly, LG officials highlight a frequent discrepancy between formal criteria of the dashboard and on-the-ground reality. For instance, some dashboard criteria evaluate the existence of specific documents (e.g., development plans, legislation, statuses), which local officials believe are not reflective of the actual situation in the municipality. In some cases, a desirable activity might be undertaken without corresponding documentation, or it might fall under a broader piece of legislation. In other cases, the existence of relevant documentation does not need to entail implementation. A related concern is voiced about the reliance on outdated data (one, two, but even three or more years old). In response, they propose stronger reliance on regular qualitative feedback from LGs, to supplant the quantitative measurement. Thirdly, while some criteria and sub-indicators have been found to be lacking in relevance, others were suggested as missing or underdeveloped. The importance of maintaining the dialogue with LGs can be demonstrated by the fact the MoF actively adjusts and expands the Minuomavalitsus dashboard in response to the feedback received. This includes, among others, the introduction of four new service areas – accessibility, environment and climate, business environment and open governance – in response to the information collected, for instance through the consultations described here. Consultations carried out in the third and final round (November-December 2022) did find an increased level of satisfaction with the dashboard among local public officials and an appreciation of MOF being reactive to their feedback. 55 Chapter 4: Policy recommendation 4.1 Enhancing citizen engagement with the dashboard  Develop and implement a comprehensive awareness campaign to introduce the dashboard to citizens, media, and civil society organizations. The high level of digital literacy in Estonia allows for cost-efficient and scalable promotion of the dashboard, be it on official government websites or social media. This campaign should highlight the benefits of the dashboard in promoting transparency, accountability, and citizen participation in governance processes.  Foster strong partnerships with civil society organizations by involving them in the development and governance of the dashboard. Engage these organizations in consultations and develop feedback mechanisms to ensure their perspectives and needs are incorporated into the dashboard's content and functionalities. This collaboration will promote ownership and trust among civil society stakeholders, leading to increased utilization of the dashboard.  Embed the dashboard more strongly into the policy narrative of the Estonian polity . This can be achieved by actively showcasing examples of how the dashboard has influenced policy decisions and improved governance outcomes. Integrating the dashboard into policy discussions, public speeches, and official documents can be used to demonstrate its value as a tool for evidence- based policymaking and democratic accountability.  Establish mechanisms for citizens and stakeholders to provide feedback on the dashboard's utility, data accuracy, and relevance. Regularly review and update the dashboard based on user feedback and emerging needs. This iterative approach will ensure that the dashboard remains responsive to the evolving requirements of citizens and stakeholders, enhancing its engagement and utility over time. 4.2 Supporting the interpretation of the data on the dashboard  To support management decisions among public officials, the dashboard can provide clear and concise information in an intuitive format to ensure that public officials can easily interpret the data and understand its implications.  To facilitate managerial actions in response to the dashboard's results, it is beneficial to provide recommendations or action plan templates alongside the displayed data. These recommendations can be based on best practices, benchmarking, or expert insights. By offering practical guidance and actionable steps, decision-makers can directly translate the dashboard's insights into policy actions.  The development of a community of practice comprising of professionals, experts, and stakeholders can bridge the gap between the interpretation of the dashboard and its utility for the work of decision-makers. 56  To enhance the dashboard's impact on policy decisions, an ecosystem for results should be created by linking the utilization of the dashboard to existing policy evaluation systems and performance management processes for public officials.  The dashboard should cater to the specific needs and preferences of different user groups, such as the public, local governments, mayors, and staff. Each user group may require different levels of detail, metrics, or visualizations to effectively understand and engage with the data. By providing differentiated user access, the dashboard can showcase results in a way that aligns with the preferences and priorities of each user group.  The dashboard can play a crucial role in understanding the situation of specific groups by disaggregating its current data to provide insights at a granular level. This can help reveal variations, disparities, and specific challenges faced by these groups. This information enables policymakers to develop targeted interventions and policies that address the unique needs and circumstances of different groups.  Dashboard measures over 300 sub-indicators across 79 local governments and 6 years, which might prevent time-constrained LG officials from exploring it fully. To support public officials to understand and interpret the data on the dashboard, it is recommended to provide cyclical performance reports derived from the dashboard. These reports can extract relevant data and insights into the strengths, weaknesses, and overall performance of LGs from the dashboard and present them in short (2 pages) and long reports (10-20 pages), making the numbers more accessible to the LG officials.  Over the last two years, the MoF has been conducting workshops on the dashboard for interested local government officials. While these workshops have been successful in promoting engagement with the dashboard, it is now crucial to shift the focus of these workshops towards its active utilization in management and fostering ongoing conversations on policy and performance. This will enable participants to effectively utilize the dashboard's potential in their day-to-day operations and decision-making processes.  As part of the WB’s engagement with the MOF, local government officials have been individually consulted on the utilization of the dashboard in their work. The MOF should leverage these consultations with local government officials to implement a monitoring framework that discusses and measures the actions taken in response to the survey findings as this could incentivize improvements. 4.3 Encouraging public officials to engage with citizens in the policy narrative  Actively engaging citizens in discussions and reforms based on the dashboard's findings serves democratic principles but also yields practical dividends. By involving citizens in policy conversations, decision-makers can tap into their diverse perspectives, local knowledge, and lived experiences, resulting in more informed and effective policy decisions. The dashboard reports and workshops that are provided to local public officials by the Ministry of Finance (MoF) can play a pivotal role in emphasizing the need for citizens' input. These reports and workshops should 57 explicitly highlight the significance of citizen engagement and the value it adds to the decision- making process.  A citizen engagement module should be incorporated into the reports and workshops provided to local public officials, creating a structured approach to involve citizens in the decision-making process. This module should serve as a guide or framework for local officials to actively seek and integrate citizen input, ensuring that citizens' perspectives and concerns are incorporated in policy discussions.  Citizen engagement can be fostered by creating a captivating “lens” into the dashboard data through special reports. These special reports should be designed to highlight specific themes, issues, or findings from the dashboard in a visually appealing and compelling manner to capture citizens' attention and stimulate their interest in exploring the data further. 4.4 Long-term plan to enhance engagement with and improvement of the dashboard  The MOF should formulate a comprehensive long-term plan for the dashboard, encompassing its usage and development going forward. This plan should involve active stakeholder engagement to understand evolving requirements, establishment of clear performance metrics for evaluation, and assignment of a dedicated team for governance and ownership. Additionally, the plan should emphasize the importance of continuous development, including technology upgrades, user feedback incorporation and iterative enhancements.  Clear objectives should be established to measure the impact of the dashboard, and a framework should be developed to monitor these objectives. Defining specific and measurable goals will allow for tracking the effectiveness and value generated by the dashboard. In parallel, fostering a culture of continuous innovation will enable the exploration and integration of new features and functionalities to enhance the dashboard's capabilities and maximize its long-term value to the organization.  Develop and continuously refine a community of practice centered around the dashboard, aimed at encouraging engagement with the dashboard and supporting its utilization in the day- to-day work of local government officials. By creating a collaborative space for officials to exchange knowledge, share success stories, and address challenges, the community of practice will foster a culture of continuous learning and improvement. This community will provide a platform for ongoing support, feedback, and troubleshooting, ensuring that officials feel empowered and confident in utilizing the dashboard as an integral tool in their daily work.  Strengthen, continuously refine, and regularly share the production of tailored performance reports from the dashboard. Closely aligning the reports with the specific needs and priorities of local governments can enhance engagement and maximize the value of the dashboard.  Use the survey findings to leverage the insights gained from LGs that responded favorably to the use of the dashboard. Their positive experiences can highlight the potential benefits of adopting the dashboard as a valuable tool for informed decision-making and efficient governance. 58  MOF should focus on understanding the perspective of non-government actors in order to effectively capture and capitalize upon their interest, thereby enhancing the utilization of the dashboard among such an important stakeholder group. By conducting targeted outreach, engaging in active dialogue, and identifying the specific needs and expectations of these actors, the MOF can tailor the dashboard's features and insights to align with their interests.  To ensure continuous improvement and effectiveness of the dashboard, it is strongly recommended to integrate an evaluation toolkit into the ongoing assessment process of the dashboard. By incorporating relevant evaluation metrics, feedback mechanisms, and user satisfaction surveys, we can systematically gauge the dashboard's performance, usability, and impact on decision-making.  Incorporate an annual reflection or evolution point for the dashboard to enhance its effectiveness and relevance. This practice will allow stakeholders to evaluate the dashboard's performance, identify areas for improvement, and align it with evolving organizational goals and priorities. By regularly reassessing the dashboard's metrics, visualizations, and user experience, decision-makers can ensure its continued value and adapt it to changing needs. 4.5 Continue expanding the scope of the dashboard  The MOF should continue expanding the dashboard in response to changing needs , as demonstrated by the inclusion of two new indicators (Environment Climate; Accessibility) in the 2022 version. This commitment to comprehensibility and responsiveness is commendable and should be sustained.  The dashboard should be flexible to respond to the next major shock , similar to the impact of events like Covid-19 or local government reforms in 2016. By anticipating potential disruptions and incorporating adaptable features, the dashboard can effectively support decision-making during times of uncertainty. Being proactive in preparing the dashboard for unforeseen events will enable stakeholders to monitor emerging trends, respond promptly, and make informed decisions.  Incorporate a system to record and maintain regular assessments of engagement with the dashboard. By consistently evaluating user interaction and feedback, this will enable the identification of areas for improvement and optimization. It will also ensure that the dashboard remains relevant, user-friendly, and aligned with the evolving needs of the stakeholders.  Implement an automated system to analyze the policy priorities as discussed in local council meetings. By harnessing technology to extract relevant information from local council meeting minutes, the dashboard can be updated to provide real-time insights into the policy landscape. This approach ensures that the dashboard remains up-to-date, enabling stakeholders to stay informed about the latest policy developments and make well-informed decisions. 59 Annexes Annex 1: Timeline of the project Table A1: Major milestones of the project Phase Timeline Establish client relationships in Estonian May 2020 - August 2020 government Design impact evaluation with MoF inputs May 2020 - December 2020 Prepare final survey instrument for citizen survey April 2021 English, Estonian and Russian Procure survey firm for citizen survey May 2021 Citizens’ survey data collection June 2021 - October 2021 Procure survey firm for public official survey August 2022 Interviews with well-performers July - August 2021 Preparation of diagnostic reports August 2021 and October 2022 LG consultation November - December 2022 Sign data agreement and transfer administrative May 2020 data Prepare final versions of public official survey October 2022 instrument in English, Estonian and Russian Public officials’ survey data collection November 2022 – April 2023 Data cleaning and analysis April 2023 – June 2023 60 Annex 2: Research Appendix The literature studying this type of Open Government originates in the seminal work of Charles Tiebout (1956). Tiebout presents a model in which LGs provide different levels of a range of public goods (and taxes to finance them with) and then citizens observe the LGs’ decisions and choose which LG to live in. Tiebout shows that this can lead to efficient sorting of households across LGs.23 Albert Hirschmann (1970) extends this type of framework to include the possibility that citizens actively engage with the LG to urge them to provide higher quality public goods (exercise “voice”) rather than simply moving to a different LG in response (exercising “exit”).24 Besley and Case (19953) argue that elections act as another key disciplining device on politicians raising taxes and providing public goods. They present a political economy model of yardstick competition (building on Shleifer, 19854) in which voters observe the combination of taxes and public goods politicians propose and choose who to vote for.25 This induces “yardstick” competition among politicians that leads them to exert more effort to provide public goods efficiently. Somewhat separately, there is a literature studying the optimal degree of decentralization of government, starting from the seminal work of Wallace Oates (1972).26 This literature typically focuses on the tradeoff between the benefit of being able to provide different bundles of public goods to different LGs where citizens have different preferences against the cost of making coordination and internalizing externalities more difficult when there are more jurisdictions. Notably absent from this literature, and where this study hopes to contribute, is the role of the effectiveness of the civil service in delivering public goods. While a lot of the literature has studied the incentives of politicians, bureaucrats are left in the background. Notable recent exceptions are Martin & Raffler (2020) and Raffler (2020) who study the role that bureaucrats play in enhancing or diminishing political accountability in Uganda.27 Finally, there is an extensive literature on the scope for increasing the information available to citizens and service users to allow them to hold service providers accountable and hence improve the quality-of- service provision. Despite this, direct information provision to citizens has had mixed to little success. The linkages between informing citizens of local service quality and the activities of public officials have been generally documented as weak. Some degree of insulation from public pressure is a key design feature of 23 Charles Tiebout, 1956, “A Pure Theory of Local Expenditures”, Journal of Political Economy, 64 (5), 416-424. 24 Albert O. Hirschmann, 1970, “Exit, Voice, and Loyalty: Responses to Decline in Firms, Organizations, and States”, Cambridge, MA: Harvard University Press 25 Timothy Besley and Anne Case, 1995, “Incumbent Behavior: Vote-Seeking, Tax-Setting, and Yardstick Competition”, The American Economic Review, 85 (1), 25-45. Andrei Shleifer, 1985, “A Theory of Yardstick Competition”, Rand Journal of Economics, 16 (3), 319-327. 26 Wallace E. Oates, 1972, “Fiscal Federalism”, New York, Harcourt Brace Jovanovich. 27 Lucy Martin and Pia Raffler, 2020, “Fault Lines: The Effects of Bureaucratic Power on Electoral Accountability”, American Journal of Political Science, forthcoming. Pia Raffler, 2020, “Does Political Oversight of the Bureaucracy Increase Accountability? Field Experimental Evidence from an Electoral Autocracy”, Mimeo: Harvard University. 61 most government executives. However, technological advances have reinstated questions about the optimal level of insulation. Information about service delivery levels can now be disseminated widely at low cost, potentially making public pressure more informed. 62 Annex 3: Minuomavalitsus Dashboard Figure A2. Service quality across all Estonian LGs according to the Minuomavalitsus dashboard (2022) Source: https://minuomavalitsus.ee 63 Annex 4: Methodological Notes 4.1 Study samples In the citizens’ survey, 3 LGs are not included both due to their small populations and small island status: Kihnu Vald, Ruhnu Vald, and Vormsi Vald. However, all 79 LGs are included in the public officials’ survey. 4.2 Quadratic Voting for Survey Research QVSR gives respondents a fixed budget to “buy” votes in favor a set of issues. Because the price for each vote is quadratic, it becomes increasingly costly to acquire additional votes to express support for the same issue (see Box 2 below). As a decision-making tool, QVSR is increasingly attracting the attention of both academics and policy-makers as it often provides a more nuanced evaluation of opinions (see here). The issues used in the citizens’ survey displayed were the 16 local public services sectors covered in the dashboard. The survey respondents were asked to distribute their votes towards the sectors as they would like to see them more or less prioritized in their LG (Box 2 below outlines the QVSR in detail). The most innovative component of the survey was the QVSR module, and so further details are provided in Box 2. QVSR gives respondents a fixed budget to “buy” votes in favor or against a set of issues. Because the price for each vote is quadratic, it becomes increasingly costly to acquire additional votes to express support or opposition to the same issue. Most surveys rely on some variant of the Likert item, a survey technology developed in the 1930s by the psychologist Rensis Likert. Survey respondents are asked to evaluate a policy statement by picking one of five (or seven) ordered responses ranging from strongly agree/favor to strongly disagree/oppose. To interpret survey answers as a measure of “true” preferences, researchers need to assume that respondents are sincere when they pick a response category. Yet, there are good reasons to think that respondents’ answers are shaped by additional motives and that these motives can conflict with respondents’ desire to be sincere. Cavaille, Chen, and Van der Straeten (2019, 2020) have considered the theoretical properties of QVSR and shown it can be superior to Likert in measuring preferences as reflected in real-world decisions and voting behavior. 64 Box 2. Quadratic Voting for Survey Research (QVSR) details 65