How to use the Common Assessment Framework (CAF) to improve the performance of regional executive authorities in the Russian Federation Methodological Guide Introduction Improving public administration is a prerequisite for accelerated social and economic development in Russian regions. Nowadays, high standards of public administration encourage “small governments� with operations that are informed by the best corporate practices. The Common Assessment Framework (CAF) for assessing the effectiveness of public-sector organizations was developed by the European Institute of Public Administration (EIPA). 1 The CAF is a quality management tool designed specifically for government agencies. The CAF provides standards and a coherent approach to measuring and improving the quality and performance of organizations. It enables governments to conduct an all-encompassing inventory of their quality, capacity and performance in a way that allows them to develop action plans to address identified challenges. This will in turn help governments to effectively translate resources and capacities into outcomes. The CAF promotes continuous improvement and encourage innovations. It is inherently adaptable, and its scope has been adjusted and customized to different contexts in the past, even among European countries. Using the CAF, it is possible to improve the performance of regional executive authorities (REAs) in the Russian Federation by improving leadership, developing and implementing comprehensive strategies, planning clearly, leveraging employees’ capacities, and enhancing processes to make them more effective and efficient. Improvements can be measured by the satisfaction/improved perceptions of key customers (citizens and businesses), employees of the executive authorities and society as a whole. This Methodological Guide was designed by the World Bank in collaboration with the Analytical Center for the Government of the Russian Federation. The Guide presents an approach for REAs to undertake an assessment based on the CAF model to identify its strengths and areas for development and draft action plans to improve the quality of their management, on their own and/or with assistance from experts. The proposed approach and improvement process is based on the REA’s assessment by its employees (hereinafter referred to as ‘self-assessment’). 1 European Institute of Public Administration, Common Assessment Framework http://archive.eipa.eu/en/topic/show/&tid=191 1 Key steps to improve organizations The self-assessment framework to improve the performance of government agencies consists of 10 key steps grouped into 3 main phases (Figure 1): Figure 1. Self-Assessment Implementation Steps Phase 1: The start of the CAF journey Step 1 Step 2 Step 3 Organize and plan the self- Compose one or more self- Communicate the self- assessment assessment groups assessment project Phase 2: Self-assessment process Step 6 Step 4 Step 5 Develop a report describing the Organize training Undertake the self-assessment results of the self-assessment Phase 3: Improvement plan Step 7 Step 8 Step 9 Step 10 Draft an improvement Communicate the Implement the Plan the next self- plan based on the self- improvement plan improvement plan assessment assessment report Source: CAF 2013 Participating employees first rate the performance of their REA based on criteria designed to identify strengths and weaknesses in the REA’s performance. The performance improvement plan for the REA is then developed based on the results of the self-assessment. Each of the phases outlined above are described in greater detail in the following sections. Phase 1: The start of the CAF journey At the outset, the REA’s leaders should adopt a decision to evaluate the performance of the REA and issue an internal regulation of the REA. This will ensure that the ensuing self-assessment and the implementation of the performance improvement plan are purposeful and action-oriented, preventing a cursory exercise. A working group of 10 members, including managers of the REA’s organizational units, should be established for the self-assessment. Ideally, the working group should not be led by the leader of the REA. That said, it is acceptable for the working group to be led by the deputy head of the REA. Each member of the working group is assigned one or more evaluation criteria for the self-assessment, and is responsible for developing the improvement plan for the criteria assigned to them. The key functions of the working group are: 2 • To summarize the results of the self-assessment and complete the self-assessment sheet provided in Table 1. • To draft an improvement plan, comprised of actions planned to address areas for development. A decision on the composition of the respondent pool, i.e. civil servants of the REA who will rate its performance, should also be made. Respondents can be drawn from the existing pool of employees, ensuring a sufficient mix of personnel in various positions and who perform strategic functions (rather than support specialists), as well as members of the working group. To ensure the robustness and reliability of the data collected for each criterion, data should be drawn from at least 30 completed questionnaires. This is the minimum threshold of the number of respondents that should participate in the self-assessment. It is recommended to include as many relevant employees of the REA as possible in the questionnaire-based survey. An analysis of the results of the self-assessments conducted in the pilot regions shows a correlation between the average rating of a REA’s effectiveness and the number of employees who participated in the survey. The fewer the respondents (i.e. the smaller the sample is), the higher the scores awarded to the REA, as there will always be some respondents who are concerned to identify shortcomings due to fear of reprisal. Therefore, it is important to note that the number of respondents can affect the objectivity of the scores entered in the self-assessment. The REA’s internal regulation on conducting the self-assessment should include the composition of the working group, with the full names and positions of its members, as well as a list of respondents participating in the self-assessment, with their full names, positions and e-mail addresses. The intention to conduct the self-assessment and the contents of the local regulation should be communicated to all civil servants in the REA. Phase 2: Self-assessment process In a nutshell, the self-assessment process comprises the following major steps: (i) A survey is conducted, when the respondents are offered to rate the performance of their REA, using a questionnaire, based on CAF framework. (ii) The data are reviewed, and the REA’s self-assessment working group drafts a performance improvement plan for the REA. The key objectives of the self-assessment are: a) To collect information from the REA’s employees and identify strengths and areas for improvement, drawing on the results of the questionnaire-based survey. b) To use the evidence obtained to draft a performance improvement plan for the REA. c) To establish baseline values to monitor subsequent changes and conduct further evaluations at regular intervals (depending on the activities developed to improve performance). d) To provide opportunities for bench-marking, sharing best practices and comparing different approaches to public administration. 2.1. Organize training 3 Shortly before the questionnaire is shared with respondents, face-to-face training on the basics of the self-assessment should be conducted. The training can be delivered by the leader of the self- assessment working group or by an external expert. It is an opportunity to provide respondents with the information on the goals of the self-assessment, and the survey implementation. The training is also intended to enhance the data collection process and the data itself, including the quality/depth of proposals to improve performance. During the training, it is also important to assure respondents that their responses will be anonymous and that the results will be presented in a general format that does not make it possible to attribute responses to a particular respondent. Providing this assurance will encourage more respondents to participate and to be objective. Respondents should also be encouraged to complete the questionnaire over several days rather than in a single sitting. For example, respondents could respond to 30 questions per day or spend 15-20 minutes a day responding to the questionnaire. This approach will reduce the impact of fatigue on the quality of responses and will increase the chances that the responses will be well-thought and include substantive comments. 2.2. Conduct the self-assessment 2.2.1 Assessment criteria The CAF consists of 9 criteria that relate to key areas of public administration that REAs need to take into account during the evaluation (Figure 2). The criteria are divided into two groups: a) The Enablers criteria includes practices used by the REA to achieve its targets and improve performance. b) The Results criteria includes the REA’s achievements. The criteria are broken down further resulting in a total of 28 sub-criteria which are used to conduct a holistic assessment of the REA’s performance (see Annex 1). 2.2.2. Administering the questionnaire The sub-criteria have their own set of questions in the questionnaire. To effectively assess the REA’s performance, the 9 main criteria should be complemented by one other criterion that is specific to the performance/operations of the particular REA. The specifics of the REA should be identified in the initial stages during in-depth interviews with members of the working group and/or the respondents. This will make it possible to capture important aspects, specific to the operations of the particular REA, that are not included CAF’s standard criteria. The specifics highlighted above should be in addition to the main CAF criteria which should remain unchanged along with the sub-criteria to evaluate the performance of the REA. 4 Figure 2. The CAF Criteria Source: CAF 2020. The survey questionnaire contains statements that describe the REA’s performance. To complete the survey, respondents should score each statement, on a scale of 1 to 5, to reflect the extent of their agreement or disagreement with the statement. A score of 5 indicates that the respondent “fully agrees� with the statement, while a score of 1 indicates that the respondent “fully disagrees� with the statement. If a statement is not applicable to the REA’s operations, respondents may choose the “not applicable� option (see the example in Box 1). Box 1. Example of a Statement from the Questionnaire Please, rate the performance of your REA by scoring each statement between 1 and 5 where 1 = fully disagree with the statement and 5 = fully agree with the statement Fully disagree Fully agree The REA development strategy was designed with the involvement of all 1 2 3 4 5 stakeholders (staff of the agency, beneficiaries/users of services, etc.) The statements are all framed in the positive, i.e. they characterize the performance of a REA in the affirmative. The respondent is required to indicate the extent to which he or she agrees or disagrees with the statement. In other words, the statement reflects the ideal situation and through their response/ score, respondents measure the extent to which they believe their REA complies with the ideal situation. The questionnaire will be disseminated through email and respondents could complete the survey online by accessing the link provided. Conducting the survey online will make it more accessible to and convenient for respondents. 5 The electronic questionnaire is programmed in such a way that, at this stage, respondents could only select a score, on a scale of 1 to 5, to assess the statement. In other words, it is not possible to provide a written response. Once a score had been selected, the navigation system automatically directed the respondent to the next statement. The self-assessment survey is conducted in two stages. In the first stage, respondents should only respond to close-ended statements, i.e. to score each of the statements on a scale of 1 to 5. Once all the respondents have completed the self-assessment, the system automatically calculates the average score for each statement/criteria/sub-criteria. The results of the survey are summarized by an expert, administering the electronic survey. In the second stage, respondents are required to answer open-ended questions. Under each sub- criterion, the system prompts the respondent to: a) Identify areas for improvement and articulate suggestions for the REA’s performance improvement plan. Areas for improvement can be drawn from statements that received the lowest scores (e.g. scores 1-2). b) Identify strengths of the REA’s performance. Strengths can be drawn from statements that received the highest scores. In this case, respondents are also required to provide specific documentary evidence (a regulation, report, manual, etc.) or reference a specific event/seminar/workshop to substantiate the response under a given sub-criterion (e.g. scores 4-5). There is an average threshold score for each sub-criterion. When the threshold is reached, respondents are required to identify areas for improvement and suggest actions to implement and achieve the improvements, or give examples of strengths. The threshold, which was established by the specialists from the Analytical Center for the Government of the Russian Federation and the World Bank, depends on a number of factors, including: the distribution of responses, size of the sample at a given REA, and the REA’s specifics etc. The threshold can be defined empirically, e.g. the sub-criterion-specific score cannot below 3.5 points or (or 75% on the CAF scale), or be calculated (no more than one standard deviation from the average value for a criterion). This approach enables the identification of areas for improvement, and the development of specific actions to achieve improvements under the sub-criteria with the lowest scores, in a focused manner. Respondents are only required to provide targeted proposals on aspects of the REA’s performance that have the lowest effectiveness. Similarly, respondents are only required to provide examples of strengths in cases where the average score is above the threshold, which indicates that the sub- criterion was rated higher than the others. 2.2.3 Analysis of the Results The survey results are recorded in the Assessment Sheet to reflect the scores, the assessment of the sub-criterion, and the respondents’ proposals on how to improve the REA’s performance or their descriptions of the REA’s strengths (Table 1 gives an example of the Assessment Sheet, using the Leadership Criterion). The results are recorded by a designated staff member of the REA or an external expert. 6 The sub-criterion-specific effectiveness is computed as the total score on the 1 to 5 scale divided by the maximum score, possible to achieve under the sub-criterion, and is expressed as a percentage (Box 2). Box 2. Example of Sub-Criterion-Specific Effectiveness Calculation If a respondent answers 4 questions to rate a sub-criterion, the maximum score would 20. If the total score for the 4 questions is e.g. 12, the sub-criterion-specific effectiveness would be: 12/20=60% The effectiveness rating phases are based on the original CAF scales provided in Boxes 3 and 4. The phase for the Enablers Criteria in the original CAF methodology is defined based on the following four-stage cycle described by American scholar William Deming: a) Plan: Review the process to understand the issue and why something does not work. To this end, it is necessary to involve the whole team to look at the issue from different angles and understand what could be improved and how. Then, make a time-bound plan and have it agreed with the team to establish what should be done and when. b) Do: Implement the new plan and do not fail to comply with any conditions. c) Check: Look at the result and understand whether everything has been done as intended. Evaluate the process with a view to improving something in the next cycle. d) Act: Use or adjust – if everything worked out well, continue using the new practices, sustain the process and try to further improve it. If not, return to the first stage and repeat it from the very beginning, but with correction actions2. 2 Methodological Guide for CAF 2020. 7 Table 1 - Assessment Sheet Effectiveness of Sub-criterion- meeting the sub- Sub-criterion Area for Suggestions for the Criterion /sub-criterion specific total score Strength criterion ** phase**** improvement action plan * / criterion***, % Criterion 1: Leadership Sub-criterion 1.1. Provide direction for the organization by developing its mission, vision and values Sub-criterion 1.2. Manage the organization, its performance and its continuous improvement Sub-criterion 1.3. Inspire, motivate and support people in the organization and act as a role model Sub-criterion 1.4. Manage effective relations with political authorities and other stakeholders * Calculated as the sum of scores for all the questions (statements) from the questionnaire under one sub-criterion ** Calculated as the total score for the sub-criterion divided by the maximum score for the sub-criterion, % *** Calculated as the arithmetic mean for all sub-criteria of one criterion, % **** Established depending on the total effectiveness of meeting the sub-criterion/criterion based on the CAF scale 8 Box 3 shows the phases of the sub-criteria of the Enablers Criteria depending on the effectiveness of meeting them. Box 3. Effectiveness Scale for Enablers Criteria Phase Score/effectiveness Lack of information 0%-10% Plan: understand what should be done and how (have a plan to do 11%-30% this) Do: implement the plan unfailingly 31%-50% Check: check/review what has been attained 51%-70% Act: correct mistakes, adjust the plan or the process 71%-90% Repeat the whole cycle, but drawing from the acquired knowledge over 90% Source: CAF 2013 For the Results Criteria, the phase is defined as the extent of meeting a sub-criterion (Box 4). Box 4. Effectiveness Scale for Results Criteria Phase Score/effectiveness Lack of information 0%-10% No results 11%-30% Unsatisfactory results 31%-50% Satisfactory results 51%-70% Good results 71%-90% Excellent results over 90% Source: CAF 2013 1.3. Develop a report describing the results of the self-assessment The self-assessment results are reviewed by the Working Group. The data from the completed questionnaire shown in the Assessment Sheet is verified by the Working Group and the most problematic areas that require improvement are identified. In addition, members of the working group review the suggestions made by the REA’s employees to improve performance. The review is designed to assess the relevance and feasibility of the suggestions and actions to be included in the plan. Following the review, the working group prepares a final report on the results of the self-assessment (the report can be a word document or a Power Point presentation) and a list of activities to improve the REA’s performance, which will be the basis for the Improvement Plan. The report and the list of activities are submitted to the REA’s leader. It is recommended that the results are shared with REA employees to encourage open and transparent communication and to assure all employees that their concerns and views have been heard. 9 Phase 3: Improvement plan In the final phase, the performance improvement plan for the REA is developed by building on the results of the self-assessment. Based on the results, the self-assessment working group develops an action plan to improve the REA’s performance (Table 2). The action plan reflects specific actions, the timeframe for implementation and who is responsible. It is likely that respondents will have suggested various actions for improvement, therefore, it is important to prioritize the areas to be included in the action plan. The following criteria should be considered when preparing the improvement plan: 1. Strategic weight of the action. The stronger the impact on stakeholders and on the performance of the REA, the more reason to prioritize the action. 2. Level of effort required to implement the action. The easier it is to implement the action, the more reason to prioritize it. 3. Required resources. The fewer inputs (resources) required to implement the action, the more reason to prioritize it. 4. Time required to implement the action. Actions that can be implemented in a short timeframe are more likely to be prioritized. REA’s are encouraged to include ‘quick-win’ actions (that can be implemented within 3 months). Such quick gains are likely to be perceived as victories by employees who will be motivated to seek more improvements. To facilitate the REA’s implementation of the performance improvement plan, it is advisable to ensure that the action plan is approved by a internal regulation of the REA, that responsibility for implementing the action plan is assigned to specific people. 10 Table 2 - Action sheet to develop a REA’s performance improvement plan Area for Starting Estimated Action Criterion/sub-criterion Action Deliverable improvement date deadline leader/team Criterion 1. Leadership Sub-criterion 1.1. Provide direction for the organization by developing its mission, vision and values Sub-criterion 1.2. Manage the organization, its performance and its continuous improvement Sub-criterion 1.3. Inspire, motivate and support people in the organization and act as a role model Sub-criterion 1.4. Manage effective relations with political authorities and other stakeholders 11 Annex 1 CAF Criteria and Sub-Criteria3 Enablers Criteria 1. Leadership The criterion assesses i) how the leaders of an REA elaborate its development strategy and contribute to its implementation, ii) how they identify values to guide the thinking and actions of the REA, iii) how they provide opportunities for the growth of civil servants and iv) create an enabling environment for the operation of the REA. 1.1. Provide direction for the organization by developing its mission, vision and values 1.2. Manage the organization, its performance and its continuous improvement 1.3. Inspire, motivate and support people in the organization and act as a role model 1.4. Manage effective relations with political authorities and other stakeholders 2. Strategy and Planning The criterion assesses how REAs plan and implement their documented strategies and if they are in line with the expectations of key stakeholders (citizens and businesses). 2.1. Identify the needs and expectations of stakeholders, the external environment and the relevant management information 2.2. Develop strategies and plans based on information gathered 2.3. Communicate, implement and review strategies and plans 2.4. Manage change and innovation to ensure the agility and resilience of the organization 3. People The criterion assesses the REA’s human resource management framework; in particular, it examines the performance of the human resource management system, including recruitment, career growth, staff promotion, remuneration, rewards and assignment of duties to ensure high performance. 3.1. Manage and improve human resources to support the strategy of the organization 3.2. Develop and manage the competencies of people 3.3. Involve and empower employees and support their well-being 4. Partnerships and Resources The criterion assesses the development of relations with external stakeholders and ways to plan internal resources to support the effective operation of the REA. 4.1. Develop and manage partnerships with relevant organizations 4.2. Collaborate with citizens and civil society organizations 4.3. Manage finances 4.4. Manage information and knowledge 4.5. Manage technology 4.6. Manage facilities 3 CAF 2020 12 5. Processes The criterion assesses the REA’s processes designed to meet the expectations and needs of key consumers of its services (citizens and businesses). 5.1. Design and manage processes to increase value for citizens and customers 5.2. Deliver products and services for customers, citizens, stakeholders and society 5.3. Coordinate processes across the organization and with other relevant organizations Results Criteria 6. Citizen/Customer-oriented Results The criterion assesses the results the REA is achieving in terms of customer satisfaction with its services (citizens, businesses). 6.1. Perception measurements 6.1. Performance measurements 7. People Results The criterion assesses the REA’s performance from the perspective of its employees. 7.1. Perception measurements 7.2. Performance measurements 8. Social Responsibility Results The criterion assesses the REA’s performance in terms of the satisfaction of the society (community) as a whole. 8.1. Perception measurements 8.2. Performance measurements 9. Key Performance Results The criterion assesses core performance results of the REA achieved through implementing strategic plans. 9.1. External results: outputs and public value 9.2. Internal results: level of efficiency 13