Knowledge Brief Health, Nutrition and Population Global Practice The use of satisfaction surveys to improve the delivery of health services to the population in Costa Rica Ana Lucia Rosado Valenzuela, Ashley Sheffel, Micaela Mussini, Ana Maria Lara Salinas, Laura Di Giorgio August 2023 KEY MESSAGES: • Satisfaction Surveys from the Costa Rican Social Security Fund make visible and improve users' perception of hospitalization and outpatient services. They have been successfully applied for five consecutive years, even in the face of the COVID-19 pandemic. • The application of the surveys has increased the interest of health personnel in knowing the users' perception of the service. Now, the services of different units can be improved and compared based on quality dimensions, improvement plans focused on patients' priorities are developed, and satisfaction data from indigenous populations are made visible. • With the support of senior management, communication and training, satisfaction surveys have taken an important place in the institution. • The way forward includes the sustainability of the tool over time and its redesign, but, above all, the translation of its results into improvement plans that have a positive impact on user satisfaction with respect to the service provided. model that would allow them to know the satisfaction of Introduction their users. The surveys are thus a means to achieve a For five consecutive years, even despite the disruption of more focused attention to the needs of CCSS patients. processes due to the COVID-19 pandemic, the Costa The development and redesign of this evaluation tool in Rican Social Security Fund (CCSS, for its name in Spanish, the CCSS was driven through the Program for Results Caja Costarricense de Seguro Social) has successfully (PforR), a financial instrument of the World Bank that applied and analyzed satisfaction surveys to evaluate its incentivizes the achievement of strategic objectives and outpatient and inpatient services based on user goals of programs in which governments seek to improve feedback. At a global level, satisfaction surveys have been the use of public expenditures or improve their an evaluation model that allows us to know the degree of performance using their own processes and institutions. user satisfaction with the service provided in the In Costa Rica, this program ran from 2016-2023 for the different areas of health care systems. Its results Strengthening of Universal Health Insurance. Its initial contribute to the development of programs and objectives included modernizing and strengthening the strategies for the continuous improvement of processes primary health care network, improving the quality of and increase the quality of care of the institution. services, increasing population coverage, and making the In the CCSS, satisfaction surveys began to be developed network more capable of prevention, early diagnosis, and in 2015, precisely with the aim of creating an evaluation control of diseases relevant to the local, national, and Page 1 HNPGP Knowledge Brief regional epidemiological profile. In addition, the de Servicios de Salud) of the CCSS, which is directly in objective was also to improve the institutional and charge of the satisfaction surveys, did not have sufficient financial efficiency of the CCSS. (1) resources for the application, evaluation, and follow-up of the results. Therefore, the project was not This knowledge report is part of a larger series of reports implemented at that time. (3) On the other hand, it was developed by the World Bank, and seeks to describe the not known whether the tool reflected the real needs and main reasons for success, challenges and key lessons concerns of Costa Ricans, but neither were there the learned during the redesign and implementation of the means to know this. satisfaction surveys, with the objective of providing a roadmap for other countries interested in implementing similar programs. The new Satisfaction Survey In 2016, the PforR brought renewed efforts to improve Background and revitalize satisfaction surveys. With the arrival of the The CCSS has been interested in integrating a systematic PforR, and the initial discussions between the World Bank mechanism that would allow them to know the voice of and the CCSS, the importance of listening to the voice of users and their satisfaction with the service since 2015. the various users and non-users of the institution's health For this reason, they created a satisfaction survey based services was recognized. Therefore, two of the indicators on the SERVQUAL model of service quality. This model is established in the PforR were linked to the redesign and a measurement scale that allows the evaluation of follow-up of the results of the satisfaction surveys. service user satisfaction through five dimensions (Table Redesign and Piloting 1). Within each dimension there are sub-dimensions The survey was redesigned with the aspiration of which, in the case of the survey developed by the CCSS, adapting the values of the dimensions of the comprise around forty indicators that make up the questionnaire, previously developed, to the priorities of questionnaire. Costa Ricans. For that, a third-party expert in the subject was hired so that, through focus groups and a Household Dimension Description Survey conducted in 2017, the priorities of people who Appearance of facilities and used, and those who did not use, CCSS services were Tangibility equipment understood. The results reflected that trust, ease of Reliability Provision of service as agreed obtaining appointments, length of care, and empathy were the most important aspects for Costa Ricans. This led to these indicators, from the dimensions and sub- Responsiveness Willingness to respond to needs dimensions of the satisfaction survey, having a higher Service provider's knowledge of weighting. Assurance care provided Another fundamental part for the World Bank and the Provision of personalized services CCSS was to know the opinion and priorities in the Empathy tailored to the user's understanding satisfaction of the indigenous population of Costa Rica. and wishes Therefore, the qualitative research and focus groups conducted also included this vulnerable group. With this Table 1. Dimensions of the SERVQUAL methodology. Own analysis, the questions asked in the satisfaction survey elaboration based on Verification Report. (2) allowed CCSS to explore what was most important for them, the dimension of reliability in the health system The SERVQUAL methodology provides a result about the (the time to get care and the ease of obtaining an differentiation between what users expect (expectations appointment). about the service) and what they receive (how they Once the surveys had been adapted to the population's receive and value the service) from health services. priorities, a pilot was carried out to evaluate the Despite having developed a great tool, the Institutional questionnaire's feasibility and comprehension. Directorate of Health Services Comptrollers (DICSS, for its The new tool name in Spanish, Dirección Institucional de Contralorías The redesign and piloting ensured an effective tool with Page 2 HNPGP Knowledge Brief satisfaction indicators that evaluate what is most weighting of the questionnaire and its indicators by important to Costa Ricans. The result was a survey that listening to the voices of Costa Ricans, including measures the five dimensions of the SERVQUAL model in indigenous populations. This milestone was effectively hospitalization and outpatient care, in addition to a achieved within the timeframe established in the PforR differentiated evaluation that measures the reliability plan. The second, that the surveys have been executed dimension in the indigenous population. Also included year after year since their inception regardless of the was the Net Promoter Score (NPS) measurement, a rating adversities they have faced, including the COVID-19 scale that qualifies the user's willingness to recommend pandemic and the cyber-attack of the institutional the facility based on the service provided in the last year, information system in 2022. In turn, the survey sample and which is used to understand the overall satisfaction achieves full representativeness of the population that and loyalty of patients with health services. entered the hospitals and used the CCSS outpatient services in the previous year. And in 2022 alone, 2,963 Implementation and improvement plans interviews were conducted for the hospitalization area and 18,503 for the outpatient evaluation.(5) Implementation The redesigned and piloted satisfaction surveys began to In addition, the integration of satisfaction surveys in the be applied in 2018 by the company CID Gallup. These are CCSS has generated a growing interest in the managers of conducted in the last three months of the year to all CCSS the provider units to know the voice of their users. This health service provider units (health areas, hospitals, and common language will be vital for the perfection of specialized centers). The first two years (2018 and 2019) improvement plans that, in the end, will achieve the main users were surveyed in person at the units. Due to the objective of the surveys, to increase the quality of COVID-19 pandemic, the surveys had to be conducted by services based on the users' voices. In fact, the telephone in 2020, and in subsequent years, it was implementation of specific surveys for the indigenous decided to continue with this modality. population has made it possible to make these users visible and identify them in health units where they were Results and improvement plans not previously recognized. The DICSS receives the results from the survey company, FACILITATORS informs the Board of Directors of the general results, and sends an individual report to each service provider unit The execution and continuity of the surveys resulted from with the results of all dimensions. Subsequently, each enabling factors that improved the process. These provider unit has one month to propose to the Medical included the following: Management an improvement plan for a minimum of Political and institutional will, and clearly established three indicators based on its results, giving priority to the responsibilities: In 2016, the Executive Presidency of the lowest scores. In turn, the units providing services to the CCSS issued an institutional commitment to move indigenous population must implement specific plans to forward with the redesign and implementation of the improve services based on their disaggregated results. satisfaction surveys. In addition, responsibility for On the other hand, the actions established in the plans coordinating and monitoring the project was given to the are executed by the personnel of each unit, and the units DICSS, which, in turn, integrated a multidisciplinary work must assign someone internal to follow up on the plans team made up of doctors, lawyers, statisticians and every six months. All improvement plans are recorded in systems engineers to manage progress. a standardized manner in a "Satisfaction Survey Improvement Plan Matrix" to facilitate their follow-up by Legal framework for the protection of users' rights: Law the Integrated Health Services Network Directorates and 8329 of the Legislative Assembly of Costa Rica describes comptrollers. the duties and rights to which all users of the country's health services have access. Specifically, Article 12 states EVIDENCE OF INTERVENTION SUCCESS that health service providers must "keep a record of patients' reactions to services, which will make it possible Two major results describe the successful progress of the to measure the degree of progress in achieving the goals satisfaction surveys. The first was to redesign the of continuous quality improvement."(6) This law was one Page 3 HNPGP Knowledge Brief of the main incentives for the CCSS to initially develop its the beginning of its implementation, the Executive user satisfaction surveys. Presidency held a press conference to communicate the benefits and objectives of the tool. In addition, the Public recognition of the best evaluated units: The best program was communicated in the internal media of the rated units in each of the health regions are publicly CCSS. On the other hand, training was given to network recognized through the media and social networks of the and unit directors on the rights and duties of users based CCSS (Figure 2). This inspires the units to improve their on the "Protocol of attention to the user".(7) service and increase their results to be part of the best units that users can see. Insurance system: The Medical Management, together with the CCSS Board of Directors, established possible sanctioning actions if health units failed to comply with the improvement plans. There is a perception that this serves as an incentive to take action toward improvement.(8) Fortunately, no one has been sanctioned yet, since so far all units have complied with their improvement plans. Health information system: The sample needed to survey users who used the services in recent months is easily obtained thanks to the Single Digital Health Record, which is widely implemented in Costa Rica's public health system. It also provides the contact information for interviewing users. Earmarked funding: The DICSS received a budget allocated specifically for redesigning and implementing the surveys. With these resources, it was possible to outsource the redesign of the questionnaire and the implementation of the surveys to organizations with expertise in these topics, since the CCSS did not have the installed capacity for these activities. Figure 2. Poster publication on best rated units in Being a PforR objective: The redesign of the satisfaction outpatient consultation by region. Credits: CCSS, 2022. surveys and a report on their results were two of the indicators promoted through the PforR, by common Publication of the general results of all units within the agreement between the CCSS and the World Bank. This institution: Year after year, the general results of all units ensured that the actions established were effective in can be viewed by the institution's personnel on an achieving the projected goals. internal communication platform. This promotes healthy competition through comparison and the search for CHALLENGES ENCOUNTERED AND SOLUTIONS better results in future years. APPLIED External surveyor: The CCSS decided to hire a third party Transformation addressed barriers and challenges during to conduct and analyze the surveys. This ensured the implementation of the surveys. In most cases there transparency and improved the quality of the interview was adequate execution of strategies to mitigate these by allowing users to express themselves freely. In developmental challenges. addition, the CCSS did not have the experience and Resistance from units: At the beginning there was some human resources to carry out the implementation of the resistance to the evaluation on the part of the health care surveys, and their evaluation, in a short period of time to provider units. This gradually diminished as it became an meet the timelines. indispensable tool for improving user care. Communication of the program to the units: To improve Lengthy and difficult bureaucratic processes: The the acceptance of the survey by the institution's staff at process of contracting third parties to conduct the survey Page 4 HNPGP Knowledge Brief annually is very lengthy and complex, so much so that it evidenced by the fact that during the COVID-19 has even jeopardized the timely start of the survey pandemic, when the CCSS made additional efforts to implementation. Fortunately, DICSS is aware of these serve the population and maintained active and constant lengthy internal approval processes and begins communication with its users, the satisfaction achieved negotiations well in advance. during the survey conducted in that year of 2020 was higher than in previous years. Similarly, this could be seen Non-response to the survey: Conducting the survey by by a decrease in the satisfaction observed after the cyber- telephone has slowed the progress of the surveys due to attacks on the system in 2022. Users mentioned that, due a drop in the response rate. Interviewers report that to the cyber-attack, their care in the health units was people do not want to answer because of distrust, or affected by the lack of availability of their clinical records, simply because they do not answer the phone.(9) This has negatively influencing their perception of the service. been solved by replacing the sample and extending the time to conduct the interviews. At the same time, The institution will continue to apply the surveys on an interviewers are being trained in methods to sensitize annual basis in the coming years as it has done so far. One users to accept the interview. of the changes going forward is the redesign of the questionnaire used in the survey, since the current Difficulties in reaching a representative sample size in format has been the same since its creation in 2017 and the indigenous population by health area: In the first there could be new services, such as tele-consultation, two years, the results for the indigenous population were which are not currently evaluated. In turn, the DICSS is disaggregated by health area, but it was difficult to expected to be responsible for optimizing the continue this way due to the small sample size. For this development of unit improvement plans and providing reason, the results continued to be presented at the them with closer follow-up and support to catalyze more national level. positive health service outcomes. For example, one of the Achieve superior quality improvement plans: Each unit main negative results of the surveys focuses on is responsible for identifying and executing its responsiveness (e.g., speed of care and willingness of improvement plans. There is currently no quality health care providers to solve user problems) and the improvement department to support or follow up on care and waiting times in the units, so improvement plans these plans. Therefore, there is heterogeneity between should be more focused on this. the improvement plans submitted by each unit and their On the other hand, the Health Statistics Area of the CCSS results. It is hoped that, in the future, the Comptroller's is generating statistics with an ethnic-racial focus and is Office will be able to provide closer follow-up to improve in communication with civil society organizations the quality of each center's improvement plans. representing indigenous populations to improve the COVID-19 Pandemic: The conversion to telephone measurement of their satisfaction.(10) surveys presented challenges in locating users as the The road travelled and the lessons learned (Box 1) from contact information contained in the EDUS could be out Costa Rica can serve as an example for other institutions, of date. This is improved from year to year with the health systems or countries seeking to implement a updating of EDUS information. satisfaction survey. The CCSS has already taken the first important step on the road to improving health services THE WAY FORWARD based on users' opinions by positioning satisfaction Through the conduct and analysis of the patient surveys as a key element in the institution's progress satisfaction survey, the CCSS has learned that the toward quality of service and user satisfaction. The satisfaction of its users is not directly correlated with the political will and the allocation of resources have been level of resources allocated to the health units or their key to sustain its application over the years. Its continuity infrastructure. This indicates that user satisfaction is will make it possible to optimize the redesign of health probably more influenced by the interpersonal services based on the demands of the entire Costa Rican relationships between providers and clients and does not population. depend solely on the physical inputs or the technical competence of the health personnel. This was further Page 5 HNPGP Knowledge Brief Box 1. Lessons learned • Consulting users is the essential first step in designing a satisfaction assessment tool based on people's preferences and value. • Knowing the target population of the survey helps to adapt and write questionnaires with local sensitivity. • Generating a pilot allows the tool to be adapted to the local context and to make modifications based on the challenges encountered in the field. • The hiring of an external interviewer allows the surveys to be transparent and allows users to express themselves honestly, avoiding bias in the results. • It is necessary to have a clear work plan and goals, since times evolve very fast, and it is necessary to comply with the surveys on an annual basis. • The improvement plans implemented by the units should be linked to the results of the satisfaction surveys to try to improve or resolve the needs of greatest concern. These plans should be followed up to ensure compliance. The Health, Nutrition and Population Knowledge Briefs of the World Bank are a quick reference on the essentials of specific HNP-related topics summarizing new findings and information. These may highlight an issue and key interventions proven to be effective in improving health, or disseminate new findings and lessons learned from the regions. For more information on this topic, go to: www.worldbank.org/health. Generous funding support for this series of Knowledge Briefs was made possible by The Access Accelerated Trust Fund. Page 6 HNPGP Knowledge Brief References The World Bank. Program Appraisal Document for a Strengthening Universal Health Insurance in Costa Rica. Program-for-Results. 2016. 2. Health OPdl. Final Verification Report - Indicator linked to disbursement #6/ Milestone 1: User satisfaction survey redesigned and approved by Board of Directors: Pan American Health Organization 2019. 3. Chacón Sánchez M. Interview Satisfaction Surveys - Program by Results CCSS/BM In: Rosado Valenzuela AL, editor. 2023. 4. Be Beyond. Net Promoter Score (NPS): What is it and how is it calculated? 2021. https://bebeyond.es/nps-que-es-y-como-se-calcula/ Institutional Directorate of Health Services Comptrollers. Presentation - Results of Satisfaction Surveys Outpatient and Hospitalization 2022. Costa Rican Social Security Fund; 2022. Rights and Duties of Users of Public and Private Health Services, 8329 (2002). 7. Salas AP. Interview Satisfaction Surveys - Program by Results CCSS/BM. In: Rosado Valenzuela AL, editor. 2023. Institutional Directorate of Health Services Comptrollers (DICSS). Generalities Process Application and Results, Satisfaction Surveys of Users in Outpatient and Hospitalization 2018, 2019, 2020, 2021. Caja Costarricense de Seguro Social; 2021. p. 16. Mora Cedeño YJ. Interview Satisfaction Surveys - Program by Results CCSS/BM In: Rosado Valenzuela AL, editor. 2023. Acuña Carrillo A. Analysis of the indigenous population in Costa Rica, its distribution in the attention in the Costa Rican Social Security Fund. 2023. Page 7