REVIEW ME FRAMEWORK PENSAAR 2020 1 © 2019 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW, Washington, DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved 1 2 3 4 19 18 17 16 This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colours, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions This work is available under the Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO) http://creativecommons.org /licenses/by/3.0/igo. Under the Creative Commons Attribution license, you are free to copy, distribute, transmit, and adapt this work, including for commercial purposes, under the following conditions: Translations—If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by The World Bank and should not be considered an official World Bank translation. The World Bank shall not be liable for any content or error in this translation. Adaptations—If you create an adaptation of this work, please add the following disclaimer along with the attribution: This is an adaptation of an original work by The World Bank. Views and opinions expressed in the adaptation are the sole responsibility of the author or authors of the adaptation and are not endorsed by The World Bank. Third-party content—The World Bank does not necessarily own each component of the content contained within the work. The World Bank therefore does not warrant that the use of any third-party-owned individual component or part contained in the work will not infringe on the rights of those third parties. The risk of claims resulting from such infringement rests solely with you. If you wish to re-use a component of the work, it is your responsibility to determine whether permission is needed for that re-use and to obtain permission from the copyright owner. Examples of components can include, but are not limited to, tables, figures, or images. All queries on rights and licenses should be addressed to the Publishing and Knowledge Division, The World Bank, 1818 H Street NW, Washington: DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. November 2019 2 Contents ABBREVIATIONS ...................................................................................................................................... 4 EXECUTIVE SUMMARY ............................................................................................................................ 5 Chapter 1. Introduction ...................................................................................................................... 7 1.1 Background ............................................................................................................................. 7 1.2 Objective of the Study ............................................................................................................ 7 1.3 Report Rationale ..................................................................................................................... 8 Chapter 2. Evolution of the Portuguese Water Sector and PENSAAR 2020 ....................................... 9 2.1 Institutional Reforms and Current Status Quo of the Water Sector....................................... 9 2.2 Performance Assessment and Role of ERSAR ....................................................................... 12 2.3 Monitoring and Evaluation of PENSAAR ............................................................................... 15 Chapter 3. Analysis of the PENSAAR 2020 M&E Framework ............................................................ 19 3.1 Strategic Plan and M&E Structure ........................................................................................ 19 3.2 Data and Indicator Sources ................................................................................................... 21 3.3 Issues with Current Indicator Definitions in the M&E Framework ....................................... 23 Chapter 4. Proposed Changes to the M&E Indicators ...................................................................... 25 4.1 Justification ........................................................................................................................... 25 Chapter 5. M&E Platform Design ...................................................................................................... 29 5.1 Considerations for Platform Design ...................................................................................... 29 5.2 Functional Requirements ...................................................................................................... 29 Chapter 6. Conclusions ..................................................................................................................... 32 APPENDIX A Draft Technical Specifications .......................................................................................... 33 APPENDIX B OVERVIEW OF CHANGES .................................................................................................. 49 Pillar 1—Environmental Protection and the Improvement of the Quality of Water Bodies ............ 49 Pillar 2–Improvement of the Quality of the Services Provided ........................................................ 52 Pillar 3–Optimization and Efficient Management of the Resources................................................. 55 Pillar 4–Economic, Financial, and Social Sustainability ..................................................................... 61 Pillar 5–Basic and Cross-Cutting Conditions ..................................................................................... 63 References ............................................................................................................................................ 66 3 ABBREVIATIONS AA water supply (abastecimento de água) AdP Ã?guas de Portugal APA Portuguese Environment Agency (Agência Portugesa do Ambiente) APDA Portuguese Association of Water and Wastewater Operators (Associação Portuguesa de Distribuição e Drenagem de Ã?guas) APESB Portuguese Association of Sanitary and Environmental Engineering (Associação Portuguesa de Engenharia Sanitária e Ambiental) APRH Portuguese Association of Water Resources (Associação Portuguesa de Recursos Hídricos) AR wastewater ((águas residuais) ERSAR Water and Waste Services Regulation Authority (Entidade Reguladora dos Serviços de Ã?guas e Resíduos) GAG PENSAAR 2020 Management Support Group GoP Government of Portugal https hypertext transfer protocol secure IRAR Institute for the Regulation of Water and Solid Waste (Instituto Regulador de Ã?guas e Resíduos) IWA International Water Association M&E monitoring and evaluation MTR midterm review OP operational objective OWASP open web application security project society PDCA plan, do, check, act PDSA plan, do, study, act PEAASAR Plano Estratégico Abastecimento de Ã?gua e Saneamento de Ã?guas Residuais PENSAAR Plano Estratégico de Abastecimento de Ã?gua e Saneamento de Ã?guas Residuais 2020 POSEUR Operational Program for Sustainability and Efficient Use of Resources (Programa Operacional Sustentabilidade e Eficiência no Uso dos Recursos) RASARP Annual Report on Water and Waste Services in Portugal SAR wastewater collection and treatment (saneamento de águas residuais) SRSS Structural Reform Support Service SSE State Secretariat of Environment UWWTD Urban Wastewater Treatment Directive W3C World Wide Web Consortium WCAG Web Content Accessibility Guidelines WSS water supply and sanitation 4 EXECUTIVE SUMMARY Twenty-six years since the approval of the 1993 sector policy reform, Portugal has witnessed a remarkable progress in the provision of drinking water supply and sanitation (WSS) services, closing most of the gaps it had in the provision of WSS services with other European countries and substantially increasing social well-being, public health, and environmental standards. For the past 19 years, the implementation of WSS sector reforms in Portugal has been guided by three successive strategic plans: PEAASAR 2000–06 (PEAASAR I), PEAASAR 2007–13 (PEAASAR II), and PENSAAR 2020– A New Strategy for the Water Supply and Wastewater Sector, still under implementation. While widely seen as successful, as substantiated by the international comparison in the midterm review (MTR) of PENSAAR 2020 (Deliverable A.1), PENSAAR 2020 also offers an opportunity to reflect on lessons learned to inform the next strategic plan. The European Commission has provided a Trust Fund under an Administrative Agreement with the World Bank to support the Government of Portugal (GoP) in conducting a midterm review (MTR) of PENSAAR 2020, aimed at sustaining and improving the WSS sector’s performance. The objective of work executed under the Fund are to improve performance of PENSAAR 2020 and develop the outline for the next strategic plan. The activities are structured along three lines: (A) full mid-term assessment and proposal of remedial actions; (B) improve the monitoring and evaluation framework; (C) develop the outline for the next strategic plan. This report, which corresponds with deliverable B.1, enhances the current monitoring and evaluation (M&E) framework for the implementation of PENSAAR 2020. In addition to recommendations on how to improve the current M&E framework, it provides insights into the current framework, proposes an updated framework, and provides suggestions based on these findings for future strategic plans. The report is an external input that will enable the State Secretariat of Environment (SSE) of the Government of Portugal (GoP) and the PENSAAR 2020 Management Support Group (GAG) to enhance the implementation of PENSAAR as a strategic plan for sector development. The current M&E framework of PENSAAR 2020 outlines a comprehensive set of 49 performance indicators—organized around the strategic and operational objectives—based on the system of performance indicators developed for the regulation process by the Water and Waste Services Regulation Authority (ERSAR). The framework can be considered a good performance assessment system, in particular concerning the quality and reliability of the information, although it lacks alignment with the PENSAAR 2020 strategy and there are inconsistencies (e.g., objectives without any indicators and indicators without any corresponding objectives). Further, the M&E framework presents some technical shortcomings that need to be addressed to allow for a better use and comparability of the system. The evolution of the Portuguese water supply and sanitation (WSS) sector over the past 26 years was significant. The combination and alignment of decisive public policies with capable institutions and a proactive and effective regulation have allowed the sector to achieve good results. Regulation was valuable for the sharing of information and knowledge, and in the dissemination of good practices and provision of incentives to improve the performance of the sector. To supplement PENSAAR 2020, a Management Support Group (GAG) was created. It is responsible for the M&E of PENSAAR 2020, and has tried to overcome a weakness in the first two strategic plans: the lack of an appropriate M&E framework for implementation. GAG’s role is based on an intermediate 5 and annual definition of a set of performance indicators that are compared with the baseline, and with the target set in the plan and with respective progress. PENSAAR 2020’s strategic plan relates to the state of the sector and has a significant number of layers. The strategic plan comprises five fundamental strategic pillars, and has 19 operational objectives. For the operational objectives, 48 measures and 133 actions are defined. This could be translated into a pyramid structure, with the strategic level on top (pillars) and the tactical (operational objectives) and operational (actions and measures) levels below. If the number of stakeholders, data sources, and performance measures in PENSAAR 2020 are considered, every element (down to the 181 measures and actions) should be monitored for progress, cost, and effectiveness. The coherence of the monitoring framework with the structure of PENSAAR 2020 influences the outcomes. For example, the absence of metrics for pillars or the fact that some indicators at the level of operational objectives are not addressing the objectives have important consequences. Data sources and how the indicators are obtained are considered reliable. Although some improvements might be required, the current system—based mostly on data collected by ERSAR—is reliable since the quality of information is assessed and data are audited. Changes to the current M&E system are proposed, based on the relevance of the indicators and their relation to the targets. Suitable alternative indicators are suggested, and others are simply removed, considering the existing constraints. These changes have gained consensus among a group of key stakeholders. Changes to M&E indicators for the current PENSAAR 2020 are proposed. Of the current 49 performance indicators, 20 should be be removed and four should be created. Of the remaining indicators, 10 should continue to be computed in the same way, and for the remaining (19 indicators), their computation formula is proposed to be altered. To be able to capture the information and quickly disseminate technical specifications for a new software platform to support the M&E framework were prepared. The tool should improve access to information and facilitate M&E of PENSAAR by GAG and help the GoP in the analysis of the sector toward key strategic decisions. The changes proposed in this deliverable represent an evolution and fine tuning of the current M&E framework, allowing for its implementation in the months left in PENSAAR 2020. The remaining issues in this deliverable should be considered at the drafting stages of the next strategic plan. 6 Chapter 1. Introduction 1.1 Background For the past 19 years, the implementation of the water supply and sanitation (WSS) sector reforms in Portugal has been guided by three successive strategic plans: PEAASAR 2000–06 (PEAASAR I), PEAASAR 2007–13 (PEAASAR II), and PENSAAR 2020–A New Strategy for the Water Supply and Wastewater Sector, still under implementation. The monitoring committee to draft the latter strategic plan was established by the then Secretary of State for Environment (SSE) and Spatial Planning through order No 9304/2013 of July 6, 2013. The motto of this plan is “A strategy to serve the people: quality services at a sustainable price.â€? The European Commission and the World Bank share the objective of building competitive and sustainable economies and reducing poverty and social exclusion—the goals of the Europe 2020 Agenda, which emphasises smart, sustainable, and inclusive growth to achieve these. The European Commission and the Bank concur that direct interaction is beneficial to both institutions and ultimately benefits their client Member States. This is particularly true for the provision of analytical, advisory, and knowledge services and technical assistance. Hence, the European Commission has provided a Trust Fund under an administrative agreement with the World Bank to support the Government of Portugal (GoP) in conducting a midterm review (MTR) of PENSAAR 2020, aimed at sustaining and improving the WSS sector’s performance. 1.2 Objective of the Study The objective of the work to be executed under the Fund was to improve performance of PENSAAR 2020 and develop the outline for the next strategic plan. The activities are structured along three lines: (A) full midterm review and proposal of remedial actions; (B) improve the monitoring and evaluation framework; (C) develop the outline for the next strategic plan: A. Improve the implementation of the Water Supply and Wastewater Sector Plan - PENSAAR 2020 by carrying out a midterm review of the implementation of PENSAAR 2020, that will enable the State Secretariat of Environment (SSE) of the Government of Portugal (GoP) to enhance implementation of the Sector Plan; B. Enhance the current monitoring and evaluation (M&E) framework for the implementation of PENSAAR 2020 and design of a new M&E platform to follow performance of the current and future strategic plans; and C. Prepare the outline of the next WSS strategic plan (for the period 2021-2027) focusing on specific areas and issues that need to be addressed to improve sector performance in line with good international and European practices. This deliverable reflects the findings on the M&E framework. It strengthens the annual report on the performance of PENSAAR 2020 and supports the design of the current and M&E framework for the next strategic plan. It is an external input that will enable the SSE and the Management Support Group (GAG) to enhance the implementation of PENSAAR as a strategic plan for sector development. The current report is the third of the series and corresponds to deliverable B.1, focusing on the second objective. Activity B.1, based on the findings of Activity A.1 (the MTR), is structured around three steps (IBRD/World Bank 2018): 7 Step 1: Relevance of indicators Step 2: Data collection improvement Step 3: Information processes A separate document, draft technical specifications for a new M&E platform for the Portuguese WSS sector, has been prepared in March 2019. These technical specifications can be found as appendix A in this report and address the third step: information processes. 1.3 Report Rationale The assessment and monitoring of progress against established objectives are critical part of any strategic plan. Analyzing the success of individual actions and their contribution to achieving the strategic objectives allows planners to improve the allocation of resources, determine the critical success factors, and realign the strategy when necessary. The PENSAAR 2020 M&E framework comprises 49 indicators organized around the project’s hierarchical structure of pillars, objectives, measures, and actions. The foundations are based on the system of performance indicators developed by the Water and Waste Services Regulation Authority (ERSAR), incorporating refined definitions and data quality assessment and control procedures. The goal is to propose an updated PENSAAR 2020 M&E framework. As such, this report intends to do the following: • Review the PENSAAR 2020 M&E framework • Review the relevance and relation of the PENSAAR 2020 indicators with the strategic and operational targets. • Propose technical specifications for an M&E platform. Deliverable A.1, PENSAAR 2020 MTR, is an external input that enables the SSE to enhance the implementation of PENSAAR as a strategic plan for sector development. The MTR examined (i) the appropriateness of the design of PENSAAR 2020; (ii) whether the Plan is still relevant; (iii) whether it is on track to achieve its targets; and (iv) the issues affecting good implementation. The deliverable finds that some of the targets are not likely to be achieved and that M&E can and should be improved to be more effective and contribute toward PENSAAR 2020s implementation. Therefore, the updated PENSAAR 2020 M&E framework considers the analysis and conclusions of the MTR. 8 Chapter 2. Evolution of the Portuguese Water Sector and PENSAAR 2020 2.1 Institutional Reforms and Current Status Quo of the Water Sector In the past 26 years, Portugal has seen significant progress in the water supply and sanitation (WSS) services, with a substantial improvement of key performance indicators (KPIs), either concerning coverage (physical access to the services) in water supply and wastewater collection and treatment or concerning environmental quality and quality of service. Although there are many challenges, including sustainability in its various dimensions (financial, social, and environmental), the results and the evolution of the sector are noteworthy. The implementation of clear and determined public policies have contributed to this state of affairs through appropriate regulation and adequate planning instruments focused on a well-defined strategy for the sector. The combination and alignment of decisive public policies with capable institutions and a proactive and effective regulation have allowed the sector to achieve these strong results. Figure 2.1 presents the evolution of the water supply services coverage (panel a), wastewater services coverage (panel b), the evolution of the quality of drinking water (panel c), and the quality of coastal bathing water (panel d) between 1993 and 2016. Figure 2.1 Evolution of Water Service Coverage, Wastewater Service Coverage, Drinking Water Quality, and Coastal Bathing Water Quality, 1993–2016 a. Access to drinking water supply services Source: Based on ERSAR documents, RASARP (AA01 Retail indicator) 9 b. Access to wastewater management services Source Based on ERSAR documents, RASAPR (AR01 Retail indicator) c. Drinking water quality Source: Based on ERSAR documents, RASARP (AA04 Retail Incidator) 10 d. Coastal bathing water quality Source: Based on APA documents, REA (State of the Environment Report) Within Portugal’s public policies for the water sector, the legal (and regulatory) framework published from 1993 onward must be highlighted. It started a revolution in WSS market structure through the separation between bulk and retail systems, and the possibility of private sector participation, and a plurality of management models for the WSS provision, which allowed healthy competition and provided incentives for improved performance. The planning instruments—through implementing strategic plans—provided clear policies and strategic guidelines for the development of the water sector. The three generations of plans, PEAASAR I (2000–06), PEAASAR II (2007–13), and PENSAAR 2020 (2014–20), with different concerns toward the Portuguese water sector development, have guided involved actors for accomplishing public policy. With the reform of the legal and institutional framework, institutions were created and roles of the main stakeholders were clarified: (i) Ã?guas de Portugal (AdP), the main operator of the bulk systems and responsible for providing national infrastructure; (ii) municipalities, the holders of the retail systems; and (iii) regulators, both of the environment (including water resources) and of water services. Moreover, regulation has proved to be valuable for the sharing of information and knowledge and in the dissemination of good practices and provision of incentives to improve the performance of the WSS sector. Regulation, however, continues to be unbalanced and has only recently become universal. Regulation has promoted and encouraged improvement of Portugal’s overall performance: private companies were regulated first, then the bulk companies immediately after, and finally of the municipal companies, promoted and encouraged the improvement of the overall performance. Although the water sector development results are, overall, very positive, there are still major challenges, in particular to overcome asymmetries. For example, the larger operators and the ones in coastal areas show a high performance in contrast to that of rural ones. The latter are very relevant in 11 number and area covered, but small in terms of percentage of the population served, and their performance is poor because they have not been able to take advantage of the reforms. The main challenges the sector faces include (i) the operational efficiency improvement and the optimization of systems; (ii) the financial sustainability of the operators and, in particular, the financing of the water sector; and (iii) the enhancement of environmental efficiency along with the robustness and resilience of existing infrastructure. The success of the institutional and development model would not be possible without the investment of billions of euros; yet they are mostly from European Union (EU) subsidies that will not be available in the future. Also, the institutional issues, which were well designed and aligned to provide the right incentives to the coast and major cities, still need to be fully implemented throughout Portugal through more homogeneous regulation and effective enforcement. It is a critical issue that the sector become more cohesive and inclusive, in which no sharp differences exist between the small towns of the countryside and the main cities along the coast. In addition, new institutional challenges have emerged, such as the problems of rainwater (especially its business and financing models), and new water sources and resources such as reclaimed water, sludge and nutrient management, and the circular economy. All these problems and challenges of the water sector are included in PENSAAR 2020, but they will not be overcome, on the whole, during the remaining time. The task is to identify what goals are still possible under the current strategic plan (Remedial Actions, Deliverable A.2) and prepare the new PENSAAR based on the experience and lessons learned from the implementation of PENSAAR 2020. 2.2 Performance Assessment and Role of ERSAR The regulation of WSS services has helped to leverage and improve the performance of the water sector in Portugal. Among the many positive contributions of the Water and Waste Services Regulation Authority (ERSAR)—initially Institute for the Regulation of Water and Solid Waste (IRAR)1—two deserve attention. The first relates to the availability and sharing of information. Since inception, one of ERSAR’s main functions is the collection of information, including the activity of the operators. ERSAR has published Annual Report on Water and Waste Services in Portugal (RASARP) since 2004,2 which disseminates information about public water supply, urban wastewater, and urban waste management services. It is a collaboration with entities that intervene in those sectors. All the information about the Portuguese water sector is on the Internet. The data and variables are standardized (individual, global, and per cluster) for the convenence of any stakeholder. In addition to the benefits of the transparency associated with water services management and its public scrutiny, the quality of the information is controlled not only by regulatory authority audits but also by the possibility of checking the robustness and accuracy of the information. The second relevant and positive aspect of the water services regulation is related to the quality of service model adopted by ERSAR, based on the comparison, disclosure, and public discussion of the performance shown by the operators (known as “sunshine regulationâ€?). There are now 14 performance indicators for water supply and 14 performance indicators for wastewater, regardless of 1 ERSAR replaced IRAR in 2009 by means of the Decree-Law No. 277/2009, of October 2, which expanded its duties and competences. 2 In 2002 and 2003, the first two versions of RASARP were published, but they include only drinking water quality. 12 the system featuring the bulk or retail component. The performance indicators assess operators’ sustainability in three dimensions (social, financial, and environmental). Through this benchmarking exercise and yardstick competition, operator performance is evaluated and sorted through a pictorial system. Utilities that receive a green score have a good performance; a yellow score highlights an acceptable performance, but with ability and obligation to improve; and a red score indicates a poor performance. This model of performance evaluation has been applied since 2004 and is now accepted by the operators. It is unanimously acknowledged as valuable and encouraging for improving the performance of the water sector. The performance evaluation model of ERSAR is in the third generation of performance indicators.3 Figure 2.2 is an example of performance evaluation model for the retail water supply service. Evaluation of each performance indicator is compared with the target set; the model also shows the evolution of that indicator over the years and comments, if necessary. 3 ERSAR (previously IRAR) has changed its performance indicator framework every five years. The first proposal, developed in 2003, consists of 20 performance indicators, the second has 16, and the third, as displayed in figure 2.2, comprises 14 performance indicators for WSS services (and for the bulk and retail systems). 13 Figure 2.2 Sample Regulatory Model of Quality of Service of ERSAR for the Retail Water Supply Source: ERSAR. The implementation of regulation in the water services and the activity of ERSAR (until 2009 IRAR) were very positive, because they created and encouraged a national culture of performance evaluation in the sector. Most operators (especially those from the coastline and major cities) calculate and include performance indicators or performance metrics in their activity reports and use performance indicators in their daily routine for management and decision making. The same process happens with other stakeholders in the water sector. Although many of the performance indicators are distinct, depending on the interests and perspective of each entity or group of entities, they all 14 use ERSAR’s data and information (some aspects are obtained from the Portuguese Environment Agency [APA]). Hence, stakeholders of Portugal’s water sector agree that the information to be used for different purposes, either in quantity or quality, must be those assembled by ERSAR. However, there is no such consensus concerning the performance indicators since the perspective of each user is unique and, therefore, performance indicators relevant for ERSAR, for example, may not be so for the utilities, consumers, or the financial institutions. Moreover, there are differences in the calculation formulas even when the indicators are the same. For example, one of the most important performance indicators concerns the coverage or the physical access to the water supply or wastewater services. It is relevant for all stakeholders, but it is calculated differently by each entity, and, in general, there is disagreement as to the choice of calculation on the basis of the number of households, as ERSAR does. ERSAR assesses the sustainability of WSS services in social, financial, and environmental dimensions with its performance evaluation system related to the quality of service regulation. The Secretary of State for the Environment (SSE) of the Government of Portugal (GoP) wants the monitoring and evaluation (M&E) of the water sector plan (in this case PENSAAR 2020, the strategic plan for the water services for mainland Portugal) to evaluate the implementation of the plan, particularly public policy. The M&E of PENSAAR 2020 relies on performance indicators and metrics applied to the water sector and to its WSS services to evaluate the accomplishment of the operational objectives. It only by chance uses the performance indicators adopted by ERSAR, even though it uses ERSAR’s data and information. By evaluating the degree of success of the metrics materialization concerning the operational objectives, it will be ensured, at least qualitatively, the success of several axes of public policies for the water sector, which with PENSAAR 2020 corresponds to its pillars. Thus, given the nature of PENSAAR 2020, the performance indicators are not the same as in the system adopted by ERSAR. They are more numerous because they are more comprehensive; involve other themes and other protagonists (not only the operators); and look especially for a more global perspective, rather than the individual, as appropriate for a regulator. 2.3 Monitoring and Evaluation of PENSAAR One of the weaknesses acknowledged in the first two strategic plans of the WSS services (PEAASAR I and PEAASAR II) was the lack of an appropriate M&E framework for their implementation. These strategic plans defined the public policy for the sector through objectives and metrics, whose success was effectively analyzed and verified only when a new plan (i.e., PEAASAR II and PENSAAR 2020) was prepared. Two obvious and negative consequences were (i) the effectiveness of the implementation of the public policies and the plan’s appropriateness were checked only at the end of the plan without proper monitoring; and (ii) it did not allow the adoption of interim measures and actions to improve their effectiveness or even change the objectives and the corresponding actions and measures based on the changing reality. Thus, M&E encompasses important management support instruments for these plans, allowing the validation of the strategy, operational objectives, and performance indicators, or correcting them as events on the ground require. M&E results can lead to adjustments of the original measures and actions so goals can be achieved. This fragility associated with inadequate M&E for the first two strategic plans (PEAASAR I and PEAASAR II) was acknowledged when preparing PENSAAR 2020. The GoP created a support group to manage 15 PENSAAR 2020. Order No. 6747/2015, of June 8, by the SSE created the Management Support Group (GAG) of PENSAAR 2020. Among its roles are the following: • Define the responsibility of the measures and actions proposed in the Plan of Action of PENSAAR 2020. • Suggest priorities for all the proposed measures in line with the critical aspects, priority actions, and themes defined. • Propose a timetable for the implementation of those measures in accordance with the priorities. • Promote and monitor the implementation of the measures with the entities involved. • Review the progress in the development of priority actions and anchor projects. • Evaluate the performance of PENSAAR 2020 through the targets set in the plan and, if necessary, propose adjustments to the values. • Evaluate the progress of the indicators and accomplishment in the water sector set out in the Operational Program for Sustainability and Efficient Use of Resources (Programa Operacional Sustentabilidade e Eficiência no Uso dos Recursos [POSEUR]). • Accomplish the evaluation process and continuous review of PENSAAR 2020, proposing the necessary adjustments with a view to the implementation of the measures and objectives of PENSAAR 2020 and the fulfilment of the national targets. GAG members nominated a president and coordinator of a working group to manage activities. As required by the order that created it, GAG should present publicly each year PENSAAR 2020 evaluation reports to allow analysis of its implementation. This evaluation report will consider annual information available from APA and ERSAR concerning the water sector of mainland Portugal. GAG has produced reports in December 2016, December 2017, and December 2018. They analyze (i) the responsibility of the measures and actions proposed in the action plan of PENSAAR 2020; (ii) priorities for all the measures; (iii) the timetable for implementation of those measures in line with defined priorities; (iv) the promotion and monitoring of the measures consistent with the defined priorities; (v) the promotion and monitoring of the measures’ implementation; (vi) the analysis of the progress in the development of priority actions and anchor projects; (vii) the performance evaluation of PENSAAR 2020; (viii) the evaluation of the progress of the indicators set out in POSEUR; and (viii) the continuous assessment and review of PENSAAR. GAG’s role in M&E of the implementation of PENSAAR 2020 is based on an intermediate and annual definition of a set of performance indicators that is compared with the baseline, the target, and its respective progress. This makes it possible to evaluate the degree of implementation of the operational objectives and the effectiveness of the measures and actions. Due to the large number of performance indicators—and that not all of them have equal importance and effectiveness in the monitoring of the implementation of public policies prescribed in PENSAAR 2020—priority is given to a limited set of indicators. This prioritization considers a set of critical aspects and the corresponding actions. The critical aspects include the poor performance of many utilities, the lack of ability of the direct management without autonomy model4 of WSS services to perform its roles, economic and financial nonsustainability, 4 Management model as described in the MTR (Quantitative Analysis). 16 dependence of the sector on financial resources, and the risk of exceeding the acceptable limit of social inclusiveness. GAG has also evaluated the status quo of the implementation of the measures, actions by stakeholders, and implementation of anchor projects, according to PENSAAR 2020. Despite the positive step in creating the GAG there is room for improving its performance through a more effective M&E of the implementation of PENSAAR 2020. This report aims to give contributions in this sense, with the objective of some being applied within this plan period and others being incorporated and adopted in the monitoring of the future strategic plan. Two types of performance improvements would enhance the performance of the actions of GAG. One is associated with the operation and governance of GAG, and a second—at a more operational and technical level—concerns the performance indicators that monitor implementation, including their relevance, data collection improvement, and information processes. This second level is the main purpose of this report, and the object of intense and detailed analysis and discussion in the following chapters. Operation and governance. GAG lacks comprehensive representativeness and stakeholder involvement, especially from municipalities. Municipalities are responsible for the implementation of a very significant set of measures and actions included in PENSAAR 2020 and for the performance of part of the water sector. Some actions can improve this situation, including political (coordination between the SSE and local governments), technical (involving technical associations, such as the Portuguese Association of Water and Wastewater Operators [APDA], the Portuguese Association of Sanitary and Environmental Engineering [APESB], and the Portuguese Association of Water Resources [APRH]), and institutional (through regulatory and other administrative authorities). Second, there should be communication improvements, particularly with the local government and other key stakeholders. Information must be available on the Internet so it can reach all interested parties. Implementation of “naming and shamingâ€? policies are often very effective. Third, more resources and logistic conditions for acting should be assigned to GAG. Not only the future platform will be of great importance but also other human and material resources should be granted. Human resources associated with GAG should be paid, without which a proper accountability will not exist. Performance indicators. The M&E of GAG is based mainly on checking the progress of the performance indicators defined for each operational objective in the plan. However, most of the following critical issues are identified as follows: • Too many indicators removes the focus of the analysis and evaluation of the performance of the plan implementation and of its operational objectives. • The need for aggregation of some indicators to get a higher-ranking performance metrics to match the operating objective (or pillar) is controversial, discretionary, and sometimes jeopardizes the credibility. • Several performance indicators lack a baseline, so it is not possible to observe the progress and effectiveness of the measures and actions. • Some less well-defined indicators do not allow evaluating or do not contribute to the evaluation of the measures and actions’ effectiveness. • The baselines of several indicators correspond to the target, or their accomplishment does not reflect an effective improvement since they are very close to it, making them ineffective. 17 They penalize the evaluation because they remove the focus and give the illusory idea that they can compensate for others that possibly yield a lower performance. • Some operational objectives lack performance indicators or associated metrics so it will not be possible to evaluate the effectiveness of the measures and actions. • Priorities for the monitored indicators do not always match those that contribute most or that are useful to check the improvement and effectiveness of the achievement of the operational objectives, especially ones associated with critical aspects. There are other important issues to address in this scope, such as ones related to data collection and information processes. These areas are bottlenecks and weaken the role and performance of GAG. These and other related issues will be analyzed and discussed in the next sections. 18 Chapter 3. Analysis of the PENSAAR 2020 M&E Framework This chapter provides an analysis of overall structure of the monitoring and evaluation (M&E) framework of PENSAAR 2020 (a preliminary analysis based on the midterm review [MTR] is in chapters 2 and 3.) The first section corresponds to the analysis of PENSAAR 2020’s structure and the monitoring framework. One cannot be understood or developed without having the other in mind, and any lack of coherence between the two is the most significant outcome of this analysis. The second section deals with data sources and how the indicators are obtained. It addresses issues raised in the objective for the assignment step 2: “data collection improvement.â€? The origin of the data used in PENSAAR 2020 is analyzed and improvement paths are considered. The third section identifies issues common to the entire M&E framework. They are the basis for many of the indicator change proposals outlined in chapter 5. 3.1 Strategic Plan and M&E Structure PENSAAR 2020 (figure 3.1) consists of three levels: strategic pillars, operational objectives, and measures or actions. The hierarchy contains five fundamental strategic pillars; for each pillar, operational objectives are established (19 total), and each objective encompasses a series of measures (48) and actions (133). This can be translated into a pyramid structure, with the strategic level on top and the tactical and operational levels below. Figure 3.1 Strategic Pillars (Axes) and Operational Objectives Source: PENSAAR 2020, Volume 2. 19 Figure 3.2 Interpretation of the PENSAAR 2020 Hierarchy - Pillars Strategic - Strategic objectives - Long term - “big pictureâ€? - Operational objectives - Medium term Tactical - Paths to achieve the strategic objectives - Actions & measures - Specific, budgeteable actions to contribute to the operational Operational objectives - Shorter term - Rapidly changing Source: World Bank elaboration. Figure 3.2 shows an interpretation of the PENSAAR 2020 hierarchy. Although the terminology can be disputed (e.g., operational objectives are under the tactical level), the actions and measures feed into operational objectives, and those into pillars. Considering the number of stakeholders, data sources, and performance measures in PENSAAR 2020, every element (down to the 181 measures and actions) needs to be monitored for progress, cost, and effectiveness. However, one of the key shortcomings of the PENSAAR M&E is the lack of alignment between the strategy and the monitoring framework. While the PENSAAR framework hierarchy (pillar, objective, measures, actions) is—in theory—clear, in practice it becomes more difficult to follow. The definitions are not always strictly followed, and some actions could be considered measures or vice versa. Sometimes objectives contribute to pillars they are not linked to. This lack of clarity has a significant impact on the M&E framework, as it becomes difficult to establish metrics for the different elements and, above all, establish how individual metrics contribute to the success of the different objectives. As a matter of fact, the M&E framework is even more challenging to interpret because there is no one- to-one correspondence between the elements in the hierarchy and the assessment elements in the M&E framework. The lower the levels in the hierarchy of PENSAAR 2020, the more complicated they become. The pillars represent the strategic objectives on a medium-term time frame (six years). The operational objectives would usually correspond to a tactical level (below the strategic one), yet their names do not help to clarify this. Finally, measures and actions are at an unclear third level. In principle, they would correspond to an operational, day-to-day level. However, the difference between measures and actions is not well defined, and they do not provide enough detail or specific targets. This confusion extends to the monitoring system. Ideally, the M&E framework allows policy makers and others to determine the success of each category in the project structure. Therefore, with an adequate development of indicators and performance metrics, it should be possible to state whether 20 an action has been completed, an operational objective has been reached, and the level of completion of the strategic objective of a pillar (e.g., completed for 85 percent). As a result, the M&E framework fails to deliver on one of its key functions: evaluating the progress and success of the strategy. Even though the indicators, measures, operational objectives, and pillars make perfect sense individually, the strategy would benefit from a closer link between them and a coherent set that enables a simple and systematic assessment and feedback on the established strategy. This is a quite common issue, especially in those systems that result from participation of many stakeholders, requiring a consensus and coordination of a significant number of opinions. Usually, projects organized under such circumstances will yield very sensible individual indicators, but often those performance assessment systems will lack the necessary discipline that facilitates the analysis of those indicators at a later stage. There are now gaps in the performance assessment system and a certain lack of coherence that makes the system more difficult to interpret. First, there are no metrics for the pillars (strategic objectives), and the progress toward the achievement of those pillars is not assessed. Consequently, there are no targets, and one cannot determine quantitatively the progress toward achievement of the strategic objective. (One possible approach could be to set up an index based on fulfilling operational objectives.) At the operational objective level, some indicators do not address the objectives, but rather the processes leading to the achievement of the objectives. Examples are the unit cost indicators for pollution removed in OP1.1, or water abstraction per person in OP3.6. These indicators measure issues relevant to the objective and will influence the progress toward its achievement, but they are not aimed at assessing whether the objective is fulfilled. Further, these indicators do not necessarily address the measures or actions for that same operational objective. There are no indicators for measures and actions. Some may not be needed if the measures and actions are implemented in the short term and can be assessed with a yes/no question, but longer-term measures should have monitoring attached. These are all structural issues within the M&E framework that cannot be addressed at the current stage and would require major changes to the framework and its philosophy. As a short-term (and far from perfect solution), for the current plan, a higher correlation is needed between objectives, measures, actions, and indicators. A single indicator or index should be linked to each objective. This is the only way to determine in a standardized and consistent way whether the objective has been fulfilled or if progress is being made toward it. 3.2 Data and Indicator Sources The current performance assessment system is heavily reliant on the performance indicators developed and collected by the Water and Waste Services Regulation Authority (ERSAR). A few indicators are also obtained from the Portuguese Environment Agency (APA). In developing the current M&E system, it seems that planners tried to force the indicators and their definitions to fit with PENSAAR 2020. This would, in principle, be against the theory that each indicator 21 should be custom selected and designed according to the monitoring and assessment needs of the strategy. As a consequence, some of the concepts that the current indicators are measuring are not fully aligned with the operational objectives. However, the quality of data collected by ERSAR is extremely reliable. This statement is based upon two facts: Quality of information is assessed. The ERSAR system is one of the few regulators in the world that includes information on the quality of data used to calculate the indicators. This has been a long- standing recommendation by the International Water Association, because considering the metrics alone can be misleading. Utilities reporting to ERSAR declare the degree of uncertainty for each data element used in calculating the performance indicators. For instance, the age and size of domestic meters will have an impact on the uncertainty of the volumes measured with them. ERSAR takes this uncertainty into account. Having to report on data quality makes each utility aware of the importance of data quality, the potential sources of data, and the need to improve data collecting and handling procedures. Data is audited. Auditors appointed by ERSAR visit each utility and audit the data submitted to the regulator. Thus, the data displayed by these indicators are not only as accurate as they can be but also have undergone a greater quality control than any other potential indicators. Audits assure that definitions are applied homogeneously by all service providers, guaranteeing a great consistency at a national level. Data collection improvement is step 2 of this deliverable, and the ERSAR performance assessment system is very good. It does not present significant problems, gaps, or bottlenecks. Data are collected periodically; they are reported along the corresponding quality of information grade; and they are independently audited. However, ERSAR data represent a problem for the remaining indicators. No other indicator in the PENSAAR 2020 system includes any quality of data assessment. Besides the environmental indicators collected by APA, the remaining indicators are collected by ERSAR. However, no additional structure was created for the remaining PENSAAR 2020 needs. Performance assessment projects require the allocation of clear responsibilities and motivation among the involved parties. One problem is that the Portuguese Ministry of Environment (MoE) seems to have only limited leverage over municipalities to submit additional data. Municipalities are already overwhelmed with requests for data from several administrations. In addition, as highlighted in the midterm review (MTR), municipality engagement in preparing PENSAAR 2020 was insufficient. Even if the MoE could convince municipalities to submit additional data, it would still need an internal structure with such responsibilities as (i) who would coordinate data submissions; and (ii) who is responsible in each municipality for the reporting and setting up of basic quality control mechanisms to guarantee that definitions are applied correctly and the data flow guarantees the quality of the calculated indicators. Otherwise, if municipalities reported with the current (nonexistent) structure, the indicators would never be of the same quality or consistency as the ones collected by ERSAR (or APA). Providing a similar standard of quality requires a significant amount of time and resources. The current objectives sometimes lack alignment with ERSAR indicators. However, creating new indicators to be aligned with the strategy requires resources and determination. In PENSAAR 2020 this has resulted in a strategy that often matches ERSAR’s performance measures, and the existence of a few situations in which the indicators loosely match (or do not match at all) the items in the strategic plan. This situation creates an additional difficulty when drafting the next strategic plan. A choice will 22 need to be made between remaining with objectives aligned with the (excellent) indicators from ERSAR or creating complementary performance assessment capabilities that allow the appropriate monitoring and evaluation of the strategy. Any new data collecting initiative should be preceded by clear mechanisms to drive or facilitate reporting. Those entities required to report should be strongly compelled to do so, and adequate carrots and sticks would need to be in place. Otherwise, the M&E framework would exist only to make the lack of good data periodically reported more evident. It is strongly recommended to continue to use as much data from ERSAR as possible, even considering that the match between objectives and indicators may not be perfect. In future frameworks, using ERSAR’s reporting system should not be taken to mean that ERSAR indicators should be used as-is. ERSAR collects a significant number of variables to produce its indicators, and access to these raw data could result in possibilities to produce metrics that are a closer match to PENSAAR objectives. 3.3 Issues with Current Indicator Definitions in the M&E Framework A detailed analysis of the 49 indicators used in the current PENSAAR framework has been performed. Individual changes and amendments are suggested in the following pages, but overarching comments can be applied to a significant number of these indicators. ERSAR’s discrete evaluation per number of households should be avoided. ERSAR has a traffic light system that qualifies the value of each indicator in one of three levels. This method is aligned with the role of a regulator, because it lets stakeholders know its opinion on the quality of the service provided for each indicator. A significant number of PENSAAR indicators use these discrete performance intervals to define the M&E indicators. As a result, the formula “% of households covered by operators with satisfactory evaluation on the indicator…â€? is used. This formulation is far from intuitive, especially for those not familiar with the ERSAR system. The formula used for these indicators measures the percentage of households that have reached a satisfactory level of quality of service within ERSAR’s scale for the indicator. Indicators defined this way are different in nature, and post-process data in a significant way when compared to more traditional ratios. In a way, these indicators do not measure reality, but, rather, the assessment of performance that ERSAR performs in its regulatory function. For instance, this could mean that instead of monitoring the levels of leakage in a network, the indicator would be monitoring whether ERSAR believes that those levels are satisfactory. While it makes sense to ascertain how many households receive “satisfactoryâ€? service, this formula is not the best option to monitor and evaluate PENSAAR. By defining indicators this way, the original data are masked, because the indicator does not express the actual performance but ERSAR’s assessment on such performance. As a consequence, data resolution is lost and the possibilities to analyze the data and obtain conclusions are limited. Because performance is expressed in discrete steps (as opposed to a continuous function), changes in actual performance may not be reflected in the indicators (data behind an indicator may improve their 23 value, but as long as they are classified in the same performance category, the indicator will show the same result). The worst possible case for this phenomenon could result in the aggregation of the indicator at a national level. For instance, a generalized increase in performance at a national level could be represented by this type of indicators as a performance getting worse. A small change in performance may lead to a change in the traffic light indicator (i.e., green to orange). In such a case, if many utilities improve significantly their performance within the green range, yet one utility decreases its performance slightly from green to orange, the total performance would be expressed as “worse than in the previous period.â€? However, in absolute terms, it could have improved. The satisfactory formula is not recommended for PENSAAR and should be replaced by indicators that are built using utilities’ raw data. The only exception would be the quality of service objectives that require combining several indicators into a quality of service index. Since the ERSAR system is well known, accepted by the industry, and provides a common denominator to aggregate very different metrics, an index based on aggregating traffic lights could be used for this purpose. New definitions of ERSAR-based indicators should be adopted. The current definition of performance indicators used by ERSAR should be used for all PENSAAR indicators. These are included in the third generation of the performance assessment system, published March 15, 2018. The ERSAR codes and definitions used in the documentation for this review (up to the MTR from this year) are from the previous generation of indicators. For instance, former indicator AA14ab (compliance with abstraction licenses) now does not even have a code (it is a system profile indicator). The new definitions are often aligned with international standards, and it may be convenient to update all indicators to their newer versions in ERSAR. While this updating could create issues when trying to determine the time evolution of the indicators, these gaps could also appear in the future if ERSAR should decide not to collect the old variables anymore, which would lead to the absence of reliable data to calculate the old indicators. At present, there is no indication that ERSAR has plans to stop collecting these data. Bulk and retail indicators should be unified and merged. In the current system, some indicators are duplicated, with one version for bulk and one for retail, yet have different definitions (e.g., for real losses, length of pipe is used as a denominator for bulk, and service connections for retail). This is a direct transposition of ERSAR’s system, in which alternative definitions of the same indicator are provided when needed (they share the same indicator code and display letter a for bulk, or letter b for retail). This duplication is not useful at the PENSAAR level. It is complex for users of the M&E framework, and complicated to aggregate bulk and retail indicators at a national level. For instance, even though it may be very interesting for PENSAAR to monitor real losses in both retail and bulk systems, this should not be done using different indicators for bulk and retail. A suitable solution would be to use the bulk indicator—per kilometer of network—for all systems and obtain an aggregated figure for the entire country). If the intention is to monitor performance of retail and bulk systems separately, both indicators should be used with independent targets (and address separate objectives). 24 Chapter 4. Proposed Changes to the M&E Indicators 4.1 Justification The following sections contain general guidelines and proposed changes for updating the current system of performance indicators. Most of the suggested changes are justified with one of the following reasons: • There is a lack of alignment between objective and indicator. Either the indicator does not match the objective or there is no indicator for that objective (in which case one is suggested). • There is a new definition of the indicators (or their variables) from the Water and Waste Services Regulation Authority (ERSAR). The new definition should replace the old one. • The current indicator follows the “number of households with satisfactory evaluationâ€? formula. The proposed indicators will be built using raw data collected from utilities. • The current indicator is duplicated, or two similar indicators address the same objective. • The current indicator has not been calculated and there is no target, or there is not a clear definition of the indicator. • A new definition can improve interpretation or international comparisons (particularly in those indicators defined per household, because most of them could be defined per population). The proposed changes do not address all possible issues. However, they are proposed following a pragmatic approach. Because PENSAAR 2020 will conclude by the end of 2020, any significant changes imply finding out baseline values from 2014, determining intermediate values, and transposing any existing targets to similar ones. It is not feasible to create data collection or processing mechanisms to support new indicators not collected by current data providers. In addition, benefits from updating current indicators are limited in terms of implementing new policies or reassessing the strategy. Many of the lessons learned during the review of the current indicator system should be taken into account when discussing the next strategic plan. The suggested changes are detailed in the following sections, organized by pillars and per operational objective. All codes and descriptions in the tables are from ERSAR-LNEC (2018). For a proper understanding of the changes to the indicators and framework, this guide should be used as reference.5 Appendix B, provides the detailed explanation of the proposed changes to the M&E indicators for PENSAAR 2020. In table 5.1 a summary of the changes is presented. Of the 49 performance indicators, 20 are proposed to be removed and four new ones created. Of the remaining indicators, 10 are suggested to be kept the same way they are currently computed, and the remaining (19 indicators) are proposed to be changed in their computation formula. One indicator has been moved from OP1.1 to OP1.2. 5 The guide is available at ERSAR website, http://www.ersar.pt/pt/publicacoes/publicacoes-tecnicas/guias. 25 Table 5.1 Summary of Proposed Changes to Indicators Proposed Operational objective Indicator changes Pillar 1: Environmental protection, improving the quality of water bodies OP1.1: compliance with 1. Improving the quality of water bodies Keep regulation 2. Number of litigation processes Remove 3. % of households covered by operators with satisfactory Move to OP1.2 evaluation in the Indicators Wastewater analysis (AR14) and and change compliance with discharge parameters (AR15) 4. % of households covered by operators with satisfactory Keep assessment at the appropriate wastewater destination (AR12) OP1.2: reduction of 1. Improving the quality of water bodies Remove urban pollution in water 2. Unit cost of pollution removed Remove bodies OP1.3: increase in 1. % of households covered by operators with satisfactory Change physical accessibility to evaluation in the indicators physical accessibility of the service the waste water service (AR01) 2. % of households covered by satisfactory individual Change wastewater solutions in respect of the total of households without physical accessibility to the service (dAR09b/(dAR11b-dAR08b-dAR07b)) Pillar 2: Improving the quality of services rendered OP 2.1: improvement of 1. AA04-safe water (%) Change the quality of the AA 2. % of households covered by operators with satisfactory Change service assessment in the occurrence of failures in supply (AA03) 3. % of households covered by operators with satisfactory Remove assessment of the occurrence of malfunctions Conducts N. º/100 km. ANO) (AA11) 4. Greywater users’ satisfaction of the water supply service Remove 5. Response to complaints and suggestions New OP2.2: improving the 1. % of households covered by operators with satisfactory Change quality of the SAR assessment in the occurrence of floods (AR03) service 2. % of households covered by operators with satisfactory Change assessment in the occurrence of structural collapses in sewers (AR09) 3. Level of satisfaction of wastewater service users Remove 4. Response to complaints and suggestions New Pillar 3: Optimization and efficient management of resources OP3.1: optimization of 1. % of households covered by operators with satisfactory Change the utilization of assessment of adherence to the service (AA07) installed capacity and 2. % of households served by operators with satisfactory Remove increased adherence to assessment in the adequacy of treatment capacity (AA09) the service 3. % of households covered by operators with satisfactory Change assessment of adherence to the service (AR06) 26 4. % of households served by operators with satisfactory Remove assessment in the adequacy of treatment capacity (AR07) OP3.2: reduction of % of households covered by operators with satisfactory Change water losses assessment of the actual losses of Water (AA13) OP3.3: control of undue 1. % of households covered by operators with an action plan for Remove influences the control of infiltrations and undue influences to the public wastewater systems implemented 2. % of undue affluent to public wastewater systems in Remove operators with a plan of action implemented for the control of infiltrations and undue inflow 3. % of the treatment capacity used under conditions of overuse Remove (%) (AR07adapted) 4. % of households served by operators with satisfactory Remove assessment in the control of emergency discharges (AR13) 5. % of households served by operators with satisfactory Remove assessment in the occurrence of floods (AR03) 6. % of volume of treated infiltrations and undue inflows New OP3.4: efficient asset 1. % of households covered by operators with satisfactory Change management and assessment in the rehabilitation of pipelines (AA10) increased rehabilitation 2. % of households covered by operators with satisfactory Change evaluation in the rehabilitation of sewers (AR08) OP3.5: valorization of 1. Reused residual water/treated residual water (%) Change resources and by- ((dAR25i+dAR25ii)/(dAR24) products 2. % of sludge valued relatively to volumes produced: V (%) = Keep ((quantity of treated sludge applied in agriculture + quantity of sludge treated by composting + amount of sludge energetically valued)/total quantity of sludge) * 100 3. Own production of energy/energy consumption (%) Change (DAA28/dAA29 and dAR27/dAR28) OP3.6: allocation and 1.% of households covered by operators with satisfactory Change efficient use of water assessment in compliance with the licensing of funding (AA14) resources 2. Water abstraction per person (lpcd) Keep (dAA20ab/(dAA07b x N. º hab. per households)) 3. Consumption per person (billed) (lpcd) Keep (dAA16b/(dAA07b x N. º hab. per households)) 4. % of population served by operators with recognized merit in Remove the efficient use of water Pillar 4: Economic, financial, and social sustainability OP4.1: sustainable 1. % of households covered by operators with satisfactory Keep recovery of assessment of the economic accessibility of the service (AA02) expenditures 2. % of households covered by operators with satisfactory Change assessment of expenditure coverage (AA06) 3. % of households covered by operators with satisfactory Keep assessment of the economic accessibility of the service (AR02) 4. % of households covered by operators with satisfactory Keep assessment of total spending coverage (AR05) 5. % of households covered by EGs with social tariff Keep 27 OP4.2: reduction % of operators with operating costs satisfactory within a Remove and/or optimization of reference band of efficient unit operating costs operating expenses OP4.3: reduction of % of households covered by operators with satisfactory Change nonrevenue water assessment in nonrevenue water (AA08) Pillar 5: Basic and cross-cutting conditions OP5.1: increased Infrastructural and asset management knowledge index Change availability of (DAA44) information Infrastructural and asset management knowledge index (dAR45) New OP5.2: innovation 1. Number of programs/partnerships between public and Remove private entities with research and teaching centers OP5.3: improvement of 1. % of lodgings with satisfactory quality (good + median)— Keep the operational indicators of the partnership agreement framework, 2. % of households covered by management entities serving less Change management, and than 10,000 households provision of services OP5.4: climate change, % of households covered by operators with contingency plans, Remove natural disasters, risk- energy efficiency, security, and adaptation of approved reduction, adaptation infrastructures (or actions) and in implementation OP5.5: externalities: 1. No. of jobs created in the private sector Remove employment, 2. No. of tenders launched and awarded Remove competitiveness, 3. Volume of contracts awarded to national companies in the Remove internationalization AA and SAR sectors in other countries Note: AA = water supply; SAR = wastewater collection and treatment (saneamento de águas residuais). 28 Chapter 5. M&E Platform Design 5.1 Considerations for Platform Design The issues in step 3 of this deliverable are to improve timely access to information; enable the identification of outliers; and enable the key water supply and sanitation (WSS) stakeholders to make fully informed strategic decisions. A significant part of the solution is in the new monitoring and evaluation (M&E) Platform that will be developed with Structural Reform Support Service(SRSS) funding for PENSAAR 2020. This does not mean that a type of software is the answer to all problems, or that by making the new platform available, all data access and analysis issues will be solved immediately. However, the platform can become an enabler or an obstacle. An appropriate design complements the data reporting procedures, strategic framework structure, and responsibilities and the analysis methodologies needed for PENSAAR. An adequate platform development should show updated information, and it needs a clear structure of administrators and users (data facilitators and data users). During the process of data collection from municipalities, the platform should facilitate individual data submission and later aggregation, as well as the quality control from a centralized account. If different users were to see different indicators and required different reports and queries, the platform should be flexible enough to show each user what is relevant. A significant part of the platform’s success will come from allowing users to explore and analyze the data. It need to be flexible to support significant analysis or strategy adaptations. This can be done by expanding indicators into subindicators (e.g., nonrevenue water [NRW] into its components), clustering data sources and excluding those submitting poor data, or by allowing the creation of user- generated dynamic graphs or reports that can be updated automatically as new data are collected. 5.2 Functional Requirements The concepts described in section 6.1 are in the draft technical specifications prepared in March 2019 for the design of the M&E platform (appendix A), based on the following international good practices: • “One-stop shopâ€? approach. Provides support during conception until deployment of the live version including training and coaching of key users. • Flexibility of data collection and links to information systems . Ensures autonomy in management and processing of data; the platform should allow for editing and creation of new datasets according to the requirements of a next strategic plan. • Data hub. On which data analysis and dissemination of information can occur solely for the purposes of the applicable sector strategic document; integration with information systems of key partners such as the Water and Waste Services Regulation Authority (ERSAR), Portuguese Environment Agency (APA), and Ã?guas de Portugal (AdP) should be a priority. • Dissemination of information should consider platform audiences. Expected outputs should include dashboards, Excel exports, PDF reports, or the visualization of a limited and controlled set of information for external purposes. Functional requirements for the platform included in the technical specificiations are: ✓ Integration of PENSAAR 2020 M&E framework and existing information systems used for collection of WSS sector performance data into web-based IT platform solution; features should include: 29 • Collection of up to 200 variables for the calculation of up to 50 PENSAAR 2020 indicators. The contractor should allow flexibility of variable selection and calculation of strategic indicators. • Links to existing systems (e.g., ERSAR, APA, AdP) for utilization of raw data (variables) and a separate workflow to enable data collection capability by using transport files (no direct upload in the platform by stakeholders). • Flagging if information is missing by specified date and who has to provide this information (systems or separate system stakeholders, e.g., ERSAR, APA, AdP). • Synchronization of data with existing systems at predetermined periods. Most variables will be obtained (information synchronized) annually, but some variables would be updated more frequently depending on the capabilities of the underlaying systems (e.g., data on sludge utilization and water abstraction quantities). • Reflecting PENSAAR 2020 reporting requirements, although the platform will rely on raw data from existing systems such as ERSAR, APA, AdP, which has their own process, time frame, verification, and audit mechanisms. • Integration of new variables and calculation of new indicators (since PENSAAR 2020 indicators will change). ✓ Platform analytical capabilities: • Breakdown of information, clustering, and web visualization and producing of tables, graphs, maps, and other presenting options. This will help the PENSAAR 2020 monitoring committee to understand and present information in the annual report on reasons for performance or nonperformance of indicators that monitor objectives’ achievement. • Application programing interface so internal users can create dashboards and specific data analysis. • Quick check option by registered user of entry number, type of data of available, data by the same user, previous entries, historical performance data. • Information presentation as dashboards, Excel exports, PDF reports, or visualization of a limited and controlled set of information for internal and external purposes. • Developing and visualizing in the platform performance information for each indicator for public users. • Predetermined customized report following the requirements of the annual reporting. ✓ Platform users: • Options for public and internal user interfaces of the platform, allowing the exercise of different functions divided by users. • Workflow enabling data collection capability of the system in which data providers have unique username and access password (issued by the system to each user). • All users must have authorized access to platform functionality. Unauthorized users should have access only to annual reports and data available for public view. ✓ Data storage and statistics: • Structured storage of electronic data used for calculation of indicators (e.g., data file, XML), allowing to access historical data and re-create M&E framework indicators as presented in the annual report. 30 • Generating visit rate statistics for the platform and optimizations for easy search by Internet search engines. The contractor should assess ongoing efforts to upgrade existing systems that are operated and maintained by WSS sector stakeholders, and allow for future links to such systems (e.g., APA is developing a new water resources platform). 31 Chapter 6. Conclusions The current monitoring and evaluation (M&E) framework of PENSAAR 2020 presents a comprehensive set of performance indicators organized around the strategic and operational objectives. The indicators are well defined, collected, and audited in most of the cases by the regulator, and include quality data. It is a good performance assessment system, especially regarding the quality and reliability of the information it provides. One of the key shortcomings of the PENSAAR M&E is the lack of alignment between the strategy and the monitoring framework. The framework does not measure progress of the strategy at the different levels (strategic, operational and actions). Further to this structural issue it also does not collect crucial information (financials) to determine the efficacy of the strategic plan. The framework was designed taking into account the available metrics (mainly from the regulator) instead of creating a structure to cater the needs of the strategic plan. The system presents technical challenges that could be overcome, resulting in better aggregation of the indicators at a national level, international comparability, and overall usability of the system. The alignment between the M&E framework with the PENSAAR 2020 strategy is not fully coherent: gaps can be observed in objectives with no indicators and indicators with no corresponding objectives. This is probably due to the participation of a large number of stakeholders in the process without adequate facilitation to reach an adequate technical solution. As existing datacollection is used, in most cases where the data have not been collected by ERSAR or APA, there are no data at all. This is the result of a lack of a data collecting structure and a robust platform that can integrate the different data sources and facilitate integrated reporting. The changes proposed in this report represent an evolution and fine-tuning of the current system. These changes presented in chapter 5 have been discussed through a preliminary consultation process with key stakeholders. The evolution of the system is not perfect, and it is aimed to address significant issues that should and can be corrected before the end of the strategic plan period in December 2020. However, other issues that are more structural need to be addressed at the early stages of the next strategic plan. The next strategic plan should have a one-to-one alignment between each objective (at different levels) and a corresponding metric. For each indicator, clear definitions, aggregation procedures, and the entity responsible for data collection should be specified. The structure should enable users to quickly assess each strategic, tactical, and operational objective, determining whether they have been achieved or on-track to be achieved. Within the monitoring framework also secondary data should be collected especially on financial resources. Further to the design of the framework also a clear feedback loop should clarify how adjustments will be made to either the strategic plan, the goals to be achieved or the required resources to achieve the goals. Additionally, the data reporting framework should be significantly strengthened, with allocation of responsibilities for data submission and a single reporting platform that can centralise the information from the diffrerent data sources and facilitate a consistent and repeatable assessment of the progress of the strategic plan. 32 APPENDIX A Draft Technical Specifications Background Context During the past two decades, the water supply and sanitation (WSS) sector reforms in Portugal have been guided by three successive strategic plans: PEAASAR 2000–06, PEAASAR 2007–13, and PENSAAR 2020. PENSAAR 2020 incorporates results-based assessments of previous plans, identifying the achievements and bottlenecks derived from their implementation. From these assessments, it should be highlighted: • Reduction of infringement cases for noncompliance with European and national legal acquis. • Positive contribution of the WSS sector toward overall compliance with the environmental objectives set for the water bodies, including protected areas (bathing waters). • Overall achievement by utilities of satisfactory levels of service quality provision based on performance indicators set by the Water and Waste Services Regulation Authority (ERSAR). • Little progress has been achieved in the private sector involvement. • Not enough progress to address, among others, water services optimization and full cost recovery, asset management, and nonrevenue water (NRW). • Need to promote management models through horizontal and vertical integration approaches to improve service efficiency and quality, promote economies of scale, and ensure the sustainability of the sector. The broader context of the implementation of PENSAAR 2020 raises special challenges for ensuring that the expected targets are achieved. In the aftermath of the financial crisis, funds available for the WSS in Portugal had to be curtailed due to budgetary constraints and limitations on the levels of tariff increases that the population could accept. As a consequence, investment levels have been below what is needed for sustainable WSS management. The move toward full cost recovery has been curtailed, affecting the financial sustainability of the sector. PENSAAR 2020’s focus on efficiency shall have a positive impact on the fiscal situation of local authorities in charge of WSS services, increase the creditworthiness of WSS operators and future investment levels, and reduce the magnitude of the need for future tariff increases. Beyond the focus on efficiency—both at services provision level and resources use—the successful implementation of PENSAAR will decisively contribute to the compliance with the European Union (EU) Environmental Acquis, including with the Urban Wastewater Treatment Directive (UWWTD). It will also have a significant impact on economic and social cohesion policies by guaranteeing the populations´ access to water supply services, taking in due consideration of quality, physical access, and affordability issues, and improving systems’ overall efficiency and economic effectiveness (e.g., economies of scale, tariff convergence schemes, and reduction of inequalities between regions). 33 Specific Information about the Strategy The vision of the sector, is described as “a sector at the service of the population and the economy of the country providing services of good quality and sustainable in environmental, economic, financial and social terms.â€? Development is based on good governance in partnership and sustained by the strategy. The strategic framework established by PENSAAR 2020 relies on five key pillars: • Environmental, protection, improvement of the quality of the water bodies • Improvement of the quality of the services provided • Optimization and efficient management of resources • Economic, financial, and social sustainability • Basic and cross-cutting conditions These five pillars have been built out to establish 19 operational objectives (see blue boxes in figure A.1). Figure A.1 Strategic Pillars (Axes) and operational objectives Source: Frade 2014. The strategy takes into account the national legal framework, European Union (EU) objectives for the WSS sector, and other national-level plans and programs, although some of these plans have faced delays that have impacted the ability for the water strategy to align with them fully. The strategic framework considers community needs through consultations. PENSAAR 2020’s action plan was formulated through a set of 48 measures and supported by 133 distinct actions. Each operational objective (OP) in figure A.1 has been built out based on actions required to achieve these strategic plans. These measures and actions have cost estimates in PENSAAR 2020, using the best available knowledge, and a financial analysis measures the economic, financial, and social sustainability of the proposed investments. PENSAAR also identifies the capacity of the sector to generate the required resources. The total cost of the measures is estimated to be €3,705 34 million. Most of the cost is for optimization of the sector and efficiency measures, followed by environmental measures and improvement of quality of service provision (table A.1). Table A.1 Total Cost of the Measures for PENSAAR 2020 Pillar Cost Environmental, protection, improvement of the 918 quality of the water bodies Improvement of the quality of the services 739 provided Optimization and efficient management of 1,828 resources Economic, financial, and social sustainability 12 Basic and cross-cutting conditions 208 Total 3,705 €, millions The results show the limited ability of the sector to generate these revenues through monthly charges. This confirms the importance of the strategic choice of investments, especially those that demonstrate significant added value, either as a catalyst for the creation of incentives, support to increase the performance of service providers, or where clear social needs are being met (e.g., meeting the needs of low-income populations, which may not generate revenue). Some proposals relate to eligibility criteria and prioritization in the new Operational Program for Sustainability and Efficient Use of Resources (POSEUR). The Strategic Plan is intended to guide the sector to the new paradigm, establishing clear leadership and consensus across sector institutions—including industry partners—with a focus on (i) restructuring the sector; (ii) financing the sector; (iii) improving service delivery; (iv) and monitoring and review of the Plan. See figure A.2 for an overview. Figure A.1 PENSAAR 2020 Management Plan Source: Frade 2014. 35 Objectives The contractor should develop, test, launch, and train Ministry of the Environment (MoE) staff on the operation and maintenance of the platform. The system should demonstrate operation with data for 2017 and reproduce key performance indicators presented in the last Management Support Group (GAG) report. The technology to develop the platform should be open and based on known operating systems or open source codes. It will employ the following good international practices: • “One-stop shopâ€? approach. Provides support during conception until deployment of the live version including training and coaching of key users. • Flexibility of data collection and links to information systems . Ensures autonomy in management and processing of data; the platform should allow for editing and creation of new datasets according to the requirements of a next strategic plan. • Data hub. On which data analysis and dissemination of information can occur solely for the purposes of the applicable sector strategic document; integration with information systems of key partners such as ERSAR, Portuguese Environment Agency (APA), and Ã?guas de Portugal (AdP) should be a priority. • Dissemination of information should consider platform audiences. Expected outputs should include dashboards, Excel exports, PDF reports, or the visualization of a limited and controlled set of information for external purposes. Required Tasks The contractor will develop an M&E platform following the requirements; test the system to ensure its operability using the links to existing databases; test the workflow of its data collection module; and train MoE staff on the operation and maintenance of the platform. The platform should assist PENSAAR 2020 monitoring committee during the preparation of annual report on the implementation of the strategy by automatization of reporting on the M&E framework. M&E Platform Development Functional Requirements ✓ Integration of PENSAAR 2020 M&E framework and existing information systems used for collection of WSS sector performance data into web-based IT platform solution; features should include: • Collection of up to 200 variables for the calculation of up to 50 PENSAAR 2020 indicators. The contractor should allow flexibility of variable selection and calculation of strategic indicators. • Links to existing systems (e.g., ERSAR, APA, AdP) for utilization of raw data (variables) and a separate workflow to enable data collection capability by using transport files (no direct upload in the platform by stakeholders). • Flagging if information is missing by specified date and who has to provide this information (systems or separate system stakeholders, e.g., ERSAR, APA, AdP). • Synchronization of data with existing systems at predetermined periods. Most variables will be obtained (information synchronized) annually, but some variables would be 36 updated more frequently depending on the capabilities of the underlaying systems (e.g., data on sludge utilization and water abstraction quantities). • Reflecting PENSAAR 2020 reporting requirements, although the platform will rely on raw data from existing systems such as ERSAR, APA, AdP, which has their own process, time frame, verification, and audit mechanisms. • Integration of new variables and calculation of new indicators for the next Strategic Plan (since PENSAAR 2020 indicators will change). ✓ Platform analytical capabilities: • Breakdown of information, clustering, and web visualization and producing of tables, graphs, maps, and other presenting options. This will help the PENSAAR 2020 monitoring committee to understand and present information in the annual report on reasons for performance or nonperformance of indicators that monitor objectives’ achievement. • Application programing interface so internal users can create dashboards and specific data analysis. • Quick check option by registered user of entry number, type of data of available, data by the same user, previous entries, historical performance data. • Information presentation as dashboards, Excel exports, PDF reports, or visualization of a limited and controlled set of information for internal and external purposes. • Developing and visualizing in the platform performance information for each indicator for public users. • Predetermined customized report following the requirements of the annual reporting. ✓ Platform users: • Options for public and internal user interfaces of the platform, allowing the exercise of different functions divided by users. • Workflow enabling data collection capability of the system in which data providers have unique username and access password (issued by the system to each user). • All users must have authorized access to platform functionality. Unauthorized users should have access only to annual reports and data available for public view. ✓ Data storage and statistics: • Structured storage of electronic data used for calculation of indicators (e.g., data file, XML), allowing to access historical data and re-create M&E framework indicators as presented in the annual report. • Generating visit rate statistics for the platform and optimizations for easy search by Internet search engines. The contractor should assess ongoing efforts to upgrade existing systems that are operated and maintained by water supply and sanitation (WSS) sector stakeholders, and allow for future links to such systems (e.g., APA is developing a new water resources platform). Architecture of the M&E Platform Accessibility. The platform should be accessible on the Internet through a stand-alone domain and integrated with the MoE website. The platform’s domain must meet MoE requirements and be free. 37 At least three domain variants shall be offered in advance by the contractor and approved by MoE. The selected domain shall be registered, retained, and assigned to MoE for at least 10 years. Structure. The platform should be developed in modules. The modules should focus on the most critical ones for the current PENSAAR 2020. The platform will be adjusted for the next strategic WSS sector document and modules will added as needed. A simplified chart of the proposed platform design is presented in figure A.3, panels a and b.6 Figure A.3 Simplified Platform Design, Variables and Indicators Subsections a. Variables 6 The final design might change. Here it is simplified to illustrate the goals. 38 b. Indicators The platform should include as a minimum the following modules: (i) PENSAAR 2020 M&E framework, (ii) data collection, and (iii) communication and reporting. The PENSAAR 2020 M&E framework module should allow: • Flexible definition of indicators, including new formulas and input variables. System should also allow to include context information. • Flexible definition of nonnumerical performance practices (e.g., existence of a certain facility: yes/no). • Management of operators (including operator creation, deletion, modification, and management of security). • Online portal for data uploads from the users, including the possibility of bulk uploads using CSV/Excel format templates or similar, and the editing of individual data to correct errors. • Built-in inconsistency checks for variables. • Automatic calculation of indicators for all operators based on variable values. • If the collection period is inferior to one year, the platform should be able to calculate aggregate annual indicators (either averaging or adding partial values). • Historic database, consolidating values from previous periods. • Reporting engine, allowing to query data from different operators and periods. • Graphing engine, allowing to produce graphs corresponding to the different queries. • Automatization of recurrent graphs or queries to facilitate reporting. • Export to common formats (graphic exports in vector or jpeg format, data exports in CSV/Excel format). The data collection module should allow for a separate workflow enabling data collection on variables that are not collected by existing platform or systems for the calculation of new indicators. The 39 approach should use transport files (no direct upload in the platform by stakeholders), following simple rules on data quality, audit, and acceptance. The communication and reporting module should allow generating references and information on the fulfilment of PENSAAR 2020 monitoring committee obligations. This will include (i) preparation of an annual report on the implementation of the strategy. The content should be predetermined, following the current structure of the GAG report, and allow boxes for writing an analysis, explanation, or providing context to the information presented in visuals; (ii) preparation of summarized information in relation to performance of indicators; (iii) provision of summary information on the implementation of the strategy for public users. Both monitoring committee and public dashboards can be placed hare. In discussion with MoE, the contractor shall determine the rights of access to types of historical performance reports and data presentation: public (no system login), semi-public (no system login), and private (system login; reserved for specific user rights). Technical Requirements to the System It is the understanding of the Bank that MoE is proposing that APA will host the M&E framework, but the contractor should reassess the selected option and discuss with the ministry which institution under MoE is best positioned (capacity and hardware capabilities) to host the new M&E platform. The portal solution should meet the following minimum technical requirements: Interface and programming language. The platform should be developed as a web-based online service, allowing to complete the full data cycle through a web browser. This would overcome potential compatibility problems between different operating systems and environments. The platform system will be developed based on the following technologies for the various components (see table A.2). Table A.2 Interface and Programming Language Requirements Database Open source priority options: PostGresSQL or MySQL Proprietary option: Microsoft SQL server 2017 (or newer version). Virtual cloud engine with proprietary version of Microsoft SQL server that can be used to host the database Integrated development PHP or Java Framework developments, according with APA environment hosting infrastructure optimized environment Programming languages Client page programming languages: HTML, CSS3, JavaScript, PHP Web programming language: HTML, CSS3, JavaScript, PHP Server-side programming language: PHP or Java (option) Data base programming language: SQL 40 Remote access to client’s When running the server provided by the client, during the server development and maintenance phases, remote access and access to files under server’s administrator authority is required Preferred methods of access: Remote desktop protocol (RDP) Classic FTP access Software for client The system will be web-based and designed to work with any machines of the newer Internet browsers. Since there are innumerable versions of browsers, the focus will be on three versions covering over 90% of the market: Google Chrome Mozilla Firefox Internet Explorer version Note: APA = Portuguese Environment Agency. Reporting engine. The platform should facilitate access to the data through custom queries and reports. Specifically, it should allow to: • Exclude certain operators from the reporting. • Create unlimited groups or clusters with the operators, classified under certain headers (clustered by size: small, medium, large). • Include any indicator, variable, context information, or practice in the query. • Automatically extend queries to all clusters in a category (same query for big, small, and medium utilities). • Allow automatization of exporting query results in standard formats. Automatic confirmation mechanisms. These shall be integrated in the platform workflow to enable data collection by using transport files (no direct upload in the platform by stakeholders). These mechanisms will include (i) structural verification: numeric values must be numbers, and required mandatory fields must be filled in; and (ii) value checks (preset maximum and minimum values; amplitude of year-to-year variations); and sequence checks between variables (in terms of sums; lower or higher, given the relationship between variables). See table A.3 for additional information. Table A.3 Additional Explanations for Working with the Platform Temporary data WSS utilities should be able to retain registered data that has not yet retention by utilities been sent, including data that do not comply with structural or value checks (i.e., errors or warnings will not prevent WSS utilities from temporary retaining of data). Data submission to WSS utilities will not be allowed to send to the platform data that fails the platform by to meet all structural checks of the transport file; utilities WSS utilities will not be allowed to send to the platform data that fails 41 to meet consistency checks (sums; lower or higher depending on the relationship between variables). Data confirmation or APA will ONLY be able to accept or reject ALL provided data (not one at rejection by the a time), providing different comments in a specific field/sphere: data platform confirmation/rejection one by one is not planned, because it will lead to complex issues in terms of validation interdependent data values (variables and related subvariables) and management of the data submission process. To avoid complete data set rejection due to several invalid data (which are not part of the mandatory ones) that cannot be corrected, WSS utilities will have to delete them when attempting to resubmit. Note: APA = Portuguese Environment Agency; WSS = water supply and sanitation. Graphic engine. This should allow to produce dynamic graphs with the data: • Graphs created from queries of the reporting engine. • Graph types: scattered plots (x versus y), box-whisker, bar graphs, line graphs, pie charts. • Dynamic graphs should be automatically updated If data is updated. • Graph definition for an operator or a cluster, may be extended for all operators or clusters. • Graph configuration needs to be dynamic and edited without the need of starting the graphing process from zero. Usability requirements. The user interface of the portal application must comply with the following minimum requirements: • Independence of the user interface functionality from user Internet browsers (e.g., MS Edge and Internet Explorer, Google Chrome, and Mozilla Firefox, as well as enabled mobile devices such as tablets and smartphones). • Multilanguage support for all user interface elements. • The user interface must meet requirements for design, usability, and accessibility for people with disabilities. • Portal accessibility policy must comply with Web Content Accessibility Guidelines (WCAG) 2.0 of the World Wide Web Consortium.7 • The user interface must be built using the latest technologies: HTML5 and CSS4, according to the standards set by the World Wide Web Consortium (W3C),8 or even newer technologies. • Verification messages or error messages should be displayed next to the enter box. Table A.4 details the roles and rights of each participant with different functionalities. The table is not exhaustive, and the contractor can agree to additional permissions with MoE. Table A.4 Roles and responsibilities 7 See the WCAG 2.0 requirements, https://www.w3.org/TR/WCAG20/. 8 See the W3C website, https://www.w3.org/. 42 PENSAAR 2020 Other platform users (monitoring committee) Raw data related rights DETERMINES a list of variables for which YES NO data has to be collected Can determine PERIODICITY for each of the NO: periodicity is annual by NO variables default (unless otherwise agreed with MoE) Can ENTER raw data NO YES: only following through the separate workflow enabling data collection by using transport files Can ENTER missing data to supplement NO Once sent to the what has already been sent to EWRC or platform validated NO: a special request to APA is required to be sent Can ALTER or DELETE all data sent to EWRC NO Once sent to the or validated platform NO: a special request to APA is required to be sent Can have a VISUAL of all primary data YES NO: only to its own available at the platform data (see ENTER raw data) Indicator related rights Can DETERMINE and apply new indicators in YES NO the system Report/reference related rights Can DETERMINE its requirements for YES NO reports/references Can DETERMINE level of access (rights) YES NO regarding its reports Functionalities Can ADD new or DELETE YES NO 43 Can ADD new or DELETE data originators YES NO Can ADD or DELETE new user YES: only monitoring YES: only internal committee users utility’s users Can BLOCK any user access to the platform YES YES: only utility’s users Generate and change passwords YES: only for monitoring YES: only utility’s committee users users Can DETERMINE values to be applied to YES NO primary data for performing consistency checks Note: APA = Portuguese Environment Agency; MoE = Ministry of Environment. Security Requirements. A user structure should be defined to guarantee that different user roles have access to adequate parts of the software (e.g., administrators, operators, data managers). See figure A.4. 44 Figure A.4 Example for an Architecture Covering Permissions, Groups, Profiles, and Users PERMISSIONS GROUPS PROFILES USERS Permission 1 Permission 2 Group 1 Profile 1 USER 1 Permission 3 Group n Profile 2 Permission n USER 2 Front end Permission 1 Group 1 Permission 2 Profile 1 Permission 3 Group 2 Profile 2 Permission 4 Group n Permission n Back end 45 • Contractor should identify potential threats, level of probability of their occurrence, and offer adequate protection of the system. • Contractor should try to ensure information security at each stage of system implementation: design, development, testing, and deployment. • Adhere to recommendations of the Open Web Application Security Project Society (OWASP)9 for protection of web applications and web services. • Portal application program code must be protected from attacks such as cross-site scripting and SQL injection. • The access to the portal must be on a Hypertext Transfer Protocol Secure (https) protocol with TLS 1.2 as a mandatory minimum. • When accessing the portal with user name and password, the password must be stored in encrypted form and use a robust encryption algorithm. • When accessing the portal with qualified electronic signature or personal indentification, a check must be made on its validity. In its proposal the contractor may justify and propose additional measures for enhancing the security and protection of the new M&E platform. System Testing The contractor must perform usability tests of the platform, including as a minimum: • Bench testing. Carry out a series of preliminary tests with selected users (as per MoE’s instructions) and develop a plan to address the needs for improvements. • Connectivity testing. Perform preliminary tests to ensure connection and utilization of raw data from existing systems and platforms. • Product. Periodically carry out product tests during the finalization of the platform with a sample (focus group) of future electronic service users (PENSAAR 2020 monitoring committee) to establish service usability and to eliminate difficulties and inconsistencies with the assignment and expected results of the platform. Training and Documentation The contractor must train the hosting authority, PENSAAR 2020 monitoring committee, and data originators (see data collection module), and organize a seminar for the public on the new M&E platform. The contractor should also ensure: • Provision of technical support, training, and transfer of skills to the MoE team with the aim of future autonomous management of the platform. 9 See the OWASP website, https://www.owasp.org/index.php/Main_Page. 46 • Provision of technical support and training of data originators for the first collection of data through the platform. • Three trainings should be organized (covering northern, central, and southern regions with up to 100 people in each) as well as one national seminar to present functionality and requirements of the new M&E platform. The contractor should provide all relevant documentation for the development and implementation of the platform, as well as a brief manual to the hosting authority to enable future autonomous management of the system. Active Commissioning of the M&E Platform The contractor should commission the M&E platform as per the timing in the next section, and ensure that it is fully operational and performs all relevant functionalities in line with the current design and specifications. All tests under 3.2. should also be performed to confirm operability of the M&E platform. Deliverables and Timing The expected start date is [insert date]. The conclusion of the contract is nine months later. Timely submission of acceptable deliverables and performance of activities as per the following timetable will be a prerequisite for payment. The contract will be procured by SRSS on behalf of the MoE. SRSS and the contractor will sign a contract for this assignment; however, the work will be delivered under close collaboration with World Bank and MoE teams and supervision by SRSS. Additional requirements might be included by SRSS to comply with its requirements and policies. The contractor will receive written comments from the MoE and SRSS on its activities, which need to be performed as per the timetable within 14 business days and will have 10 business days to revise and deliver final deliverables and activities. For M&E platform development and implementation the contractor should comply with the timetable shown in table A.5. Table A.5 Timetable Activities Indicative deadline from contract effectiveness (required time span) 1 M&E platform development Two months 2 Platform testing, including input of Three months required data from other existing platforms and systems 3 Training and documentation Four months 4 Active commissioning of the platform Five and a half months Note: M&E = monitoring and evaluation. 47 Resources and Payment Schedule The contract shell be a lump sum contract as detailed in table A.6. However, the contractor’s offer must include and daily rates. Development and implementation of a new M&E platform: €60,000. Table A.6 Payment Schedule Activities Payment at completion and approval of activity (% of total, €60,000) 1 M&E platform development 30 2 Platform testing, including input of 25 required data from other existing platforms and systems 3 Training and documentation 15 4 Active commissioning of the platform 30 Note: M&E = monitoring and evaluation. 48 APPENDIX B OVERVIEW OF CHANGES Pillar 1—Environmental Protection and the Improvement of the Quality of Water Bodies OP1.1. Regulatory compliance. This operational objective is aimed at compliance with environmental regulations in wastewater discharge with the environment. It presents four indicators (although there are actually five), a large number that should be reduced in the future. OP1.1 Regulatory compliance Indicator 1 Source: APA Definition % of water bodies at national level that meet the classification criteria for good ecological status Aim Measure the quality of water bodies and, therefore, the environmental sustainability Comments None Action No action required Additional notes None OP1.1 Regulatory compliance Indicator 2 Source: PENSAAR Definition Number of agglomerations in litigation in three cases (normal areas, sensitive areas, and small agglomerations) Aim Measure the compliance with the environmental legislation and, therefore, the environmental sustainability Comments - Litigation occurrences address past performance and not current one - It is only a proxy indicator because it measures the consequence of (not) achieving the objective rather than the objective itself - Instead of measuring the litigation, the indicator should focus on the data reported to the EU on treatment levels for the different areas (normal and sensitive) Action Remove indicator Additional notes None OP1.1 Regulatory compliance Indicator 3 Source: ERSAR (AR14 and AR15) Definition % of households covered by operators with satisfactory evaluation on the indicator AR14 (wastewater tests) % of households covered by operators with satisfactory evaluation on the indicator AR15 (discharge parameters) 49 Aim Measure the compliance with the environmental legislation and, therefore, the environmental sustainability Comments - These are two separate indicators, but they can be weighted - Although ERSAR has changed codes and definitions (replaced by new indicator AR13ab), the concept has not changed - ERSAR continues to collect these data - These indicators (weighted indicators) are more suitable for the OP1.2 Action Move the weighted indicator to OP1.2 Additional notes None OP1.1 Regulatory compliance Indicator 4 Source: ERSAR (AR11ab) Definition % of households covered by operators with satisfactory assessment at the appropriate wastewater destination (AR11) Aim Measure the compliance with the environmental legislation and, therefore, the environmental sustainability Comments - Although an initial assessment may suggest that by combining the bulk and retail indicators from ERSAR, households may be counted twice, ERSAR already calculates the combined indicator appropriately - Since it is a ratio, the % of households is the same as the % of inhabitants Action No action required Additional notes None OP1.2 Reduction of the urban pollution of the water bodies. This operational objective deals with urban impact in water bodies. Caution should be exerted because measuring the general pollution of water bodies may not be addressing the objective (dealing only with the urban contribution to such pollution). Although the recommendation is to remove both indicators presented, O.P1.1. indicator 3 should be incorporated into this objective. OP1.2 Reduction of the urban pollution of the water bodies Indicator 1 Source: APA Definition % of the water bodies at national level, whose concentration of the general physicochemical elements supporting biological elements meet the classification criteria for good ecological status Aim Measure the quality of the water bodies and, therefore, the environmental sustainability Comments - Virtually the same as OP1.1. indicator 1 (the definition is slightly different, but in practice the baseline, target, and intermediate values are the same) - It would better address OP1.1., but its twin indicator is already there 50 Action Remove indicator Additional notes None OP1.2 Reduction of the urban pollution of the water bodies Indicator 2 Source: PENSAAR Definition Cost of: CBO5 removed/inhabitant N removed/inhabitant P removed/inhabitanta Aim Measure the easiness of reducing the urban pollution of the water bodies and achieving the environmental sustainability Comments - Unit costs do not address the objective of reducing urban pollution in water bodies - No values were reported in 2017, and no targets are set for 2020 - Wide consensus (within GAG) that this indicator with several elements is not appropriate for PENSAAR 2020 Action Remove indicator Additional notes None a. Although this is considered one indicator, it has three elements, which makes it impossible to compute. OP1.3 Increase the physical access to the wastewater services . The two proposed indicators are the most suitable (although the new ERSAR codes need to be used). In future strategic plans, it would make sense to combine these indicators into a single one (that could be disaggregated if needed into subindicators). OP1.3 Increase the physical access to the WW services Indicator 1 Source: ERSAR (AR01) Definition % of households covered by operators with satisfactory evaluation in the indicators’ physical accessibility of the service (AR01) Aim Measure the coverage of the wastewater service with sewerage networks Comments - Physical access does not guarantee connection to the service However, this is not a KPI issue (it is related to the objectives, and this review does not address the objectives) - “Access through bulk supplyâ€? does not make sense; only the retail version is needed - The original indicator (without the “satisfactory evaluationâ€? formula) can be used Action Replace indicator with AR01b “Acessibilidade fiÌ?sica do serviço atraveÌ?s de redes fixas (%)â€? Additional notes None 51 OP1.3 Increase the physical access to the WW services Indicator 2 Source: ERSAR (calculated from variables) Definition % of households covered by satisfactory individual wastewater solutions in respect of the total of households without physical accessibility to the service Aim Measure the quality of the individual wastewater solutions Comments The variables used to calculate this indicator have been renamed by ERSAR Action Redefine the indicator with the new ERSAR variables (e.g., dAR09b to dAR16b) Additional notes None Pillar 2–Improvement of the Quality of the Services Provided OP2.1. Improvement of the quality of the water supply service. The indicators proposed in this objective need to be revised. They need to assess fulfilling the objective (the quality of the service) or the progress of actions taken to achieve the objective. In addition, a clear definition of quality of service is needed, including the importance of the issues. For instance, if over five years leakage is reduced, but the quality of the supplied water decreases, what has been the evolution of the quality of service? Because the definition is somewhat subjective, the index to measure it needs to replicate such definition. The proposal is to use all ERSAR indicators related to quality of service in water supply (that are not included in other PENSAAR objectives) and combine them in a single index: (i) AA03b Service interruptions; (ii) AA04b Safe water; and (ii) AA05b Replies to complaints and suggestions. To combine indicators into a single index, the ERSAR traffic light system (qualifying the service) can be used. Each utility would have three levels for each indicator with numerical values assigned to each level: • Green = 10 • Yellow = 5 • Red = 0 They can be further weighted by population (with all three indicators having the same importance) to obtain an overall assessment of quality of service for the nation, ranging from 1 to 10. OP2.1 Improvement of the quality of the water supply service Indicator 1 Source: ERSAR (AA04) Definition % non-AA04 indicator Aim Measure the safe water supplied and, therefore, the quality of service (drinking water) provided Comments None Action Maintain indicator using AA04b Use ERSAR traffic light system (avoid the “satisfactory assessmentâ€? formula) Additional notes This indicator should be part of an index to assess quality of service in water supply 52 OP2.1 Improvement of the quality of the water supply service Indicator 2 Source: ERSAR (AA03) Definition % of households covered by operators with satisfactory evaluation on the indicator AA03 Aim Measure the continuity of the water supply and, therefore, the quality of service provided Comments None Action Maintain indicator using AA03b Use ERSAR traffic light system (avoid the “satisfactory assessmentâ€? formula) Additional notes This indicator should be part of an index to assess quality of service in water supply OP2.1 Improvement of the quality of the water supply service Indicator 3 Source: ERSAR (AA10) Definition % of households covered by operators with satisfactory evaluation on the indicator AA10 (main failures) Aim Measure the quality of service provided Comments Indicator does not address the objective Action Remove indicator Additional notes None OP2.1 Improvement of the quality of the water supply service Indicator 4 Source: PENSAAR Definition Level water users’ satisfaction with the water supply service Aim Measure the users’ satisfaction with the water supply service Comments Indicator does not have a baseline and it is difficult to measure in a homogeneous way Action Remove indicator Additional notes None OP2.1 Improvement of the quality of the water supply service NEW INDICATOR Source: ERSAR (AA05b) Definition Response to complaints and suggestions Aim Measure the users’ satisfaction with the water supply service Comments None Action Add indicator Use ERSAR traffic light system (avoid the “satisfactory assessmentâ€? formula) Additional notes This indicator should be part of an index to assess quality of service in water supply 53 OP2.2 Improvement of the quality of the wastewater service. This objective is the equivalent of OP2.1 in the wastewater service, and a similar approach can be used. ERSAR indicators related to quality of service in wastewater service (that are not included in other PENSAAR objectives) need to be combined in a single index: (i) AR03 Sewer flooding; (ii) AR04 Replies to complaints and suggestions; and (iii) AR08b Structural collapses in sewers. To combine indicators into a single index, the ERSAR traffic light system (qualifying the service) can be used. Each utility would have three levels for each indicator with numerical values assigned to each level: • Green = 10 • Yellow = 5 • Red = 0 They can be further weighted by population (with all three indicators having the same importance) to obtain an overall assessment of quality of service for the nation, ranging from 1 to 10. OP2.2 Improvement of the quality of the wastewater service Indicator 1 Source: ERSAR (AR03) Definition % of households covered by operators with satisfactory assessment in the occurrence of floods (AR03) Aim Measure the number of floods and, therefore, the quality of service provided Comments None Action Maintain indicator using AR03b Use ERSAR traffic light system (avoid the “satisfactory assessmentâ€? formula) Additional notes This indicator should be part of an index to assess quality of service in water supply OP2.2 Improvement of the quality of the wastewater service Indicator 2 Source: ERSAR (AR09) Definition % of households covered by operators with satisfactory assessment in the occurrence of structural collapses in sewers (AR09) Aim Measure the number of collapses in sewers and, therefore, the quality of service provided Comments None Action Maintain indicator using AR09b Use ERSAR traffic light system (avoid the “satisfactory assessmentâ€? formula) Additional notes This indicator should be part of an index to assess quality of service in water supply OP2.2 Improvement of the quality of the wastewater service Indicator 3 Source: PENSAAR Definition Level of satisfaction of wastewater service users Aim Measure the users’ satisfaction with the wastewater service 54 Comments Indicator does not have a baseline and is difficult to measure in a homogeneous way Action Remove indicator Additional notes None OP2.2 Improvement of the quality of the wastewater service NEW INDICATOR Source: ERSAR (AR04b) Definition Response to complaints and suggestions Aim Measure the users’ satisfaction with the wastewater service Comments None Action Add indicator Use ERSAR traffic light system (avoid the “satisfactory assessmentâ€? formula) Additional notes This indicator should be part of an index to assess quality of service in water supply Pillar 3–Optimization and Efficient Management of the Resources OP3.1 Optimization of the utilization of installed capacity and increased adherence to the service OP3.1 Optimization of the utilization of installed capacity and Indicator 1 increased adherence to the service Source: ERSAR (AA07) Definition % of households covered by operators with satisfactory assessment of adherence to the service (AA07) Aim Measure the effective access to the water supply service Comments - “Satisfactory assessmentâ€? formula should not be used - New ERSAR indicator version should be used - Indicator should be measured by population, not household Action Replace indicator AA07 by new AA011b Calculate by population Additional notes None OP3.1. Optimization of the utilization of installed capacity and Indicator 2 increased adherence to the service Source: ERSAR (AA09) Definition % of households served by operators with satisfactory assessment in the adequacy of treatment capacity (AA09) Aim Measure the optimization of the treatment facilities Comments ERSAR no longer uses this indicator Action Remove indicator Additional notes None 55 OP3.1 Optimization of the utilization of installed capacity and Indicator 3 increased adherence to the service Source: ERSAR (AR06) Definition % of households covered by operators with satisfactory assessment of adherence to the service (AR06) Aim Measure the effective access to the wastewater service Comments - “Satisfactory assessmentâ€? formula should not be used. - Indicator should be measured by population, not household - Use only retail indicator Action Continue using AR06b Use the original ERSAR indicator Additional notes None OP3.1. Optimization of the utilization of installed capacity and Indicator 4 increased adherence to the service Source: ERSAR (AR07) Definition % of households served by operators with satisfactory assessment in the adequacy of treatment capacity (AR07) Aim Measure the optimization of the treatment facilities Comments ERSAR is not using the indicator anymore Action Remove indicator Additional notes None OP3.2 Reduction of physical water losses OP3.2 Reduction of physical water losses Indicator 1 Source: PENSAAR Definition % of households covered by operators with satisfactory assessment of the actual losses of water (AA13) Aim Measure the water losses and, therefore, the operational efficiency of the water supply service Comments - “Satisfactory assessmentâ€? formula should not be used. - New ERSAR indicator version should be used - Use only retail indicator (bulk is not so relevant, cannot be easily added per service connection, prevents international comparisons) Action Replace old indicator AA13 by new AA012b (m 3/service connection) Additional notes None 56 OP3.3 Control of rainwater to foul sewerage OP3.3 Control of rainwater to foul sewerage Indicator 1 Source: ERSAR (AA13) Definition % of households covered by operators with an action plan for the control of infiltrations and undue inflow to the public wastewater systems implemented Aim Measure the willingness to deal with infiltrations and undue inflow Comments Indicator does not seem to be collected, have a baseline or a target Action Remove indicator Additional notes None OP3.3 Control of rainwater to foul sewerage Indicator 2 Source: PENSAAR Definition % of undue inflow to public wastewater systems in operators with a plan of action implemented for the control of infiltrations and undue inflow Aim Measure the effectiveness of dealing with infiltrations and undue inflow Comments Indicator does not seem to be collected or have a baseline or target Action Remove indicator Additional notes None OP3.3 Control of rainwater to foul sewerage Indicator 3 Source: ERSAR (AR07 adapted) Definition % of the treatment capacity used under conditions of overuse (%) (AR07 adapted) Aim Measure the overuse of the treatment facilities Comments Indicator seems relevant at utility level, but loses it when aggregated Action Remove indicator Additional notes None OP3.3 Control of rainwater to foul sewerage Indicator 4 Source: ERSAR (AR13) Definition % of households served by operators with satisfactory assessment in the control of emergency discharges (AR13) Aim Measure the quality of the control of emergency discharges Comments Indicator seems relevant at utility level, but does not provide useful information when aggregated Action Remove indicator Additional notes None 57 OP3.3 Control of rainwater to foul sewerage Indicator 5 Source: ERSAR (AR03) Definition % of households served by operators with satisfactory assessment in the occurrence of floods (AR03) Aim Measure the level of occurrence of floods Comments Indicator does not address the objective Action Remove indicator Additional notes None OP3.3 Control of rainwater to foul sewerage NEW INDICATOR Source: PENSAAR Definition (dAR54_Billed wastewater (m3/year) x AR11_Physical accessibility to treatment (%)) / (dAR53_Wastewater treated in WWTP (m 3/year) + dAR51_Exported raw wastewater (m3/year)) Aim Measure the level of treated infiltrations and undue inflows and, therefore, the operational efficiency Comments None Action Add new indicator Additional notes None OP3.4 Efficient management of assets and increase its rehabilitation OP3.4 Efficient management of assets and increase its Indicator 1 rehabilitation Source: ERSAR (AA10) Definition % of households covered by operators with satisfactory assessment in the rehabilitation of pipelines (AA10) Aim Measure the suitability of the rehabilitation practices and, therefore, the infrastructure sustainability in the water supply Comments - “Satisfactory assessmentâ€? formula should not be used - New ERSAR indicator version should be used Action Replace old indicator AA10 by new AA09 Use ERSAR original indicator (avoid “satisfactory assessmentâ€? formula) Additional notes None OP3.4 Efficient management of assets and increase its Indicator 2 rehabilitation Source: ERSAR (AR08) Definition % of households covered by operators with satisfactory evaluation in the rehabilitation of sewers (AR08) 58 Aim Measure the suitability of the rehabilitation practices and, therefore, the infrastructure sustainability in the wastewater Comments - “Satisfactory assessmentâ€? formula should not be used - New ERSAR indicator version should be used Action Replace old indicator AR08 by new AR07 Use ERSAR original indicator (avoid “satisfactory assessmentâ€? formula) Additional notes None OP3.5 Upgrade resources and subproducts OP3.5 Upgrade resources and subproducts Indicator 1 Source: ERSAR (combined variables) Definition Reused wastewater/treated wastewater (%) ((dAR25i+dAR25ii)/(dAR24) Aim Measure the reused wastewater and, therefore, the sustainability of the water resources use Comments ERSAR variables have been updated Action Update variable dAR25iab with dAR55ab Update variable dAR25iiab with dAR56ab Update variable dAR24 with dAR52ab Additional notes None OP3.5 Upgrade resources and subproducts Indicator 2 Source: ERSAR (combined variables) Definition % of sludge valued relatively to volumes produced: V (%) = ((quantity of treated sludge applied in agriculture + quantity of sludge treated by composting + amount of sludge energetically valued)/total quantity of sludge) * 100 Aim Measure the valued sludge and, therefore, the sustainability of the natural resources use Comments None Action No action required Additional notes None OP3.5 Upgrade resources and subproducts Indicator 3 Source: ERSAR (combined variables) Definition Own production of energy/energy consumption (%) (DAA28/dAA29 and dAR27/dAR28) Aim Measure the energy independence and, therefore, the environmental sustainability by using less natural resources and less polluting ones Comments ERSAR variables have been updated Action Update variable dAA28 with dAA63 Update variable dAA29 with dAA64 59 Update variable dAR27 with dAR59 Update variable dAR28 with dAR60 Additional notes None OP3.6 Allocation and efficient use of the water resources. The nature of this objective is to control the efficient use of water resources in the urban environment. OP3.6 Allocation and efficient use of the water resources Indicator 1 Source: ERSAR (AA14) Definition % of households covered by operators with satisfactory assessment in compliance with the licensing of the abstractions (AA14) Aim Measure the water resources license, use, and, therefore, the sustainability of the water resources Comments Use inhabitants instead of households Action Maintain indicator Use ERSAR original indicator (avoid “satisfactory assessmentâ€? formula) Additional notes None OP3.6 Allocation and efficient use of the water resources Indicator 2 Source: ERSAR (combined variables) Definition (dAA20ab/(dAA07b x No. inhabitants per households)) Aim Measure the water resources use per capita and, therefore, the sustainability of the water resources Comments None Action No action required Additional notes None OP3.6 Allocation and efficient use of the water resources Indicator 3 Source: ERSAR (combined variables) Definition (dAA16b/(dAA07b x No. inhabitants per households)) Aim Measure the water consumption per capita and, therefore, the sustainability of the water resources Comments None Action No action required Additional notes None 60 OP3.6 Allocation and efficient use of the water resources Indicator 4 Source: PENSAAR Definition % of population served by operators with recognized merit in the efficient use of water Aim Measure the water resources efficient use and, therefore, the sustainability of the water resources Comments No baseline, intermediate value, or target. Action Remove indicator Additional notes None Pillar 4–Economic, Financial, and Social Sustainability OP4.1 Sustainable cost recovery OP4.1 Sustainable cost recovery Indicator 1 Source: ERSAR (AA02) Definition % of households covered by operators with satisfactory assessment of the economic accessibility of the service (AA02) Aim Measure the affordability of the water supply service customers Comments Although it currently reads at a constant 100%, tariffs are expected to be raised if a greater percentage of the costs is recovered; therefore, the indicator needs to be kept to monitor if the service continues to be economically accessible Action No action required Additional notes None OP4.1 Sustainable cost recovery Indicator 2 Source: ERSAR (AA06) Definition % of households covered by operators with satisfactory assessment of expenditure coverage (AA06) Aim Measure the financial sustainability of the water supply services Comments - The percentage can be aggregated at national level - “Satisfactory assessmentâ€? formula should not be used Action Use original ERSAR indicator Aggregate percentage nationally Additional notes None OP4.1 Sustainable cost recovery Indicator 3 Source: ERSAR (AR02) Definition % of households covered by operators with satisfactory assessment of the economic accessibility of the service (AR02) 61 Aim Measure the affordability of the wastewater service users Comments Although it currently reads at a constant 100%, tariffs are expected to be raised if a greater percentage of the costs is recovered; therefore, the indicator needs to be kept to monitor if the service continues to be economically accessible Action No action required Additional notes None OP4.1 Sustainable cost recovery Indicator 4 Source: ERSAR (AR05) Definition % of households covered by operators with satisfactory assessment of total spending coverage (AR05) Aim Measure the cost recovery and, therefore, the financial sustainability of the wastewater services Comments None Action No action required Additional notes None OP4.1 Sustainable cost recovery Indicator 5 Source: PENSAAR Definition % of households covered by EGs with social tariff Aim Measure the inclusiveness of the social tariff Comments Social tariff conditions are designated by law; therefore, the indicator is significant Action No action required Additional notes None OP4.2 Optimization of operating costs OP4.2 Optimization of operating costs Indicator 1 Source: PENSAAR Definition % of operators with operating costs satisfactory within a reference band of efficient unit operating costs Aim Measure the cost efficiency and, therefore, the financial sustainability of the services Comments - Indicator not collected - No baseline value and no current value Action Remove indicator Additional notes None 62 OP4.3 Reduction of nonrevenue water OP4.3 Reduction of nonrevenue water Indicator 1 Source: ERSAR (AA08) Definition % of households covered by operators with satisfactory assessment in nonrevenue water (AA08) Aim Measure the nonrevenue water and, therefore, the financial sustainability of the services Comments Avoid the use of “satisfactory assessmentâ€? formula Action Use AA08 original ERSAR indicator and aggregate it nationally as a ratio including only the water entering the bulk system Additional notes None Pillar 5–Basic and Cross-Cutting Conditions OP5.1 Increase the availability of information OP5.1 Increase the availability of information Indicator 1 Source: ERSAR (DAA44) Definition Infrastructural and asset management knowledge index (DAA44) Aim Measure the knowledge about its water supply assets Comments Code has changed to dAA32 Action Change indicator to dAA32 Additional notes None OP5.1 Increase the availability of information Indicator 2 Source: ERSAR (dAR45) Definition Infrastructural and asset management knowledge index (dAR45) Aim Measure the knowledge about its wastewater assets Comments Code has changed to dAR41 Action Change indicator to dAR41 Additional notes None OP5.2 Innovation OP5.2. Innovation Indicator 1 Source: PENSAAR Definition Number of programs and partnerships between public and private entities with research and teaching centers Aim Measuring the innovation-related activities Comments - Innovation is difficult to measure - Quantity does not imply quantity 63 - Indicator not collected. No baseline value and no current value Action Remove indicator Additional notes None OP5.3 Improvement of the operational framework, the management, and the provision of services OP5.3 Improvement of the operational framework, the Indicator 1 management, and the provision of services Source: PENSAAR Definition % of households with satisfactory quality (good + median) —indicators of the partnership agreement Aim Measure the improvement in the incentives created and in contracting out of the water supply and wastewater services Comments None Action No action required Additional notes None OP5.3 Improvement of the operational framework, the Indicator 2 management, and the provision of services Source: PENSAAR Definition % of households covered by management entities serving less than 10,000 households Aim Measure the improvement in the market structure and, therefore, the sustainability of the services Comments - Objective of aggregation is based on population and not households - The reference value for this indicator should be 40,000 inhabitants Action Change indicator definition to 40,000 inhabitants Additional notes None OP5.4 Climate change, natural disasters, risks—reduction, adaptation OP5.4 Climate change, natural disasters, risks —reduction, Indicator 1 adaptation Source: PENSAAR Definition % of households covered by operators with contingency plans, energy efficiency, security, and adaptation of approved infrastructures (or actions) and in implementation Aim Measure the planning and other good practices related to climate change, natural disasters, risks—reduction and adaptation Comments - Indicator not used 64 - No baseline, measurement or target Action Remove indicator Additional notes None OP5.5 Externalities: employment, competitiveness, internationalization OP5.5 Externalities: employment, competitiveness, Indicator 1 internationalization Source: PENSAAR Definition No. of jobs created in the private sector Aim Measure the employment creation in the sector Comments Indicator is not fully aligned with the objective Action Remove indicator Additional notes None OP5.5 Externalities: employment, competitiveness, Indicator 2 internationalization Source: PENSAAR Definition Number of tenders launched and awarded Aim Measure the wealth creation in the sector in the national market Comments - Indicator is not fully aligned with the objective - Indicator not collected - No baseline value and no current value Action Remove indicator Additional notes None OP5.5 Externalities: employment, competitiveness, Indicator 3 internationalization Source: PENSAAR Definition Volume of contracts awarded to national companies in the AA and SAR sectors in other countries Aim Measure the wealth creation in the sector due to the international market Comments - Indicator is not fully aligned with the objective - Indicator not collected - No baseline value and no current value Action Remove indicator Additional notes None 65 References ERSAR/LNEC (Water and Waste Services Regulation Authority /Laboratório Nacional de Engenharia Civil). 2018. Technical Guide 22: Guide for the Assessment of the Quality of Service Provided to the Users in Water and Solid Waste: ERSAR/LNEC. Frade, J. 2014. PENSAAR 2020: A New Strategy for the Water and Sanitation Sector in Portugal— Highlights: Governo de Portugal. http://www.ppa.pt/wp-content/uploads/2014/10/PENSAAR- 2020-Highlights_José-Frade.pdf. IBRD (International Bank for Reconstruction and Development)/World Bank. 2018. PENSAAR 2020: Midterm Review Inception Report. Washington, DC: World Bank. Moen, R. Foundation and History of the PDSA Cycle. Accessed on July 26, 2019. https://deming.org/uploads/paper/PDSA_History_Ron_Moen.pdf WKKF (W.K. Kellogg Foundation). 1998. Logic Model Development Guide. Battle Creek, MI: W.K. Kellogg Foundation. www.wkkf.org. 66