Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00005210 IMPLEMENTATION COMPLETION AND RESULTS REPORT Loan Number 8201-BR ON A LOAN IN THE AMOUNT OF US$ 350 MILLION TO THE State of Parana FOR THE SWAp for Parana Multi-sector Development Project May 29, 2020 Governance Global Practice Latin America And Caribbean Region CURRENCY EQUIVALENTS (Exchange Rate Effective {May 26, 2020}) Currency Unit = Brazilian Real BRL 5,3701 = US$1 FISCAL YEAR January 1 - December 31 Acting Regional Vice President: J. Humberto Lopez Country Director: Paloma Anos Casero Regional Director: Robert R. Taliercio Practice Manager: Adrian Fozzard Task Team Leader(s): Kjetil Hansen ICR Main Contributor: Kjetil Hansen, Angela Nieves Marques Porto ABBREVIATIONS AND ACRONYMS CGE Office of the Comptroller General (Controladoria Geral do Estado) CPF Country Partnership Framework CPS Country Partnership Strategy DLI Disbursement Linked Indicator EEP Eligible Expenditure Program GDP Gross Domestic Product GOP Government of Parana HDI Human Development Index IAP Parana Environment Institute (Instituto Ambiental do Paraná) ICMS Value-Added Tax on Sales and Services (Imposto sobre Circulação de Mercadorias e Serviços) ICR Implementation Completion Report IDEB Index of Primary Education Development (Indice de Desenvolvimento da Educacao Basica) IPARDES Institute of Economic and Social Development (Instituto Paranaense de Desenvolvimento Econômico e Social) IPVA Motor-Vehicle Property Tax (Imposto sobre a propriedade de Veículos Automotores) IRR Internal Rate of Return ISR Implementation Status & Results Report ITCG Land, Mapping and Geosciences Institute (Instituto de Terras, Cartografia e Geociências) ITCMD Tax on Inheritances and Donations (Imposto de Transmissão Causa Mortis e Doação) NRM Natural Resource Management MTR Mid-Term Review PAD Project Appraisal Document PDO Project Development Objective PIU Project Implementation Unit SWAp Sector-Wide Approach RWS Rural Water Supply Systems TA Technical Assistance TCE Court of Accounts of the State of Parana (Tribunal de Contas do Estado do Paraná) SEED State Secretariat of Education (Secretaria da Educacao e do Esporte) WHO World Health Organization TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 6 A. CONTEXT AT APPRAISAL .........................................................................................................6 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION .............................................................. 14 II. OUTCOME .................................................................................................................... 16 A. RELEVANCE OF PDOs ............................................................................................................ 16 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 18 C. EFFICIENCY ........................................................................................................................... 28 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 30 E. OTHER OUTCOMES AND IMPACTS (IF ANY)............................................................................ 30 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 31 A. KEY FACTORS DURING PREPARATION ................................................................................... 31 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 32 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 33 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 33 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 34 C. BANK PERFORMANCE ........................................................................................................... 37 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 38 V. LESSONS AND RECOMMENDATIONS ............................................................................. 38 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 41 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 65 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 67 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 68 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P126343 SWAp for Parana Multi-sector Development Project Country Financing Instrument Brazil Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency State of Parana State Secretariat of Planning Project Development Objective (PDO) Original PDO The objective of the Project is to make access to economic and human development opportunities more equitable and environmentally sustainable in the Borrower s territory through the modernization of the Borrower s public sector and revenue management. Revised PDO The objectives of the Project are to: (i) increase the survival rate in the final cycle of fundamental education; (ii) improve maternal and emergency health care services; (iii) promote business initiatives and improve natural resources management in rural areas;and (iv) contribute to building capacity in public administration, and environmental and disaster risk management. Page 1 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 350,000,000 333,626,367 333,626,367 IBRD-82010 Total 350,000,000 333,626,367 333,626,367 Non-World Bank Financing 0 0 0 Borrower/Recipient 363,239,777 0 0 Total 363,239,777 0 0 Total Project Cost 713,239,777 333,626,367 333,626,367 KEY DATES FIN_TABLE_DAT Approval Effectiveness MTR Review Original Closing Actual Closing 06-Nov-2012 13-Jan-2014 04-Apr-2016 30-Nov-2017 30-Nov-2019 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 31-Dec-2015 136.96 Change in Results Framework Change in Components and Cost Change in Disbursements Arrangements Change in Legal Covenants Change in Institutional Arrangements Change in Financial Management 26-Apr-2017 220.84 Change in Project Development Objectives Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Change in Implementation Schedule Other Change(s) KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Satisfactory Modest Page 2 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 22-Jan-2013 Satisfactory Satisfactory 0 02 27-Apr-2013 Satisfactory Satisfactory 0 03 06-Nov-2013 Satisfactory Moderately Satisfactory 0 04 17-May-2014 Satisfactory Satisfactory 50.00 05 04-Nov-2014 Satisfactory Satisfactory 99.94 06 22-Apr-2015 Satisfactory Moderately Satisfactory 99.94 07 21-Oct-2015 Moderately Satisfactory Moderately Unsatisfactory 136.96 08 06-Mar-2016 Moderately Satisfactory Moderately Unsatisfactory 136.96 09 06-Jun-2016 Moderately Satisfactory Moderately Satisfactory 199.90 10 14-Dec-2016 Moderately Satisfactory Moderately Satisfactory 220.84 11 19-Jun-2017 Moderately Satisfactory Moderately Satisfactory 223.61 12 19-Dec-2017 Moderately Satisfactory Satisfactory 271.56 13 15-Jun-2018 Satisfactory Satisfactory 300.33 14 16-Dec-2018 Satisfactory Satisfactory 303.94 15 27-Jun-2019 Satisfactory Satisfactory 316.72 16 11-Dec-2019 Moderately Satisfactory Satisfactory 329.55 SECTORS AND THEMES Sectors Major Sector/Sector (%) Agriculture, Fishing and Forestry 22 Other Agriculture, Fishing and Forestry 22 Page 3 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Public Administration 44 Sub-National Government 19 Other Public Administration 25 Education 12 Other Education 12 Health 22 Health 22 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Private Sector Development 4 Enterprise Development 4 MSME Development 4 Public Sector Management 3 Public Administration 3 Administrative and Civil Service Reform 3 Human Development and Gender 47 Health Systems and Policies 23 Health System Strengthening 23 Education 24 Access to Education 12 Education Financing 12 Environment and Natural Resource Management 48 Environmental Health and Pollution Management 48 Air quality management 16 Water Pollution 16 Soil Pollution 16 Page 4 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) ADM STAFF Role At Approval At ICR Regional Vice President: Hasan A. Tuluy J. Humberto Lopez Country Director: Deborah L. Wetzel Paloma Anos Casero Director: Rodrigo A. Chaves Robert R. Taliercio Practice Manager: Arturo Herrera Gutierrez Adrian Fozzard Task Team Leader(s): Fanny Weiner Kjetil Hansen ICR Contributing Author: Kjetil Hansen Page 5 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. At the time of project preparation and project approval in 2011, the State of Parana was the fifth largest economy in Brazil with income and social indicators above the national average. The State economy represented 5.9 percent of the country's GDP (R$251.5 billion or US$149.5 billion1). With a total population of 10.44 million, Parana was home to 5.4 percent of the country's total population, with 85.3 percent living in urban areas. Its GDP per capita, the fifth highest in the country, was 75 percent above the national average during project preparation in 2011-12. Overall, Parana had achieved a high level of economic and social development. Social indicators, such as income inequality, illiteracy and child mortality pointed to higher levels of social development than the national average: the human development index was the fifth highest in 2011 (0.761) and continued to increase over the project period (0.792 in 2018). 14.5 percent of the population lived under the poverty line (US$ 5.5 per day), the fifth lowest in Brazil. This fell slightly to 13.7 percent at the end of 2018. Figure 1. Real GDP Growth (2008-19) Figure 2. Real GDP per capita 12.0 45,000 10.0 40,000 8.0 35,000 6.0 30,000 4.0 25,000 2.0 20,000 0.0 15,000 200820092010201120122013201420152016201720182019 10,000 -2.0 (est) -4.0 5,000 -6.0 0 2010 2011 2012 2013 2014 2015 2016 2017 Paraná Brasil Paraná Brazil Source: IPARDES 2. Parana’s economy began to slow down in 2011. Growth slowed from 9.9 percent in 2010 to zero growth in 2012, recovering to 3 percent in 2013, before falling to -3.5 in 2015 and 2016 and then recovering to 1.3 percent in 2017. This pattern largely mirrored that of Brazil. In 2018, Parana again saw economic contraction, with growth at negative 0.4 percent, while the national average remained at 1.3 percent. Growth for 2019 rebounded to 0.7 percent. Real Gross Domestic Product (GDP) per capita is 1 Parana Institute of Economic and Social Development ( Instituto Paranaense de Desenvolvimento Econômico e Social - IPARDES). Page 6 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) significantly higher than the Brazilian average (Figure 2) . 3. Parana had significant regional inequalities. Strong performance in the Western, Northern and the Eastern regions of Parana contrasted with poverty and economic stagnation in the Central Region (Figure 3). The three stronger regions concentrate population, economic activity and institutional capacity. Municipalities in Parana’s Central Region consistently lag behind the rest of the State in terms of employment, income, and human development. Moreover, the Metropolitan Region of Curitiba, the State's capital, is also marked by pockets of high poverty and low access to services. While Parana held the seventh and ninth positions for poverty and extreme poverty rates, respectively, when compared to other states, the lagging Central Region would be closer to the 19th position, with 30 percent of total households considered poor.2 This demonstrated that there was still significant room for improvement in social development. Figure 3. HDI by Municipality Source: IPARDES 4. In the Central Region of Parana, smallholder agriculture was the main source of income and shaped the social and economic structure of most municipalities. There was still relatively little agricultural processing, few linkages with agri-business or economic activities with other regions of the State. Recently, rural tourism has begun to generate some income and shows potential for expansion. However, the Central Region offered few job opportunities and low wages, especially for women and youth, leading to a continuing exodus to Parana’s growing cities and to neighboring States. 2The poverty line considered in the PAD was twice the extreme poverty line, an estimate of the value of a food basket with minimum calories required to adequately supply a person, based on recommendations from the FAO and the WHO. Page 7 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) 5. Four main factors impeded the Central Region from participating in the economic growth of the rest of the State: physical environment and low productivity; capacity constraints and low innovation; limited physical infrastructure; and political history and land tenure issues . To address these issues, the Government strategy for the revitalization of the economy of the Central Region put emphasis on increasing the competitiveness of key value-chains, improving the quality of life in rural areas, promoting access to land and land regularization, and promoting sustainable management of natural resources. 6. The sustainability of the agricultural economy in Parana was threatened by widespread soil degradation, inadequate and excessive use of pesticides, degradation of water quality, and the loss of forest cover and biodiversity. These issues also affected the quantity and quality of water for human consumption and the energy sector, while increasing the risks of floods and landslides. To address these issues, the Government strategy emphasized modernizing tools for the management of rural space and working with agricultural producers at the micro-catchment level to improve rural soil and water management. 7. A central pillar for ensuring a sustainable development path for Parana relates to the Government's environmental management capacity and its ability to manage risks and respond to natural disasters. Overall, the State had a very good regulatory framework and capacity to address environmental issues. However, unclear and complex operating procedures for environmental compliance contributed to delays and to private sector dissatisfaction. An analysis undertaken by Government agencies (based on Civil Defense Registries) indicated an increase in disaster events in the State. These events included floods, storms, winds, landslides and others such as oil spills and toxic substances. The State had a Civil Defense System more oriented towards response activities and lacks a more proactive, preventive capacity. 8. Parana had achieved near universal enrollment in fundamental education and was one of the best performing states in Brazil but 23.8 percent and 25.6 percent of the children enrolled in State schools in fundamental and secondary education, respectively, were at least two years older than the normal age-grade level. Repetition and dropout rates remained high, despite recent improvements, especially in the second cycle of fundamental education: 13.6 percent for repetition and 4.4 percent for dropout in 2010. Notwithstanding the overall good performance and quality of education, there were disparities of student achievement among regions and schools. Some schools in Curitiba had already achieved the 2021 Ministry of Education targets, while others such as in Loanda - just 200 km from Curitiba had an IDEB score similar to the worst performing schools in Brazil. The same gaps were observed between public and private schools, the latter having already achieved the 2021 targets. 9. Parana’s changing epidemiological and demographic context hindered posed a challenges for the health condition of the population. Although the State had broadly adequate health care coverage, there were concerns over the quality of health services provided in three areas: the reversal in the maternal mortality trends of the past years; the 18 percent increase in congenital syphilis cases from 2007 to 2009; and the low capacity of hospitals and the inability of health units to provide necessary services. While each of the 22 health administrative regions had a referral hospital, most of these were small and unable to absorb the demand for more complex healthcare procedures. The health sector in Parana also faced key challenges related to distorted supply, weak management mechanisms, and poor information for decision-making and organization of healthcare provision. Oversupplied, underused, misplaced and inequitable concentration of medical equipment reinforced accessibility issues, while also affecting quality and contributing to increased cost of service provision. Overall, these challenges had undermined the efficiency pf the State health system and led to suboptimal service provision despite the broadly Page 8 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) adequate coverage. 10. An inefficient and ineffective public administration hampered the State's ability to address inequalities and improve its economic performance. Unlike some other Brazilian states, which had launched reforms and programs to restore fiscal balance and improve public sector performance, Parana's public administration had not undertaken significant modernization efforts. While the Government of Parana (GOP) had adopted some isolated modernization measures, it had not yet made a more comprehensive reform effort to shift the focus from formalistic procedures and input controls to results- based management. 11. Seeking to transform this situation, the State Government proposed an aggressive reform program to modernize its public sector and increase fiscal space for investment in order to promote integrated social and economic development. The GOP's own assessment recognized that its ability to promote integrated development depended on introducing significant public administration reforms to increase public sector efficiency and effectiveness, and improve access to quality public services, especially among the poorest population groups and lagging regions. Increasing fiscal space for investment, modernizing and simplifying bureaucratic procedures were also central to improve conditions for private investments in the State. Finally, fostering economic activities in lagging regions and better management of risks and vulnerabilities were vital for the State to reclaim its position in the Brazilian economy. 12. The goal of the Government of Parana was to introduce a new approach to public sector leadership focused on results and the needs of citizens. Parana’s administrative structure dates back to 1978 and there was a growing concern that it was no longer fit for implementing the new policy priorities. It was burdened by overly bureaucratic and cumbersome procedures that further aggravated existing inefficiencies and lacked a results orientation. The incoming Government in 2011 made firm commitments to restore administrative efficiency, adjust the State's public finances and introduce results-based management through the establishment of performance agreements and sectoral targets. GOP identified three broad areas reflected in the three pillars of the Government's strategy: fiscal quality; institutional modernization (including the introduction of new tools and practices in the areas of planning, budgeting and management); and more strategic and efficient human resources management. The Parana SWAp operation was designed to support the government’s objectives in these areas. 13. In mid-2014 shortly after the project became effective, Brazil experienced one of the most severe economic and political crises in its history. A confluence of factors including falling international commodity prices and domestic economic policy errors caused GDP to contract by 3.5 percent in 2015 and 3.3 percent in 2016, before beginning a slow and long recovery in 2017. In parallel, a massive corruption scandal (Lava Jato – or Operation Car Wash) was unfolding, involving a large swathe of the Brazilian political class, eventually bringing about a political crisis and leading to the impeachment of the sitting president. Mass unemployment and popular dissatisfaction with the political class led to popular unrest and further instability, and the recession impacted the federal government and states’ fiscal space for reforms. Hence the political and economic environment in which the project was implemented was extremely challenging for most of the project period. Theory of Change (Results Chain) 14. The project appraisal document did not outline an explicit theory of change. Figure 4 below depicts the Theory of Change inferred from the original Project Appraisal Document (PAD) linking the interventions included in each of the two components (activities/inputs) to their expected outputs, Page 9 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) outputs to their corresponding outcomes and finally, the outcomes to the expected longer-term impact – using the information from the Results Framework. Figure 4. Theory of change Inputs/activities Outputs Outcomes Long-term impact Education Training of teachers Student learning assessment system Increased graduation rate in final Increased survival Technical assistance for in place cycle of primary school to [from rate in the final preparation of school University based in-service teacher 78.8 to 85.9] cycle of maintenance program training, education development plan Improved management of fundamental (EDP) physical infrastructures in education School rehabilitation and expansion 85% of teachers participating in peer- schools Preparation of IT platform for to-peer in-service teacher training Improved management of Increased quality of school maintenance monitoring 750 teachers in indigenous schools education sector based on education (Obras Online) covering all trained better use of performance data state schools 450 state schools rehabilitated Learning as measured by the and/or expanded index of primary education development (IDEB) Health Construction, expansion and Information materials for pregnant 38% Decrease in maternal Improved maternal equipping of primary care units women, protocols for primary health mortality ratio [from 65.11 to and emergency in 399 Municipalities care unit staff 47.18 per 100K] health care services 90% of pregnant women Training for 30,000 primary Fully trained and equipped regional identified with high risk of health care staff on prenatal emergency health care teams (SAMU) complications referred by care covering 90% of State within max 1 primary care units to a hospital hour response that is part of the infant and maternal healthcare network 100 Ambulances, 3 heliports, 4 (Mãe Paranaense) helicopters, 1 airplane, related 90% of live births to pregnant urgent care medical equipment women who attended more for 12 regional urgent care than 7 antenatal check-ups teams, 72 advanced urgent care units, 207 basic care units 6% Decrease in mortality rate for and training on equipment use deaths through external causes, except violence [from 51.48 to 48.57] 5% decrease in mortality rate for cardiovascular diseases 100% of state population with access to emergency health services Improved management of urgent and maternal health care programs Rural Development Training and technical support Business proposals prepared and Increased sales for small Promote business to producer groups producer groups financed (75) producer groups initiatives and Road maintenance equipment 21,000 Farmers trained or receiving improve natural Page 10 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Inputs/activities Outputs Outcomes Long-term impact purchased TA Increased number of people in resources Technical assistance for 60 management plans for rural roads rural areas with access to management in preparation of Micro- improved water sources rural areas 6,000 Beneficiaries of legally catchment plans [28,000] concluded land regularization Drilling and preparation of processes Improved watershed wells management 8 indigenous lands with productive projects supported 250 Micro-catchment plans prepared 750,000 Hectares supported by the micro-catchment action plans interventions New wells functioning with water testing and connections Public sector management Purchase and installation of Integrated tax management system Tax arrears collection [from 1.99 Increased capacity new tax management system implemented and available to 5 % of outstanding arrears] in public Training program for civil throughout the state of Parana Enhancing core competencies of administration and servants 12,000 civil servants certified through civil servants environmental and f2f and online training modules disaster risk State Disaster Risk Strengthening internal control management Management policy framework Internal control system operation in system established 50 government entities 15% reduction of sick Acquisition and installation of Occupational health policy prepared leave/absences due to work equipment for the disaster risk 60% of landslide risk prone areas related illness or injury of monitoring and warning system mapped education sector and establishment of a disaster Improved identification of Strengthening of the disaster risk management center disaster risks monitoring capacity and warning Adoption of new techniques system for hydro-meteorological Improved capacity for delivering and procedures for events environmental registration environmental licensing licensing and oversight services 25 Municipalities with a decentralized environmental and licensing and monitoring system Implementation of a water resources sub-system of the integrated environmental management and water resources system Project Development Objectives (PDOs) 15. The objective of the project as described in the Loan Agreement (p. 7) is as follows: The objective of the Project is to make access to economic and human development opportunities more equitable and environmentally sustainable in the Borrower's territory through the modernization of the borrower's public sector and revenue management. 16. The PDO was subsequently revised as follows: The objectives of the Project are to: (i) increase the survival rate in the final cycle of fundamental education; (ii) improve maternal and emergency health care services; (iii) promote business initiatives and improve natural resources management in rural areas; and Page 11 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) (iv) contribute to building capacity in public administration, and environmental and disaster risk management. Key Expected Outcomes and Outcome Indicators 17. The (revised) key performance indicators were directly linked with each of the four sub-objectives in the revised PDO (education, health, rural business initiatives and natural resources management, capacity building and environmental and disaster risk management) notably:  Increase the survival rate in the final cycle of fundamental (primary) education - Increase primary school graduation rate from 78.4 % to 85.5%  Improve maternal and emergency health care services - Decrease the maternal mortality rate from 65.11 per 100,000 live births to 47.18 per 100,000 live births - Decrease mortality rate from non-violent external causes from 51.48 to 48.57 per 100,000 habitants  Promote business initiatives and improve natural resources management in rural areas - 750,000 Hectares supported by the micro-catchment action plans interventions - 75 rural producer groups supported through the project with business initiatives implemented  Contribute to building capacity in public administration and environmental and disaster risk management - Increase the recovery of tax arrears from 1.99 percent to 5 percent of total stock of arrears Components 18. The project used a Sector-Wide Approach (SWAp) modality with two complementary components. A US$ 314.125 million results-based component was structured around Eligible Expenditure Programs (EEPs) and Disbursement Linked Indicators (DLIs) in the areas of Rural Development, Environmental and Disaster Risk Management, Education and Health. A US$ 35 million Technical Assistance component provided targeted support in the form of equipment and studies in the areas of: environmental management; disaster risk management; land use planning; public sector training/capacity development; rural development; regional planning; internal control systems; tax management systems; evaluation of educational outcomes; school infrastructure management; and public asset management. Component 1: Fostering Equitable and Environmentally Sustainable Economic and Human Development 19. Component 1 is a results-based component structured around DLIs in the areas of Rural Development, Environmental and Disaster Risk Management, Education and Health. 20. Subcomponent 1.1. Sustainable Rural Development supported the GOP’s rural development strategy and territorial development program (Pro-Rural) and water and soil management program (Microbacias), with a focus on: assisting small rural producers in improving their business practices; financing of business proposals and targeted training and technical assistance before and during implementation; preparation of territorial development plans; and improved water and soil management in micro-catchments. 21. Subcomponent 1.2. Environmental and Disaster Risk Management supported the Page 12 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) implementation of the State government’s strategy for environmental and disaster risk management, including: modernizing the State’s environmental licensing system; and strengthening the State’s system for monitoring and response to disasters. 22. Subcomponent 1.3. Education Supported the State’s education sector strategy to ensure access to good quality education, including: strengthen the learning assessment system; improvement of the skills and proficiency of teachers through in-service and university-based teacher training; and rehabilitation, repairs and expansion of the existing schools, science laboratories, libraries, and strengthening systems for maintenance and monitoring of state school infrastructure ( Renova Escola). 23. Subcomponent 1.4. Health supported the strengthening of the State’s emergency health care system, in particular: management and technical training for hospital and frontline staff; expansion of trauma care, transport and specialized health services for treating trauma, myocardial infraction and strokes; expansion of pre-hospital emergency health care; implementation of performance-based contracts; strengthening of the patient referral system; strengthening of the infant and maternal healthcare network through expansion of the infant and maternal health care and construction, expansion and equipping of primary care units of selected municipalities. Component 2: Technical Assistance for more efficient and effective public management 24. The technical assistance component provided targeted support to studies, equipment and advice in the following areas. 25. Subcomponent 2.1. Fiscal Quality originally included technical assistance to strengthen the assessment and collection of two of the three major taxes collected at state level, notably: review the motor vehicle property tax assessment and collection process; the assessment and collection process of the inheritance and donations tax; the process of collection of overdue taxes; and all segregated funds. 26. Subcomponent 2.2. Institutional Modernization supported a review of organizational structures and procedures for modernizing the State’s public administration; strengthening of the budget process; introduction of results-based management and M&E capacity; and strengthening internal control procedures. 27. Subcomponent 2.3. Strategic and efficient Human Resources Management included: strengthening personnel expenditure controls; strategic HR management including personnel/talent management in center of government institutions (planning, finance); modernization of the HR system and the occupational health service; and strengthening of the public administration school and training. 28. Subcomponent 2.4. Mitigating the Environmental Impact of the Agriculture Sector supported a review of existing studies on the impact of key agricultural sub-sectors (on water, biodiversity, greenhouse gas emissions) and potential mitigation measures including available financing instruments for mitigation at the local level. 29. Subcomponent 2.5. Support to the Modernization of the Environmental Licensing System included studies and analysis related to environmental monitoring and enforcement, modernization of existing IT systems, and training for users of the new system. 30. Subcomponent 2.6. Support to the development of a State Disaster Risk Management System included Financing of diagnostics, methodologies, plans and procedures for the management of risk and disaster in the Borrower’s territory and related capacity building. Page 13 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) 31. Subcomponent 2.7. Education included: design and setting up of a new learning assessment system; impact evaluation on the Borrower's education development plan to improve teacher effectiveness; and development of an integrated system for the management of the Borrower's school’s infrastructure and equipment. 32. Subcomponent 2.8. Health included: design and structuring of an information system to monitor the health care network including planning, budgeting, epidemiological indicators and monitoring and evaluation modules; development and implementation of a communication plan driven by users and health professionals; and patient satisfaction surveys to evaluate the quality of services provided by Health Service Providers. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION 33. The project was restructured twice; a level 2 restructuring in 2015 to account for the delay in effectiveness due to late approval of the project by the Senate, and a level 1 restructuring in 2016 following the mid-term review. Revised PDOs and Outcome Targets 34. The project underwent a level 1 restructuring in 2016 which included a change in the PDO and a number of indicators. The restructuring was based on the findings of a mid-term review carried out in early 2016. The review proposed a change in the PDO to better reflect the spirit of the project, a two-year extension of the closing date to November 2019, and adjustments to several indicators and indicator targets. The revised PDO is presented in para 15. Revised PDO Indicators 35. The level 1 restructuring in 2016 removed one PDO indicator (improved identification of disaster risks); introduced two new PDO indicators (hectares supported by the micro-catchment action plans interventions, and mortality rate for death through external causes, except violence); adjusted the description and methodology for one PDO indicator (producer groups supported through the project with improved access and integration to output markets business initiatives implemented); and modified the targets of two PDO indicators (revenue from tax arrears as a share of stock of tax arrears, and producer groups supported through the project with improved access and integration to output markets business initiatives implemented). Table 1 summarizes changes to indicators. Page 14 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Table 1. Changes to PDO indicators and targets Legend: Deleted New Actual target achieved Revised Components 36. The project components remained the same for the duration of the project Other Changes 37. A level 2 restructuring in 2015 simplified the disbursement arrangements and adjusted the results framework to account for a one-year delay in project effectiveness due to delayed approval of the project by the Brazilian Senate. The restructuring also replaced six indicators, altered the targets of nine indicators, and added two indicators to monitor the implementation of the indigenous people’s safeguards. 38. The 2016 level 1 restructuring reallocated budget amongst Technical Assistance (TA) activities Page 15 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) to account for the addition of new and significant activities. The new activities were: preparation of two regional development plans; and the development and implementation of a new integrated tax administration system. There were adjustments to four DLIs, four intermediate indicators were added and two were deleted. Rationale for Changes and Their Implication on the Original Theory of Change 39. During the period between Board approval and the start of project implementation, the recession led to a reduction in eligible expenditures under the EEPs. To mitigate the foreseen risks and taking into account lessons learned from other SWAp operations in Brazil, a level 2 restructuring in 2015 simplified the disbursement arrangements originally agreed to ensure that stronger emphasis was placed on the achievement of development objectives rather than on financial execution. The delay in project effectiveness also led to activities being carried out with the Government's own funds which in some cases led to the achievement of final targets before the expected completion date. To address these changes, the restructuring also modified the results framework as noted above. 40. Based on the outcomes of the mid-term review in 2016, the level 1 restructuring sharpened the PDO to better reflect the intended results outlined in the PDO indicators . A number of TA activities and intermediate indicators and DLIs were changed or adjusted in line with the government and team’s assessment of needs, priorities and realism at the time of the mid-term review. The project theory of change did not change as a result of the restructuring. II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating Rating: Substantial 41. At the time of the ICR, the relevance of the project development objectives was substantial, despite perhaps being too wide in scope with its four distinct sub-objectives across several sectors. The results areas continued to be closely aligned with state priorities as laid out in their medium-term program and specific sectoral programs, and indicators and targets were aligned with these. The World Bank’s Country Partnership Strategy (CPS) (2012-15) had four main objectives: Increase the efficiency of public and private investments; Improve quality and expand provision of public services for low income households; Promote regional economic development through strategic investments and policies; and Improve sustainable natural resource management and climate resilience (see Figure 5). The actions of the project were closely aligned with the CPS and supported all four areas, in particular sub area 1.1 Improved Fiscal and Public Sector Management, sub area 2.2 Improved quality of education for low income groups, 2.3 Improved access to health care for low income households, 3.1 improved policy coordination at territorial level, 3.3 improved transport infrastructure and management, 4.1 integrated water resources management, 4.3 Improved environmental management, biodiversity conservation and climate change mitigation, and 4.4 More effective disaster risk management. 42. The PDO objectives continued to be aligned with the Bank’s new Country Partnership Framework (CPF) (FY2018-22). The new CPF focused on three broad areas: Fiscal Consolidation and government effectiveness; Private sector investment and productivity growth; and Inclusive and Page 16 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) sustainable development. The current CPF is a slight shift from the CPS, with a more focused program on fewer areas and an increased emphasis on fiscal management, public sector effectiveness in education and health and private sector investment through regulatory reforms. However, all of the areas supported through the project are linked to the focus areas outlined in the CPF, in particular Focus areas 1, which focused on strengthening fiscal management at all level of government, and increased effectiveness of service delivery in education and in health and focus area 3 which aimed to promote socio economic development of small rural producers and vulnerable groups. Relevance of DLIs 43. The 13 DLIs were selected in consultation with the authorities and technical staff in each relevant area. The DLIs are integrated in the results framework and the links between results-based activities and their intended outcomes are well defined. The DLIs were measurable with clear protocols developed for their monitoring which was the responsibility of the respective technical agencies (see Table 2). Disbursements were subject to clear verification protocols as outlined in the PAD (table A3.3, p63). Table 2. Disbursement-Linked Indicators (DLIs) DLI Targets and verification protocols DLI1: Number of business proposals approved and Scalable indicator. End target 75, disbursements based on annual to be financed. targets. List of agreements signed with associations and sample contracts published. Subject to six operational criteria. DLI2: Number of micro-catchment action plans Scalable indicator. End target 250., disbursements based on annual elaborated targets. Micro-catchment plans in accordance w operational manual. DLI3: Implementation of a water resources sub- Sub-system implemented in five distinct steps: TORs elaborated, system of the integrated environmental Contract Signed, Report on project design, at least one system module management and water resources system operational; reduction in avg processing time for issuing water rights. DLI4: Student Learning Assessment System System implemented in three steps: pilot and two learning assessments operational published; fourth assessment; fifth assessment. All published. DLI5: Teachers participating in peer-to-peer in- Scalable indicator measured as percentage of teachers recorded on service teacher training payroll. End target 85%. DLI6: State schools rehabilitated and/or expanded Scalable indicator. End target 340. Works completed with a certificate signed by secretary of education, company and works technical supervisor. DLI7: Percentage of pregnant women identified Scalable indicator. Target of 80%. Disbursements based on four annual with high risk of complications referred by primary lagged targets. Calculated with data from state health system and care units to a hospital that is part of the hospital information system. borrower’s infant and maternal healthcare network (Mae Paranaense) DLI8: Proportion of live births to pregnant women Maintenance Target of 80% measured annually through information who attended more than 7 antenatal check-ups system on live births (SINASC) DLI9: Enhancing core competencies of civil Disbursement based on four operational steps: signature of contract for servants payroll audit; competencies development plan prepared; number of civil servants trained reported by gender. DLI10: Strengthening of internal control systems Target: system purchased, implemented and operational in 50 government entities. 7 Disbursements based on TORs; Contract; Assessment report; strategic plan; system implementation. DLI11: Strengthening state capacity to monitor and Instruments for monitoring of public policies, including: logframes for evaluate public policies all EEP areas; baseline report on land regularization; impact evaluation reports in areas tbd. DLI12: Improvement of the borrower’s tax Target: system fully operational as measured by proof that processes management system on restitution, rectification, exemptions, immunity requests done Page 17 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) through integrated system. 7 Disbursement based on: TORs; consultant contract; progress report; action plan; progress report on action plan; progress report on action plan. DLI13: Establishment of state disaster risk Measured by: official order establishing the DRM working group; DRM management policy framework work plan; establishment of state civil defense and protection committee; guidelines for preparation of DRM plan approved by Council: State DRM plan approved. Rating of overall relevance: Substantial 44. The overall relevance is rated as Substantial, based on the assessment of the relevance of the PDO, DLIs, and intermediate indicators and the strong linkage with government’s own medium-term program and the Bank’s Systematic Country Diagnostic (SCD) and CPF . B. ACHIEVEMENT OF PDOs (EFFICACY) 45. The project development objective was partially achieved if measured by the PDO level indicators but closer to being fully achieved when other indicators and results are accounted for . The PDO level indicators for the health sector exceeded their targets, the education sector objective was achieved, while public administration capacity and rural development targets were not fully achieved (see Table 3). However, this result is partly due to the choice of PDO level indicators which were not always representative of the wider objective of the results area in question. This was the case for the public administration capacity indicator (percentage of tax arrears) which did not capture the broader objectives of modernizing the public sector. The same applies to the rural development objective. When taking into account the fuller list of intermediate level indicators and DLIs and some additional results not captured through any project indicators, a fuller and more nuanced picture emerges, and the achievements of the project are decidedly more comprehensive. These results are presented in the following sections. Table 3. PDO indicators Target Actual PDO Indicators Baseline 2019 2019 Status Increase the survival rate in the final cycle of fundamental education Increase primary school graduation rate (percentage) 78.4 85.5 86.9 Achieved Improve maternal and emergency health care services Decrease the maternal mortality rate (share of 100,000) 65.11 47.18 43.3 Exceeded Decrease mortality rate for deaths through external causes except violence (share of 100,000) 51.48 48.57 46.34 Exceeded Contribute to building capacity in public administration, and environmental and disaster risk management Revenue from tax arrears as a share of the stock of tax arrears (percentage) 1.99 5 2 Not achieved Promote rural business initiatives and improve natural resources management in rural areas Hectares supported by the micro-catchment action plans Partially interventions (number) 0 750,000 728,000 achieved Producer groups supported through the project with improved Partially access and integration in output markets (number) 0 75 24 achieved Page 18 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Assessment of Achievement of Each Objective/Outcome Objective 1: Increase the survival (graduation) rate in the final cycle of fundamental (primary) education Rating: Satisfactory 46. This subcomponent aimed to: implement the learning assessment system; strengthen teacher development through in-service and university-based training (implemented with government’s own resources); carry out rehabilitation, repairs and expansion of existing schools, science laboratories, libraries (not implemented); and strengthen systems for maintenance and monitoring of state school infrastructure (implemented with government’s own resources). Technical assistance activities foreseen in this area included: design and implementation of a new learning assessment system; impact evaluation of the borrower’s education development program; and development of an integrated system for borrower’s school infrastructure and equipment management. 47. Learning assessment system. The student learning assessment system (Sistema de Avaliação da Educação Básica do Paraná – SAEP) was developed and implemented. Its main objective was to monitor primary school test scores in Mathematics and Portuguese through periodic examinations. A total of five learning outcome assessments in Math and Portuguese were carried out as planned. The systematic testing program covered 1,091,057 primary school students as of end 2019 (76.8 percent of the primary student population of the State). The PDO indicator for this objective “Survival rate in the final cycle of fundamental education” was achieved and at 86.9 percent surpassed the end target of 85.5 percent. Test results are now publicly available and are being used by teachers and school administrations in the management of the State school system. Results for schools and students are shared with teachers, students and parents and form the basis for dialogue at the individual school level. However, the State has not made use of the extensive data collected to inform the formulation of public policies and other government interventions. The results are presented at an aggregate level and there is a tendency to avoid benchmarking and singling out weaker performing schools to hold teachers and managers accountable for results. 48. Teacher training. The project also supported the educational development program (Programa de Desenvolvimento Educacional – PDE) through which teacher’s pedagogical and technical skills are strengthened through a two-year training program involving a year full-time in university and a second year of part-time training while working/teaching at 75 percent. The university based in-service teacher training program covered a total of 9,835 teachers against a target of 7,650. The State court of accounts (Tribunal de Contas do Estado – TCE) and State Parliament have both questioned the cost (over BRL200 million per year) and fiscal sustainability of the program and have proposed an evaluation of the program to determine whether it had a significant impact on learning outcomes. An outcome evaluation was foreseen as part of the technical assistance component under the project but was dropped due to time constraints. The largest cost/fiscal impact is related to the increased wage costs related to the automatic promotion following participation in the program, although this latter element was subsequently changed. However, participation in the training program remains the only requirement for career advancement for teachers in the State system. The project supported a peer-to- peer teacher training program (Formação em Ação) and the indicator on peer-to-peer teacher training was achieved at 86.3 percent (surpassing the end target of 85 percent), reaching 9,835 teachers. 49. School renovation. The school renovation program (Renova Escola) achieved its targets. A total of 466 schools were rehabilitated and/or expanded, against an end target of 450 (DLI). An additional Page 19 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) 1,322 schools received various equipment including basic furniture, IT equipment for computer centers, laboratory equipment and library materials. In addition, an integrated IT system was put in place to catalogue, manage and monitor the school infrastructure to ensure that financial resources are only channeled to where they are needed. The system ( obras online) covers all State schools and includes maps, pictures of buildings and facilities, equipment, and an assessment of their functional/physical condition and maintenance needs. It also includes a planning/analytical module allowing for better budget estimates and operational planning. Objective 2: Improve maternal and emergency health care services Rating: Highly Satisfactory 50. This subcomponent supported the strengthening of the State’s emergency health care system, in particular: management and technical training for hospital and frontline staff; expansion of trauma care, transport and specialized health services for treating trauma, myocardial infraction and stroke; expansion of pre-hospital emergency health care; implementation of performance-based contracts; strengthening of the patient referral system; strengthening of the infant and maternal healthcare network through expansion of the infant and maternal health care and construction, expansion and equipping of primary care units of selected municipalities. The technical assistance originally foreseen under this subcomponent – the development of an integrated information system; communication plan; and patient satisfaction surveys – were dropped from the project and replaced with neo-natal and emergency certifications. However, due to delays in the preparation of TORs for these activities and difficulties in determining the selection method these activities were dropped from the TA component and instead carried supported through the DLIs. 51. Emergency health care. Activities supported the State’s emergency medical services program (Rede de Urgência e Emergência), which aims to improve the timeliness and coverage of emergency services through investments in emergency care services, including the purchase of new equipment, expansion or installation of ICUs, a broad training program for over 12,000 medical personnel, and incentive schemes for hospitals. The objective related to trauma and emergency care was to ensure that patients were identified, reached and treated within the critical time to reduce mortality (within one hour). There were several indicators related to this results area, including the PDO indicator “Decrease in mortality rate for deaths through external causes except violence” which saw a very significant reduction, from 51.48 per 100,000 to 46.34 against a target of 48.97 (and preventing over 1,000 deaths over the project period). The indicator “Mortality rate for cardiovascular diseases” was reduced from 80.39 per 100,000 to 68.49, surpassing the reduction target (76.2 per 100,000). The last indicator “Percentage of population with access to urgent and emergency health services” was not fully achieved, although it doubled from 44 percent in 2013 to 90 percent in 2019, falling short of 100 percent coverage as planned. 52. Maternal health. Activities supported Parana’s maternal health program (Rede Mae Paranaense) as it invested in upgrading primary care clinics, with a focus on improving maternal health services and continuous pre-natal and post-partum care throughout the State. Support was organized in line with an updated risk profile based on World Health Organization (WHO) standards involving intensified monitoring for those considered to be at risk. Nearly 60,000 medical professionals received training to strengthen their ability to identify risk factors and provide appropriate care, including referrals for specialized medical attention for those at higher risk. Hospitals received additional funding based on risks in order to incentivize them to provide pre-natal services and post-partum services. Results for this Page 20 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) objective are excellent. Maternal mortality was reduced from 57.11 per 100,000 in 2015 to 43.35 per 100,000 in 2019, significantly exceeding the target of 47.18 per 100,000. Infant mortality was also reduced from 12.17 to 10.3 per 1,000 births, exceeding the target of 10.5 per 1,000 births. The share of pregnant women subject to at least 7 pre-natal visits increased from 80 to 100 percent, above the 90 percent target. 100 percent of pregnant women identified with high risk of complications are now referred by primary care units to a hospital for closer follow up and care, against the end target of 90 percent. The State health authorities continue to have challenges in servicing indigenous populations where both infant and maternal mortality is higher. Objective 3: Promote business initiatives and improve natural resources management in rural areas Rating: Moderately Satisfactory 53. This results area included support to: GOP’s rural development strategy and territorial development program (Pro-Rural), with a focus on assisting small rural producers in improving their business practices, through financing of business proposals and targeted training and technical assistance to strategic partners including municipal councils, NGOs and local leaders, research and innovation, land regularization, rural roads rehabilitation, preparation of territorial development plans; and the State’s micro-catchment program (Microbacias) which aimed to improve water and soil management through the adoption of improved methodologies and techniques, environmental education and the adoption of sustainable agricultural and natural resource management (NRM) practices. The Pro-Rural program supported by the Bank was focused on the 132 poorest Municipalities in the State located mainly in the central region, as identified in the project document (PAD) and outlined in the operational manual. 54. Small rural producers. Activities in support to small rural producers aimed to support small scale producer’s groups with increased access to markets through capacity building and through the preparation of business proposals/projects, complemented by ongoing technical assistance and matching grants to business organizations (associations and cooperatives). The initial target was to support 210 projects. The project mid-term review (MTR) noted the limited number of proposals submitted for financing (27 as of 2016), which it ascribed to the low capacity of community associations. During the MTR the target was reduced to 75 and direct technical support and training in project preparation was added to the activities. As a result, a total of 52 business proposals were eventually prepared and financed, supporting 40 percent of all associations and cooperatives in the area (130) and approximately 5,200 agricultural families. The PDO indicator protocol outlines six conditions that must be in place to fully satisfy the indicator targets: the producer groups had to receive at least 5 technical assistance visits per year; have a functioning accounting system in place; have a maintenance and operations plan; a reserve fund; have received training on cooperatives and management; and have made sales to at least one new buyer. As of project closing only 24 of these satisfied all six conditions. The Institute of Economic and Social Development (IPARDES) is undertaking an impact evaluation to determine the results of the support including compliance on the six criteria. Despite a failure to reach the indicator target, a large number of producer groups received training and technical assistance. A total of 75,784 agricultural producers had received technical support and training through Pro-Rural as of end 2019, a significantly higher number than the original target of 21,000. In addition, many of the projects were slow to get started but are currently under implementation. 55. Land title regularization (Regularização fundiaria). The lack of formal land titles has prevented small rural agricultural producers from investing in their land. Hence Pro-Rural aimed to map and register a number of the more than 100,000 unregistered properties in the state. The project supported this Page 21 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) process, with a goal of supporting 6,000 land users to finalize the regularization/formalization of their properties. While geo-referencing was carried out for 6,907 properties, only 2,702 properties had had been fully regularized as of project closing, or 45 percent of the target (November 2019). Another 3,500 processes are in still awaiting a decision by the judicial authorities, who continue to have limited capacity and large backlogs of such requests. An important finding from this process is that there are no standard procedures in place for the process of land regularization: different judges are demanding different justifications for regulating land ownership. There is evidence that greater emphasis is currently being placed on regularization as a result of the project activities. The Institute of Land, Cartography and Geology (ITCG) now has a specialized department dealing with such requests. 56. Rural Roads. Many rural areas/communities and producers are isolated from markets due to poor road access. In order to remedy this problem, the project supported the purchase of maintenance equipment to be managed by local organizations made up of private and public organizations and coordinated by municipal authorities. The activities under this component took longer to implement than initially planned due to the complexity of managing international bids. Eight lots of roads maintenance equipment were purchased and have been transferred to inter-municipal consortia covering 40 municipalities (rural maintenance groups). Approximately 675Km of rural roads had been upgraded or maintained using this equipment as of project closing. The project aimed to support the preparation of 60 management plans for rural roads, but none had been completed by the project closing date. The following activities were carried out: completion of the mapping through the identification of official municipal rural roads according to Master Plans and Municipal Laws (exclusion of private roads); survey of the quantitative and qualitative information of the municipal roads; and workshops, meetings and public consultations for development of Management Plans. 57. Soil and Water Management: Through the Microbacias program the project supported the adoption of improved methodologies and techniques for sustainable soil and water management in rural areas, environmental training, promotion of sustainable agricultural and NRM practices by rural producers in select micro-catchment areas through the development of micro-catchment development plans, grants to associations or municipalities for rural producers, grants to producer associations for agro-ecological business initiatives and improved access to potable water for poorer communities in priority areas. The State had suffered a drought in 2011 and 2012, affecting mainly rural areas and the agriculture production. During project preparation it was agreed to include retroactively the investments of 185 rural water supply systems supported by the State in response to the drought. Most of the works were implemented between 2011 and 2014. 58. A total of 321 micro-catchment plans were developed by the project, surpassing the target of 250 plans. The PDO indicator “Hectares supported by the Micro-catchment Action Plans” was almost fully achieved with 728,000 hectares covered against a target of 750,000 hectares. The Project originally planned to implement 480 rural water supply systems (RWS) and after the restructuring in 2016 the target was revised to 350 systems. A new intermediate indicator “Number of people in rural areas provided with access to improved water sources under the project” was included with an end target of 28,000. A field visit carried out in late 2019 to all 185 rural water supply systems found that 37 systems were delivering treated water, 57 were delivering bulk untreated water, 31 were dry wells without enough water to meet demand, 50 systems were incomplete, missing distribution networks or in need of other equipment, while 10 systems had water quality issues at the source and were deemed inappropriate for human consumption. A total of 6,644 families (19,267 people) were receiving water from these systems. An action plan was prepared to address areas of concern. From November 2019 to March 2020, a final Page 22 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) assessment was conducted by IPARDES and the Parana Agricultural Development Company (CODAPAR). The assessment found that only eight systems were delivering treated water to approximately 1,395 families (21 percent of the families) or 4,046 people, and 28 systems were delivering bulk untreated water (66 percent of the families). This is clearly below the project target of 28,000. 59. The relatively weak results in water supply investments under this subcomponent can be attributed poor institutional coordination between communities, municipal authorities and the State water company (AGUASPARANA) and limited involvement of communities in the planning and implementation of the RWS. The municipal authorities were responsible for financing the water distribution after the project had financed the drilling and preparation of the wells. Agreements were signed to this effect, however municipalities claimed to not be aware of the associated costs and had not budgeted resources. Communities were responsible for the operation and maintenance including water treatment. While all communities received information and training on water treatment, the training may not have been sufficient to ensure regular treatment. Half of the plans did not involve any community participation and therefore the willingness of the communities to continue with operation and maintenance was limited. Objective 4: Contribute to building capacity in public administration and environmental and disaster risk management Rating: Moderately Satisfactory 60. Activities under this results area included: support to strengthen the tax management system and management of arrears; modernization and strengthened results focus of the public sector through tools and training; strategic and efficient human resources management; strengthened internal controls; modernization of the natural resources management system; establishment of a disaster risk management system. There were adjustments to the planned activities and challenges on the tax administration reforms. However, the State achieved significant results in the implementation of a system to manage state assets, training for a large number of public servants, strengthened internal controls, creation of a modern licensing system for water use rights and establishment of a highly effective disaster risk management unit and system. 61. Fiscal quality. The project supported the State tax authorities in modernizing their tax management system and strengthening tax arrears collection. The original activities under this subcomponent were to provide technical assistance for reviewing the assessment and collection process for the motor vehicle tax and the inheritance tax, as well as the process for the collection and assessment of tax arrears and a review of all segregated funds. However, these activities were undertaken internally by the Secretariat of Finance, and the client requested the project to instead finance the purchase, installation and training on a new integrated tax administration software/system, to replace a number of outdated mainframe systems developed separately over the previous decade. The main objective was to make the system more user-friendly for both the tax administration and the taxpayers, including the automation of some standard processes, increased analytical capabilities (i.e. integrated overview of each taxpayer), integration with other systems (including property registries) and by providing an improved public interface where taxpayers can file and pay online. 62. An international bidding process was conducted and the contract signed at the end of 2017 with a consortium of domestic and international companies. Work started in early 2018 with the planning, conceptualization, installation, testing and adjustment of the system, one module at a time. The system is structured around the three main taxes collected at the State level, notably the vehicle circulation tax Page 23 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) (IPVA), the inheritance tax (ITCMD), and the consumption tax (ICMS). As of project closure one module was fully operational and another module was being tested. The third and most complex module (ICMS) was still at the planning state at the end of 2019. 63. The indicator on tax arrears (revenue from tax arrears as a share of the stock of tax arrears) was kept as the main PDO level indicator for this results area. No progress was made on this indicator over the period of the project: the baseline in 2013 was 1.99 percent and the result at the end of the project was 2 percent, against a target of 5 percent. New arrears have stayed relatively stable at between 3 to 4 Bn R$ per year: in 2019 3.9 bn R$, against 35.9 Bn R$ in tax receipts. Tax arrears accrue almost entirely on the sales tax (98 percent of total arrears stock) which requires a monthly declaration from companies. There are several reasons for the lack of progress on this indicator. First, the regular practice of giving tax amnesties and ‘discounts’ for the settlement of arrears created a perverse incentive whereby instead of settling liabilities in due time, some taxpayers let their liabilities go into arrears, in the expectation that Government would repeat its tax amnesty program. Second, the absence of a modern system for classifying and managing arrears prevented the authorities focusing efforts on those arrears that are more likely to be recovered (or economically beneficial to recover) and writing off arrears in a timely manner (some arrears date from the 1980s). Third, companies and individuals can appeal tax assessments directly to the revenue service’s administrative appeals courts, where cases can drag on for years. If a company loses, it can then take its case to the judicial system, which adds more years to the process. 64. Institutional modernization. Some of the initial planned activities were carried out with Government own funds while others were replaced due to changing priorities. The initial activities were as follows: review of organizational structures and procedures for the modernization of the borrower’s public administration (dropped); strengthening of budget processes (dropped); introduction of results- based management and strengthening of monitoring and evaluation capacity; and strengthening of internal control procedures. 65. A new activity introduced was the update of the State’s property registry ( Cadastro Imobiliario do Estado), with the objective of increasing the efficiency in the management of State assets, including the identification of potential savings in the use of assets and/or sales of assets. Technical assistance financed under component 2 of the project involved a mapping and classification of all State property, with geo- referencing, photos, an evaluation of all property’s physical condition and an updated assessment of the property value. An IT system/database was developed with information on all 4,090 properties, replacing the older paper file system that had limited and outdated information. As a result of this activity the State now has a more complete and updated overview of its physical assets to use for strategic planning purposes and more active asset management. As an example, several unoccupied properties have been sold, raising over BRL5 million for the State, and other sales are under consideration. A review of the State’s vehicle fleet found that many were not being used optimally and the state asset management office revised its fleet composition and introduced cost savings measures such as outsourcing of maintenance contracts, leasing of vehicles and vouchers for use with private transport providers. Finally, the State’s balance sheet was updated with the new valuations (BRL 9.2 billion), bringing the State into line with national legislation with regards to the regular update of property valuation and the calculation of asset depreciation. The next step is to prepare a strategic asset management strategy including options for more efficient property management. 66. Another new area of engagement was the technical assistance provided for the preparation of regional development plans as part of the State’s regional development program. The project financed Page 24 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) the preparation the Plano Metropole Norte covering fifteen municipalities in the interior north, which examines how this part of the State can build on its status as an industrial center; and the Plano Litoral covering the seven coastal municipalities, which looks at options for stronger links between the ports hub and local businesses and communities. Both activities started in early 2018 and were finalized at the end of 2019. Both plans enjoy strong buy-in and ownership by the authorities at State and municipal levels and provide a solid blueprint for future public investment programs. The process included a significant amount of community mobilization and participation of non-governmental groups, an element that was instrumental in generating wide ownership and support of the plans beyond the local authorities. In addition, the data collection process produced a considerable amount of information that is being used in municipal and regional planning and has been made available to civil society and the public3. 67. Internal Controls. The internal control unit was established in 2007 to carry out internal audits across the public sector in Parana. The unit remained relatively inactive until 2011 when it was combined with the units charged with disciplinary actions against civil servants ( Corregedoria) and interphase with public and civil servants receiving complaints (Ouvidoria). In 2013 the Controladoria Geral do Estado (CGE) was created, with a broader mandate including being a standard-setter and resource center for other internal control units throughout the State apparatus. The project financed technical assistance to review and update the unit’s work processes, standard products, key performance indicators, and risk assessment methodology. An integrated IT system was prepared to manage the unit’s work program, including modules to manage caseloads, monitor and assess progress and outcomes including a business intelligence capability. A comprehensive training and capacity program for the new unit work program and use of the new system was delivered, including 19 courses in the areas of basic internal controls, transparency, complaints handling, and audits benefitting 66 institutions and over 670 personnel. 68. Strategic and Efficient Human Resources Management. The project had planned to support the following activities: strengthening controls over personnel expenditures; improving the attraction and retention of personnel; modernization of the borrower’s HRM directorate and its units throughout the State; modernization of the borrower’s occupational health service; and strengthening the Borrower’s School of Government and promoting a results-based public administration. Most of these activities were carried out with Government own funds while awaiting project effectiveness, including the payroll audit and acquisition of IT equipment for the HRM unit, and some technical studies related to the occupational health strategy. 69. The project supported the Parana School of Government (Escola de Gestão) with technical advice and equipment to strengthen performance-based management in public administration. A broad training needs assessment was carried out across the public sector institutions of Parana and a series of training courses were prepared and delivered. This included 109 face to face courses and another 76 workshops, presentations and online courses (available through Moodle online open source platform4), in several areas including: leadership; planning; project management; results-based management; client-based public management; innovation; and numerous others including sector specific courses in education, health. 25,000 civil servants (57 percent women and 47 percent men) participated in the training courses and received course certifications, over twice the target of 12,000. In addition, the school also supported research and innovation through the production of 187 research 3 http://pdslitoral.com/ 4 https://www.ead.pr.gov.br/course/index.php Page 25 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) papers and notes on good practices in various public sector areas/topics and 209 academic papers for peer reviewed journals. Activities related to the State’s occupational health service were initiated and financed by the State itself, including the diagnosis of staff exists and absences due to work-related illnesses, preparation of a Work Safety Report and a Program for the Prevention of Environmental Risks at Work for the Health and Education sectors. However, the collection and monitoring of data and information on the related indicator “Reduction of sick leave and absenteeism due to work-related illness or injury” was not followed up on and hence there is no data available to assess the impact of these activities. 70. Mitigating the Environmental Impact of the Agriculture Sector. The initially planned activities included: review of existing studies on the impact of key agricultural subsectors on the environment and complementary analysis; analysis of options available to address the negative impacts of sub sector activities identified above; and analysis of financing instruments available to mitigate the impact of the agricultural sector on the environment and how they could be used at the state, municipal and other administrative subdivision levels. Due to changes in government priorities none of the planned activities were financed by the project. An impact evaluation for the land regularization/titling activities was planned, and TORs were prepared but the bid was never launched. The planned studies on soil and water quality were carried out with Government financing. 71. Support to the Modernization of the Environmental Licensing System. This area included more than 15 activities with the following results: expansion of the automatic air quality stations network; expansion and modernization of intelligence electronic data and devices – SIGARH and IDE; rehabilitation of Government buildings to provide better service for internal and external clients; decentralization of some environmental licensing responsibilities to municipalities; strengthening the implementing agencies’ management capacities through training of staff and equipment; enhancing the capacity to process, analyze and respond to licensing requests, particularly with regard to water use rights. Technical assistance was provided for the development and implementation of the integrated environmental licensing system (SIGARH). Other planned TA support included: studies and analysis related to environmental monitoring and enforcement; modernization of the information technology system of the environment directorate; and training related to the strengthening of the environmental management approach. 72. The project financed the investments and technical assistance for the development of an integrated IT system to handle the management of water use rights, from the request to issuance of licenses and concessions and their follow-up/monitoring and control. The system (Sistema Integrado de Gestao de Recursos Hidricos – SIGARH) is managed by AGUASPARANA and provides an integrated platform for the management of water use rights, monitoring of water use by small and medium enterprises (SME) and management of water and air quality monitoring stations around the State (also purchased by the project). An important feature of the new integrated system is the ability to better manage water use rights and the actual water consumption of users. The system is integrated with the land use system (Sistema de Gestão Ambiental – SGA) which is managed by the environmental institute (Instituto Ambiental do Parana – IAP) through a shared database. The local state-owned IT company of Parana (Celepar) maintains the software.5 The system was rolled out to municipalities and is currenting implemented in 30 municipalities meeting the related indicator end target of 25. The system has led to a significant reduction in the average time for processing of water use rights from 180 to 90 as 5 See: http://www.aguasparana.pr.gov.br/pagina-3.html Page 26 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) planned for more complex requests, and in some cases temporary user licenses are obtained immediately online, allowing small scale users to advance with their activities while obtaining additional documentation. The project was also meant to finance TA to review and revise the processes for land and water use licensing. However, resistance from various institutions caused the activity to be dropped. 73. Disaster Risk Management: The project supported the development of a disaster risk management (DRM) framework, purchase of equipment, and establishment of a state-of-the-art disaster risk monitoring and management unit for the State. This included the preparation of a state policy on civil defense and protection and an operational plan, but the latter was never published as it depended on a National/Federal master plan which is still awaiting publication. High definition satellite images (at a resolution of 2 meters) was purchased for over 200,000 square kilometers of the State. The images were used to update new multi-use maps including urban and land use planning. The satellite mapping was followed by an analysis and identification of risk areas (86 risk areas) and the preparation of a risk profile for each area. A forestry inventory is also underway, and data is still being processed. The work focused mainly on urban and semi-urban areas and will need to be expanded to cover rural and less densely populated areas. 74. The disaster monitoring unit is fully operational and provides ongoing real-time monitoring of disaster risks in the State. As part of the disaster risk strategy, fifteen rapid response teams were created, trained and equipped to respond to cases of flash floods and other natural risks/disasters and an automated system was put in place to alert, locate and communicate with citizens in risky areas. An incident command system was put in place, built on the US Model. Lives have been saved by the rapid response teams since their establishment, and the units have been deployed to other States to support rescue efforts and help set up similar structures (i.e. Minas Gerais, Para). The project also helped to facilitate integration and coordination amongst related institutions working on environmental and risk management in the State. The project had planned to purchase several X-ray radars, but the process was cancelled given the time required to undertake an international bidding process. Justification of Overall Efficacy Rating 75. The project efficacy is rated as Substantial: the operation partially achieved its PDO indicators and some remaining targets are expected to be achieved after completion of this ICR . Out of six PDO level indicators, three achieved their targets fully, out of which two exceeded their targets significantly.. Two indicators were partially achieved with one almost entirely achieved and another expected to be fully achieved beyond project closure. One indicator was not achieved. 76. The development objectives of the project were largely achieved when considering other results in addition to the PDO indicators. The graduation rate in the final cycle of primary education increased significantly from 78.4 percent in 2012 to 83.3 percent as of end 2019. Maternal and emergency health care services were strengthened as evidenced by the drastically reduced maternal mortality rates from 65.11 to 43.35 per 100,000 and deaths from external causes except violence from 51.48 to 46.34 per 100,000. Business initiatives in rural areas were strengthened, not only through the full support to small producer groups by project closure, which were fewer than planned (26 out of 75 by end 2019 but ongoing) but also through training and technical advice to over 75,000 small farmers and producers. The natural resources management in production units of selected micro-catchments of the State was improved, including through access to improved water sources for over 4,000 people, the regularization of 2,700 properties, and 321 micro-catchment plans covering over 728,000 hectares of land (against a target of 750,000 hectares). Public administration capacity was strengthened through the purchase and Page 27 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) installation of the integrated tax management system, the update of the state asset management strategy, the preparation of a stronger internal control system and support to the public training institute training program. Public sector capacity for environmental and disaster risk management was significantly strengthened through the setting up of a highly effective disaster risk monitoring unit, mapping and risk- profiling of high risk areas, and an integrated system to automate the process of submitting and approving environmental and water use licenses, generating significant time-savings for small producers and individual users and improved monitoring and control by the state environmental agencies through automated water use monitoring. C. EFFICIENCY Assessment of Efficiency and Rating 77. The efficiency of the project is rated as substantial. A 14-month effectiveness delay caused an initial disbursement lag, which picked up in 2014 following effectiveness in Q2, reaching US$ 100 million by Q2 2015. Disbursements were temporarily put on hold between Q1 and Q3 of FY16 due to the initial findings of the 2014 Audit report, while further information on the findings was collected. There were additional delays in disbursement related to the implementation of the technical assistance component, where several changes to the planned activities and the preparation and approval of TORs and bidding processes took a long time. A complicating factor in project implementation of the technical assistance component was the significant changes in the exchange rate over the life of the project, causing the amount available in Brazilian Reais to nearly triple. The project remained close to its disbursement profile as it progressed, until Q2 of 2018 when the disbursement gap increased again (271 vs 311) reaching the largest gap in Q2 of 2019 (303 vs 350 planned), before disbursements again started catching up towards the end of the project. Total disbursements were US$ 333,630,000 (out of the planned US$ 350 million), a disbursement rate of 95.32 percent. Component 1 (results-based component) was fully disbursed, while component 2 (technical assistance) was 45 percent disbursed. A total of US$ 16,370,000 from the technical assistance component was cancelled at project closing. 78. Project expenditures appear to achieve value for money. Each contract underwent a competitive bidding process where relevant comparators were used to prepare the technical documents and to assess and evaluate bids. Efficiency and cost considerations were taken into account in the negotiation of contract amendments where these happened. As an example, the cost of the integrated tax management system (at 27 million Reais, or 8.5 US$ million) is much lower than the 62 million Euro cost of the Finnish system (also a commercial off the shelf solution), even when taking into account the price differences between the two countries. However, it is difficult to find meaningful cost comparisons, given that the specificities of different systems might differ substantially on important cost elements. For example, while the Parana system only covers the three state taxes, the Finish system covers all levels of taxes in the country. Economic and Financial Analysis 79. The economic and financial analysis conducted at the time of appraisal did not calculate economic or financial benefits. The PAD simply referred to likely positive impacts on economic growth in the sectors covered by the project and overall fiscal/budgetary benefits from strengthened public sector management, but without any attempt to quantify or link these improvements to specific project activities or targets. Subsequently, a detailed economic analysis was carried out at the time of the Mid-Term Review in July 2016. This analysis quantified economic benefits and internal rates of return for each results Page 28 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) area of the project, and it benefitted from having updated targets and indicators and partial project implementation. The analysis concluded that all the subcomponents were economically viable with returns higher than the 6 percent used as the minimum internal rate of return (IRR). Education and health represented 80 percent of total investment cost and showed the highest returns of 19 percent and 11 percent respectively. Conservative estimates of project benefits were 73 percent above project costs. 80. The economic analysis was updated with information from project closing at the end of 2019 . The expected economic returns from the technical assistance component were also calculated and added to the analysis, including the investment in the new integrated tax system and the mapping of state assets. The 2016 economic analysis did not include estimated benefits from the reduced mortality rates from the improved emergency health services and this element was also added to the analysis. In addition, some basic assumptions were adjusted, for example using the median income instead of the minimum wage to calculate the expected average economic returns to lives saved through the health interventions. Total program costs increased substantially from the time of the economic analysis, which had an impact on the expected economic rate of return. 81. The ICR economic analysis calculates returns from investments in maternal health, education, disaster risk management, rural development, and public sector management. For maternal health, benefits are from a decrease of mortality rates of pregnant women and newborn babies and savings in treatment costs. The expected net benefits from this results area at 109 million Reais, equivalent to an IRR of 11 percent. The expected net economic benefits from the program of emergency health care is estimated at 194 million Reais, equivalent to an IRR of 12 percent. For the education results area, benefits were estimated for students and trained teachers, in the form of increased productivity and salaries. The value of expected benefits was 14 million Reais, 62 percent of corresponding to benefits from increasing the survival rate, and the remaining 38 percent to the teachers’ training. Benefits from strengthened disaster risk management include reduced property damage and reduction of lives lost, with lower bound estimates slightly higher than the costs of the intervention and the return at 6 percent. Upper boundary estimates showed benefits 8 times higher than costs. Support to producer groups in the form of training and grants yielded average internal rates of return ranging from 7.3 to 20.6 percent based on a sample of producer groups in 2016 and interviews with small scale agricultural producers and mainly based on benefits from increased revenues and agricultural yields. The expected benefit from the integrated environmental licensing system was the largest in relation to the value invested, at over 450 percent IRR, stemming from the immediate time savings to individuals and businesses requesting water use licenses. There were also positive net benefits from the investment in the integrated tax system from time savings in filing and paying the vehicle tax through the new system. The asset mapping and registry will likely not generate positive net benefits within the 5 years used for the economic analysis. Table 4. Summary of Economic Analysis Present Value of Flows (R$) IRR Costs Benefit Net Benefit % Integrated Rural Development Sustainable Rural Development (17,242,839) 18,449,837 1,206,998 7.3% Environmental & Disaster risk management (104,494,950) 105,717,150 1,222,200 6.3% Education (teacher training and evaluation) (136,597,000) 309,627,000 173,030,000 19% Health Emergency Health Care (1,382,000,000) 1,597,000,000 214,000 16% Infant and Maternal Health Care Network (1,039,000,000) 1,108,000.00 69,000 7% Public Sector Management Tax System, Asset Mgmt, Env Mgmt System (54,323,623) 111,528,068 57,204,445 105% Page 29 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) D. JUSTIFICATION OF OVERALL OUTCOME RATING 82. The overall outcome rating for the project is Satisfactory . This rating is based on assessments of project relevance, average efficacy of the four objectives, and efficiency. The final rating noted in Table 5 below is based on the methodology provided in Annex H of the updated World Bank Guidance Note for Implementation Completion and Results Reports for Investment Financing Operations, issued September 27, 2018. The last ISR from November 2019 had downgraded the progress towards achievement of PDO to Moderately Satisfactory based on the data available at the time for the PDO indicators. The ICR process allowed for an assessment of significant additional performance data – thus providing a more comprehensive assessment of project results as outlined above. For example, since project closing, the number of producer groups supported through the project with improved access to markets increased from 0 to 24 and will increase further, the new Water Licensing system was rolled out to more municipalities than planned and has significantly reduced the time to obtain licenses from more than 180 days to less than 90 days on average, and the progress on implementing the new tax system was further advanced with two modules fully operational, leading to a reduction in the time that taxpayers spend on declaring and paying the inheritance and vehicle circulation taxes. Table 5. Summary of ratings Outcome Component Rating Relevance Substantial Overall Project Efficacy Substantial Increase the survival rate in the final cycle of fundamental Satisfactory education Improve maternal and emergency health care services Highly Satisfactory Promote business initiatives and improve natural resources Moderately Satisfactory management in rural areas Build capacity in public administration and environmental Moderately Satisfactory and disaster risk management Efficiency Substantial Overall Outcome Rating Satisfactory E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 83. An important objective of the project was focused on gender specific activities, notably the support to reduce maternal mortality through strengthened maternal health services, ensuring a closer follow up of expectant mothers (through at least 7 prenatal visits). This is also one of the areas with the strongest results, with maternal mortality reduced from 65.11. to 42.8 per 100,000. A specific focus on indigenous women (target of 350 women) ensured that these generally marginalized members of society also benefitted from the program. In addition to the many deaths prevented through the Mãe Paranaense program, regular access to medical services and close follow-up during pregnancy is likely to have improved the quality of pregnant women’s lives in several other ways. The public sector capacity building program benefitted women, with 57 percent of trained certified public sector personnel being female. Page 30 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Institutional Strengthening 84. Institutional strengthening was the core theme of the multi-sector operation, through support to specific training and capacity building such as through the school of Government, but also through the implementation of other programs where technical capacities were strengthened through interaction with World Bank staff and/or interaction with international experts mobilized to provide advice and technical assistance. Institutional changes, like governance, accountability, and results-based management are processes that require a long-term commitment and have a steep learning curve. As the operation showed, the process is ongoing and evolves over time, requiring constant adaptation and refinement. The inter-sectoral participatory approach used with a results-based framework in tackling some of the key issues across sectors has created an important transformation process within the State and introduced innovative concepts and state of the art technologies for results-based policies, M&E, as well as accountability and ownership. Mobilizing Private Sector Financing Not applicable Poverty Reduction and Shared Prosperity 85. Activities supported by the project contributed to poverty reduction. The Pro-Rural program was specifically designed to focus on the 132 poorest municipalities in the State. Some of the most vulnerable groups and regions were targeted through actions that included: training and technical assistance to small associations in some of the poorest municipalities including indigenous communities; increased access to improved water sources; rehabilitation of rural roads including donation of roads maintenance equipment to inter-municipal consortia; improvements in water quality that improved the quality of scarce resources and allowed for participatory monitoring, creating benefits for the society at large, but especially for the areas where the quality of water was particularly poor. The activities financed under the health component also had a strong poverty focus. Slightly over half of the 380 primary health units that were refurbished and/or equipped were in the municipalities with the lowest human development index in the state, reaching the poorest 15 percent of the state’s population. Finally, the Renova Escola program prioritized schools in most need of upgrading, renovation or additional equipment, which tended to be in poorer areas. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 86. The authorities had limited experience in working with the World Bank. The project was the first World Bank operation with the Government of Parana in nearly ten years. Previous engagements with the state had focused primarily on the agriculture sector.6 87. The results-based modality was aligned with the authorities’ development strategy. The Government’s public sector reform program sought to introduce a “new way of governing”, based on strong leadership, cooperation within government and the agreement of development objectives. This results-based approach was well-suited to the SWAp modality operation that was being successfully used 6The previous operation with the state was the Parana Rural Poverty Alleviation & Natural Resources Management Project (P037828), approved by the Board in November 1995 and closed in March 2006. Page 31 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) in other States like Ceara and Minas Gerais7 and the design of the project was built on the lessons from these operations. The project was designed around nine existing Government programs, with 13 Disbursement Linked Indicators that would provide an incentive for the achievement of agreed results. 88. The programs selected were at the core of the Government’s approach and revolved around two broad thematic areas: integrated development and human development. Under the first area, the project supported initiatives to promote green growth in the State with a focus on reducing regional inequalities and based on social, environmental and economic sustainability. The second area supported the improvement of access and quality of fundamental public services, with a focus on health and education. 89. The SWAp modality allowed for a technical assistance component to support implementation of key elements of the Government’s strategy . World Bank specialists from all sectors, many based in Brazil, worked with State officials and technical staff in the design of indicators, interventions and activities to ensure full alignment with the Government strategy. B. KEY FACTORS DURING IMPLEMENTATION 90. The deep macroeconomic crisis affected project implementation by reducing eligible expenditures and so delaying disbursement under component 1 (DLIs). Another key exogenous macroeconomic factor was the strong depreciation of the Brazilian Real from 2 BRL=1 USD to almost 4 BRL=1 USD over the period of implementation. This increased the local currency valuation of project funds and the implementation of the technical assistance component of the project. 91. Inter-agency cooperation and coordination was assured through a project coordination committee, consisting of all implementing institutions which met regularly to monitor progress and discuss operational issues. A very experienced Project Implementation Unit (PIU) provided guidance and overall coordination support to this arrangement. 92. Delays in the approval of the project and of the level 1 restructuring had a cumulative effect on disbursements and implementation. The project took over one year to become effective pending approval in the Senate. A number of indicator targets had been achieved by the time the project became effective, due to ongoing efforts under the programs. The second restructuring (level 1), in 2016, took one year to receive approval and signature of new Loan Agreement. The result of these delays was that the first technical assistance contracts, which had timelines that went beyond the original project closing date, were only signed in mid to late 2017 and the majority began implementation in 2018. In the meantime, several activities planned under the technical assistance component had been financed by the Government. 93. The low technical capacity of some executing institutions, lack of experience with international competitive bidding and perhaps low priority given to some of the technical assistance activities led to significant delays in preparing and finalizing TORs, requiring significant assistance from World Bank procurement and technical staff. Delays in the preparation of TORs prevented technical assistance activities being carried out for education and health, and the need for additional quality control related to complex IT systems such as the integrated tax system, the water resource management system, and the IT tool for the Controladoria Geral do Estado (CGE) delayed the activities. The majority of contracts 7Ceara Multi-sector Social Inclusion Development (P082142), Ceara Inclusive Growth (SWAp II) (P106765) and Minas Gerais Partnership II SWAp (P101324). Page 32 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) needed one or more extensions to account for poor quality of the first products and other unforeseen delays, which affected the disbursements calendar. The Brazilian public sector tends to place a heavy focus on ex-ante controls and requires several clearances from the State attorney general’s office (Procuradoria Geral do Estado – PGE), further adding to delays at every step, including the clearance of technical specifications/TORs, Requests for Proposals (RFPs), evaluation reports and contract proposals. 94. The audit report for 2016 raised concerns regarding the implementation of the Renova Escola program. The findings included suspicion of fraud and corruption in school construction works. Even though it was unclear whether project funds were directly involved, disbursements were temporarily suspended to allow for further investigation. The investigations did not find evidence of fraud and corruption in project funds but nevertheless, the World Bank and the PIU decided to exclude the expenditures related to the Renova Escola program from the project. The audit report also found evidence that the wells built with project funds were not connected to households. A number of missions were carried out to further investigate this issue and mitigation measures were proposed and carried out during the following years as described above. These issues caused an overall delay in project disbursements, as well as the downgrading of some project ratings. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design Rating: Modest 95. The M&E design at project preparation was adequate. The Results Framework was detailed, including annual targets, frequency of reporting, data source, methodology and responsible institution. The Results Framework was complemented by a series of logical frameworks prepared for each EEP by the Parana Institute for Economic and Social Development (IPARDES) in collaboration with implementing agencies. However, the PAD did not include a clear results chain since it was not a requirement at the time. Some indicators lacked a certain cohesiveness, and a few had no direct relation to an activity supported by the project. This problem was remedied to a large extent through regular project supervision, ongoing technical support and the two restructurings. The original indicator targets were not always realistic and required later adjustments and the results monitoring and evaluation section of the PAD provided relatively little information about the process, structures and framework in place for monitoring results. The Disbursement Linked Indicators and disbursement targets were clearly outlined at project design. Each of the 13 DLIs had clear disbursement targets and a protocol for verification. M&E Implementation Rating: Modest 96. A M&E module was added to the Project Management and Monitoring System (SIGMA-PP) and used effectively to monitor progress against indicator targets in the later years of implementation. The logical frameworks developed by IPARDES were very detailed, with a total of 132 indicators, 34 of which constituted the project Results Framework. The complexity of the M&E framework and the number of agencies involved in the monitoring of indicators were a challenge particularly in the first years of project implementation. The development of a M&E module in the SIGMA-PP significantly improved the collection of data and monitoring and reporting of results. Although complex and creating an added Page 33 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) burden in monitoring efforts, the logical frameworks/M&E exercise proved useful in training counterparts on how to think more methodically about results management. 97. The original PDO was considered too broad and general to allow for effective monitoring of progress towards its achievement and was subsequently changed to better reflect the key performance indicators of the project as outlined in the PDO level indicators, DLIs and intermediate indicators. However, as project activities changed, especially under component 2 (technical assistance), the results indicators and targets were not always adjusted. For example, the purchase and installation of the integrated tax system, while necessary and useful to the tax authorities, could not be expected to influence the collection of tax arrears, yet that was maintained as a PDO level indicator throughout the life of the project. M&E Utilization Rating: Substantial 98. There is evidence in project documentation that shows that available data was used strategically or operationally to inform project implementation, policymaking and to engage the client . As noted above, close monitoring of indictors was generally used proactively to adjust indicator descriptions and indicator targets during two project revisions. Monitoring was also used to ensure additional technical support to resolve bottlenecks to achieving planned results targets. Justification of Overall Rating of Quality of M&E Overall quality of M&E: Modest 99. While the M&E framework was not particularly strong at design and appraisal (although adequate), it was substantially strengthened during project implementation, and there is strong evidence of proactive monitoring of results and the use of results information to strengthen project design and adjust necessary technical and supervisory support. The culture of monitoring and evaluation in the State was strengthened significantly due to the experience of implementing this complex operation, and evidence suggests that the lessons learned have been utilized in subsequent planning and formulation of public policies and other Government interventions. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE Environmental 100. At the time of appraisal, the project was classified as Category B because the expected environmental impacts were limited in scope, localized, temporary and reversible. The project triggered the following environmental safeguard policies: Environmental Assessment (OP/BP 4. 01), Natural Habitats (OP/BP 4. 04), Forests (OP/BP 4. 36), Pest Management (OP 4.09), and Physical Cultural Resources (Op/BP 4.11). Measures taken to address environmental safeguard policy issues included the preparation of an environmental assessment and environmental management framework (Environmental Management Framework – EMF) and institutional arrangements for supervision and oversight of environmental compliance. Due to the multisectoral nature of the project, each sector had to comply with the overall project safeguard framework and with sectoral federal and state laws. Adequate budget and technical assistance were allocated for environmental safeguards aspects. 101. Overall, supervision missions and reports have confirmed general compliance with environmental safeguards polices. During project implementation, no significant negative environmental Page 34 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) impacts were identified. The overall activities supported by the project led to positive impacts on the environment, such as improvement of environmental management and water resource management. Compliance with safeguards policies was assessed regularly by the World Bank team and reflected in the ISRs throughout the life of the project. No grievances were submitted during project implementation regarding environmental risks and impacts. The environmental safeguard compliance was consistently rated Satisfactory during project implementation. Social Safeguards 102. The project triggered OP 4.10 (Indigenous Peoples) and 4.12 (Involuntary Resettlement). Social risks were related to activities in health, education, and agriculture. The risk rating was low until year 2016, after which it was raised to moderate due to the greater potential for resettlement issues related to the rural water systems and the rural roads maintenance works. The safeguards performance remained satisfactory for OP 4.10 (indigenous peoples) in the health and education areas but social safeguards performance was rated moderately unsatisfactory from October 2017 due to lack of accurate information on the management of potential involuntary resettlement. 103. At the time of project closure the World Bank was informed by the Borrower that project activities did not involve resettlement as foreseen in the Loan Agreement , and that objectives of the project’s Resettlement Policy Framework – including ensuring information disclosure and consultation to affected people of the supported works – have been achieved. A pending commitment still remains at the time of ICR completion for the Borrower to provide the Due Diligence Report. This report should describe and evidence how compliance with the principles set in the Resettlement Policy Framework were achieved for all eight rural road upgrading interventions and 137 basic water systems built with project support. Likely impacts of involuntary resettlement on peoples’ livelihoods were expected to be negligible because the works would require only small strips of land that would be voluntarily donated by beneficiaries and, whenever physical displacement could be needed, the engineering designs would be changed to avoid these adverse impacts. The Task Team will continue working with the Borrower to meet this requirement by June 15, 2020 and ensure there are no pending issue related to involuntary resettlement in the project. Financial Management 104. Financial management (FM) supervision missions mostly classified the rating as Moderately Satisfactory. There were two periods when the performance rating was downgraded to Moderately Unsatisfactory. 105. In September 2015, FM was downgraded to MU when the audit report identified that existing control systems and procedures were insufficient to adequately control, account, report and manage loan funds. The findings of the TCE pointed to: suspicion of fraud and corruption under the Renova Escola program, although it was not clear whether project funds were directly involved; and unfinished works, with respect to water supply systems in rural areas executed under the Microbacias program. In the audit report for the year 2016, the TCE reinforced the findings related to the Renova Escola program and left the decision to the World Bank as to whether to include expenditures related to this program as part of the EEPs. The World Bank considered it prudent to exclude these expenditures from the project, to avoid any issues moving forward. The FM specialist agreed to resolve the issue from a disbursements point of view. Issues concerning the Renova Escola program were corrected via disbursement request number 10 and the TCE-PR confirmed its acceptance of the mitigating measures taken to address these issues in the 2018 audit report In the audit reports from 2014 to 2018, the TCE pointed out issues of non-functioning Page 35 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) wells under the Microbacias program. In the 2018 report, auditors emphasized that in the Aide Memoire dated March 18, 2019, a two phased action plan was agreed to be implemented by the project closing date. The results of the first phase were presented to the auditors during the audit review period; however, the second phase had only just been initiated, and only after its conclusion would it be possible to issue an opinion on the matter. A request for clarification on this point was included in the audit review letter sent to the PIU on November 20, 2019, for which the response is currently pending. 106. In June 2018 FM was downgraded to MU due to the problems presented by the new financial management system (SIAF). In the case of (b) above the existing State financial management information system (SIAF) was discontinued on December 31, 2017, when the old SIAF was transformed in a database (storing the information recorded until December 31, 2017). The launching of the new SIAF took place on January 1st, 2018. Unfortunately, there were unexpected issues in the implementation of the new system which prevented its full usage. These issues lasted for almost a year and only in the supervision mission in August 2019, was the World Bank was able to assess the new SIAF and the quality of its integration with the Project Management and Monitoring System (SIGMA-PP), reconciling the information downloaded from both systems and confirming the reliability of information. This review found that SIGMA-PP reflects the project’s structure, which is also reflected in the LDO (Law of Budgetary Directions) and LOA (Annual Budgetary Law) and that SIGMA-PP is duly integrated with the new SIAF since December 2018 and is updated with information from the new SIAF daily. 107. The project’s FM risk rating was Moderate until August 2015. In September 2015 it rose to High (due to the audit findings); in June 2016 it changed to Substantial and in November 2016 it was changed back to Moderate. In December 2017, the risk rating was raised again to Substantial (due to the audit findings), which was maintained until the end of the project. given the issues with SIAF. 108. Audit reports were all received after the due date, due to the audit reports having to be approved by the TCE plenary. All Interim Financial Reports (IFRs) received during the life of the project were considered acceptable and were generally received with some delay. Continuous support has been provided to the PIU for the submission of correct IFRs. All reports submitted so far have been considered acceptable. The 2014 and 2016 reports expressed qualified opinions, while the 2015 and 2017 reports expressed unqualified opinions. The 2018 Audit Report expressed an unqualified opinion; however, it highlights that there was margin for the improvement of the internal controls related to the project. The 2019 and 2020 reports have not yet been received. 109. There have been no cases of ineligible expense claims so far. All expenditures indicated as non- financeable by the TCE or by the World Bank were promptly substituted by other eligible expenditures. Procurement 110. One of the main challenges in the project was ensuring compatibility between the World Bank’s procurement systems and requirements with State and Federal legislation. This included finding an acceptable manner to embed antifraud and corruption clauses in the bidding and contract documents. The Government agencies had well-functioning procurement teams, with the relevant experience in procuring goods and services using Brazilian law and rules on procurement but with limited experience with World Bank guidelines and no prior experience with processes to hire consulting services. Although the key staff received substantial training and support from the World Bank, it took longer than planned to process procurement planning, preparation of bidding documents, terms of references, and evaluations. Page 36 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) 111. The Project’s implementation arrangements did not address implementation challenges, particularly on procurement issues. The multi-sector Project involved as beneficiaries and implementing agencies the Secretariats of Finance, Planning, Education, Agriculture, Health, Environment, among others. The large number of sectors and implementing agencies complicated procurement monitoring by the PIU, which was responsible for overall procurement aspects under the project and centralized the procurement of all TA activities. The PIU was in charge of providing cross-support to all staff involved in the project, as well. 112. Despite initial challenges in applying the World Bank’s procurement policy, procurement rules were complied with and there were no major procurement issues throughout implementation. The application of the World Bank’s guidelines in the selection of consultants and firms seems to have had a positive impact on the quality of the service providers. However, the turnover of high Government officials in the secretariats caused difficulties, especially delays in definitions and preparing technical documents. This affected the procurement of several project activities and was particularly evident in the implementation of the TA activities. 113. Overall, the policy advice and technical assistance provided by the World Bank team helped to strengthen institutional capacity. By project closing, the PIU´s staff have strengthened capacity to perform procurement processes for all consulting activities, fiduciary responsibilities, planning, monitoring and evaluation. Moreover, the project provided a number of trainings to key technical personnel on procurement and management, which is especially relevant for the GOP and municipalities in the context of few prior relevant experience. C. BANK PERFORMANCE Quality at Entry 114. The project was well designed and benefited from a highly capable PIU. The preparation of the operation drew on lessons learned from the implementation of the previous SWAps, which had introduced results-based methodologies in other Brazilian states. Prior experience demonstrated the need for a strong and extensive technical assistance component to provide support to public sector modernization activities and key elements necessary to carry out the GOP’s strategy in each of the sectors. Implementation arrangements were complex, given the number of actors involved in the execution of the various programs supported under the project. Overall coordination was the responsibility of the State Secretariat of Planning and Coordination as the implementing agency. To better manage coordination during implementation, a Project Management Committee was set up, comprised of representatives of all executing secretariats. Technical focal points for each of the EEPs were appointed, responsible for the timely execution and adequate monitoring of their respective EEP’s and technical assistance activities supported under the project. Implementation of the technical assistance component was centralized at the PIU, which was intended to allow for greater specialization of the procurement team in Bank procurement procedures, reducing mistakes and accelerating implementation. The Quality at Entry is rated as Moderately Satisfactory. Quality of Supervision 115. Project implementation was well supported by the task team and progress was adequately monitored and documented. The task team showed significant proactivity in adapting the project activities and targets in response to changing needs and rebalancing of priorities, including through a thorough mid-term review followed by a level 1 restructuring in 2016. The team generally responded Page 37 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) quickly and effectively when issues arose, as was the case with the audit report for the year 2016. Supervision missions occurred regularly to support the implementation of the project and provided technical support to all sectors. The task team also successfully facilitated knowledge exchange by bringing internal experts to Parana to support the authorities on activities under the TA component. 16 Implementation Status & Results Report (ISR) were filed at a bi-annual frequency between 2013 and 2019. ISRs captured well the status of the indicators and progress of the project. Three changes in task team leadership during the lifetime of the project created some challenges and delays, but continuity of the procurement team and the majority of sector specialists based in Brasilia helped to ensure continuous support through regular technical missions and close supervision. The supervision of social safeguards was less frequent after 2016 and this may have contributed to the more difficult communication surrounding the approach to application of the RPF. The Quality of Supervision is rated as Satisfactory Justification of Overall Rating of Bank Performance 116. The overall Bank performance is considered to be Satisfactory. This assessment is made on the basis of a review of the PAD and accompanying documents, standard monitoring documentation (ISR), the two restructuring documents and the extensive Mid-Term Review and the Economic and Financial Analysis carried out in 2016. These all show a proactive and technically competent Bank team that provided timely and appropriate support to the authorities to prepare, supervise and revise the project. D. RISK TO DEVELOPMENT OUTCOME Rating: Moderate 117. Risks were reviewed throughout the implementation period and were systematically assessed by supervision missions as Moderate. The political and governance, macroeconomic and fiduciary aspects were identified as having the highest risks to the achievement and sustainability of results under the project. Of these, only macroeconomic risks materialized through depreciation of the exchange rate and its impact on project funds under component 2. Other risks that materialized were technical in nature. These included the lack of engagement by municipal water authorities to ensure the preparation (linkages to distribution networks) and maintenance of the wells prepared under component 1 and limited community mobilization and training on water treatment. Other risks include the ability of the authorities to continue providing the necessary technical support and training to associations and cooperatives for their business initiatives. The sustainability of the teacher training program may also be in question, pending evaluation of the program to determine whether the highs costs are justified by the results in terms of learning outcomes. V. LESSONS AND RECOMMENDATIONS 118. A multi-sectoral operation at the sub-sovereign level is complex to prepare and implement. The following lessons and recommendations represent some reflections from different parties involved in the project, based on experiences from the project design and implementation. 119. Results based instruments can work effectively to support improvements in public sector management and reforms. The results-based component was successful in motivating reforms. The selection of well-designed DLIs focused attention on outcomes rather than outputs or budget execution. However, some institutions did not fully understand the concept of DLIs and expected Bank Page 38 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) funding to be additional to their budget. It is important to ensure a common understanding amongst all stakeholders of how a results-based instrument works in practice from the concept and preparation stage. 120. Disbursements based on the achievement of results focused the dialogue on outcomes and helped to increase inter-agency coordination to solve problems. Improved coordination practices can be sustainable after the end of the project if the respective stakeholders see the value of enhanced communication and cooperation in tackling complex problems, but this requires sustained engagement and technical support to multiple stakeholders. 121. Use of a technical assistance component to complement the results-based lending can improve sustainability of reforms – but should be selective. High quality TA delivered to the agencies engaged in carrying out the results-based component facilitated progress towards the PDO targets and enhanced sustainability by institutionalizing the underlying changes in public sector management. However, technical assistance contracts require additional attention and support from the World Bank in the design and preparation of TORs and quality control during implementation. Future operations should be selective with regards to technical assistance activities, focusing only on areas with clear gaps on the client side, and where there is a very clear value added from Bank engagement throughout the process of developing technical specifications and ensuring quality control of the end product. 122. Setting meaningful targets requires prior technical work on realistic projections and periodic adjustment during program implementation. Several indicators vastly exceeded their targets or reached their target levels early in the project. This points to weak technical preparation, or overly modest target setting. The Bank’s technical experts may need to conduct more in-depth reviews of the counterpart’s upstream preparatory work, take regular stock of progress including projections, and make adjustments during project implementation. For example, benchmarking Parana with other Brazilian states at similar level of development or income could have helped to better determine realistic targets and projections for their achievement. 123. Ensure that the indicators reflect Government priorities, that there are concrete activities in support of their achievement, and that these remain relevant and realistic over the duration of the project. The tax results indicator (PDO indicator 4) was disconnected from a clear government program and specific activities to ensure follow-up and close monitoring of progress. This could have been avoided or remedied by a deeper dialogue, both at project preparation and during implementation. 124. Project design benefits from assessment of the maturity and realism of the client’s reform program through sustained dialogue at project preparation stage . Reforms under the public sector management results area could have benefitted from further dialogue and technical support. This results area did not have a clear program as such but rather a set of separate activities, and while these activities were priorities at the time, they did not have a strong program behind them, meaning that the attention to the achievement of targets and benchmarks may have been weaker than other results areas. 125. Proactive and hands-on project supervision and periodic adjustment in project activities is critical for project success. The task team worked proactively to adjust the project activities and indicators, replacing activities in line with shifting needs or where the Government carried out actions outside the formal boundaries of the project but which nevertheless contributed to the project’s objectives. This same proactivity led to the effective handling of the irregularities uncovered by the financial audit, without compromising project activities. Sector and fiduciary specialists based in Brasilia were able to engage easily and quickly with counterparts throughout the life of the project. However, the Page 39 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) differences with the authorities over the safeguards procedures related to the rural roads and wells should have been discussed and procedures agreed on earlier. 126. Managing complex multi-sector operations requires generous supervision budgets . Results were excellent where the Bank was able to provide on-demand support when needed (for example DRM). Results could have been improved in other areas (ie Rural Development) if there had been additional ongoing support. A multi sector project should be treated as several different interventions and budget allocated accordingly to ensure sufficient Bank technical support to the client during implementation. . Page 40 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Increase the survival rate in the final cycle of fundamental education Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Survival rate in the final cycle Percentage 78.40 80.00 85.50 86.90 of fundamental education in State Schools 15-Jun-2012 31-Aug-2017 30-Aug-2019 26-May-2020 Comments (achievements against targets): Achieved and targets surpassed. Objective/Outcome: Improve maternal and emergency health care services Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Decrease in Maternal Number 65.11 47.53 46.58 43.30 Page 41 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Mortality Ratio 30-Aug-2010 31-Aug-2017 30-Aug-2019 26-May-2020 Comments (achievements against targets): Achieved and targets surpassed. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Decrease in Mortality Rate Number 51.48 48.57 46.34 for deaths through external causes, except violence 30-Aug-2011 30-Aug-2019 26-May-2020 Comments (achievements against targets): Achieved and targets significantly surpassed. This indicator was added after the MTR. Objective/Outcome: Promote business initiatives and improve natural resources management in rural areas Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Hectares supported by the Number 0.00 750000.00 728000.00 Micro-catchment Action Plans' interventions 15-Jun-2012 30-Aug-2019 26-May-2020 Page 42 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Comments (achievements against targets): Partially achieved. This indicator was added after the MTR. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Producer groups supported Percentage 0.00 60.00 75.00 24.00 through the project with improved access and 15-Jun-2012 31-Aug-2017 30-Aug-2019 26-May-2020 integration to output markets Comments (achievements against targets): Partially achieved. Of a total of 52 financed projects,13 were finalized (physical and financial execution) by project closing, but the evaluation of the criteria as established for the indicator was pending. The remaining financed projects were being finalized and hence the evaluation required additional time to complete. By the time of the ICR, 24 producer groups were supported through the project and had improved access and integration to output markets. During the second half of 2018, the following actions were carried out to enable the monitoring of the results of the indicator: (a) formalization of the Technical Cooperation Agreement No. 002/2018; (b) completion of the study with the characterization of approved organizations; (c) conclusion of the theoretical study of the Subcomponent, generating a logical model, where the indicators used to monitor and evaluate the proposals will be presented according to the criteria presented in the review of the Indicator for the period 2016-2018; and (d) finalization of the activities to prepare the Initial Diagnosis of the Organization with the objective of obtaining the basic information on the entity and the current situation (zero mark) which will be used to guide the work of ATER and for evaluation of the implementation of the Proposal of Business Initiative. Successful verification of the results will depend on the finalization of the physical-financial execution of the Terms of Development. Objective/Outcome: Contribute to building capacity in public administration, and environmental and disaster risk manage Page 43 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Revenue from tax arrears as a Percentage 1.99 2.00 5.00 2.00 share of the stock of tax arrears 30-Aug-2013 31-Aug-2017 30-Aug-2019 26-May-2020 Comments (achievements against targets): Not achieved. In 2016, when SEFA implemented a late tax recovery program with cash benefits which enhanced the results. In the following years, no similar programs were implemented, thus the increase was not as large as had been anticipated at the time of project mid term review. One of the reasons for the continued low tax arrears recovery rate is the lack of a proper segmentation of tax arrears into categories, for example arrears under dispute, arrears linked to non-operational entities (with low likelihood of recovery). A.2 Intermediate Results Indicators Component: Component 1: Fostering equitable and environmentally sustainable economic and human development Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of Farmers trained Number 0.00 21000.00 21000.00 75784.00 or receiving technical assistance 15-Jun-2012 31-Aug-2017 30-Aug-2019 26-May-2020 Comments (achievements against targets): Achieved and targets surpassed. Page 44 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of business Number 0.00 150.00 100.00 52.00 proposals approved and to be financed 15-Jun-2012 31-Aug-2017 30-Aug-2019 26-May-2020 Comments (achievements against targets): Partially achieved. The final DLI (for the year 2017) target was 40 business proposals approved and to be financed, and was achieved in 2017 as planned. The remaining targets were not linked to disbursements. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Beneficiaries of legally Number 0.00 6000.00 2702.00 concluded land regularization processes 15-Jun-2012 30-Aug-2019 26-May-2020 Comments (achievements against targets): Not achieved. This indicator was added after the MTR. The achievement of this indicator has been affected by the failure of the Public Defender's Office and the State Court of Appeals to conclude land regularization processes. By December 31, 2018, the ITCG carried out occupational and cartorial surveys, the geo-referencing of real estate and organization of documents for the investigation of processes to benefit 6,907 families of farmers. However, the Page 45 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Public Defender's Office and the Court of Justice have only been able to file actions that will benefit 2,702 families, or 39 percent of the total. If not for the delay in the Public Defender's Office and the Court system, the indicator would have been fully achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Management Plans for rural Number 0.00 60.00 0.00 roads prepared 15-Jun-2012 30-Aug-2019 26-May-2020 Comments (achievements against targets): Not achieved. This indicator was added after the MTR. To achieve this results indicator the following activities were carried out: (i) completion of the mapping through the identification of official municipal rural roads according to Master Plans and Municipal Laws (exclusion of private roads); (ii) survey of the quantitative and qualitative information of the municipal roads; and (iii) workshops, meetings and public consultations for development of Management Plans. However, the Management Plans were not completed by project closing. According to the road mapping data compiled by Parana Cidade (2006), the State of Parana has about 290,000km of unpaved rural roads, which represent 84 percent of the state's road network. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of Micro-catchment Number 0.00 350.00 250.00 321.00 actions plans elaborated 15-Jun-2012 31-Aug-2017 30-Aug-2019 26-May-2020 Page 46 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Comments (achievements against targets): Achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of people in rural Number 0.00 28000.00 3338.00 areas provided with access to Improved Water Sources 15-Jun-2012 30-Aug-2019 26-May-2020 under the project Comments (achievements against targets): Not achieved. This indicator was added after the MTR. The State authorities visited all the water systems from Apr to Nov 2019, defining an action plan to resolve the problems identified for each system and starting the implementation of those action plans. Following the results of the visits, from the 185 systems: only 20 percent (37) are delivering treated water; 30 percent (56) are delivering untreated water (trainings were conducted for those cases); 27 percent (49) are incomplete (SANEPAR will implement the works in the following months); 17 percent (31) had a dry well (the State is studying how to resolve those cases); and the other 5 percent (12) have singular characteristics (being defined an ad-hoc action plan). Aguasparana and SANEPAR have just signed a two year Technical Cooperation Agreement to address the problems identified after the project closing date. The State conducted a final audit to assess: the final results of the RWS investments at closing date; and the main challenges and lessons learned of the RWS in all the phases (from identification and planning of investments until operation of the systems). Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 47 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Strengthening of the Disaster Text None Monitoring System Increased capacity Capacity (qualitative Risk Monitoring capacity and and Warning System (qualitative and and quantitative) of Warning System for Hydro- operational quantitative) of the the Warning System meteorological events Warning System, i.e. increased through equipment, models, new equipment and processes, among implementation of others. new models and processes 15-Jun-2012 31-Aug-2017 30-Aug-2019 26-May-2020 Comments (achievements against targets): Achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Implementation of a water Text No integrated One of the four Average processing Three of the four resources subsystem of an environmental and modules of the time for issuing of modules of the Integrated Environmental water resource subsystem is water rights has been subsystem are and Water Resources management system implemented. reduced to 90 days implemented Management System in place (from 180 in 2016). 27-Mar-2015 31-Aug-2017 30-Aug-2019 26-May-2020 Comments (achievements against targets): Achieved. Page 48 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Student Learning Assessment Text None Fifth Assessment Fifth Assessment Five assessments System operational carried out carried out carried out 15-Jun-2012 31-Aug-2017 30-Aug-2019 26-May-2020 Comments (achievements against targets): Achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Teachers participating in Percentage 57.00 90.00 85.00 86.30 Peer-to-peer in-service Teacher Training 15-Jun-2012 31-Aug-2017 30-Aug-2019 26-May-2020 Comments (achievements against targets): Achieved. Page 49 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion University-based in-service Number 2351.00 7830.00 7689.00 9835.00 Teacher Training, Education Development Plan (EDP) 15-Jun-2012 31-Aug-2017 30-Aug-2019 26-May-2020 Comments (achievements against targets): Achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion State Schools Rehabilitated Number 92.00 340.00 450.00 466.00 and/or Expanded 15-Jun-2012 31-Aug-2017 30-Aug-2019 26-May-2020 Comments (achievements against targets): Achieved. The final DLI target (for the year 2017) was 360 state schools rehabilitated and/or expanded. The remaining targets were not linked to disbursements. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 50 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Percentage of pregnant Percentage 5.50 80.00 90.00 100.00 women identified with high risk of complications referred 15-Jun-2012 31-Aug-2017 30-Aug-2019 26-May-2020 by primary care units to a hospital that is part of the Borrower's Infant and Maternal Healthcare Network Comments (achievements against targets): Achieved and targets significantly surpassed. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Proportion of live births to Percentage 80.00 80.00 90.00 100.00 pregnant women who attended more than 7 ante- 15-Jun-2012 31-Aug-2017 31-Aug-2017 26-May-2020 natal check-ups Comments (achievements against targets): Achieved. The final DLI target (for the year 2017) was 80%. The remaining targets were not linked to disbursements. Page 51 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Decrease in Infant Mortality Number 12.17 10.50 10.30 Ratio 30-Aug-2010 30-Aug-2019 26-May-2020 Comments (achievements against targets): Achieved. This indicator was added after the MTR. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Decrease in Mortality Rate Number 80.39 76.20 68.49 for Cardiovascular Diseases (by age group) 30-Aug-2011 30-Aug-2019 26-May-2020 Comments (achievements against targets): Achieved and targets surpassed. This indicator was added after the MTR. It replaced the indicator "Percentage increase in the number of ICU beds in the State of Parana". Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 52 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Percentage of the population Percentage 43.83 100.00 90.25 with access to Urgent and Emergency Health services 30-Aug-2013 30-Aug-2019 26-May-2020 Comments (achievements against targets): Partially achieved.This indicator was added after the MTR. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of teachers who Number 437.00 750.00 807.00 work in indigenous schools trained 10-Feb-2014 30-Aug-2019 26-May-2020 Comments (achievements against targets): Achieved. This indicator was added after the MTR. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Indigenous lands with Number 0.00 8.00 2.00 productive projects supported 10-Feb-2014 30-Aug-2019 26-May-2020 Page 53 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Comments (achievements against targets): Not achieved. This indicator was added after the MTR. One of the Indigenous Land associations with a productive project approved for financing in 2017 did not present the required documentation. Failure to launch a new call for proposals in 2019 negatively impacted the achievement of this indicator. Component: Component 2: Technical Assistance for more efficient and effective public management Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Enhancing core Text No online training 4000 civil servants are Competencies of civil Competencies of civil competencies of civil platform in place and certified through the servants have servants have servants need for additional participation in the increased (as reported increased training for civil face-to-face and in the final report of servants online training the consultancy) modules 27-Mar-2015 30-Aug-2019 31-Aug-2018 26-May-2020 Comments (achievements against targets): Achieved. The final DLI target (for the year 2017) was 12,000 civil servants civil servants certified through the participation in the face-to-face and online training modules. The remaining target was not linked to disbursements. 187 public sector management best practice cases were produced and registered in the Observatory of Innovation in Public Management in Paraná. 22.000 civil servants were trained (46 percent male, 54 percent female) and 260 events were held (60 percent in-person). Page 54 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Strengthening of Internal Text Internal Control Unit Information system Internal control Internal control Control Systems recently created and for the internal information system information system lacking formal control systems implemented and implemented and methods and implemented. operational in 50 operational in 117 procedures government entities government entities and 100% of users trained 15-Jun-2012 31-Aug-2017 30-Aug-2019 26-May-2020 Comments (achievements against targets): Achieved. The final DLI target (for the year 2018) was 80% of users of the system have been trained on using the new system. The remaining target was not linked to disbursements. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Strengthening State Capacity Text Weak capacity to Report with Report with Report with to monitor and evaluate monitor and evaluate recommendations on recommendations on recommendations on public policies public policies the sustainability and the sustainability and the sustainability and continuation of the continuation of the continuation of the land regularization land regularization land regularization activities. activities. activities. 15-Jun-2012 31-Aug-2017 30-Aug-2019 26-May-2020 Page 55 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Comments (achievements against targets): Achieved. This indicator was slightly revised after the MTR, from the original "strengthening state capacity for evidence based policy making". Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Reduction of sick leave and Text The State has started Reduction of sick The implementation absenteeism due to work- to define an leaves/absences due of occupational health related illness or injury occupational health to work-related illness activities is being done policy or injury by 20% using own funds. 10-Feb-2014 30-Aug-2019 26-May-2020 Comments (achievements against targets): Not achieved. This indicator was introduced after the mid-term review and the first targets were met. The following activities were carried out with SEAP's own financial and human resources: diagnosis of exits and absences due to work-related diseases; prioritization of the health and education sectors for the implementation of occupational health strategy; elaboration of Work Safety Report and the Program for the Prevention of Environmental Risks at Work for these two sectors. However, the subsequent targets were not achieved and the percent reduction was not investigated since the monitoring of the number of medical examinations and absences and exits due to work-related illness or injury was not carried out. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 56 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Improvement of the Text Lack of enforcement Evaluation, Processes of Borrower's tax management in tax arrears declaration and restitution, system collection and collection of the rectification, and adequate resources to ITCMD is 90% exemption and administer tax automated. immunity requests are receipts done through the integrated system. 15-Jun-2015 30-Aug-2019 26-May-2020 Comments (achievements against targets): Achieved. This indicator was added after the MTR. It replaced the original indicator "revision of tax management processes". The final DLI target (for the year 2018) was processes of restitution, rectification, and exemption and immunity requests are done through the integrated system. The remaining target was not linked to disbursements. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Establishment of State Text Publication of State State Law establishing State Plan for The State Plan for Disaster Risk Management Decree establishing the State DRM Policy Protection and Civil Protection and Civil (DRM) Policy Framework the DRM working Framework published Defense is under Defense has been group in the Borrower’s implementation drafted and was Diario Oficial presented to the state council for protection and civil defense in August 2016. However, the State Plan is dependent on Page 57 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) the National Plan which has yet to be published. 15-Jun-2012 31-Aug-2016 30-Aug-2019 26-May-2020 Comments (achievements against targets): Achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of Municipalities Number 0.00 22.00 25.00 27.00 with a Decentralized Environmental Licensing and 15-Jun-2012 31-Aug-2017 30-Aug-2019 26-May-2020 Monitoring System Comments (achievements against targets): Achieved and targets surpassed. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Improved Management of Text The management of Monitoring System for Monitoring System for Page 58 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Physical Infrastructures is physical infrastructure Physical Physical schools for SEED in the education Infrastructures is Infrastructures is sector is inadequate operational operational 15-Jun-2012 31-Aug-2017 26-May-2020 Comments (achievements against targets): Achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Capacity strengthening of Text The capacity for Report with Report on the third Evaluation report on SEED to evaluate teacher teacher training recommendations of dimension of PDE the implementation of training programs. evaluation is improvements in EDP the third dimension of inadequate as a result from the the Educational evaluation. Development Plan - PDE has not be prepared 15-Jun-2012 31-Aug-2017 30-Aug-2019 26-May-2020 Comments (achievements against targets): Not achieved. Page 59 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Page 60 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) B. KEY OUTPUTS BY COMPONENT8 Objective/Outcome 1: Increase the survival rate in the final cycle of fundamental education 1. Survival rate in the final cycle of fundamental education in State Outcome Indicators Schools 1. Student Learning Assessment System operational 2. Teachers participating in Peer-to-peer in-service Teacher Training 3. University-based in-service Teacher Training, Education Intermediate Results Indicators Development Plan (EDP) 4. State Schools Rehabilitated and/or Expanded 5. Number of teachers who work in indigenous schools trained Component 1 1. Five Student Learning Assessments carried out 2. 86.30% of teachers participated in Peer-to-peer in-service Teacher Key Outputs by Component Training (linked to the achievement of the Objective/Outcome 1) 3. 9835 teachers trained through University-based in-service Teacher Training, Education Development Plan 4. 466 State Schools Rehabilitated and/or Expanded 5. 807 teachers working in indigenous schools trained Objective/Outcome 2: Improve maternal and emergency health care services 1. Decrease in Maternal Mortality Ratio Outcome Indicators 2. Decrease in Mortality Rate for deaths through external causes, except violence 8 All public documents are available at http://www.planejamento.pr.gov.br/Pagina/Projeto-Multissetorial-Banco-Mundial Page 61 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) 1. Percentage of pregnant women identified with high risk of complications referred by primary care units to a hospital that is part of the Borrower's Infant and Maternal Healthcare Network 2. Proportion of live births to pregnant women who attended more than 7 ante-natal check-ups Intermediate Results Indicators 3. Decrease in Infant Mortality Ratio 4. Decrease in Mortality Rate for Cardiovascular Diseases (by age group) 5. Percentage of the population with access to Urgent and Emergency Health services Component 1 1. 100% live births to pregnant women who attended more than 7 ante-natal check-ups Key Outputs by Component 2. 100% of pregnant women identified with high risk of complications (linked to the achievement of the Objective/Outcome 2) referred by primary care units to a hospital that is part of the Borrower's Infant and Maternal Healthcare Network Objective/Outcome 3: Promote business initiatives and improve natural resources management in rural areas 1. Hectares supported by the Micro-catchment Action Plans' interventions Outcome Indicators 2. Producer groups supported through the project with improved access and integration to output markets 1. Number of Farmers trained or receiving technical assistance 2. Number of business proposals approved and to be financed 3. Beneficiaries of legally concluded land regularization processes Intermediate Results Indicators 4. Management Plans for rural roads prepared 5. Number of Micro-catchment actions plans elaborated 6. Number of people in rural areas provided with access to Improved Water Sources under the project Page 62 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) 7. Indigenous lands with productive projects supported Component 1 1. 52 producer groups projects financed 2. 2702 beneficiaries of legally concluded land regularization processes 3. Micro-catchment actions plans elaborated Key Outputs by Component 4. 3338 people in rural areas provided with access to Improved Water (linked to the achievement of the Objective/Outcome 3) Sources under the project 5. Two indigenous lands with productive projects supported 6. 728000 hectares supported by the Micro-catchment Action Plans' interventions 7. 75784 Farmers trained or received technical assistance Objective/Outcome 4: Contribute to building capacity in public administration, and environmental and disaster risk manage Outcome Indicators 1. Revenue from tax arrears as a share of the stock of tax arrears 1. Enhancing core competencies of civil servants 2. Strengthening of Internal Control Systems 3. Strengthening State Capacity to monitor and evaluate public policies 4. Reduction of sick leave and absenteeism due to work-related illness or injury 5. Improvement of the Borrower's tax management system Intermediate Results Indicators 6. Strengthening of the Disaster Risk Monitoring capacity and Warning System for Hydro-meteorological events 7. Implementation of a water resources subsystem of an Integrated Environmental and Water Resources Management System 8. Establishment of State Disaster Risk Management (DRM) Policy Framework 9. Number of Municipalities with a Decentralized Environmental Licensing and Monitoring System Page 63 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) 10. Improved Management of Physical Infrastructures is schools for SEED 11. Capacity strengthening of SEED to evaluate teacher training programs Component 1 1. Capacity (qualitative and quantitative) of the Warning System increased through new equipment and implementation of new models and processes 2. Three of the four modules of a water resources subsystem implemented Component 2 1. 22.000 civil servants were trained (46 percent male, 54 percent female) 2. 187 public sector management best practice cases produced and Key Outputs by Component registered in the Observatory of Innovation in Public Management in (linked to the achievement of the Objective/Outcome 4) Paraná. 3. 260 training events for civil servants 4. Internal control information system 5. Report with recommendations on the sustainability and continuation of the land regularization activities 6. Integrated tax administration system 7. State Plan for Protection and Civil Defense 8. Monitoring System for Physical Infrastructures 9. Two regional development plans 10. 4,090 state properties evaluated 11. Integrated environmental licensing system 12. Disaster risk monitoring and management unit for the State Page 64 of 86 The World Bank SWAp for Parana Multi-sector Development Project (P126343) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Fanny Weiner Task Team Leader(s) Sinue Aliram De Souza Procurement Specialist(s) Maria Joao Pagarim Ribei Kaizeler Financial Management Specialist Judith M. Lisansky Social Specialist Gunars H. Platais Social Specialist Maria Bernadete Ribas Lange Social Specialist Supervision/ICR Kjetil Hansen Task Team Leader(s) Sinue Aliram De Souza Procurement Specialist(s) Susana Amaral Financial Management Specialist Fernanda Balduino de Oliveira Financial Management Specialist Frederico Ferreira Fonseca Pedroso Team Member Angela Nieves Marques Porto Procurement Team Daniel Ortega Nieto Team Member Leandro Oliveira Costa Team Member Ezau Pontes Team Member Davide Zucchini Team Member Satoshi Ogita Team Member Barbara Cristina Noronha Farinelli Team Member Maria Bernadete Ribas Lange Environmental Specialist Michele Martins Team Member Graciela Sanchez Martinez Social Specialist Page 65 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Roberto F. Iunes Team Member B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY11 16.944 129,280.16 FY12 35.535 181,110.27 FY13 15.734 65,963.93 Total 68.21 376,354.36 Supervision/ICR FY13 13.995 70,534.33 FY14 16.948 75,426.97 FY15 31.612 170,284.50 FY16 41.331 161,634.77 FY17 31.891 155,605.26 FY18 29.203 152,436.39 FY19 20.375 113,421.58 FY20 34.419 208,970.99 Total 219.77 1,108,314.79 Page 66 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Component 1: Fostering equitable and environmentally sustainable 0 315.00 0 economic and human development Component 2: Technical Assistance for more efficient 0 35.00 0 and effective public management Total 0.00 350.00 0.00 Page 67 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) ANNEX 4. EFFICIENCY ANALYSIS 1. The project was evaluated from an economic perspective and complemented with sensitivity analysis in 2016. The analysis was updated based on updated information at the project closing date as of November 2019. The approach to measure economic benefits varies according to the results area. For health, the economic benefits were calculated on the basis of lives saved through the emergency and maternal health interventions. For education, benefits were measured as increased earnings for student and teachers resulting from earlier graduation and teacher certification. Disaster risk management was evaluated by simulating the breakeven point of averted losses that makes the intervention viable and a benchmarking technique using international comparisons. The rural development component was evaluated showing marginal increased returns for farmers. Finally, public sector efficiency is measured by the returns from the investment in the integrated tax system as measured by the reduction in the time for tax payers to file and pay their vehicle circulation tax, the returns from the updated state property registry in the form of properties sold and maintenance savings and the integrated water management system returns are measured by the time saved in obtaining water use licenses. 2. Impact on Economic Development. Results show that the project has been a worthwhile investment as the investments are contributing positively to the development of the State of Parana and will continue to do so into the future. All sub-components are economically viable with internal economic returns higher than 6%. The overall project yields a 15% return and US$ about 226M of net benefit. Total benefits are 73% higher than the costs, giving ample room for uncertainties along the lifecycle of the interventions. Methodology 3. Cost benefit analysis was used for the evaluation of all the interventions under the project. The cost- benefit analysis calculates the net benefits generated by each subcomponent on an incremental basis. The benefits of each intervention equal the difference between the incremental benefits and the incremental costs of two scenarios: with and without the project. The with-project scenario considers actual and expected achievements from the interventions. The without-project scenario corresponds to business as usual. Benefits and costs of both scenarios were projected during the lifetime of the works using a discount rate of 6%9. Other benefits cannot be measured and so they are mentioned and a description of their potential effects is provided. The lifetime of the economic returns to project investments were estimated as follows: a) for the health component 35-years period; b) for education 15 years; c) for disaster years 10; and d) for rural development 5 years and ( e) public sector 5 years. 4. The benefits were measured based on the expected targets established in the results framework of the original project in 2012, and revised in 2015 and the final reported results at the time of project closing in November 2019. The following indicators were selected from the results framework. For health interventions: a) decrease in maternal mortality ratio; b) percentage of pregnant women identified with high risk of complications referred by primary care units to a hospital that is part of the Borrower’s; c) proportion of live births to pregnant women who attended more than 7 ante-natal check-ups; and d) mortality rate for deaths through external causes except violence. For Education: a) survival rate in the 9 According to the guidelines established by the Sustainable Development Group of the World Bank in February 2016. Page 68 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) final cycle of fundamental education in State School; and b) Number of teachers trained. For Rural Development: producer groups supported through the project with improved access and integration to output markets. For disaster, better risk management. For Public Sector Management: the original indicator focused on the reduction in the stock of tax arrears. However due to changes in the related project component, the economic valuation was instead assessed on the basis of the expected reduction in time for taxpayers to register, file and pay taxes through the new integrated tax system financed in part by the project. Components of the Project and associated costs 5. The foreseen cost at appraisal was 1.3 Bn Reais (USD 678M), 80% of it was assigned to education and health, and the remaining 20% to rural and environmental and disaster risk management. However, total disbursements as of end 2019 end was over double the amount at 2.8 Bn R$ (2012 prices). To transform the prices of activities already implemented, disbursements from the loan were transformed to 2012 prices using the exchange rate at time of disbursement and then adjusted by inflation. Counterpart funds were transformed to 2012 prices using the CPI index from the time of appraisal to the date when funds were used. From 2012 to 2019 the exchange rate went from R$ 2.00: 1US$ to R$ 3.94: 1USD, which means 65 percent loss of the Brazilian currency against the USD. Each disbursement and payment was affected differently according to the time of payment. For this evaluation an average annual inflation rate of 6% was used. Figure 1. Program Disbursements R$ and US$ Nominal prices, exchange rate, and CPI index 700,000,000 10.00 9.00 600,000,000 8.00 500,000,000 7.00 6.00 400,000,000 5.00 300,000,000 4.00 200,000,000 3.00 2.00 100,000,000 1.00 0 0.00 2013 2014 2015 2016 2017 2018 2019 Total Disbursements (Real) Total Disbursements (USD) Real/USD Exchange Rate Inflation Table 1. Investment Costs from 2013-2019 (R$ - Nominal) 2013 2014 2015 2016 2017 2018 2019 Total Rural Development 15,006,870 31,430,269 9,935,099 33,947,899 23,864,743 37,745,791 3,973,872 155,904,543 Rural Development 4,469,447 2,296,217 5,799,878 27,535,757 6,452,519 15,436,048 2,112,923 64,102,789 Page 69 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Water Management 10,537,423 29,134,052 4,135,221 6,412,142 17,412,224 22,309,743 1,860,949 91,801,754 DRM and Environment 9,308,201 2,101,810 1,807,675 11,926,372 13,399,915 922,557 1,372,146 40,838,676 Environmental Licensing 3,994,681 1,149,880 721,000 6,335,000 12,200,561 Disaster Risk Management 5,313,520 951,930 1,086,675 5,591,372 13,399,915 922,557 1,372,146 28,638,115 Education 87,036,885 75,114,622 34,298,307 23,144,176 28,868,552 31,933,666 17,623,191 298,019,399 Evaluation Learning 5,712,927 3,058,928 966,193 12,738,468 22,476,516 Teacher Training 11,045,067 11,767,796 18,659,412 8,184,318 8,519,534 2,309,586 6,951 60,492,664 School Maintenance 70,278,891 60,287,898 14,672,702 14,959,858 20,349,018 16,885,612 17,616,240 215,050,219 Health 141,363,426 251,870,492 393,364,714 549,516,934 578,222,673 509,858,087 372,917,184 2,797,113,510 Urgent Care 100,581,706 154,474,091 190,830,128 344,086,080 286,485,015 285,854,470 264,452,843 1,626,764,333 Maternal Health 40,781,720 97,396,401 202,534,586 205,430,854 291,737,658 224,003,617 108,464,341 1,170,349,177 Public Sector Management 4,385,783 15,567,293 24,787,530 20,416,230 65,156,836 Total Disbursements (Real) 252,715,382 360,517,193 439,405,795 618,535,381 644,355,883 580,460,101 395,886,393 3,291,876,128 6. The costs were transformed to 2012 adjusting actual and planned works by inflation and exchange rate fluctuations. Results vary per intervention: a) In the rural development sub-component actual costs were almost 40% lower than expected; b) environmental & disaster management was 25% lower; c) education component is 34% lower; while d) health component is much higher, nearly four times higher for the emergency centers, and 275 percent higher for the infant and maternal health care network intervention. Table 2. Expected and Actual costs ( 2012 prices) Expected costs at Actual costs 2012 Actual/ Million R$ appraisal prices Expected Sustainable Rural Development 221.5 132.5 0.59 Territorial Economic Development-PRORURAL 130.9 54.4 0.41 Improved Water Soil Management Micro-Catchments 90.6 78 0.86 Environmental & Disaster risk management 44.9 34.7 0.77 Environmental Licensing System Modernization 17.2 10.3 0.60 Natural and Man-made disaster risk management 27.8 24.3 0.87 Education 373.1 249.3 0.66 Learning Assessment System 15.4 19.1 1.24 Teacher Development Program 96.0 51.4 0.53 Upgrading of school facilities 261.7 178.8 0.68 Health 717.0 2,794 3.89 Emergency Care 357.2 1,382 3.86 Infant and Maternal Health Care Network 359.8 994.7 2.76 Public Sector 75.6 55.1 0.72 Total 1,356.5 2,849 2.1 7. To conduct the economic analysis, the costs used are expressed at 2012 prices. They were transformed to economic prices using a conversion factor of 0.85 to eliminate imperfections caused by market distortions such as subsidies and taxes10. Maintenance costs were estimated as 1% of the investment costs. 10 VAT of 17% included in the estimation of the conversion factor Page 70 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Health 8. This subcomponent supported the implementation of two Healthcare Programs (referred to as networks – redes in portuguese) covering the 22 health administrative regions of the State: (i) the infant and maternal health care network (Programa Mae Paranense) and (ii) the emergency health care network. 9. Maternal Health Program: From the time when the program started, the maternal mortality rate 11 declined from 65.1 in 2010 to 43.35 by the end of the implementation period in 2019 and it is expected to decrease further to 41 in the next 2-3 years. The child mortality rate12 of newborns decreased from 12.4 in 2010 to 10.3 in 2019. The reduction of mortality rates is the result of the actions implemented under the project. More pregnant women had regular check-up visits during the prenatal period. The proportion between the number of babies born alive and the number of pregnant women who attended at least 7 medical visits during the prenatal period increased from 80% in 2010 to 85% in 2019. Through check-up visits the potential risks at delivery were identified and mitigated with special medical care. The percentage of pregnant women with high risk at delivery sent from primary care centers to hospitals belonging to the Programa Mae Paranense increased from 5.5% in 2010 to 99.9%, in 2019 surpassing the target set at appraisal of 80%. 10. The benefits estimated are: a) decrease of mortality rates of pregnant women and newborn babies; and b) savings in treatment costs when more attention is given to pregnant women and babies through primary care instead of urgency centers. Reduction of mortality rates will improve the quality of family life bringing along additional social and economic benefits. The benefits of reduction of mortality rates were estimated as the avoided cost of premature death. For the calculation, the number of deaths were projected for both scenarios: with and without project. For the without project scenario, the mortality rates were kept constant at 2010 levels, that is, 12.4 newborn babies dead per one thousands babies born alive; and 65.1 dead pregnant women per 100 thousand babies born alive. For the with project scenario, the actual mortality rates achieved to date were included, as well as expected by the end of implementation period (10.4 for the child mortality rate, and 43.35 for the pregnant women mortality rate) 11. The number of years lost due to the premature death were calculated using the life expectancy at Brazil (75 years according to WHO), minus actual age of date. According to health statistics from the Secretariat of Health, the average age of date of pregnant women varied from 28 years to 31 years in the period 2010-2015. This means that about 45 years were lost per pregnant death. For newborn babies who died, the number of years lost was estimated from their productive life (16 years and on) until the life expectancy, which results in about 59 years. The value per year lost was estimated using the median wage of R$ 1,075 per month (2012 prices) discounted at 3% during the projection period. Results show an average present value of premature death of pregnant women of R$ 315,458 (US$ 157,729) and R$ 354,825 (US$ 177,413) per newborn baby dead for an average 45 years of productive life. 11 The maternal mortality rate is measured as number of pregnant women dead per 100 thousand children born alive 12 Child mortality rate is measured as number of newborns dead per 1000 children born alive Page 71 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) 12. The number of deaths of pregnant women were projected for both scenarios: without and with project. The without project scenario was projecting keeping constant the mortality rate of pregnant women at time of appraisal, 65.1 per 100 thousand babies born alive. Under the with project scenario the mortality rate decreases to expected levels by the end of the implementation period, that is 43.3 and left constant thereafter. The incremental benefit corresponded to the difference of cost of premature death from both scenarios. 13. The number of deaths of newborn babies was projected using information from SINACS (Information system for newborn babies), which shows that about 65% of deaths was preventable and 53% of these were related to hospital and outpatient care. The intervention focuses in this specific group of babies and so this is the number used for the projection of both scenarios: with and without project. The without project scenario was forecasted keeping constant the mortality of newborn babies at time of appraisal, 12.4. The with project scenario decreased the mortality with the levels already attained, 10.4 and left constant after. The incremental benefit corresponds to the difference of cost of premature death from both scenarios. 14. Benefits related to savings on medical treatment cost were also estimated using information from the SUS and from medical center and specialists from Secretariat of Health. The savings of treatment cost results when more primary care is offered and as a result less urgent care is needed. Total benefits corresponded to the sum of the avoided cost of premature deaths from pregnant women and newborn babies plus savings on treatment costs. The economic evaluation show that benefits are higher than the costs of the intervention. Net benefits are expected to be US$ 21.4M, and the internal rate of return is 7 percent over 35 years. Results are on the conservative side as the cost of premature death was estimated with the median wage and does not include the importance of a life of a mother in the quality of family life and all the social and economic consequences that come along. Sensitivity analysis shows that the intervention will still be viable even if no additional improvement is attained and the target already achieved remains unchanged. 15. The emergency medical care network program was the largest investment of the project at over 1.3 Billion R$. It invested in emergency health infrastructure and equipment including ground and air emergency transport and extensive training on trauma care. It set up regional emergency units covering the entire state at a maximum of 250 Km radius, with the objective of being able to attend to any medical emergency within 1 hour. As a result of this program the “mortality rate for deaths through external causes except violence” was reduced from 51.48 per 100,000 to 26.53, essentially halving the mortality rate, and saving on average 212 lives per year. For the purposes of this analysis, it was assumed that lives saved through the emergency services network would add on average of 15 additional productive years to a persons life, at a NPV life value of R$191,919 calculated using the 2012 median state income of R$1,075. At this rate, the program would generate a net benefit of R$214 million and an IRR of 16 percent over 35 years, taking into account the initial investment of 1.3 Bn R$ and 1 percent O&M costs and a discount rate of 3 percent. Table 3. Results of the economic evaluation Health Sub-component Present Value of Flows (000 R$) IRR Costs Benefits Net Benefit % Infant and Maternal Health Care Network 1,039 1,108 69 7% Page 72 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Emergency Care Network 1,382 1,597 214 16% Education 16. The main challenges being addressed by this sub-component are: a) improving the quality of instruction; b) reducing student over-age, therefore increasing retention; and c) improving the schooling environment in State School in fundamental and secondary education. These challenges have been addressed through the following activities: a) learning assessment system; b) teacher development program; and c) upgrading of school facilities. 17. Implementation of learning assessment system accompanied by implementation of standardized evaluation on regular basis, has helped the State to increase its capacity and its means to measure learning outcomes and disseminate evaluation results with a view to quickly and effectively implement pedagogical remedial intervention. Teacher development program has improved the skills and proficiency of teachers in the State through in-service and university-based teacher training. Upgrading of school facilities through rehabilitation, expansion, and addition of facilities such as laboratories, libraries, furniture, and equipment, has helped improve quality of education, reduce overcrowded schools, and increase the capacity of some of the schools. 18. Costs. The costs included in the evaluation consists of the actual investment costs of each activity expressed at 2012 and adjusted by the conversion factor. Maintenance costs was included. Actual investment cost is about US$ 134M, of which, 71% corresponds to upgrading schools, 23% to train the teachers, and the remaining 5% to the learning assessment system. Among the indicators selected for monitoring the implementation of this sub-component are: a) the survival rate (cohort) in the final cycle of fundamental education in State Schools; b) University-based in-service teacher training; and c) number of schools upgraded. The survival (graduation) rate (cohort) in the final cycle of fundamental education in state schools measures the percentage of students in the beginning year of fundamental education who reach the final year at the end of the required number of years. This indicator measures not only the retention capacity of the state schools but also the internal efficiency of the education system. 19. This intervention will have direct and indirect benefits. Direct and primary beneficiaries will be the students, and the teachers who got training: a) Students will benefit when they graduate at the right age at fundamental schools, since they will more likely graduate from secondary school and either continue superior education or get to the labor market earlier. When survival rate(cohort) improves the kids will graduate earlier, getting earnings in average two years in advance; b) Teachers will improve their salaries when they get training. Teachers’ salary is established based on their education level, and school tier. To be eligible for training the teacher has to be at least at level II in tier 8. After the training, the teacher reaches level III tier 1. Before the training, the monthly salary for teaching 20-hours per week was R$ 2,490 (level II tier 8); after the training it raised to R$ 3,027, which means an increase of R$ 537 per month. Every two years the teacher gets a performance evaluation and if it goes well it advances a tier. For this evaluation the increase of R$ 537 was adjusted by inflation to expressed it in 2012 prices. 20. Net benefits were estimated as incremental benefits from two situations, ‘with’ and ‘without’ interventions. In the ‘without’ intervention scenario, survival rates at time of appraisal remained and no Page 73 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) training for teachers was included. In the ‘with’ intervention scenario, survival rate and trained teachers increase to actual levels achieved by 2015 and left unchanged afterwards. The benefits for students were estimated as those obtained from higher survival rate. The survival rate at the baseline (2012) was 78.4, which means that 78% of the kids who started the final cycle of fundamental education in state schools reached the final year at the end of the required number of years. The remaining 22% either dropout or stayed at school behind in courses credits. In the without project scenario this percentage was kept unchanged. In the with project scenario the survival rate increased to reach actual achievement of 83.3% by 2015 and left unchanged after. 21. Benefits were estimated as higher earnings that come when graduating at the right time instead of staying at school for one, two, or more years. For this evaluation, benefits were estimated assuming that the number of students who graduate on time as a result of the increase of the survival rate will increase their lifetime income by two-years of early wages. The increase of income was estimated using the minimum wage in Brazil at 2012 prices (about R$ 500) for a period of two years. Actual increase may be higher as the student could pursue further education or could get a better paid job. 22. The number of students who benefit from a better survival rate was estimated as the incremental number of students who graduate at the right time when comparing two scenarios: without and with project. The calculation was done applying the survival rate to the number of students in the last year of FE. It is interesting to notice that the student population in FE and SE in the whole state has been in decline in the past years, and so the number of students graduating has decreased as well. However, when comparing the without and the with project situation, there is a gain of 5,700 students per year who are graduating at the right time thanks to the project. The number of students along the lifetime of the intervention -estimated as 15 years- is projected using the decline rate shown in the previous years (-4.6% per year). Table 4. Students graduating at the right age in FE Increase Baseline (2012) 2015 2012-2015 Students enrolled in FE and SE in the State 2,026,3431 1,920,0201 Students enrolled in FE and SE in State Schools 1,067,9531 965,5701 % of those enrolled in FE 62%1 58%1 Students enrolled in second cycle of FE in State Schools 664,6161 557,9041 % of those enrolled in in the 9th grade of FE 24%2 24% Students in last 9th grade of FE at State Schools 159,508 133,897 % of those who finish at the right time (survival rate) Without project 78.4% 78.4% With project 78.4% 82.7% 4.3% Students who graduate at the right age in FE Without project 125,054 104,975 With project 125,054 110,733 5,758 1 Source Seed/Sude/Diplan - Coordenação de Informações Educacionais 2 It corresponds to the percentage of students enrolled in the ninth year of FE in the 350 schools upgraded under the project. 23. The benefits for teachers were estimated as the number of teachers trained under the program (9,835) multiplied by the increase of salary when changing the category. Additional income that will come from performance evaluation were not included in this evaluation. Present value of expected benefits is Page 74 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) US$ 341M, 59% of them correspond to benefits from increasing the survival rate, and the remnant of 41% to the teachers’ training. Table 5. Benefits from the Sub-component Education (Million US$) Present value of Flow of Benefits Benefits from the Increase of survival rate 202 59% Benefits from teachers’ training 139 41% Total benefits 341 100% 24. Results of the Economic evaluation of the education subcomponent . The benefits of the teacher training program are linked to the activities implemented under the teacher development program, and so this specific intervention was evaluated by itself. Benefits from the increasing of the survival rate are due to the activities implemented in the whole sub-component and as such it was evaluated as a comprehensive program including all activities under the sub-component. 25. Results of the activities under the teacher development program show that benefits are well above the costs and the return of 33% is significantly higher that the discount rate of 6%. The costs can increase about four times and this particular intervention would still be viable. Actual benefits are expected to be higher as it contributes to achieve higher survival rates, and teachers can get good performance evaluation and increase their salaries even more. 26. Benefits from increasing survival rate and benefits from the teacher’s training program were compared to the costs of all activities under the sub-component. Results show that present value of net benefits of the program will be US$ 173M and the return generated as high as 19% 27. Results are reassuring given that the sub-component will yield positive returns even taking out the benefits from teachers’ training, and including only benefits from survival rate. In this case the return is 11%, and benefits 40% higher than the costs. Table 6. Results of the economic evaluation - Education Sub-component Present Value of Flows (000 USD) IRR Costs Benefit Net Benefit % Teacher development program 31,553 117,388 85,835 33% Total Subcomponent Including benefits from survival rate & teachers’ training 136,597 309,627 173,030 19% Including only benefits from survival rate 136,597 192,238 55,642 11% 28. Sensitivity analysis shows ample room for unexpected changes on the variables over time. The remaining cost of investment can increase two fold and the sub-component would be viable. Even if the unemployment rate in Brazil is applied to take into account that some of the students may not get job, the project will still be viable. Actual benefits are higher than those included in this evaluation, as only Page 75 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) direct benefits were accounted for. Additional benefits that will come from graduating at the right time, and the associated decreasing dropout rate, will impact positively the society, yet they were not quantified in this evaluation. Disaster risk management 29. This subcomponent focused on (i) restructuring of governance for DRM), (ii) increasing understanding of risk, and (iii) enhancing response capacity. The State Government has been implementing the activities under this sub-component. An early warning system through the installation of alarms and automatic text messages to warn citizens in risk zones is now in place and fully functional. Previous to the project there was no system in place, and by 2016, about 1,500 alarms systems were installed and the alerts issued through SMS have increased significantly. The response time to disasters has improved from 24 hours before the project started to 8 hours or even less according to the Civil Defense office. 30. The benefits of the early warnings systems are important as they give people time to flee from a flash flood, or a landscape, or a fire, or a hazard zone where a catastrophe happened, etc. It also enables local authorities to evacuate or shelter large number of people in advance to a disaster event. It provides information on the occurrence of a hazard; and it enables a faster response to problems of food and water insecurity. Timely warnings also enable people to protect their properties and infrastructure. Effective warning systems reduce substantially deaths, injuries and damages from severe events. This is especially important in the State of Parana, where according to the Civil Defense office about half a million people reside in flooding and landslides’ prone areas. 31. While the cost of the preparedness and better response to disaster events included in the intervention is known; its benefits are more difficult to determine. Benefits accrue over the lifetime of the mitigation actions and might not be completely demonstrated unless a hazard event occurs. Two ways of estimating risks can be distinguished13: a) a forward-looking impact-based assessment. In order to calculate benefits using a forward-looking risk assessment, estimates about the frequency and intensity of a potential hazard event are needed, as well as an assessment of vulnerability (potential infrastructure damage under different hazard scenarios and potential causalities or injuries) and potential exposure to the event. It is then possible to calculate a community’s overall risk associated with a particular hazard and demonstrate how that risk would be reduced under different mitigation scenarios. From this, the overall benefits of a particular mitigation activity can be estimated; and b) backward-looking risk estimation, which relies on damage reports from past events and adjusts those for possible changes in exposure and vulnerability, to arrive at a current risk estimate. Then, as with forward-looking assessments, total risk reduction of various mitigation activities is calculated and the associated benefits are derived. The benefits are discounted over time. 32. There are essentially four steps to analyzing the costs and benefits of interventions to reduce risk: a) risk analysis; b) identification of risk management options and their associated costs; c) risk analysis with and without implementation; and d) estimation of costs and benefit probabilities. All of this should be done in a probabilistic manner, i.e in relation to the probability that hazard events with a given intensity will occur at specific frequencies. For this evaluation there was not enough information to run the risk 13 Mechler, R (2005). Cost-benefit analysis of natural disaster risk management in developing countries. Working paper. Deutsche Gesellschaft fuer Technische Zusammenarbeit (GTZ), Eschborn Page 76 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) analysis and no simulation models to estimate the impact of the intervention. The sub-component was evaluated using the information that was available to make a rough estimate of the benefits. Two approaches were used: a) a partial backward-looking approach; and b) a benchmarking technique. 33. A Backward-looking approach was selected for this evaluation. using damage assessments carried out by the Civil Defense from events that occurred over the 2006-2015 period. Damages varied according to type of event, intensity, frequency, etc. The Civil Defense registers the type of events by frequency of the event, people affected (evacuated, people displaced, hurt, and other kind of affectation), people dead, dwellings damaged or destroyed, and estimates the total damage cost of properties and infrastructure. Damages associated with people affected or value of life are not estimated by the Civil Defense. The following events are assessed on regular basis: floods, landslides, storms, erosion, drought, forest fires, fires in properties, hazardous substances released, building collapse, events from accidents occurred when transporting hazardous and non-hazardous materials. In the period 2006-2015, the average number of disasters caused by climate related and anthropogenic events were 447 per year, which affected one million people, killed 17, damaged 28,000 dwellings and registered a damage cost of R$800M per year. Table 7. Disaster Events and affectation 2006-2015 # of people affected Dwellings Total damage costs of Frequency of (Evacuated, displaced, injured, damaged or properties and events other) Deaths destroyed infrastructure(000 R$) 2,006 251 483,484 2 19,827 1,014,795 2,007 312 90,147 7 10,156 134,219 2,008 269 265,227 17 21,818 261,704 2,009 428 875,425 8 23,725 1,008,820 2,010 405 479,712 10 25,460 432,742 2,011 462 1,522,550 36 31,447 451,707 2,012 512 2,824,842 13 11,856 2,570,902 2,013 616 1,904,192 29 38,631 620,512 2,014 567 1,465,866 26 49,500 995,553 2,015 650 426,145 24 50,733 523,773 Average 447 1,033,759 17 28,315 801,473 34. The impact of the intervention on disaster risk management will be obtained mostly by the installation and well managed early warning systems to send alarms in advance to the events to the communities. The type of hazards that can be predicted are those related with climate change events, such as, flooding, landslide, erosion, storms, drought, and in some cases forest fires. Therefore, the benefits from the project will mainly come from a reduction of damages and associated costs caused by climate related events. The information from disasters was then separated by type of event, and only the climate related one was considered in this evaluation. About 80% of all disasters registered by the Civil Defense corresponds to climate related events on average, which are responsible for half of the deaths, and almost all the damages and people affected. Table 8. Climate Related Events and affectation 2006-2015 Percentage of climate related events to total events Frequency of # of people affected Dwellings Total damage costs of events (Evacuated, displaced, injured, Deaths damaged or properties and Page 77 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) other) destroyed infrastructure(000 R$) 2,006 60% 100% 50% 100% 100% 2,007 48% 100% 14% 99% 96% 2,008 68% 63% 71% 100% 99% 2,009 82% 98% 75% 100% 99% 2,010 75% 99% 60% 100% 100% 2,011 76% 74% 39% 100% 100% 2,012 89% 99% 46% 100% 100% 2,013 76% 74% 24% 100% 98% 2,014 86% 98% 50% 100% 100% 2,015 88% 96% 54% 100% 100% Average 78% 89% 46% 100% 100% 35. Benefits will be reflected in the form of losses that are averted from the mitigation measures implemented under this sub-component. Given the uncertainty of the impact of the intervention on reducing the damages, sensitivity to different reduction of losses was conducted ant the breakeven point to make the project viable was estimated. Results show that for averted losses higher than 2% of damage costs, the project would be viable. The higher the averted losses the bigger the returns. If damage costs reduce by 5% the intervention would yield 28% return and net benefits of US$ 52M. If damages reduce by 2.5%, net benefits would be close to US$ 9M and the yield return would be 11%. Table 9. Results of the economic evaluation - Environmental and DRM activities Present Value of Flows (000 USD) IRR Costs Benefit Net Benefit % If damage cost reduces by: 5% 16,100 85,353 67,983 49% 3% 33,173 51,212 33,842 32% 2.5% 33,173 42,676 25,307 26% 2% 33,173 34,141 16,771 20% 36. Benchmarking technique. This approach evaluates potential benefit according to results of the Hallegatte study (2012)14 for this type of intervention. The study found that the hydro-meteorological information and early warning systems installed in Europe have saved several hundreds of lives per year, avoided between Euros 400M and 2.9 billion of disaster asset losses per year, and produced between 3.4 and 34 billion of additional benefits per year through the optimization of economic production in weather- sensitive sectors. For developing countries, the study found that benefits included: i) between USD 300 million and 2 billion per year of avoided asset losses; ii) an average of 23,000 saved lives per year, valued in the range of USD 700 million to 3.5 billion per year using the Copenhagen Consensus guidelines; and iii) from USD 3 to 30 billion per year of additional economic activities. 37. Hallegatte’s study states that to achieve the assumed benefits, investment should include: (1) local observation systems; (2) local forecast capacity; (3) increased capacity to interpret forecasts and translate 14Hallegatte, Stéphane, 2012. A Cost Effective Solution to Reduce Disaster Losses in Developing Countries: Hydro-Meteorological Services, Early Warning, and Evacuation, World Bank Policy Research Working Paper #6058 Page 78 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) them into warnings; (4) communication tools to distribute and disseminate information, data, and warnings; and (5) institutional capacity building and increased decision-making capacity by the users of warnings and hydro-meteorological information. The project supported the State of Parana in all of these critical areas. 38. Hallegate estimated benefits from early warning and preparation measures in three categories: a) reduction of asset losses; b) reduction of human losses; and c) other economic benefits such as productivity enhancements. Regarding the reduction of asset losses and other economic benefits, the study found that a well-functioning modern early warning system in a high income country reduces disaster-related asset losses from 0.003% to 0.017% of GDP; and add adds gains of about 0.1 to 1% in weather-sensitive sectors, which in GDP terms corresponds to 0.0025 to 0.025% on average. The study concluded that a) for low-income countries only around 10% of the benefits were achieved on average; b) for lower middle income countries - 20%; c) upper middle income, 50% of benefits (in relation to high income countries). Applying the results of the Hallegate’s study to this assessment, the expected benefits from the early warning systems and hydro-meteorological information could be around 50% of those obtained in a high income country, in the range of 0.0028% of GDP to 0.021%. Table 10. Expected Benefits from the DRM intervention as percentage of the GDP Well developed Country Upper middle Income Country (% GDP) (% GDP) Low Estimate High Estimate Low Estimate High Estimate Expected benefits: Reduction of Asset losses 0.003% 0.017% 0.0015% 0.0085% Gains from economic activity 0.0025% 0.025% 0.0013% 0.0125% Total benefits 0.0055% 0.0420% 0.0028% 0.0210% 39. Applying the resulting percentages to the 2012 GDP in the State of Parana of US$ 145.5 (IBGE), the expected annual benefits for this intervention would be in the range of US$ 4M to US$ 30.56M. Table 11 Expected Annual Benefits from the DRM intervention (million US$) Annual Benefits (million US$) Present Value Benefit (million US$) Low Estimate High Estimate Low Estimate High Estimate Expected benefits: Reduction Asset losses 2.18 12.37 Gains economic activity 1.82 18.19 Total benefits 4.00 30.56 27.5 212.2 40. The present value of benefits over 10 years is in the range of US$ 29 to US$ 212M. Results of the evaluation of the intervention using the benchmarking technique estimates benefits to be slightly higher than the costs of the project and a return of 6 percent. The high boundary estimates show benefits up to 8 times higher than costs. Results obtained with both approaches are similar. Results from the benchmarking technique show that under low estimates the intervention is viable economically and the Page 79 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) resulting benefits correspond to 2.0% of the average annual damages costs registered by the Civil Defense, which is similar to the breakeven point obtained in the backward-looking approach. Table 12. Results of the economic evaluation of the Environmental and DRM (using Benchmarking technique) Present Value of Flows (000 USD) IRR Costs Benefit Net Benefit % Low Estimates 33,173 33,561 388 6% High Estimates 33,173 256,284 223,112 196% Sustainable Rural Development 41. Territorial Economic Development (Pro-Rural) focusses on improving the competitiveness of small rural producers in the Central Region. Pro-Rural is helping small producers improve their productivity, their associative and marketing techniques, and foster connections with partners and markets in more dynamic regions, through technical and commercial partnerships. The activities that are being implemented are: a) technical assistance to help them identifying new economic opportunities and capacity building; b) provision of public service key for rural competitiveness, such as: roads, land regularization, pilot initiatives for new technologies, and new processing and market techniques; and c) support to sustainable business initiatives through financing subprojects submitted by small producer groups (Associations). 42. Improved Water and Soil Management in Micro-Catchments. This activity is promoting the adoption of sustainable agricultural and natural resources management practices by rural producers through the preparation of micro-catchment development plans, provision of incentives for implementation of priority activities. The program has also improved access to potable water for poorer communities in priority areas. By the end of 2019: a) 17,338 small farmers got training, 51 businesses plans presented by farmers’ associations were approved by the Secretariat of Agriculture for further support, and 81% of farmers associations supported by the project increased their revenue from sales of products or services by at least 7%; b) under the program of micro-catchment, 112 basins had actions plan elaborated, and some of them had implemented activities on soil conservation and improvement of water quality and quantity. 43. The benefits of these actions have been important to the farmers, some have increase their income, ranging from 6% to 20 percent (2016 study). Many have employed additional workers; have increased the productivity, and the plans for micro-catchments have impacted the productivity of the sector in general, and some water sources have been appropriately put in use for human consumption with quality and quantity required. Some farmers were interviewed during preparation of this evaluation and expressed their satisfaction with the program. Among the benefits they mentioned are: better farming practices learned on the training, improvement of soil and water usage, better application of fertilizers, diversify crops, use of equipment and tools that increased their productivity. Page 80 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) Table 13. Returns of cultivating crops in Small Farms in the micro-basins Present Value of Flows during 5-years period (000 R$) After implementation Costs Revenues Net Profit IRR Juquia-Goioxim 967 1,040 73 7.3% Sapucai-Anahy 3,256 3,290 34 6.4% Horionte-Tupassi 604 701 98 20.6% 44. The farmers’ associations received technical assistance to prepare business plans. The Secretariat of Agriculture reviewed them and accepted 52, which will receive additional support. The associations proposed different kind of business activities, such as: a) upgrade a warehouse to receive, classify, storage, and transport the produce to the markets; b) milk collection and production of dairy products; c) production and processing vegetables; and d) production of grapes and grape juice processing. Results of some of the business plans presented by the association show returns from 21% to 136%. Table 14. Results of Business plans prepared by Farmers’ Associations Present Value of Flows during 5-years period (000 R$) Costs Revenues Net Profit IRR Upgrade of warehouse to receive, process and distribute produce 1,166 1,402 235 21.0% Milk collection and processing dairy products 9,045 9,962 917 23.3% Milk collection, refrigeration, and distribution 89,928 93,324 3,397 64.9% Production and processing organic vegetables 3,990 5,723 1,734 96.6% Production of Grapes & grape juice processing 22,217 24,130 1,914 25.2% Processing vegetables 400 1,062 662 136.5% Public Sector Management 45. This component supported the strengthening of public sector capacity through the following activities (i) curriculum development and training for the civil servant training institute (ii) strengthened tax management through the purchase and installation of an integrated tax management system (iii) improved management of environmental and water licensing through the development of an integrated water licensing system (iv) strengthened internal controls through the development of an integrated internal control system (v) mapping and updating of the state asset registry. 46. An upgrade of the state’s tax system was necessary, as functionality of the legacy system was deteriorating and maintenance costs were unsustainable. As of project closing, only two modules of the new tax system were ready and being tested, and running in parallel to the old system. The remaining module will be finalized in 2020. The most immediate and significant economic impact from the purchase and installation of the new integrated tax system will come from the automation of the vehicle circulation tax. With over 7.5 million vehicles in circulation in the state, even a minimum of time saved on the filing and payment of the vehicle tax has a potentially significant impact on taxpayers. For example, a reduction of just half an hour in the time spent to register, file and pay the vehicle circulation tax would yield over Page 81 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) 182 million Reais in savings over 5 years, against the 27 million Real investment in the system. The system is expected to generate savings in the inheritance tax module as well. 47. The new water license management system (SIGARH) has significantly reduced the time required to obtain water licenses for companies and citizens. This has an important impact on the cost of citizens, in the form of time spent on preparing requests and the speed at which requests can be obtained, thus avoiding delays in the execution of investment projects or ongoing business activities. It has also generated internal efficiency gains for the environmental directorate whose staff can re-direct their skills to more strategic support for larger and more complex projects while smaller requests and dealt with automatically, only requiring limited staff engagement in the initial stages. The calculated benefits for this analysis come from the time saved in obtaining a water use license, translated into additional days of productive activities. The values are calculated using the average time saved per type of water license issued through the system (compared with the average time required to obtain the same category of licenses before the system) multiplied by the 2012 median income (R$, 1075 per month) for one person discounted at 6 percent per year for five years. The returns are very large: 600 million R$, from an investment of 8.1 million R$, or a return of 7000 percent. Reducing the monthly income to minimum wage (500 R$) reduces the return to 185 million, still over 2000 percent return. Further reducing the number of productive additional days to a quarter of the average days saved brings a return of 45 million and an IRR of 450 percent. These are very conservative estimates, since 48. The asset mapping exercise helped the state to update its real estate asset registry (over 4,000 real estate assets worth over 9.2 Bn Reais) and move to a more modern system of monitoring its asset portfolio. The asset management unit within the secretariat of planning aims to move to a more strategic approach to property management in order to generate savings through more efficient use of its properties. In a first stage, 3 lots of real estate assets that were not in use were identified and sold off for around 5 million Reais. Savings on property maintenance is expected in the order of around 1 million R$ per year. Additional disposal of state property is likely to happen in the future, including lease/rental and other options and is expected to generate additional revenue. The net benefit for this investment was negative. Table 15: Results of the economic evaluation - Public Sector Component Summary of Benefits Cost (R$) Benefits (R$) Net Benefit IRR CBR Integrated Tax System -27,949,620 56,362,095 28,412,475 102% 2.0 SIGARH -8,159,570 45,064,463 36,904,893 452% 5.5 Asset Mapping -18,214,433 10,101,510 -8,112,923 -45% 0.6 Total -54,323,623 111,528,068 57,204,445 105% 2.1 Page 82 of 83 The World Bank SWAp for Parana Multi-sector Development Project (P126343) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS The draft ICR was shared with the State authorities on May 13, 2020. The team subsequently received some specific comments and editorial suggestions directly in the draft document and mainly related to the project activities and indicators. The authorities agreed with the analysis and overall rating. Page 83 of 83