Document of The World Bank Report No: ICR2778 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-75100) ON A LOAN IN THE AMOUNT OF EURO 32.0 MILLION (US$ 46.4 MILLION EQUIVALENT) TO THE REPUBLIC OF SERBIA FOR THE DELIVERY OF IMPROVED LOCAL SERVICES (DILS) PROJECT November 24, 2015 Health, Nutrition and Population Global Practice South Eastern Europe Country Management Unit Europe and Central Asia Regional Office  i    CURRENCY EQUIVALENTS (Exchange Rate Effective November 2015) Currency Unit = New Serbian Dinar New Serbian Dinar 1.00 = US$ 0.00944 US$ 1.00 = 105.94 New Serbian Dinar FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS CPS Country Partnership Strategy MOLEVSP Ministry of Labor, Employment, Veterans and Social Policy CIDA Canadian International Development MOP Family Allowance (Materijalno obezbedenje Agency porodice) CSW Center for Social Work MPALS Ministry of Public Administration and Local Self- Government EC European Commission NGO Non-governmental Organization ECA Europe and Central Asia (S)NPI (Serbia) National Investment Plan EMF Environmental Management Framework OECD Organization for Economic Cooperation and Development EMP Environmental Management Plan PAT Project Administration Team EU European Union PCU Project Coordination Unit FM(S) Financial Management (Specialist) PHC Primary Health Care FSU Financial Services Unit PIC Project Implementation Committee GDP Gross Domestic Product PISA Programme for International Student Assessment GOP Ministerial strategic plan with three-year POGM Project Operational and Grants Manual projections GoS Government of Serbia PPB Project Policy Board HBS Household Budget Survey PRS Poverty Reduction Strategy HIF Health Insurance Fund RSO Republic Statistical Office IBRD International Bank for Reconstruction SCTM Standing Conference of Towns and Municipalities and Development ICT Information Communication Technology SDP School Development Plan IDP Internally Displaced Person SEIP Serbia Education Improvement Project LFS Labor Force Survey SIF Social Innovations Fund LSG Local Self Government (municipal SLSS Serbia Living Standards Survey authorities) LSMS Living Standards Measurement Survey TIMSS Trends in International Mathematics and Science Study MOESTD Ministry of Education, Science and UNDP United Nations Development Program Technological Development MOF Ministry of Finance USAID United States Agency for International Development MOH Ministry of Health WHO World Health Organization   Senior Global Practice Manager: Timothy Grant Evans Country Director: Ellen A. Goldstein Practice Manager: Enis Barış Project Team Leader: Ana Holt   ICR Team Leader: Ana Holt   ii      SERBIA Delivery of Improved Local Services (DILS) Project CONTENTS Data Sheet ...................................................................................................................................................... i  A. Basic Information .................................................................................................................................. i  B. Key Dates .............................................................................................................................................. i  C. Ratings Summary .................................................................................................................................. i  D. Sector and Theme Codes ...................................................................................................................... ii  E. Bank Staff ............................................................................................................................................. ii  F. Results Framework Analysis ............................................................................................................... iii  G. Ratings of Project Performance in ISRs ............................................................................................ viii  H. Restructuring (if any) ........................................................................................................................ viii  I. Disbursement Profile ............................................................................................................................ x   1. Project Context, Development Objectives and Design ............................................................................. 1  2. Key Factors Affecting Implementation and Outcomes ............................................................................. 6  3. Assessment of Outcomes ........................................................................................................................ 12  4. Assessment of Risk to Development Outcome ....................................................................................... 19  5. Assessment of Bank and Borrower Performance.................................................................................... 20  6. Lessons Learned...................................................................................................................................... 22  7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ......................................... 23   Annex 1. Project Costs and Financing ........................................................................................................ 24  Annex 2: Outputs by Component............................................................................................................... 25  Annex 3: Economic and Financial Analysis ............................................................................................... 50  Annex 4. Bank Lending and Implementation Support/Supervision Processes ........................................... 55  Annex 5. Beneficiary Survey Results ......................................................................................................... 56  Annex 6. Stakeholder Workshop Report and Results ................................................................................. 57  Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ................................................... 58  Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ..................................................... 60  Annex 9. List of Supporting Documents .................................................................................................... 61  Annex 10. Revisions in Results Framework and Monitoring after Level 2 Project Restructuring of June 2012 ............................................................................................................................................................ 62  Annex 11. Revisions to Project Description after Level II Project Restructurings..................................... 67  Annex 12. Project Results, as per the Amended Results Framework (June 26, 2012 and October 30, 2013) .................................................................................................................................................................... 75  Annex 13. Project Results Framework, as per Project Appraisal Document .............................................. 78        Data Sheet     A. Basic Information Delivery of Improved Country:  Serbia  Project Name:  Local Services Project  Project ID:  P096823  L/C/TF Number(s): IBRD-75100  ICR Date:  11/24/2015  ICR Type: Core ICR  Lending Instrument:  SIL  Borrower: GOVERNMENT  Original Total USD 46.40M  Disbursed Amount:  USD 42.16M  Commitment:  Revised Amount:  USD 46.40M      Environmental Category: B  Implementing Agencies: Ministry of Health (MOH) Ministry of Education, Science and Technological Development (MOESTD) Ministry of Labor, Employment, Veterans and Social Policy (MOLEVSP)  Co-financiers and Other External Partners: n/a B. Key Dates Revised / Actual Process  Date  Process  Original Date  Date(s)  Concept Review:  07/21/2006  Effectiveness: 03/10/2009 03/10/2009  06/26/2012 11/30/2012 Appraisal:  04/10/2007  Restructuring(s):    10/30/2013 08/07/2014  Approval:  03/18/2008  Mid-term Review:   10/20/2011      Closing:  12/31/2012  03/31/2015  C. Ratings Summary C.1 Performance Rating by ICR  Outcomes:  Moderately Satisfactory Risk to Development Outcome:  Moderate Bank Performance:  Moderately Satisfactory Borrower Performance:  Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank  Ratings Borrower Ratings  Moderately Quality at Entry:  Government:  Moderately Satisfactory Unsatisfactory  Implementing Quality of Supervision:  Moderately Satisfactory Moderately Satisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Moderately Satisfactory Performance:  Performance: i      C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators  Rating   Performance  (if any) Potential Problem Project Quality at Entry No  None  at any time (Yes/No):  (QEA):  Problem Project at any Quality of Yes  None  time (Yes/No):  Supervision (QSA):  DO rating before Moderately     Closing/Inactive status:  Satisfactory  D. Sector and Theme Codes   Original Actual  Sector Code (as % of total Bank financing)      Central government administration  20 10  General education sector  13 10  Health  13 35  Other social services  14 15  Sub-national government administration 40 30        Theme Code (as % of total Bank financing)      Decentralization  29 20  Education for all  14 10  Health system performance  14 25  Municipal governance and institution building 29 30  Social Safety Nets/Social Assistance & Social Care 14  15  Services  E. Bank Staff Positions  At ICR At Approval  Vice President:  Cyril E. Muller Shigeo Katsu Country Director:  Ellen A. Goldstein Jane Armitage Practice Manager:  Enis Barış  Arup Banerji Project Team Leader:  Ana Holt  Truman Packard ICR Team Leader:  Ana Holt    ICR Primary Author:  Suzana de Campos Abbott    ii      F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The project will help to increase the capacity of institutional actors and Beneficiaries in order to improve access to and the efficiency, equity and quality of local delivery of health, education and social protection services, in a decentralizing environment. Revised Project Development Objectives (as approved by original approving authority)1 (a) PDO Indicator(s) Original Actual Value Target Values Formally Revised Achieved at Indicator  Baseline Value  (from Target Values  Completion or Target approval Years  documents) Primary health care financing allocated according to capitation- and output-based Indicator 1:   formula  PHC providers Value PHC providers paid PHC providers paid paid according quantitative or   according to output- only by salary  to output-based Qualitative)   based formula  formula Date achieved  06/2012  12/2013   03/2015  Comments (incl. % Achieved  achievement)   Indicator 2:   Central and local per capita funding formulae in the education sector piloted Piloting   Theoretical piloting Value completed in started but actual quantitative or Non-existent  15 piloting was not Qualitative)   municipalities  conducted in any of the municipalities Date achieved  06/2012  12/2013    03/2015  Comments (incl. % Not Achieved  achievement)   Percent of children from vulnerable groups in project schools (vulnerable groups according to OECD classification: Category A – children with disabilities; B- Indicator 3:   children with learning difficulties; C-children from socio-economically disadvantaged groups) Value 3.2% - 3.6%  7.5%    6.56%  quantitative or                                                              1  The PDO and Intermediate Outcome Indicators presented in this Section are those following the June 2012 restructuring, which aimed at improving specificity, measurability, and relevance of the indicators to the PDO. The original indicators as presented in the Results Framework in the Project Appraisal Document (PAD) have not been presented here, as the Results Framework contained neither baseline data nor targets for the indicators, nor was progress towards these indicators ever measure. Annex 10 presents original and revised indicators and the rationale for the revisions as mentioned earlier.   iii      Original Actual Value Target Values Formally Revised Achieved at Indicator  Baseline Value  (from Target Values  Completion or Target approval Years  documents) Qualitative)   Date achieved  06/2012  12/2013    03/2015  Comments Partially Achieved (the actual value achieved reached 8.44% in 2010/11; the final (incl. % actual value achieved is in comparison with 4.11% for non-grant schools)  achievement)   Percent of MOLEVSP financing allocated for Disabled Peoples Organizations Indicator 4:   (DOPs), allowing for equal access and improved transparency and based on results Value quantitative or 20%  100%    100%  Qualitative)   Date achieved  06/2012  12/2013   03/2015  Comments (incl. % Achieved  achievement)   Number of Roma children receiving vaccinations through the Roma health mediators Indicator 5:  program  Value quantitative or 0  18,795    30,018  Qualitative)   Date achieved  06/2012  12/2013   03/2015  Comments (incl. % Exceeded  achievement)  Percent of primary health care centers receiving at least a 3-year certificate of Indicator 6:  accreditation  Value quantitative or 0%  25%    40%  Qualitative)   Date achieved  06/2012  12/2013    03/2015  Comments (incl. % Exceeded  achievement)  (b) Intermediate Outcome Indicator(s) Original Actual Value Target Values Formally Revised Achieved at Indicator  Baseline Value  (from Target Values  Completion or approval Target Years  documents) Indicator 1:   Legislative framework allows for capitation- and output-based formula   Value Law passed Law passed and by- No legislative (quantitative and by-law   law adopted to enable framework  or Qualitative)   adopted to capitation  iv      Original Actual Value Target Values Formally Revised Achieved at Indicator  Baseline Value  (from Target Values  Completion or approval Target Years  documents) enable capitation Date achieved  06/2012  12/2013    03/2015  Comments (incl. % Achieved  achievement)   Indicator 2:   Improved allocation framework for equalization funds developed  Framework with Framework with No clear methodology methodology Value methodology and and criteria for and criteria for (quantitative   criteria for allocation equalization fund allocation of or Qualitative)   of equalization funds allocation  equalization developed  funds developed Date achieved  06/2012  12/2013   03/2015  Comments (incl. % Achieved  achievement)   Indicator 3:   Central and local formulae developed Central formulae Value Formulae nearly completed, quantitative or Non-existent    developed  local formulae Qualitative)   development initiated Date achieved  06/2012  12/2013   03/2015  Comments (incl. % Partially achieved  achievement)   Number of educational institutions (schools and preschools) by type of grants Indicator 4:   received  37 EIs for 0 EIs for schools 560 schools o/w 37 schools without violence EIs for schools without without violence violence Value 330 EIs for quantitative or 0 EIs for inclusive   330 EIs for inclusive inclusive Qualitative)   education education education 196 EIs for 0 EIs for inclusion of 193 EIs for inclusion inclusion of Roma children  of Roma children  Roma children Date achieved  06/2012  12/2013   03/2015  Comments Achieved  (incl. % v      Original Actual Value Target Values Formally Revised Achieved at Indicator  Baseline Value  (from Target Values  Completion or approval Target Years  documents) achievement)   Number of medical staff and associated trained to recognize needs of vulnerable Indicator 5:   groups  Value quantitative or 0  2,000    2,000  Qualitative)   Date achieved  06/2012  12/2013   03/2015  Comments (incl. % Achieved  achievement)   Percent of primary health care centers that have completed quality accreditation Indicator 6:   process  Value quantitative or 0  50%    51%  Qualitative)   Date achieved  06/2012  12/2013   03/2015  Comments (incl. % Achieved  achievement)   Indicator 7:   Number of PHC centers that have adopted clinical pathways Value quantitative or 0  50    50  Qualitative)   Date achieved  06/2012  12/2013   03/2015  Comments (incl. % Achieved  achievement)   Indicator 8:   Number of schools that have undergone a school performance external evaluation Value quantitative or 0  60    100  Qualitative)   Date achieved  06/2012  12/2013    03/2015  Comments (incl. % Exceeded  achievement)   Number of training participants among education staff that have undergone Indicator 9:   development training organized by the project Value 23,000 quantitative or 0  participants   23,387 participants  Qualitative)   (14,000 staff) Date achieved  06/2012  12/2013   03/2015  Comments (incl. % Achieved  achievement)   vi      Original Actual Value Target Values Formally Revised Achieved at Indicator  Baseline Value  (from Target Values  Completion or approval Target Years  documents) Indicator 10:   Number of inter-sectoral committees trained Value quantitative or 0  150    150  Qualitative)   Date achieved  06/2012  12/2013   03/2015  Comments (incl. % Achieved  achievement)   Indicator 11:   Percent of PHC centers with fully operational HMIS platform at the PHC level Value quantitative or 0  85%    96%  Qualitative)   Date achieved  06/2012  12/2013   03/2015  Comments (incl. % Exceeded  achievement)   Central management information system (MIS) fully operational in all social Indicator 12:   protection institutions MIS established Value and introduced MIS system quantitative or Non-existent    in all social developed and tested  Qualitative)   protection institutions Date achieved  06/2012  12/2013   03/2015  Comments (incl. % Partially Achieved  achievement)   Number of staff in social protection institutions trained and certified in the use of the Indicator 13:   centralized MIS  600 (2 to 3 Value persons quantitative or 0  in/across all   600 persons  Qualitative)   locations in the country Date achieved  06/2012  12/2013    03/2015  Comments (incl. % Achieved  achievement)   Indicator 14:   Centralized procurement of pharmaceuticals initiated Health Health Insurance Value Insurance Fund awarded quantitative or None  Fund awards   framework Qualitative)   framework agreements for 50 agreements for vii      Original Actual Value Target Values Formally Revised Achieved at Indicator  Baseline Value  (from Target Values  Completion or approval Target Years  documents) at least 50 most frequently drugs from the dispensed drugs  B list Date achieved  06/2012  12/2013   03/2015  Comments (incl. % Achieved  achievement)   G. Ratings of Project Performance in ISRs Date ISR Actual Disbursements No.  DO  IP  Archived  (USD millions) 1  06/30/2008  Satisfactory Satisfactory 0.00 2  06/10/2009  Satisfactory Satisfactory 0.00 3  12/30/2009  Satisfactory Moderately Satisfactory 2.36 4  04/15/2010  Satisfactory Moderately Satisfactory 2.88 5  11/21/2010  Satisfactory Moderately Satisfactory 5.13 6  07/09/2011  Satisfactory Moderately Satisfactory 11.80 7  03/13/2012  Satisfactory Moderately Satisfactory 20.32 8  12/19/2012  Moderately Satisfactory Moderately Satisfactory 24.28 9  06/22/2013  Moderately Satisfactory Moderately Unsatisfactory 27.68 10  10/19/2013  Satisfactory Moderately Satisfactory 29.25 11  12/17/2013  Satisfactory Moderately Satisfactory 30.19 12  06/25/2014  Moderately Satisfactory Moderately Satisfactory 36.04 13  11/24/2014  Moderately Satisfactory Moderately Satisfactory 39.79 14  03/17/2015  Moderately Satisfactory Moderately Satisfactory 40.27 H. Restructuring (if any) ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s)  Key Changes Made  PDO Change  DO  IP  in USD millions A Level 2 Project Restructuring took place to: (a) improve the specificity, measurability and relevance of project indicators to 06/26/2012    S  MS  20.32  strengthen the Project’s results and monitoring framework; (b) reorganize the Project Description to better align the Project’s structure viii      ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s)  Key Changes Made  PDO Change  DO  IP  in USD millions with its implementation arrangements and outcomes; (iii) add the design and implementation of communication campaigns and of measures to improve knowledge as activities and corresponding non- consultant services as an expenditure category critical to build and sustain reform momentum; (iv) add people with disabilities to the list of excluded groups explicitly mentioned as beneficiaries of grants to develop outreach services; and (v) simplify the disbursement schedule by closing of three categories of expenditures and reallocating the remaining funds under these categories into a single, new Category (7) to facilitate the effective use of loan proceeds. A Level 2 Restructuring extended 11/30/2012    S  MS  20.32  the Loan’s Closing Date by one year to December 31, 2013. Level 2 Restructuring took place to introduce an additional intermediate outcome indicator to reflect support to the GoS’ reform on improving efficiency through centralized 10/30/2013    S  MS  29.25  procurement of pharmaceuticals. It also extended the Loan’s Closing Date by an additional nine months to September 30, 2014 (for activities implemented by the MOH only). A Level 2 Restructuring extended the Closing Date by an additional six months to March 31, 2015 to allow the GoS time to cover 08/07/2014    MS  MS  36.04  remaining health activities, utilizing uncommitted loan funds to cover priorities emerging from damages that resulted from severe flooding in May 2014.  ix      I. Disbursement Profile x    1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. Strategic Context. At the time of Appraisal of the Delivery of Improved Local Service Project (DILS, the Project), the Republic of Serbia (Serbia, GoS) had outlined its 2008 National Plan for Integration (NPI). Its NPI was a strategic document that defined Serbia’s agenda for integration with the European Union (EU)—the country’s overarching development objective— and clearly laid out the priorities for harmonization of Serbia’s institutional and legal framework with EU requirements. One of its key elements was increasing employment and living standards and encouraging more balanced regional development. Decentralization, which entailed greater local autonomy and involvement in service delivery, provided strong fiscal incentives through the allocation of resources to local levels of government. It was considered important for European accession and integration, and enjoyed broad political support. 2. Decentralization of responsibility for service delivery was also key to implementing the GoS’ poverty reduction strategy and for improving human development. It was key to sector strategies adopted by the Ministry of Education, Science and Technological Development (MOESTD), the Ministry of Health (MOH) and the Ministry of Labor, Employment, Veterans and Social Policy (MOLEVSP) to improve education, health and welfare outcomes, and was seen by the GoS--together with an accompanying reform of the intergovernmental public finance system— as a critical step in strengthening of local public administration, to make local self-governments more accountable and responsive to the needs of households. A framework law on Local Self- Government Finance that increased the resources and fiscal autonomy of local self-government (LSGs) was passed in Parliament in July 2006, but awaited implementation. 3. Social and Poverty Indicators. Serbia’s social indicators and achievements presented a mixed picture. Health indicators (life expectancy, under-five mortality, infant mortality, maternal mortality) had improved, but still lagged behind those of the EU-15. Serbia also faced problems in providing quality education and was failing to equip young people with the skills and knowledge needed by the labor market. Pre-school education reached only 33 percent of eligible children. Serbia’s safety net for the poorest was very limited—only three percent of poor households received targeted cash benefits, and vulnerable, excluded groups (such as Roma, refugees and internally displaced persons (IDPs)) failed to receive benefits or services. The main social assistance program, the materijalno obezbeđenje porodice (MOP), was well targeted, but its coverage was extremely limited and its benefit levels low. The MOLEVSP Disability Fund, financed with proceeds from Serbia’s State Lottery, was the primary mechanism for supporting the provision of social services for disabled people. The risk of poverty was especially high among children and youth, and vulnerable groups including the disabled, children, Roma, IDPs and refugees showed the highest poverty rates. 4. Progress with Decentralization. Delivery of health, education and social protection services had been mainly a State level responsibility. In education, local governments were responsible for financing the operating costs of primary and secondary schools, but staffing patterns and salaries for teachers and other staff---the bulk of sector spending—were set and paid for by the State. LSGs had virtually no formal role in financing health care. They assisted in the administration of social assistance, but their budgetary responsibility was limited to providing supplementary benefits and services over and above those required by national legislation, provided there was flexibility in their budgets. This system resulted in a significant variation in 1      social services provided between poorer and wealthier municipalities. Despite active capacity- building support by several donor agencies, the capacity of LSGs and other local service providers to manage increased responsibilities varied considerably. The Ministry of Public Administration and Local Self-Government (MPALSG) was expected to assume a critical role in assisting LSGs to manage increased responsibilities. 5. At the time of Appraisal, there had been some degree of decentralization of responsibility for service delivery. Starting in January 2007, in health, selected decision-making authority had been transferred from the State level to provincial and municipal governments, including for the provision of primary health care (PHC), The ability of municipalities to appoint PHC directors, was expected to provide LSGs increased “voice” in running the facilities. For most municipalities, however, funding would still be provided on a line-item basis by the Health Insurance Fund (HIF, a payroll-tax-financed social health insurance institution that was expected to begin implementing capitation-based financing for selected PHC institutions. This h was expected to provide incentives for PHC physicians to provide high quality care that met the needs/demands of users, since users could change doctors if their demands were not met. 6. In education, municipalities were responsible for provision and financing of pre-school services. Primary and secondary school directors were appointed by school boards, consisting of representatives of the municipality, teachers and parents, school directors, selected school staff. Schools were required to prepare and implement development plans reflecting local priorities. The GoS was piloting a “school report card” to enhance accountability of performance to local stakeholders. However, salaries were paid by the GoS and allocated on the basis of centrally- determined teaching loads; plans to increase local management responsibilities were under discussion, including giving LSGs the responsibility to finance staff salaries based on a grant from the GoS (based on per-student financing) and greater local control of the school network. 7. Decentralization of responsibilities for the delivery of social assistance benefits and services was expected to be limited. The MOP was delivered by de-concentrated agencies affiliated with the MOLEVSP, the Centers for Social Work (CSWs), while child allowances were delivered by LSGs. The GoS assigned priority to reforming the finance and delivery of services for the disabled, transitioning out of the model of incremental financing towards one of competitive grants awarded to innovative projects. LSGs were expected to assume greater responsibility for delivering cash and in-kind benefits (that would still be financed from the GoS budget), and the GoS aimed to increase the role of non-governmental, community organizations in the provision of high quality in-kind social services.   8. The Rationale for World Bank Involvement was strong. Decentralization was moving rapidly, and the World Bank and other agencies had provided technical assistance in this area. The World Bank was well positioned to support decentralization and help strengthen local public administration in view of then ongoing, complementary World Bank-financed investment projects and policy operations that supported systemic reforms in each sector. Support through analytical work on poverty, including in the development of the Poverty Reduction Strategy, had helped identify obstacles to increasing access to vulnerable and marginalized groups. Also, the World Bank’s ability to work across sectors was seen as facilitating inter-ministerial coordination required for effective decentralization of service delivery, especially important since there was no inter-ministerial body for coordinating decentralization other than the Commission for Local Self Government Finance that had a narrow mandate to focus on the fiscal aspects of decentralization. The Project was also consistent with the Priority II of the World Bank’s Country Partnership 2      Strategy (CPS) FY08-11, that aimed at providing opportunities and broadening participation in growth and its key indicators were consistent with the CPS outcomes. 1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) 9. The PDO was to increase the capacity of institutional actors and beneficiaries in order to improve access to and the efficiency, equity and quality of local delivery of health, education and social protection services, in a decentralizing environment. The Results Framework, as presented in the PAD, is provided in Annex 13. The following outcome indicators were defined to measure progress towards the PDO: (a) Allocation of financing from State government to local self- governments for health and education services, made according to the Law on Local Self Government Financing; (b) Percentage of grants awarded by the Project to support innovations in inclusion, that have achieved success in increasing inclusion of marginalized groups (the poor) to health, education and social protection services; (c) Share of MOLEVSP financing for disabled groups that is allocated according to a new program-based model of allocation; (d) Satisfaction with the quality of delivery of services among users of health facilities and disability services in municipalities benefiting from the Project, as proxied by responses from exit surveys of users in these municipalities; (e) Number of local service providers (PHCs, schools, CSWs and non- governmental organizations) who have gone through a quality accreditation process, designed and administered by the relevant institution in their sector; and (f) Establishment of information management systems linking local service providers (PHCs, schools, CSWs and non- governmental organizations) with the relevant line ministry in their sector, and where relevant to the successful delivery of services, that allow information sharing across sectors. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 10. The Project’s PDO was not revised. The Results Framework was revised first in a Level 2 restructuring dated June 26, 2012. Changes to the Results Framework and indicators under this restructuring were introduced simply to improve specificity, measurability and relevance of the indicators to the original PDO. The ICR team did not apply a disbursement weighted rating methodology as required by Appendix B of the ICR Guidelines because: (a) the restructuring did not involve a change in the PDO; and (b) no key associated outcome targets were raised or lowered. A matrix presenting the Project’s original and revised Key Indicators, as well as the rationale for the revisions is presented in Annex 10. Another Level 2 Project Restructuring dated October 30, 2013 introduced the following additional intermediate outcome indicator to reflect support to the GoS’ reform on improving efficiency through centralized procurement of pharmaceuticals: “Centralized procurement of pharmaceuticals initiated”, with an end of Project target of “Health Insurance Fund awards framework agreements for at least 50 drugs from the B-list”. Details on the Key Indicators and their status at Closing are presented in Annex 12. 1.4 Main Beneficiaries 11. The Project Appraisal Document (PAD) did not identify specifically the Project’s main beneficiaries. Nevertheless, inferring from the description of the Project’s components, its main beneficiaries included: users of the GoS’ primary health and social assistance services that would benefit from more efficient delivery and higher quality services, persons with disabilities that would benefit from more effective and responsive programs funded through a restructured Disability Fund, and excluded groups (Roma, internally displaced persons, elderly, persons with 3      disabilities) that would benefit from more responsive services, through special programs to expand quality social services coverage to ensure their access to these services. The Project would also improve the capabilities of LSGs, CSWs, PHCs, schools, as well as state ministries (MOE, MOH and MOLEVSP) to deliver more efficient, coordinated and responsive social services to their populations through technical assistance to put into effect new procedures, strategies and information systems and carry out evaluations of their efforts. 1.5 Original Components 12. The DILS Project comprised five components, as described below: Component I: Making Fiscal Decentralization Work. (Total Costs, including contingencies, US$7.0 million; Bank financing, US$7.0 million). This component was to support investments (goods, technical assistance and training) for line ministries and LSGs to develop and adopt mechanisms for earmarking and allocating funds to investment programs at the local level, in accordance with the 2006 Law on Local Self-Government Finance. Activities under this component would: (a) develop new “funds-follow-the-user” formulae and financing framework with the line ministries and the Commission for Local Self-Government Finance, and (b) provide training and capacity building for sector actors at the State, local and service provider levels in applying the new financial framework. Component II: Improving Outreach and Access through Development and expansion of Innovations in Service Delivery. (Total Costs, including contingencies, US$12.2 million; Bank financing, US$12.2 million). This component was to support investments (goods, technical assistance, training and grants) to: (a) increase incentives for local service providers (LSGs, PHCs, schools, CSWs and NGO and community organizations) to develop outreach and inclusion services to excluded groups (such as Roma, IDPs, and refugees) by awarding grants (channeled through the responsible line ministries) to service providers that implement inclusion activities; (b) reform the financing and delivery of services for disabled groups by assisting the MOLEVSP, the CSWs and staff of LSGs responsible for these services to develop and implement a new model of financing and delivering services, based on a system of competitive review of proposals and the awarding of grants to the most promising projects, and implement structures to monitor the performance and evaluate the impact of activities supported by the Fund; (c) build LSG capacity to address the special needs of excluded vulnerable groups through activities that help them identify and implement alternative service delivery arrangements, and (d) improve knowledge and establish institutions to safeguard citizens’ rights. Component III: Supporting a New, Regulatory, Oversight and Quality-Assurance Role for State level Ministries. (Total Costs, including contingencies, US$7.4 million; Bank financing, US$7.4 million). This component was to support investments (goods, technical assistance, training and grants) to: (a) develop and support the training required for State level ministries to shift to a regulatory, oversight and quality assurance role; (b) create mechanisms to foster and assure quality, including the development of structures for accreditation, licensing and accountability to enable the GoS to monitor service delivery and to ensure compliance with procedures and standards, as well as accountability in management and service delivery; (c) define regulations and standards for local service delivery, including the specification of the new roles and responsibilities of municipal staff and service providers in the decentralized system; (d) support the implementation of accreditation, licensing and regulatory standards, and awarding grants to local service providers (PHCs, schools, municipal units responsible for social services and CSWs) to implement those activities. This sub-component would also support minor civil works in PHCs to improve quality of care; and (e) develop mechanisms and capacity of line ministries to ensure quality 4      improvements through piloting, monitoring, impact evaluation and performance analysis, to ensure the effectiveness of specific project investments and to support evidence-based decision making and accountability. Component IV: Improving Capacity of LSGs and other Local Public Institutions as Service Providers. (Total Costs, including contingencies, US$16.5 million; Bank financing, US$16.5 million). This component was to support investments in goods (information and communication technology hardware and software and special technical services, technical assistance, training and grants to schools, PHCs and municipal authorities) to: (a) support capacity building, tailored to LSG and local providers’ needs, to build the minimum competencies for the delivery of services being decentralized; (b) provide grants for quality improvement in schools, mainstreaming a grant facility established by the GoS (with World Bank support under the Serbia Education Improvement Project) that finances quality enhancement proposals in school development plans (SDPs) and for minor civil works to renovate existing schools; (c) improve the use of information, statistics and Information Communication Technology (ICT) equipment to provide better information needed at all levels for policy making, planning, and budgeting of services, as well as for operating, managing and monitoring the delivery of health, education and social protection services. Component V: Project Implementation Support. (Total Costs, including contingencies, US$3.2 million; Bank financing, US$3.2 million). This component was to support a Fiduciary Services Unit (FSU), as well as implementation support for Project Administration Teams (PATs) in the MOE, MOH and MOLEVSP, and the Bank Loan’s Front End Fee. 1.6 Revised Components 13. The Level 2 June 26, 2012 Project Restructuring (Section 1.3) modified the Project description to: (a) better align the Project’s structure with its implementation arrangements and outcomes; (b) add the design and implementation of communications campaigns and of measures to improve knowledge as activities and non-consultant services as a disbursement category under Components 1, 2 and 3; (c) add people with disabilities to the excluded groups targeted by Grants and (d) correct minor inaccuracies in references to governance and implementation arrangements. The revision combined Components 3 and 4 since the GoS faced difficulties in planning and reporting results by component, as most activities under these components were being carried out jointly both by State ministries and LSGs and contributed to the same results. The merging of the two components did not entail any changes to their content, resolved these ambiguities and better aligned the project structure with implementation arrangements and outcomes. The October 30, 2013 Restructuring (Section 1.3) introduced a minor revision to the Project Description to reflect changes to Sub-component (e) of Component 3 to include support to the reform on centralized procurement of pharmaceuticals and efficiency improvements. The revisions after both of these Level 2 Restructurings are presented in Annexes 11 and 12. 1.7 Other significant changes 14. The June 26, 2012 Restructuring also provided for changes to the Project’s disbursement schedule to facilitate the effective use of loan proceeds. Given the uncertainty surrounding the final composition of activities and their prices, the effective use of loan proceeds would have eventually required frequent reallocations. The changes included the closure of three of six original disbursement categories (1 to 3), and the reallocation of remaining funds into a single, new Category 7 that comprised “Goods, Consultants’ Services, including the preparation of the audit under Part 4 of the Project, Non-Consultants Services, Grants (other than Grants for Civil Works 5      under Part 3(f) of the Project, Training and Operating Expenses”. A Level 2 Restructuring on November 30, 2012 extended the Closing Date by one year to December 31, 2013. The October 30, 2013 Restructuring extended the Closing Date by an additional nine months to September 30, 2014 (for activities implemented by the MOH only). A last Level 2 Restructuring dated August 7, 2014 extended the Closing Date by an additional six months to March 31, 2015, representing a cumulative extension of 27 months. This final extension was approved to allow the GoS time to cover remaining health activities, utilizing uncommitted loan funds to cover priorities emerging from damages that resulted from severe flooding in May 2014 (Section 2.2). 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 15. Project Preparation. The Project was prepared in slightly less than two years from concept discussion through approval by the World Bank’s Board. The World Bank’s preparation team was large, and included specialists from each of the three sectors in addition to others. Key lessons were identified in the PAD, but it is not clear that these were adequately addressed in the Project’s design. The team arranged a Roundtable with the Government, including representatives from the Ministry of Finance (MOF), the Ministry of Education, Science and Technological Development (MOESTD), the MOH, the MOLEVSP and the Standing Committee of Towns and Municipalities (SCTM). This group, further strengthened by representatives of the Ministry of Public Administration and Local Self Government, met consistently and worked closely with the World Bank on preparation. This team presented and discussed the Project’s concept and proposed activities with relevant stakeholders, and SCTM routinely provided their feedback to GoS ministries and the World Bank team. The PAD identified four risks—complex institutional arrangements, lack of implementation capacity, a possible politically controversial rationalization process and uncoordinated donor activity. While appropriate, these were not detailed as justified by a complex operation, and did not envisage mitigation measures. 16. Political and Economic Environment. The Project was approved in the midst of challenging internal political developments, and in the midst of the 2008-2009 global economic and financial crisis. In January 2008, Serbia had held first round presidential elections, and in February a candidate from the pro-European Union Democratic Party was elected with a majority. Two weeks after the election, Kosovo declared independence leading to the dissolution of Parliament and the scheduling of new parliamentary elections for May 2008. The EU Democratic Party scored a significant victory but fell short of an outright majority. The formation of the government by a complex coalition comprised of 13 political parties created an environment that was not particularly favorable for project implementation, especially in cases where it was necessary to reach agreement among government institutions headed by officials representing different coalition partners with divergent philosophies. 17. Project Design. At the time of preparation, fewer lending instruments were available for consideration. While some of the objectives related to social service delivery were supported by the Public Expenditure DPL II, the GoS opted for an Investment Landing project. Still, the Project’s design was undoubtedly ambitious and complex, envisaging implementation of activities in three sectors, and through multiple layers of government in addition to by individual schools, PHCs, CSWs, and NGOs and other civil society organizations and associations. Activities would be implemented within a framework of coordination at the national level and eventually coordination at the level of individual LSGs to facilitate an integrated service delivery. In several 6      areas, progress in activities at the national level was needed to make progress at the local, municipal level, or at the level of individual schools or PHCs. Further, the Project envisaged the development and implementation of information systems—notoriously complex in terms of procurement and implementation--in three sectors and multiple levels of government. There were structural design issues in that a few large activities that were to take place over implementation were dependent on policy reforms. About one third of loan proceeds were to be devoted to technical assistance—very supervision intensive and slow disbursing—that were required to design, put in place and implement the activities that followed, such as information systems development and implementation (that represented over 40 percent of loan funding), and grant programs. While there is undoubtedly value in supporting this type of technical assistance activities, especially for promoting coordination and building cohesion, initial delays in these activities led to initial implementation delays across the board. Lastly, original Project design (components and sub-components) did not provide sufficient clarity about how the Project activities were to contribute to the achievement of improved efficiency, access, equity and quality in the delivery of basic services in a decentralized context in the three participating sectors (Section 2.3), an appropriate Results Framework and system for monitoring and evaluation was not in place (Section 2.3) and there were issues with the structure and design of the Project’s components, namely overlap between the activities in Components 3 and 4, and in the specification of the loan’s disbursement categories that made reporting and disbursements difficult (Sections 1.6 and 1.7). 2.2 Implementation 18. The World Bank’s Board approved the Euro 32.0 million loan for the DILS Project on March 18, 2008. The Loan Agreement was signed on April 11, 2008 and declared effective on March 10, 2009. An official Project Launch was held in October, 2009 opened by two Ministers (Health and Education), the President of the SCTM, and the WB’s Country Manager. Implementation Progress as reported in Implementation Status Reports was downgraded to Moderately Satisfactory in December 2009, further downgraded to Moderately Unsatisfactory in June 2013 and upgraded to Moderately Satisfactory four months later as it remained through completion. Its Development Objectives rating was downgraded to Moderately Satisfactory in July 2012 and remained Moderately Satisfactory throughout completion. Several factors affected its implementation, as described below. The final allocation of sector and theme codes (Basic Data Sheet) was adjusted to reflect an estimate of sectors and themes upon completion. 19. Status of Preparation. As described in Section 2.1, several actions required for early implementation were not in place when the loan for the Project was approved. The administrators for the Fiduciary Services Unit (FSU) and the Project Administration Teams (PATs) in each of the three ministries were only being hired in mid-2009, the Designated Account was being opened at the same time, and the institutional mechanisms for requesting grant proposals, including a revised grant manual, were only then being finalized. These led to delays in implementation of technical assistance activities needed to procure information technology equipment, and provision of grants under various lines. In part these delays were the result of the changes in the political environment (Section 2.1) that caused delays in loan effectiveness and early implementation, but they also reflected project readiness at approval. 20. Focus on Disbursements. By June 2010, more than two years after approval, only Euro 1.6 million had been disbursed of the Euro 32.0 million loan. Implementation support missions began to focus on accelerating implementation to increase disbursements by identifying the five largest activities in each sector and agreeing upon a plan to implement those activities faster. These 7      included the procurement of information technology hardware in health, vehicles for schools and the beginning of implementation grants in each of the sectors. 21. Mid-Term Review and Project Restructuring. The Mid-Term Review (MTR) was held on October 10-21, 2011, and included a two-day intensive workshop to: (a) share progress achieved in each of the three participating sectors; (b) discuss how to adjust outcome and output indicators to better reflect the achievement towards the PDO; and (c) exchange views on how to reorganize the presentation of the Project’s key activities in Schedule 1 of the Loan Agreement to ensure a better alignment of the Project Description with the PDOs. Agreements reached during the MTR resulted in the Level II Project Restructuring described in Sections 1.6 and 1.7. 22. Political Transitions. Several political transitions occurred throughout implementation, along with other changes in leadership of the State ministries. Throughout there were six governments, involving four ministers of health, three ministers of education, two ministers of labor and social protection. Changes at the ministerial level invariably led to changes at the level of the PATs in each of the three ministries, as well as in the Coordinator and other staff of the FSU (in 2012), resulted in the need for additional time the new teams to become familiar with the Project’s activities in their respective sector, and often involved revisions to ministries’ internal operating procedures and procurement plans. For example, in 2012, by Ministerial Decision, the MOH established a DILS Project Steering Committee (composed of seven Assistant Ministers) that had to approve every project activity in detail. 23. Government Commitment. Government commitment affected the Project in three ways. First, commitment to the Project’s activities was generally maintained through the PATs in each of the ministries, but those activities that depended upon the passage of legislation or policy reorientation were affected by the frequent changes at the ministerial level and a lack of momentum and commitment to objectives (e.g., the introduction of capitation in education). Second, the Project’s design had envisaged the establishment of a Project Policy Board (PPB) composed of all the stakeholder ministries (MOE, MOH and MOLEVSP), MPALSG, MOF and one representative of LSGs, that would meet regularly, and that would be responsible for project progress, act as a forum to coordinate activities (particularly those of a cross-ministerial nature), resolve disputes and make decisions on necessary adjustments, should the need arise. Throughout, the PPB met only three times, and did not function as contemplated. The MPALSG’s involvement was limited, with the consequence that the Project was implemented as three separate projects at the national level with inter-sectoral cooperation provided on an ad-hoc basis and mostly at the level of the LSGs and other local agencies. Finally, the GoS’s broad commitment to its stated goals in individual sectors was generally strong but intermittent. This was reflected in the passage of legislation in key areas either supported by or affecting the Project, which either provided the legal basis to move forward or led to implementation and design issues, and also in the establishment of committees and commissions that had new responsibilities over activities financed by the Project. The Law on the Foundations of the Education System was passed in 2009, creating the key legal pre-conditions for the DILS implementation, e.g., mandating the change in the allocation mechanism of funding to schools from input to per capita financing (but later, commitment to its implementation wavered). The new Social Assistance Law was eventually adopted by parliament in March 2011. However, it had been expected earlier with the effect that the design of the management information system for social protection had to move ahead in advance of and then later in parallel with the passage of the law, and as a result the system has to be retrofitted to incorporate elements introduced. Similarly, the establishment of the Commission for the Integrated Health Information System in early 2013 affected the implementation of this activity when the 8      Commission suspended activities related to the system’s development, although this has now been resolved. The adoption of a new Law on Public Procurement in late 2012 did not affect procurement procedures, per se, but did affect the processes of procurement planning in the ministries (procurement plans became integral parts of ministries’ procurement and financial plans, reducing flexibility). 24. Integrated Assistance. The Project was implemented within a framework of assistance for each of the sectors that included investment and policy lending, and technical assistance. These included the Serbia Health Project (and Additional Financing) that closed on March 31, 2012, a Programmatic Public Expenditure Development Policy Program (PEDPL), and a Second Serbia Health Project approved in February 2014. At the policy level, the PEDPL series (planned as three operations but only two went forward), aimed at, inter alia, reducing the size of the large public sector, and increasing the efficiency of expenditures in health and education spending (in part through introducing a productivity factor into the calculation of the wages of primary health professionals and per-pupil financing in education), while mitigating the social impact of the crisis and expanding coverage of social assistance programs (in part by supporting a adjustments to social assistance benefits through amendment of the Social Welfare Law in 2011). As a result, the Project both provided more focused technical assistance to policies included in the PEDPL, and benefited from the policies that the PEDPL supported, especially in health. Likewise, in health, the Project gave continuity to reforms initiated under the Serbia Health Project Additional Financing, by providing a “bridge” for continued implementation of ongoing activities under the follow-on project that is now, in turn, providing continuity and further reforms in support of the DILS Project’s objectives. In education, an Education Sector Technical Assistance (2012) complemented the Project’s activities. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 25. Design. Monitoring and evaluation was to be carried out by ministries and LSGs through regular data collection instruments such as household surveys, and existing administrative monitoring systems, such as the education information system, the health information management system, etc. Progress would be monitored according to the PAD’s outcome and intermediate outcome indicators. Design also included funding for carrying out impact evaluations of the phased implementation of decentralization (particularly the roll-out of new financing formulae and assumption of responsibilities for management of schools and PHCs), and of innovations in service delivery to vulnerable groups, particularly to disabled people. 26. There were several issues with the design of the Project’s Results Framework (RF) that in part paralleled the issues with the Project’s design (Section 2.1). Specifically, for outcome indicators: (a) all lacked baseline and target values; (b) several referred to intermediate outcomes (i.e., outputs) rather than outcomes or were difficult to measure; (c) some referred to two or three sectors (e.g., education, health, social protection), and there was insufficient clarity regarding which sector had responsibility to report on a particular indicator and how the achievement of the indicator would be measured; (d) several were not fully aligned with the PDO (e.g., those relating to Roma inclusion, while inclusion was not an objective) and (e) there were inconsistencies in the wording of several between the PAD and Supplemental Letter No. 2 to the Loan Agreement. Shortcomings with the original intermediate outcome indicators were similar to those of PDO indicators and some corresponded only loosely with PDO indicators. 9      27. Implementation and Utilization. At the MTR it became clear that the RF required amendment. Agreements during the MTR led to a revision to the RF in the June 2012 restructuring (Section 1.3). The restructuring rectified shortcomings in the RF, and clarified processes and responsibilities for data collection and reporting. It assigned indicators to each ministry, formally defined baseline and target values for outcome indicators and revised them to provide a more adequate measurement framework for the efficiency, access/equity and quality of the delivery of health, education and social protection services at the local level, and revised the intermediate output indicators. A summary of the changes to the RF, together with a justification for the revision to indicators is provided in Annex 10. Monitoring of project outcomes and intermediate outcomes were since carried out by sector, by the responsible State ministry, on the basis of their own systems and of information submitted by the LSGs. The RF was further revised in the October 2013 restructuring (Section 1.3 and Annex 11). Consultants carried out two evaluations, of the Trainings and Grant Programs for Inclusive Education and of the PHC grants. The findings of these evaluations are incorporated in this ICR, and described in Annex 2. 2.4 Safeguard and Fiduciary Compliance 28. Safeguards. The Project triggered the Environmental Assessment (OP/BP4.01) and Physical Cultural Resources (OP/BP4.11) safeguards, and was classified as an environmental category B. As some of the grant sub-projects would support building rehabilitation works and the location of these works was not known during preparation, an Environmental Management Framework (EMF) was prepared. The EMF required preparation of environmental management plans to address risks associated with air quality, noise, water quality, solid waste disposal, asbestos, toxic and hazardous wastes and medical wastes, in addition to risks to cultural heritage. The implementation support team included routinely a Safeguard Specialist to review compliance, and capacity and institutional arrangements, and generally found these satisfactory. At the team’s recommendation, a qualified safeguards specialist was hired to advise the MOE and MOH, as well as grant sub-project implementing agencies on the environmental screening process and assignment of sub-project environmental categories. Although no major issues surfaced with safeguard compliance, the, preparation of a final Environmental Compliance Report by the MOE (on grant sub-projects) was substantially delayed, and submitted long after the MOE’s activities had been concluded in December 2013. The MOE’s Environmental Specialist was no longer available to provide the individual site supervision reports that had been prepared, with the consequence that this report was not fully documented, and did not reflect the standard of good practice in supervising civil works under the Project. 29. Financial Management and Procurement. The Project’s financial management and procurement functions were centralized in the FSU, supported by the three PATs in MOE, MOH and MOLEVSP that would provide technical inputs. The FSU was to consolidate procurement plans, project accounts and disbursements, the project audit, and forward the require documentation to the World Bank. Both financial management and procurement were supervised routinely, with detailed assessments carried out during the MTR, which found that fiduciary responsibilities were being carried out satisfactorily by the FSU with inputs from the PATs. Quarterly financial reports (IFRs) were submitted to the Bank within due dates and assessed to be reliable. Transactions review and walk-through test of internal controls were conducted and no irregularities identified in the process. Audit reports were received in a timely fashion and auditors issued clean opinion on project financial statements. 10      30. Several issues stand out with respect to financial management (staffing, budgeting, accounting, internal controls, funds flow, financial reporting and external auditing) and disbursements. Towards the end of implementation of its activities, the MOE had accumulated large payment arrears that had been pending for well over two years, primarily due to weak oversight by the MOESTD. For example, there were disagreements about the quality of the deliverables, apprehension to make payments for services rendered under previous administrations, and lost institutional memory due to frequent MOESTD staff turnover. In addition, disbursement estimates were revised formally on several occasions to reflect delays in effectiveness and implementation, and especially the Project’s structure and readiness that required completion of several technical assistance and design activities for disbursements against large value activities (IT equipment, grants) to begin (Section 2.1). 31. Procurement was on occasion rated Moderately Satisfactory in ISRs due mostly to two issues: (a) delays in implementation of large activities, (b) the complexity and time required for procurement of IT hardware and systems, and (c) frequent communication problems between the FSU and the PATs, which prevented the efficient updating of the procurement plans. IT systems procurement presented challenges, as follows: (a) for MOLEVSP one large contract was envisaged, but this was eventually broken into two contracts; and (b) for MOH, the most advantageous bid had to be rejected because on member of the consortium with the lowest bid was a state company. The World Bank implementation support team recommended repeatedly that the FSU update the procurement plan more frequently, with inputs on changes in dates and activities from each PAT as soon as these took place, to take advantage of the procurement plan as a tool for managing, planning, and monitoring project implementation activities. 2.5 Post-completion Operation/Next Phase 32. There is no follow-on operation, per se. Additional support for health reform is being provided under the ongoing Second Serbia Health Project, approved in early 2014, that will continue to improve the efficiency and quality of the public health system through strengthening of health financing, purchasing, and maintenance systems and quality improvement systems and management of selected priority non-communicable diseases. It will provide continuity to important mechanisms initiated under the Project, including, improvements to further primary health care financing, to further instruments for quality improvement systems. The EU is financing hardware for the MOE’s information system. As of now, the remaining activities, including operations and maintenance, have been absorbed within the responsibilities of the relevant State ministries, and of now strengthened LSGs in accordance with the Law on Local Self Government Finance. 33. The Government of Serbia and the World Bank have identified skills development and inclusion of vulnerable groups as strategic priorities for the country. A discussion between the World Bank and the Government of Serbia has been launched in recent months on a proposed Early Childhood Development (ECD) Project. Such a project would aim at benefitting particularly children from poor and minority backgrounds and providing them with a critically needed head- start in terms of educational inclusion and skills acquisition. The project would support quality early learning in connection with other ECD measures for different age-groups: pre-school children (3 – 5.5 years); younger children (0 – 3 years) and their families, but also strengthen the transition into primary education (age 5.5/6.5 plus) with a focus on inclusive education and learning. Other aspects of the project would include expansion of access to ECD, especially for 11      children from lower socio-economic and vulnerable backgrounds, revision of ECD financing and more efficient use of the ECD and primary school network. 34. The World Bank has recently completed a Western Balkans Investment Fund (WBIF) Inclusive Education Technical Assistance in Serbia. Within this context the WBIF was used for financing analytical work to provide a stock-taking of the use of already established measures of inclusive education, the development of a roadmap institutionalization the Monitoring Framework for Inclusive Education in Serbia and the training concept for monitoring and evaluation of Inclusive Education. This work also supported capacity building activities that include training, learning and exchange of experience on Impact Evaluations. Regional consultations and a national event on Monitoring and Evaluation of Inclusive Education were further activities that were also used for dissemination of results. 35. A Jobs and Competitiveness Project was recently approved by the Board (September 2015) and aims to improve effectiveness and coordination of selected public programs to alleviate constraints to competitiveness and jobs. The project consist of four components, one of which related to labor. Labor-related component aims at: (a) Improving the effectiveness of labor intermediation services and active labor market programs; and (b) Facilitate social assistance beneficiaries’ transitioning into formal jobs. 36. At the Government’s request the World Bank is currently conducting a functional review of service delivery in the three sectors (health, education, social protection and labor). For the health and education sectors, the functional review consists of an assessment of service delivery mechanisms, efficiency and rightsizing and is focused on primary and secondary health care workforce and pre-university education and of the school network and teaching force, respectively. For social protection, the purpose of pension portion of the functional review is to improve service delivery by the Pension and Disability Fund of Serbia through improved business processes and secondarily and over the long-term, to improve overall Pension and Disability Fund staffing efficiency. The social assistance segment of the functional review aims at improving service delivery by consolidating program design and oversight and better integrating social assistance and social insurance. Lastly, the review covers the labor part with a view to improve service delivery of the National Employment Service. 37. At the Government’s request, the World Bank is also undertaking analysis of pharmaceutical policies in the health sector with a particular focus on increasing efficiency of procurement of drugs and medical devices. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation Relevance is rated Substantial. 38. Relevance of Objectives. The Project’s objective was relevant at the time it was approved, and continues to be relevant. Hence, relevance of objectives is Substantial. The Project’s objective is all the more relevant following the 2008 global economic and financial crisis, which resulted in a real decline of 3.5 percent in Serbia’s GDP 2009, requiring determined fiscal adjustment and reversing previous impressive declines in poverty starting in the early 2000s. Poverty had fallen from 13.4 percent in 2002 to 6.1 percent in 2008, but rebounded to 6.9 percent in 2009 and further 12      to 9.2 percent in 2010, according to the Household Budget Survey, placing substantial stress on Serbia’s social protection system and exacerbating long standing challenges for its most vulnerable populations. Despite improvements, Serbia still faces challenges in educating its citizens, especially when compared to its neighbors. Not enough of its young people are enrolling in school, and too many drop out before finishing. There are also issues of access to education for more vulnerable populations. Health outcomes have improved and more services are delivered at lower cost. Still, an aging population, the introduction of new, expensive pharmaceuticals and the development of new technologies are exacerbating the fiscal pressures. The increase in unemployment and poverty has reduced the HIF’s revenue base and increased the pool of vulnerable groups who must be subsidized from the general budget. As a result, the GoS is seeking ways of using resources more efficiently. The Project’s objective is fully consistent with the second pillar of the Bank’s FY12-15 Country Partnership Strategy for the Republic of Serbia: Improved Efficiency and Outcomes in Social Spending, and with the respective CPS Outcome (Strengthened fiscal performance with sustained improvement in human development outcomes through more efficient, effective social spending). 39. Relevance of Design and Implementation. While supporting priority activities in three sectors, the Project’s design was relevant when prepared, but lost relevance at the beginning of the implementation period. It was designed as an integrated, cross-sectoral operation that was to be implemented with strong coordination at the State level that promoted the same cross-sectoral coordination at the level of LSGs. The Project Policy Board was not functional, and the MPALSG’s involvement in implementation was minimal. Realigning its components and RF to this reality during implementation, and continuing to support, on a sector-by-sector basis, priority activities that would individually contribute to its objectives, with coordination only at the level of LSG, have to a certain extent brought its relevance back during implementation. Relevance of design and implementation are rated modest and substantial, respectively. 3.2 Achievement of Project Development Objectives 40. Achievement of Project Development Objectives is rated Substantial. The Project was implemented in an environment of varying degrees of decentralization of service delivery within each of the sectors, and widely varying LSG capacity to manage these increased responsibilities (Section 1.1). Nevertheless, the Project made substantial progress towards the achievement of its PDO, as measured by its Key Outcome Indicators, although progress varied by sector, as described below by sub-objective, i.e., efficiency, equity and quality of local service delivery; outputs financed that led to their achievement are presented in Annex 2. Improve the efficiency of local delivery of health, education and social protection services 41. Progress towards this sub-objective is rated substantial. The Project had a decisive impact on improving the efficiency of local delivery of health services, but was less successful in establishing the per capita financing mechanism in education as originally envisaged. By reforming the allocation of funding under the Disability Fund, it also had an impact on social protection (paragraph 46). The greatest impact was on the financing of primary health care, which is now the only sector in Serbia financed on basis of performance, i.e., productivity based pay. The capitation formula developed by the MOH and the National Health Insurance Fund under the EU- funded Support for the Implementation of Capitation in Primary Health Care in Serbia came into force in October 2012 with the approval of the Law on Salaries of Public Servants and corresponding by-laws defining performance payments in primary health. The Project provided 13      training for all employees in all LSGs and PHCs participating in the Project (500) and also others that were not participating to familiarize the participants with the capitation formula, and highlight the possibility that the formula offers in terms of improving the operation and efficiency of PHCs so that the switch to the new system could proceed smoothly. When introduced, the capitation formula allocated two percent of the salaries of primary health care providers based on performance; this proportion has been increased gradually to eight percent in 2015. Several other activities contributed to increase the efficiency of primary health care service delivery, including: (a) the presentation to staff of 27 PHCs of the step-down method for analyzing the costs and activities in PHCs, to improve administration and spending efficiency; (b) an analysis of PHC financing at the LSG level, including 33 municipalities that participated under the Project, including a review of local legislation, aimed at improving the allocation framework for equalization funds; (c) assessing the energy efficiency of 33 PHC buildings, in order to improve the structures to become more efficient; (d) support to the HIF to build capacity for the introduction of diagnosis related groups; (e) the analysis of the activities and financing of National Reference Laboratories (NRL), their equipment and staffing, with a view to improve their services and efficiency; and (f) capacity building for staff of PHCs on EU health policy and preparation of project proposals for EU financing. 42. Outcomes aimed at increasing the efficiency of education financing were developed, but not implemented. The Project aimed to develop and pilot central and local formulae that would finance pre-school and primary education on a per-pupil basis providing an incentive for LSGs and schools to find an optimal allocation to class sizes and the type and number of facilities needed to meet service needs. Progress was made on developing the central formula, and more limited progress on the development of local formulae. An external assessment confirmed that local formulae might not be necessary given Serbia’s geographic size and resource allocation modalities. Nevertheless, the GoS is considering the piloting and implementation of the central per-pupil funding as part of its broader efforts to “right size” government expenditures. Improve access and equity of local delivery of health, education and social protection services 43. Progress towards this sub-objective is rated substantial. The Project had a significant impact in improving access and equity of local delivery of health, education and social protection services. In health, the Project implemented an effective program of recruiting and financing of Roma health mediators to work in PHCs, and training of 75 mediators on public health and communication skills, hygiene and prevention of communicable diseases, vaccination, health lifestyles, etc., that reached a total of 50,754 children, 46,453 women and 43,201 men (total 140,408). This program resulted in the vaccination of 30,018 children, and in providing for Roma: (a) 2,340 personal documents and health cards; (b) medical examinations to 650 pregnant women and new mothers; (c) 1,496 screening examinations to women; (d) 1,144 mammograms; (e) 11,371 women selecting their gynecologists; (f) 2,998 Roma selecting their primary care providers; (g) health education work for 3,529 persons; (h) 58,961 visits provided and (i) 260 children enrolled in school. Grants aimed at increasing access to healthcare for vulnerable population groups (Annex 2) were equally effective reaching 44,250 persons, including persons over 65, youth, persons with disabilities and Roma. Unofficial feedback highlights an increase in the number of newly diagnosed cardiovascular diseases, elevated blood sugar, elevated triglycerides and cholesterol, and addition to other non-communicable diseases, which should lead to improved health outcomes. Palliative care provided to the elderly undoubtedly improved the quality of life of the terminally ill and of those that depend on care and assistance. Single elderly were provided with access to nurses, in cooperation with CSWs, or with volunteers from NGOs. Specialist referrals, 14      transportation, guidance on obtaining health insurance where warranted, were additional outputs that should improve health outcomes of vulnerable groups. The Project trained 2,000 health professionals on the needs of vulnerable groups in 42 PHCs. 44. The MOE was equally effective in improving access and equity in pre-school and primary education, through sub-projects awarded under two grant programs: Strengthening Schools for Inclusive Education and Education Inclusion of Roma. The inclusive education program, intended to improve the quality of education for disadvantaged populations (violence at schools, Roma, students with special needs, rural schools and students requiring motivation). Details on these grants, criteria for their award and implementation arrangements are provided in Annex 2. In addition, four thematic trainings were directed at inclusive teams in 313 schools (Annex 2). The External Evaluation concluded that “Based on the results and opinions of the different stakeholders, it can be concluded that the DILS school grants program has achieved its goal. The results of the evaluation revealed that schools that benefited from grants achieved “remarkable progress” in relation to the control group. As a result, the share of children from vulnerable groups in project schools increased from 3.2 percent (presumably in 2009, according to the restructuring paper from June 2012) to 6.6 percent when the education activities closed in December 2013. The 2010/11 data indicate that the share of children from vulnerable groups in project schools reached a high of 8.44 percent before the decline, which reflects the precariousness of this indicator. It should also be noted that project schools had nearly 60 percent more students from vulnerable groups than non-project schools (4.11 percent). With respect to training, the evaluation found that the greatest effect or range of impact was attributed to the module on strategies and methods of adapting instruction for children with disabilities and gifted children, especially in increasing the sensitivity of teachers of students from vulnerable groups. 45. Sub-project grants awarded under the Education Inclusion of Roma program aimed to contribute to the creation and implementation of the Roma integration policy at the municipal level. Details on these grants, criteria for their award and implementation arrangements are provided in Annex 2. The External Evaluation found that this program was equally successful, and met its goals by contributing to better cooperation of different agencies at the local level in providing services and support to children from deprived groups, uniting various stakeholders, and enabling cooperation. The Evaluation’s found that stakeholders reported significantly reduced absenteeism and increased school achievements of students from deprived communities, although drop-out rates and primary education coverage continue to warrant attention. 46. For Social Protection, MOLEVSP satisfactorily adjusted the Disability Fund’s financing modality to one where project funding for addressing the needs of persons with disabilities are allocated under a competitive mechanism based on the quality and relevance of the proposals submitted, thereby producing the greatest results for its target population. Based on an iterative four stage process of developing a Grant Operations Manual, awarding sub-project grants, revising the Manual to incorporate implementation lessons, awarding another round of grants, etc., a final Grant Manual incorporates improved formats for applying for grants and templates for reporting on expenditures and activities implemented, and improved scales for assessing project proposals, which are transparent and public. The Manual includes clearly defined modalities of project financing, conditions, criteria and procedures for awarding grants, for monitoring funds usage, etc. The Disability Fund awarded 92 sub-project grants to NGOs, eight of which were implemented in partnership with local stakeholders (Annex 2). 15      Improve quality of local delivery of health, education and social protection services 47. Progress towards this sub-objective is rated substantial The Project also had an influential impact on improving the quality of local delivery of health, education and social protection services. In health, the Project financed activities at the State level, as well as activities with LSGs and project PHCs, especially, three types of sub-project grants aimed at helping the PHCs accomplish the Project’s objectives: (a) PHC Accreditation Grants; (b) Quality Improvements through Investment Grants; and (c) Grants for Introducing Information System Software. 48. A total of 51 percent of Serbia’s PHCs have completed the quality accreditation process, and 39.62 percent of all PHCs have received at least a 3-year certification. PHC Accreditation Grants (Annex 2) were awarded to the 42 PHCs participating in the Project, and an additional 20 PHCs based on a request by the MOH. Accreditation standards for PHCs with the process defined in scope, form and duration by the Agency for Accreditation of Health Care Institutions in Serbia, include: (a) standards for patient care in general practice, gynecology, pediatrics, specialist- consultative services and home care; (b) standards of clinical support: pharmaceutical services, laboratory diagnostics and diagnostic radiology; and (c) non-clinical standards for the environment, human resources, information management, governance and management. The External Evaluation confirms that 83 PHCs, in three cycles, implemented the accreditation process, as well as the Institute for the Health Care of students in Belgrade. An additional 33 institutions have begun the process (but without grant funding). The MOH remarked that accreditation has produced the following benefits: “the development of multidisciplinary teams, review of the institutions’ operational policies, improving data systems, generating local and national prestige, and improved networking between primary health centers in exchange of good practices.” The Evaluation found that PHC institutions recognize accreditation as one of the most important external mechanisms for improving the quality of health care. 49. Quality improvements through investment grants (Annex 2) were supported for 43 PHCs participating in the Project to support local cooperation between healthcare institutions and LSGs, and to serve the practical integration of all activities in the field of quality improvement that were implemented at the local level, ranging from strategic planning, through recommendations made in the process of accreditation to the development of an integrated quality improvement plan and its implementation. The MOH has found that for the majority of the PHCs this was an opportunity to understand how investments can be used for quality improvements, as they were able to analyze their needs, prioritize, and compete for funding, explaining the rationale with evidence to support their investment proposals. The External Evaluation reports that PHCs in general found that the investments the grants supported improved the quality and efficiency with which services are delivered, increased the interest of the respective LSGs in healthcare issues, and enhanced public trust in the system. All have gained competence in how to prioritize investments within constrained resources, how to prepare investment proposals and how to monitor and evaluate progress. 50. Finally, grants for introducing software were provided to all (158) PHCs, as well as to 3 health institutions for healthcare of students and the Republican Gerontology Institute (Annex 2). At project completion, 95.7 percent of Serbia’s PHCs had a fully operational health management information system platform. The External Evaluation highlights several positive benefits and impacts of these grants that are described in Annex 2. 51. Further, the Project achieved the target of 50 percent of PHCs having adopted clinical pathways. Under the Project, a team of regional coordinators was established and trained, seminars 16      with top management of hospitals and PHCs, clinical pathway coordinators from healthcare institutions and the MOH conducted, and: (a) developed methodologies for developing clinical pathways according to the guidelines of the European Pathway Association; (b) carried out a pilot project in 50 PHCs with the specific aim of developing methodology of development and implementation of six clinical pathways in 18 hospitals and 8 clinical pathways in 44 PHCs, and (c) created the condition for adoption of relevant legislation. 52. In education, the Project also achieved impressive results. One hundred schools participated in the Program for International Student Assessment (PISA) 2012, and in the Trends in International Mathematics and Science Study (TIMSS) 2011. The Project provided consultant services for: (a) enhancement of education policy planning and coordination; (b) developing training packages and manuals, and training to ensure compliance with the international agreed target population definitions and sampling procedures and effective administration and management of the assessments; (c) preparation activities for participation in the PISA 2012 and TIMSS 2011, and (d) reporting on results from participation in PISA 2009. The Project also provided technical assistance and training for the implementation of standards for the end of compulsory education (8th grade) and related capacity building. While it is difficult to link project investments with quality improvement outcomes in education, the student assessment structure was greatly enhanced. PISA is conducted every three years and Serbia participated in PISA 2015, which will allow for longitudinal comparison of learning outcomes of 15 year-olds. 53. In Social Protection, the centralized management information system for social protection institutions has been developed, equipped with infrastructure and software in all 265 CSWs, and 600 staff in all of the CSWs have been trained in its use. However, due to delays in its development and implementation, in part due to delays in approval of the Social Assistance Law, and later in the need to adjust the system to new requirements introduced in the approved law, the system is not yet operational. The Ministry is currently working on the revision of the Law on social protection that would be adopted soon. Once the Law is adopted the Ministry will work on software modules revision to reflect the changes in the Law. After modules are prepared, revisions to the MIS system will be operationalized. The system provides a single database that will contain the records of all approximately 700,000 social protection beneficiaries, with linkages between 256 CSWs and the MOLEVSP. It is expected that when operational, the system will reduce the processing time between the entries of a case to its resolution from 60 to 30 days. The system’s modules include: (a) basic records in the CSW, providing for their more efficient operation; (b) management of basic records, providing for case management and record keeping; (c) financial benefits, providing for more efficient processing of social assistance and benefits; (d) custody, providing for more efficient processing of custody cases for children and adults; (e) adoption, providing for efficient evaluation and processing of adoption cases, both for children and prospective parents; (f) exercise of parental rights, providing for more efficient monitoring and supervision over the exercise of parental rights; and (g) participation in court proceedings, providing for more efficient participation and monitoring. 3.3 Efficiency 54. Efficiency is rated Modest. The PAD did not provide a full economic analysis, but provided a review of the expected impacts on service delivery, fiscal sustainability using evidence from literature. Although the Project generated a number of direct and indirect multiplier impacts, due to data limitations the benefits could not be quantified and hence a traditional economic analysis has not been attempted. The Project contributed to improving overall efficiency in the 17      use of resources in health, education and social protection through implementation of a number of activities. For health, these include: (a) introduction of per capita financing of primary health care; (b) strengthening of primary health care and the capacity of LSGs to deliver it, through capacity building, efforts to reach out to unattended groups, clinical pathways among others, which should reduce higher cost services such as specialist referrals, hospital admissions, and diagnostic and therapeutic procedures; (c) health management information system that will lead to better coordination, record-keeping, and as a result, lower cost and more quality driven care; and (d) consequently, a better health status of the population, with the benefits that brings to society. For education, these include: (a) efforts to improve access and equity, inter alia, through addressing the needs of vulnerable children, inclusive education, in a cost effective manner, and (b) studies on international student assessment that should, in the longer term, lead to improvements in quality and cost effectiveness of interventions. In social protection, these include: (a) a new, more effective, model for allocating resources for PWD, that will ensure that projects and programs financed are efficient and effective in meeting the needs of the groups for which they were intended; and (b) a management information system that will centralize data, reduce transaction processing times, and ensure a better link between beneficiaries of cash benefits and relevant social services offered by other social protection institutions. Even though the Project has greatly contributed towards setting the foundation for improved efficiency in all three sectors, the following next steps could have led to greater improvement in that regard: in health: obsolete Law on Electronic Medical Records (EMRs) still prevents the sole use of EMRs, thus leading to duplication in medical record keeping (electronic and paper-based); in education: although studies on international student assessment have been completed, measures towards improvement of quality and cost-effectiveness are yet to be launched; in social protection: the management information system has been installed and piloted, but is yet to be fully operational nationwide. 55. Despite problems with its original design, when restructured, the Project was well implemented by PATs in each of the MOE, MOH and MOLEVSP, supported by a FSU that supported all three. This was especially noteworthy when considering that the Project also supported all 158 PHCs, 559 preschools and primary schools, 14,000 teachers and school administrators, 130 NGOs, and 142 CSWs. Finally, loan resources were allocated efficiently, to the sector that had the most capacity to implement investments effectively. The final allocation of resources was as follows: MOH, EUR 16.9 million; MOE, EUR 9.1 million and MOLEVSP, EUR 5.8 million. Pragmatically, the Project did not finance hardware for the education management information system, since financing for this was expected to become available from the EU. So far, the tender has been prepared but not yet contracted. 3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 56. The Project’s Overall Outcome Rating is assessed as Moderately Satisfactory. This was undoubtedly a complex project, whose original design proved difficult to implement. Through the October 2012 restructuring, it maintained its relevance, which has become all the more important given fiscal constraints that the GoS is facing, and the need to do more with less. Its outcome— efficacy—especially in view of the multitude of positive results and evaluations that have been reported, is impressive, despite some shortcomings, especially in education. The Project not only supported actions aimed at promoting efficiency in service delivery, but was implemented efficiently, especially given its complexity across sectors and levels of government, However, critical steps towards further improvement in efficiency in all three sectors remain to be addressed. 18      3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 57. The Project impacted disadvantaged groups that have traditionally not been provided access to health and education services, and that are associated with higher levels of poverty, including the disabled, Roma, youth, especially in rural, dispersed areas of the country. These impacts are detailed in Section 3.2, especially in the objective of improving access and equity. (b) Institutional Change/Strengthening 58. The Project supported several training, technical assistance and analytical activities and investments aimed at strengthening institutional capacity, most notably the design and implementation of information systems, but also through the provision of grant schemes in working across sectors, with LSGs coordinating activities implemented by PHCs, CSWs, schools, together with NGOs to provide higher quality inclusive services. But it also supported the implementation of several new policies such as the performance-based allocation of resources in primary health and, the reform of the Disability Fund to allocate resources based on performance that have now been institutionalized. These are described in detail in Section 3.2. (c) Other Unintended Outcomes and Impacts (positive or negative) 59. Not applicable. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 60. Not applicable. 4. Assessment of Risk to Development Outcome Rating: Moderate 61. The Risk to Development Outcome is assessed as Moderate. The main risk to sustainability would be the fiscal constraints that the GoS is facing at present. However, several factors bode well for sustainability, in that the activities that were financed either served to promote efficiency, and hence strengthened the GoS’ ability to deliver more and better quality services at lower cost, or proved that it is possible to deliver inclusive services that yield results with small financial commitment. The integrated health information system has contributed significantly to the efficiency, transparency and accountability of the delivery of primary health care by avoiding unnecessary diagnostic procedures, specialist referrals and hospital admissions, all while delivery higher quality care. The information system for the MOLEVSP, when fully functional at the level of all CSWs, will allow higher quality services, with shorter response times, and coordination among multiple levels of government. In education, the network established around inclusive education brought together trained school officials and teachers to provide equitable access to children with learning difficulties, disabilities, and from socio-economically disadvantaged groups. With this, it developed improved vertical coordination (among different levels of schooling) and horizontal coordination (among different schools at the same level). The Disability Fund, now grounded with new operating and results-based procedures for resource allocation, 19      helped identify those services that have the greatest impact on the lives of the disabled. All of these positive experiences should weigh heavily in resource allocation decisions at the national level, even in a resource-constrained environment. Finally, the Laws of Self Government and of Local Government Finance have broadened the scope of public revenues that belong to LSGs, giving them not only greater responsibilities in terms of service delivery, but also greater decision making authority on disposable resource allocation. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Unsatisfactory 62. Bank Performance in Ensuring Quality at Entry is rated Moderately Unsatisfactory. The preparation team helped prepare a conceptually strong project that responded to the GoS’ request for assistance to its decentralization of service delivery efforts. Nevertheless, the team underestimated severely the implementation risks of such a complex project, especially since the timing of its approval by the World Bank’s Board—two months before elections--was not optimal, the Project required passage of legislation to achieve its objectives and the capacity to implement and monitor progress on a cross-sectoral basis at several levels of government (and especially at the level of LSG) simply did not exist. It was the Project, through the activities it supported that would eventually put in place the capacity to work cross-sectorally at the various levels of government. Further, the M&E mechanisms for the Project as set up at the onset were very weak. PDO was overly complex aiming to address equity, efficiency, access and quality, and not equally adaptable with all the sectors. PDO and corresponding indicators were not fully aligned to appropriately reflect Project achievements and the Results Framework was rather weak with a number of indicators lacking baselines and proper measurement mechanisms. In addition, the indicators were not fully aligned with the PDO. (b) Quality of Supervision Rating: Moderately Satisfactory 63. Quality of Supervision is rated Moderately Satisfactory. Initially, the World Bank’s implementation support team aimed to foster inter-sectoral cooperation, especially at the State level, in order to jump-start implementation of the Project as designed. When it became clear that the Project could not be implemented as designed, the team worked closely with the FSU and the PATs to realign the Project’s description and Results Framework, within the context of its PDOs, to respond to the GoS’ priorities for decentralization in three sectors, while at the same time retaining, where feasible, the goal of promoting inter-sectoral coordination, especially at the level of LSG. But the Mid-Term Review took place late in the project implementation and the subsequent project restructuring took too long to complete, the issues that the team had identified early on (e.g., slow disbursements) were not the correct ones so that the Quality of Supervision, especially in the Project’s initial years of implementation was only Moderately Satisfactory. Implementation support became a challenge, especially since the Project in effect required strong inputs from specialists in three sectors, with knowledge of information systems design and procurement, and with oversight of activities carried out by LSGs across Serbia. Task management changed six times, as did team members, but overall strong implementation support was provided throughout. Quality of Supervision improved significantly after the project restructuring in June 20      2012, and it was through the strong implementation support provided thereafter, that the Project was able to accomplish important results. Towards the final years of implementation the team recommended correctly that upon project completion, the closing date be extended for MOH activities only, since not only were these showing the most promising results, but also, continued support under the Project would provide continuity in World Bank health sector assistance until the Second Health Project was approved. Coordination of this assistance was achieved through responsibility of task management for implementation support assigned to the same Task Team Leader. Finally, the team quickly identified health sector activities that would be required to help the GoS respond to the impact of flooding in 2014 on its population, and restructured the Project to provide this needed assistance. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 64. Overall Bank Performance is rated Moderately Satisfactory. Despite issues with the its design, the World Bank’s implementation support teams through intensive and proactive assistance, was able to address those issues satisfactorily and realign the Project’s description and monitoring framework, intensifying support in those areas where results were more promising, responding to the GoS’ priorities, and in this manner overcome the original design issues. Nevertheless, issues with the Project’s original design required strong efforts on the part of the World Bank’s implementation support team in order to keep implementation on track. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Satisfactory 65. Government Performance is rated Moderately Satisfactory. The Project was approved and implemented during a difficult period, especially in view of the economic challenges following the 2008-09 financial crisis. Despite the tight fiscal space, however, the GoS has largely pushed ahead with reforms with a view to international integration, especially EU membership for which Serbia has been recommended “candidate” status by the European Commission. The Project’s cross-sectoral implementation model that was to be supported by the Project Policy Board did not function, as had been expected. Nevertheless, the GoS remained committed to the Project’s objectives, and provided the necessary resources, staffing and support to its restructuring despite changes in government, in general, and in individual ministries, in particular, that led to wavering commitment to specific policy actions on occasion. (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory 66. Implementing Agencies Performance is rated Moderately Satisfactory. The Project Implementing Agencies were the FSU, the PATs in the MOE, MOH and MOLEVSP, and the LSGs, PHCs, schools and CSOs that implemented subproject grants. Performance varied by agency, as can be expected. Despite occasional lapses due to changes in staffing, the FSU provided timely and responsive support and coordination to implementation, consolidating and coordinating with each of the PATs. The PATs in the MOH and MOLEVSP worked generally effectively, although each was affected by ministerial changes, changes in procedures, and, in the case of the MOLEVSP, delays in information system procurement due to protracted delays in approval of the 21      Social Assistance Law. Nevertheless, both of these were able to sustain implementation progress, despite these changes and unforeseen delays. The PAT in MOE faced more routine issues throughout, especially insofar as tracking invoices and arranging payments to contractors, and providing follow-up to and delivering routine project supervision reports, such as the Environmental Compliance Report on the implementation of sub-project grants. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory 67. Overall Borrower Performance is rated Moderately Satisfactory, based on similar ratings for Government and Implementing Agencies performance. 6. Lessons Learned 68. The Project offers several lessons with regard to project design and implementation. • Projects designed and implemented across multiple sectors and among multiple layers of government and implementing agencies require a strong coordinating body at the national level (in addition to coordination mechanisms at the local level). Design had envisaged a Project Policy Board with representatives of stakeholder ministries, as well as strong involvement by the MPALSG. This Board met only twice, and the PMALSG’s involvement was minimal such that during implementation, the inter-sectoral coordination required was only provided, without much technical support, at the level of individual LSGs. As a result, the Project was implemented almost as three separate projects in each of the sectors supported, with little if any coordination among the ministries at the State level. Implementation support, mission Aide Memoires, and completion reporting were likewise carried out on a sectoral basis. • PDOs for complex, multisectoral projects should be kept simple, contained and easily measurable. This is especially true for a first multisectoral project in a country, in which commitment to adopt a rather complex approach has not yet been tested. • A well formulated Results Framework, with outputs leading to intermediate outcomes, leading to outcomes, with baseline data and targets upon completion can facilitate project management and oversight. The Project’s RF did not have a clear results chain, nor baseline data or targets. Further, it defined outcome indicators that spanned sectors, which became difficult to monitor in the absence of a coordinating body at the national level. The project restructurings amended the RF to make it compatible with implementation, establish baselines and targets, but monitoring was done sectorally, with the additional effort required to compile monitoring data from three ministries. • Disbursement schedules need to take into account the nature of the activities being financed, and the estimated time needed to and cost of implementing each of the activities. The Project devoted about one third of its funding to supervision intensive and slow disbursing technical assistance that was required up-front to help define both the policy actions, and more importantly, the grant programs and information systems that were to be financed later. Implementation support identified that these issues with design were impacting the pace of disbursements, but spent an inordinate effort in accelerating disbursements when the issue was structural. • Placing a fiduciary coordinating unit in one of the implementing agencies helps to provide some coordination of at least procedural requirements, but can create some tensions among implementing agencies. The Financial Services Unit was housed in the Ministry of Health, and the other implementing agencies could voice concern about lack 22      of attention or support to their needs. It would have been preferable to have the coordinating unit in this case housed in a central ministry that was not directly an implementing agency. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 69. The Ministry of Health, Ministry of Labor, Employment, Veterans and Social policy, and Ministry of Education have provided substantial contributions to the ICR. The Ministries of Health and Education submitted their respective Completion Reports in early 2015, and the Ministry of Labor, Employment, Veterans and Social Policy submitted an unofficial version of its report. Information included in the reports have been incorporated in this ICR; the full reports are available upon request. Comments from the Borrower on the draft ICR were received by letter from the Ministry of Finance of November 4, 2015. These comments are presented in Annex 7. (b) Cofinanciers: Not applicable (c) Other partners and stakeholders: Not applicable 23      Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Actual/Latest Appraisal Estimate Percentage of Components  Estimate (USD (USD millions)  Appraisal  millions) 2 I. Making Fiscal Decentralization Work 6.45 7.00 109 II. Improving Outreach and Access through Development and Expansion of 11.52 12.20 106 Innovations in Social Service Delivery  III. Supporting a New Regulatory, Oversight and Quality Assurance Role for 10.13 19.76 195 Central Government Ministries 3  IV. Improving Capacity of LSGs as 16.07 3.20 20 Service Providers  Total Baseline Cost 44.17 42.16 95 Physical Contingencies  -- 0.00 Price Contingencies  2.11 -- Total Project Costs 0.00 -- Front-end fee IBRD  0.12 -- Total Financing Required 46.40 42.16 91 (b) Financing Appraisal Actual/Latest Type of Estimate Estimate Percentage of Source of Funds  Cofinancing (USD (USD Appraisal  millions) millions) 2/  Borrower  0.00 0.00  -- International Bank for Reconstruction 46.40 42.16  91 and Development                                                                       2  Decrease in IBRD financing reflects exchange rate fluctuation in the Euro to dollar exchange rate. In addition, US$230,000 were cancelled at the end of the grace period.  3  Education and Social Protection components closed 15 months before Health. All the unspent funds were transferred to health and used towards improving quality of health care (equipment for cancer screening, and emergency ambulatory vehicles, and response to damage caused by severe floods in 2014).  24      Annex 2: Outputs by Component Results  Outputs  Benefits Sustainability  Comments Component 1: Transform Financing Models • PHC • Health: The Project     Capitation formula providers paid focused on capacity was developed by according to building at the local MOH and the output-based level to support these Republican Health formula activities (see below) as Insurance Fund (PDO, well as to improve the under the EU- Achieved)  functioning of the funded Support for existing financial the Implementation instruments for of Capitation in improving the delivery Primary Health of health care at the Care in Serbia. primary level.  Relevant changes, which introduced that part of the salaries of health care employees be determined based on performance came into effect on 10/1/12.   • Central and • Education: Some       local per progress was made in capita funding the development of formulae in central and local per the education capita funding formulae sector piloted in the education sector (PDO, Not and a theoretical piloting Achieved)  was started, but neither formulae was piloted in municipalities  (a) Development of “funds-follow-the-user” formulae • Legislative       See above. framework Legislative allows for framework came capitation and into effect on output-based October 1, 2012  formula (IO, Achieved)  • Improved • Health: Analysis of PHC This analysis   Focus of analysis allocation financing at the LSG contributes to a was on funds framework level including 33 better allocated by LSGs for municipalities that understanding of to PHCs and equalization participated under the situation on primary health care funds Project. The analysis the ground, in general (not on developed includes a review of indicating the funding PHCs (IO, relevant legislation.  current problems received from the Achieved)  in the financing of National Health primary health Insurance Fund). care at the local The analysis showed level, developing that transfer of the argument for PHCs to LSGs was the amendment to not accompanied by the relevant increased amounts legislation. of transfers of funds 25      Results  Outputs  Benefits Sustainability  Comments from the central to LSGs.   • Central and • Education: Central per       local capita funding formula formulae was developed by an developed international consultant; (IO, • Education: Technical Substantially and analytical work (and Achieved)  training, see below) for development of local formulae through which per capita funding could be applied, including: (a) determining all relevant data-sets and their correlation to develop an electronic matrix form to be used by municipalities for data entry; (b) development of an interactive and analytical electronic data processing form in Excel, drawing on official sources of data for education; (c) data collection and analysis in 16 municipalities; (d) development of local formulae for re- distribution of funds based on per capita model, based on standardized costs per pupil; and (e) reporting and providing presentations to the MOE and MOF.   (b) Support for the design and implementation of communication campaigns and training for all sector actors at the state, LSG and service provider levels in the application of the new financing mechanisms    • Health: Training of 200       employees in 44 LSGs and 300 staff from 48 PHCs on changes in primary health care financing--in transfer of primary health to LSGs, as well as the introduction of funding through a capitation formula that is used to determine part (10%) of the salary of members of doctors’ teams which is 26      Results  Outputs  Benefits Sustainability  Comments determined based on performance;    • Health: Application of       the step-down method in analyzing costs and activities in PHCs, through presentation in 27 PHCs that provided data for the analysis   • Health: Two-day       training for 450 staff from all PHCs on the EU Health Policy and drafting projects that involve cross-border cooperation that would be financed from EU pre-accession funds.   • Health: Four sessions of       training in which all PHC staff participated on: (a) management and financing of organizations in the healthcare system; (b) management and financing as part of efforts to improve the performance of health professionals, and (c) cost management and financial reporting   • Health: Assessing the     In response to the energy efficiency of the new Law on main PHC buildings Planning and participating in the Construction and the Project, and preparing relevant Ordinance relevant reports  that provides that all new buildings and existing buildings that are under reconstruction, rehabilitation or restoration should include the energy efficiency report as part of its technical documentation.   • Health: Support to the • The new   Previous projects Department for nomenclature financed by the WB Improvement of of health (Serbia Health Financing of Health services came Project AF) had Services of the National into effect on assisted in terms of Health Insurance Fund 1/1/14, as a the introduction of and training of about first step in diagnosis related 5,500 employees in 65 introducing groups  health care facilities that diagnosis 27      Results  Outputs  Benefits Sustainability  Comments provide acute inpatient related care on the new way of groups  reporting and diagnosis related groups    • Health: an analysis of • Proposal     the activities of National made for the Reference Laboratories sustainable (NRLs); analysis of the financing of existing method of the NRLs, the financing NRLs, of the increase in status of NRLs’ the equipment, education availability of levels of NRL staff, NRL analysis of access to services, for NRL services as well as the increase reporting and in the level of information flow related training of to NRLs  staff working in the NRLs, as well as changes in reporting with regard to NRLs   • Education: Seminar with       representatives of 16 LSGs selected for the piloting of per capita funding in the education sector to present the concept and agree upon activity plan • Education: Two day training module for LSG representatives to provide instructions on statistical data gathering and input in the electronic data base, using the Excel data processing system developed (see above) • Education: Module for representatives of 16 pilot municipalities to present, review and interpret the results generated from the data gathering exercise and to determine the elements of the future local formulae  Component 2: Improve Access and Quality • 6.56 percent • Education: See below       of children from vulnerable 28      Results  Outputs  Benefits Sustainability  Comments groups in project schools (PDO, Partially Achieved)  • 30,018 Roma • Health: Recruitment and For Roma:     children financing of 15 Roma • 2,340 received health mediators to work personal vaccinations in PHCs, and training of documents through the 75 mediators public and health Roma health health and cards mediators communication skills, provided program hygiene and prevention • 650 pregnant (PDO, of communicable women and Exceeded)  diseases, vaccination, new mothers healthy lifestyles, had medical claiming rights in terms examinations of health care and • 1,496 insurance, social care, screening women’s rights, neglect examinations and abuse, human performed for trafficking, and code of 1,496 women conduct  • 41 mammogram s carried out • 11,371 Roma women chose their gynecologists • 2,998 Roma chose their doctor • Health education work carried out for 3,529 persons • 58,961 visits conducted, and • 260 children enrolled in school (a) Provision of Grants to beneficiaries to develop outreach services to excluded groups  • 560 • Education: Grants to       educational 298 schools in the institutions program Strengthening received Schools for Inclusive grants for Education for the schools development and without implementation of violence (37), school projects to inclusive achieve two of the five education inclusive education (330) and goals, plus 9 pilots (307 Roma in total) 29      Results  Outputs  Benefits Sustainability  Comments children (193) • Education: Municipal (IO, grants to 92 local Achieved)  education institutions (pre-school and primary) and 56 NGOs in 56 municipal programs to contribute to the creation and implementation of the policy of Roma integration at the municipal level   • Health: Provision of       grants to 42 PHCs that signed a contract with the MOH for participation in partnership with local governments, NGOs, Centers for Social Work or schools; grants were implemented in three cycles (2011, 2013 and 2014), and reached a coverage of 44,250 persons  (b) Reform the financing and delivery of services for disabled people (including grants)  100% Percent of • MOLEVSP: Awarded       MOLEVSP 92 grants to NGOs, in financing four phases, in allocated for accordance with the Disabled Peoples Grant Operational Organizations, Manual that defined allowing for equal modalities of project access and financing, conditions, improved criteria and procedures transparency and for awarding and based on results monitoring how funds (PDO, Achieved)  would be spent by associations focused on activities aimed at improving life and position of PWD; 3 of those grants supported parallel grants by the MOH;  (c) Expand Borrower’s ministries’ and LSG’s capacity to address the specific needs of excluded vulnerable groups  • 2,000 medical • Health: Training of       staff and health professionals and associated associates on the needs trained to of vulnerable population recognize groups in 42 PHCs needs of involved in the Projects, vulnerable as well as other PHCs groups (ILO, that expressed Achieved)  interest/need for training 30      Results  Outputs  Benefits Sustainability  Comments in: (a) introduction to geriatrics; (b) prevention of violence against the elderly; (c) improving communication with the Roma population and sensitization of health professionals; (d) implementation of the Special Healthcare Protocol to protect children from abuse and neglect; (e) youth- friendly health system; (f) partnership for health, and (g) prevention of violence against the elderly.   • Health: Training for     The adoption of the health professionals and Palliative Care associates from 42 Strategy and Action PHCs participating in Plan in 2009 created the Project, as well as an conditions for the additional 19 PHCs on development of palliative care  palliative care at the primary health care level, within the existing home care departments of 88 PHCs in Serbia.     • Health: Procurement of       70 vehicles for palliative care to improve the working conditions of medical professionals, and to provide efficient care to users of home care services    • Education: Four sets of     Coaches were thematic trainings were trained by local conducted for members consultants and of inclusive school authors of trainings, teams or staff from 313 as well as through schools (295 primary observation of the and 18 secondary) in all training  regions. These included: (a) two day training Strategies and methods of adapting instruction for children with disabilities and gifted children (53 groups of 313 schools, a total of 1,484 participants, 51 coaches); (b) two day 31      Results  Outputs  Benefits Sustainability  Comments training Planning and Individual Education Plan (53 groups of 313 schools, 1,470 participants, 36 coaches); (c) one day training Monitoring, evaluation and revision of the Individual Education Plan (30 groups from 180 schools, 699 participants, 33 coaches); and (d) two day training Motivation for learning and psychological principles of learning, to increase the generic competencies of teachers for more efficient and effective teaching, through the implementation of various learning strategies and support to students in developing self-regulatory learning strategies and motivation for learning (206 groups, 6,006 participants, 78 coaches)     • Education: Procurement       of Assistive Technologies equipment and software, and distribution to 6 schools; Procurement of 21 school buses for 21 municipalities providing better access to education for students in rural areas, in particular Roma children; Reconstruction of school toilets in 40 schools  (d) Improve knowledge and establish institutions to safeguard citizens’ rights   • Health: In November Patient rights   Relying on the 2010, a pilot program advisors in local Project’s experience Implementation of governments have under the pilot, the Patient Rights been in place GoS adopted the Protection at the Local since December, Law on Patient Self Government Level 2013, and by Rights in May 2013, December, 2014 introducing the 32      Results  Outputs  Benefits Sustainability  Comments was initiated in 8 the municipalities institution of patient municipalities. and cities in rights advisor in • Health: Supported LSGs Serbia have local government in assuming new appointed 144 units to provide responsibilities of the patient rights information and Law on Patient Rights advisors. The advice on the rights through: (a) Standing of patients and consultations with Conference of provide protection presidents of Towns and of patients’ rights municipalities, mayors, Municipalities upon the submission directors of health care and the MOH in of objections related institutions, in order to late 2014 carried to obtaining introduce and out a joint healthcare. The implement the new law evaluation of the relevant Action Plan and policy, and to Law, based on for implementation consider steps to be submissions of of the Law was taken to assume new annual adopted in August responsibilities; (b) performance 2013  seminars for patient reports by the advisers in patient rights municipalities and cities, advisors.   as well as for representatives of the local Health Councils; (c) seminars for advisers and representatives of Health Councils for the purpose of exchanging information on two- months of their work, problems and issues; and (d) supply of printed promotional materials on the new service to all health care institutions and local governments as well as the Manual for the Law’s implementation. Component 3: Improve accountability and quality • 39.62 percent • Health: Grants, carried     Accreditation of of PHCs out in four cycles, to health care received at support accreditation of institutions as a least a 3-year first 82 PHCs, and then concept was certificate of in an additional 33 introduced to the accreditation PHCs (See below)  health system (PDO, through the Health Exceeded)  Project AF, and then further developed and institutionalized under the Project (a) Development and support of training for the Borrower’s ministries to shift to a regulatory, oversight and quality assurance role for LSGs to develop competencies in the delivery of services being decentralized • 23,387         education staff have undergone 33      Results  Outputs  Benefits Sustainability  Comments development training organized by Project (IO, Exceeded)  • 150 inter- • Health: About 800 In 2015, 33 PHCs   Local Health sectoral representatives of local developed their Councils (LHCs), an committees governments and PHCs strategic plans inter-sectoral trained (IO, were trained in several with an action (health, education, Achieved)  events on strategic plan for the year, social protection, planning on how to and can, on the NGOs, institutes define their objectives, basis of clearly and departments of activities, indicators, defined strategic public health) and lines of responsibility, to priorities, apply advisory working include vulnerable for funds from body of executive population groups in local budgets. In government or the their territory as well as the future, the municipal/city to monitor progress LHCs will be the assembly, which towards achieving the implementers of deals with health in set goals and performing strategic planning.  all policies at the their evaluations.  local level, with special emphasis on the needs of vulnerable population groups. LHCs became mandatory under the Law on Patients’ Rights. By December 2014, 122 LHCs had been established (out of 145 local government units). LHCs together with patients’ rights advisors have a central role in the preparation of the local program budgets, introduced in 2015. The training responded to this need   • All: Training grants Early benefits of   The Law on focused on capacity the inter-sectoral Foundation of the building of 150 inter- commissions Education System sectoral commissions revealed: (a) an (2009) provided for established across the increased number the establishment of country; about 700 of Roma children inter-sectoral members of these enrolled in regular commissions at the commissions received schools; (b) a local level, through two day trainings; decreased number the MOH, MOE and technical assistance for of Roma children MOLEVSP. The drafting the inter- attending special commissions sectoral by-laws  education schools; include and (c) increased representative of the 34      Results  Outputs  Benefits Sustainability  Comments participation of LSGs as well as Roma parents in representatives of claiming rights for the relevant local equal access to institutions/service basic services  providers in health, education and social protection. (b) Definition of regulations and standards for local service delivery, including the specification of the new roles and responsibilities of municipal staff and service providers in the decentralized system and creation of mechanisms to foster and assure quality, including developing systems of accreditation, licensing and accountability….  • 50% of PHCs • Health: Establishing and       have adopted training a team of clinical regional coordinators, pathways (IO, conducting seminars Achieved)  with top management of hospitals and PHCs, clinical pathway coordinators from healthcare institutions, the MOH and others, study tours for directors and coordinators to: (a) develop methodologies for developing clinical pathways according to the guidelines of the European Pathway Association; (b) carry out a pilot project with the specific aim of developing a methodology of development and implementation of 6 clinical pathways in 18 hospitals and 8 clinical pathways in 44 PHCs , and (c) create conditions for adoption of relevant legislation.    • Health: Training trainers Between 5 and 8   Results of an from 56 PHCs, for each clinical pathways evaluation of the clinical pathways that have been implementation of eventually provided implemented in clinical pathways in training to over 2,500 56 PHCs   PHCs showed an health professionals in improvement in the relevant clinical quality and pathways  efficiency of professional performance: better record-keeping, better patient education, patients leave to see a specialist better prepared, better 35      Results  Outputs  Benefits Sustainability  Comments quality and more efficient preventative services provided, an increased number of preventative examinations, more efficient patient management, service being provided equally to all patients, active doctor-patient cooperation, improvement of quality indicators, reduced cost of care, more thorough examinations and records of physical examinations, monitoring of all risk factors and their elimination.   • Health: Carrying out a     This pilot was pilot project in 50 PHCs, carried out in which were involved in coordination with the development and the EU-funded implementation of Technical clinical pathways.  Assistance to Implementation of the National Screening Programme in Serbia Project • 51% of PHCs • Health: Grants, carried   Sustainability   have out in four cycles, to and completed support accreditation of functionality quality first 82 PHCs, and then of the Agency accreditation in an additional 33 for process (IO, PHCs  Accreditation Achieved)  , and accreditation became synonymous with the activities to improve health care quality and patient safety  • 100% of • Education: Consultant     Funding for schools have services for: (a) participation in undergone a enhancement of PISA 2011 was school education policy provided by the performance planning and European external coordination; (b) Investment Bank. 36      Results  Outputs  Benefits Sustainability  Comments evaluation developing training (IO, packages and manuals; The Law on Achieved)  (c) preparation activities Foundation of the for participation in Education System Program for (2009) introduced International Student the state matura, a Assessment (PISA) standardized end of 2012 International compulsory Assessment; (d) education exam.  preparation activities for participation in Trends in International Mathematics and Science Study (TIMSS) 2011 International Assessment; and (e) report on results from participation in PISA 2009 International Assessment • Education: Technical Assistance and training for the implementation of standards for the end of compulsory education (8th grade), and capacity building   (c) Support for MoE-supported Grant mechanisms aiming at promoting locally-inspired quality improvements through the financing of Grants to schools, LSGs and education service providers…..                     (d) Improvement in the use of information, statistics and ICT…..including investments needed to improve connectivity of local service providers to the internet and national data networks…. • 95.7% of • Health: Purchase and   Funding to Investments in IT PHC centers installation of IT maintain and equipment were co- have fully equipment (over 209 provide financed between operational servers, 5275 continuous the Project and the HMIS workstations, 2,650 technical Health Project AF; platform at printers, 1,312 bar code improvement the PHC level scanners and 3,250 bar s is a Issuance of an (IO, code readers), network challenge   Ordinance on the Exceeded)  infrastructure (local area Detailed Contents of networks) and Technical and implementation of Functional software (8 out of 20 Requirements for software solutions Setting Up of the received the compliance Integrated Health certificate allowing Information System them to participate in in 2009  the procurement of software) in all PHCs; the initial focus was on all 158 PHCs, and in 2011 implementation expanded also to four primary health care 37      Results  Outputs  Benefits Sustainability  Comments institutes; (of the 158 PHCs, only 42 were selected according to defined criteria and were participants of other three components, but decision was made to include all PHCs); 1,199 (out of 1,830) health outposts and remote clinics are connected to central buildings • Health: Funds for the purchase of software to implement electronic health record in the PHCs was provided through grants • Health: Additional IT/Software purchases included: (a) ICT equipment (Disaster Recovery System remote back up, back up servers, and main routers) for the Health Data Center-- responsible for maintaining the continuity of ICT activities at the national level in over 200 facilities; (b) ICT support for the early detection of breast, cervical and colon cancer; (c) Development and implementation of IT systems, purchase of equipment and development of software solutions to pilot e- Prescription (e.g., developing and issuing electronic prescriptions) in one district; (d) implementing the Picture Archiving and Communication System (PACS) and the Radiology Information System (RIS) in the Emergency Center, Clinical Center of Serbia; (e) purchase of additional IT equipment for two hospitals and 38      Results  Outputs  Benefits Sustainability  Comments clinics; and (f) purchase of software to implement RIS in 6 hospitals.  • MIS system • SP: Purchase and A single database Maintenance To avoid delays in developed for installation of IT will contain of the system, the development of Social hardware, servers, records of all both software, the Protection scanners, , network approximately hardware and process proceeded Institutions infrastructure and 700,000 social software in parallel with the (IO, Partially software for the protection needs to be approval process of Achieved)  MOLEVSP and 265 beneficiaries, with contemplated. the Law on Social Centers for Social Work linkages between Protection and (CSW); software and among 265 Migration of bylaws. This posed modules include: (a) CSWs and the data from the considerable basic records in the MOLEVSP. It is CSWs’ challenges, in terms CSW, providing for expected that databases to of maintaining their more efficient when fully the new compatibility operations; (b) operational, it will central between the management of basic cut the processing database provisions of the records, providing for time between needs to be Law and the case management and entry of a case to completed, software.  record keeping; (c) its resolution from but presents financial benefits, 60 to 30 days, challenges providing for more plus electronic due to the efficient processing of record keeping.  need to social assistance and compensate benefits; (d) custody, previous providing for more software efficient processing of owners to custody cases for ensure children and adults; (e) software adoption, providing for code. efficient evaluation and processing of adoption A backup cases, both for children plan and and prospective parents; disaster (f) exercise of parental recovery rights, providing for environment more efficient must be monitoring and developed supervision over the and put into exercise of parental place. rights; (g) participation in court proceedings, The new Law providing for more on Social efficient participation Protection and monitoring; The amended system currently more than 20 provides for administrativ administration at the e proceedings central level, and administration at the procedures, district level or and provided administration at the for 36 new level of CSW.  documents. The financial benefits module needs 39      Results  Outputs  Benefits Sustainability  Comments to be aligned with these new requirements. Modules on placement in foster homes and placement in social protection institutions also need to be completed/am ended to conform to the new Law. For the system to become fully operational, there should be a transition period of six months during which old and new systems are run in parallel. • 600 staff in • SP: Systems users in the     Some aspects of the Social CSWs, future operators system require Protection of the Customer Service updating, and Institutions and system further training will trained and administrators in the be necessary.  certified in MOLEVSP’s IT the use of the Department have been Central MIS trained, using a train- (IO, the-trainer methodology.  Achieved)    • Education: Consultants       Services to: (a) carry out the system analysis of information needs and existing IT systems in schools and other education institutions; (b) map education institutions to be included into information system development; (c) prepare a conceptual plan for information system development 40      Results  Outputs  Benefits Sustainability  Comments (including design of system architecture and implementation plans)   • Education: Consultants     The preliminary IT- Services to: (a) develop related TA and technical specifications capacity building for Education was funded by the Information System Project; the (EIS) system analysis, procurement of IT for EIS software, and for infrastructure and infrastructure; (b) software was funded establish the interactive by the European web portal; and (c) Investment Bank.  develop technical specifications for buying the equipment for the interactive web portal (e) Development of mechanisms to reinforce the Borrower’s capacity in ensuring quality improvements to service delivery….    • Health: Development of       an Action Plan and publishing of a new Ordinance on Quality Indicators that includes as one of its key indicators the Integrated Quality Improvement Plan;    • Health: Organized a       series of workshops in which managers of all health institutions in the National Healthcare Network learned about the basic principles of both quality and integrated planning in the field of quality;   • Health: Conducting       national conferences on quality (2009, 2010, 2011) through a network of institutes of public health—annual meetings at the Institute of Public Health of Serbia in which all institutes report on the results of their efforts to improve the quality of care in their districts;    • Health: Developed a       proposal of quality indicators as criteria for awarding grants in the Second Health Project and of the scope of work 41      Results  Outputs  Benefits Sustainability  Comments of the Commission for Improvement of Quality and Patient Safety;   • Health: Produced Motivated health     together with the professionals at Commission for the all levels of Development and healthcare to use Implementation of Good evidence-based Clinical Practice recommendations Guidelines a total of 31 in their everyday good practice practice, thus guidelines, and contributing to presented them in over activities to 42 workshops for improve health healthcare professionals care quality and from all health facilities patient safety    • Health: Conducted       campaigns for the promotion of national good clinical practice guidelines, including a press conference, over 50 press releases in all national media, interviews with the President of the Commission for the development and implementation of good clinical practice guidelines, leaflets and posters printed and distributed to all health facilities.    • Health: Developed the       List of Equipment of National Interest, consisting of 303 units of equipment, based on the Global Medical Devices Nomenclature, and subsequently a database of this equipment.     • Health: Conducted the Data from this     Serbia Population survey provided: Health Survey covering (a) an overview of 6,500 households, and a the results of related media campaign policies and to inform and promote programs that public response to the were implemented survey.  since the earlier 2006 survey; and (b) an identification of priority issues and, 42      Results  Outputs  Benefits Sustainability  Comments consequently, redefinition of the goals of health policy and strategy.   (f) Provision of Grants to carry out minor civil works for the renovation of PHCs and schools    • Health: Grants to 43 This type of grant   Funds were used to: PHCs for construction was intended to purchase works, such as repairs, promote practical ambulances (11 joinery replacement, integration of all PHCs); purchase control of leakages, activities in the medical equipment removal of counters in area of improving (17 PHCs); waiting rooms to quality that were implement improve implemented at construction works communications with the local level, (22 PHCs);  healthcare users, from strategic procurement of planning through equipment and transport recommendations vehicles;  obtained through the process of accreditation to the development of an integrated quality improvement plan and its implementation using the grant proceeds.   • Health: Purchase of 30 36 emergency     ambulances and medical ambulance and dental equipment vehicles and for PHCs affected by medical and floods in 2014; delivery dental equipment of workshops for for 20 PHCs psychosocial support to purchased for citizens in flooded areas, flood affected and support for areas  improvements to development counseling in PHCs  Component 4: Support for Project Implementation Provision of TA, training and goods to the FSU and the PATs, including preparation of the audit     • Operating costs, training       (fiduciary) and audits Grants financed by the DILS Project: 1. Several activities under the Project were carried out through grants delivered by the MOE, MOH and MOLEVSP to municipalities, NGOs, and PHCs. In all cases, the provision of grants followed a similar process: finalization of the respective grant manual (including criteria and procedures for accessing and implementing grants), dissemination of availability of grant funding, provision of training on preparing grant proposals, preparing grant proposals, implementing and monitoring grants. Details on the several lines of grants provided, the activities they financed and their beneficiaries are provided below, by sector. 43      Ministry of Education, Science and Technology (MOE) 2. The MOE provided grants through two programs: Strengthening Schools for Inclusive Education (together with trainings for teachers presented in the Table above), and Educational Inclusion of Roma. 3. A total of 298 schools received grants under the Strengthening Schools for Inclusive Education program for the development and implementation of school projects aimed at achieving two of five goals: (a) improving the professional competencies of employees to work with students with disabilities and learning difficulties; (b) sensitizing school stakeholders and the local community for the acceptance and support to students with disabilities and learning difficulties; (c) enhancing cooperation with parents of students with disabilities and learning difficulties; (d) involving better students with disabilities and learning difficulties in the educational process; and (e) providing access to the school premises for students with disabilities and learning difficulties. In all, about Euro 1.2 million were allocated to schools through these grants, approximately Euro 4,000 per school. These grants were intended to improve the quality of education for disadvantaged populations (violence at schools, Roma, students with special needs, rural schools and students requiring motivation). The grants financed small civil works and materials and equipment. Four thematic trainings included: (a) Strategies and methods of adapting instruction for children with disabilities and gifted children; (b) Planning and Individual Education Plan; (c) Monitoring, evaluation and revision of the Individual Education Plan; and (d) Motivation for learning and psychological principles of learning. 4. The Education Inclusion of Roma program aimed to contribute to the creation and implementation of the policy of Roma integration at the municipal level. To implement the grant program, 56 municipalities were selected on the basis of two main criteria: (a) the level of economic development of the municipality, and the less developed municipalities were given preference; and (b) the aggregate composite index based on a series of indicators derived from a group of official record indicators related to the Roma population (e.g., number) and indicators collected by the Roma Education Fund. Municipal grants were awarded to 192 pre-school and primary schools and 56 NGOs within the overall 56 municipal programs financed. Municipalities received, on average, Euro 35,000 with the objective of increasing the educational inclusion of Roma students by increasing the availability and quality of education. Municipal mentors were appointed (26) to assist the 56 municipal teams develop and implement their projects. Participating municipalities were required to: (a) provide a complete analysis of the educational needs of Roma children and a plan of community development activities to address them; (b) remove barriers to access of the right to education for Roma children, and provide full educational coverage of Roma children in pre-school and elementary education; (c) increase significantly the participation of Roma children in pre-school education; (d) improve significantly the quality of services for Roma children at the local level; (e) strengthen the responsiveness of local authorities and others responsible for promoting social integration of Roma children in the community; (f) the inclusion of any educational institution in activities related to the policy of Roma integration; and (g) create a local action plan to ensure sustainability of the activities aimed at the inclusion of Roma students. 5. The External Evaluation of the Training and Grants Programs for Inclusive Education found that these grant and training programs have achieved their goals. The external evaluation used an on-line questionnaire to collect data for each program (and training) designed on the basis of inclusiveness indicators but tailored for each group of respondents (e.g., municipal officials, school principals, teachers trained, NGO representatives, the teams for inclusive educations, school coordinators for the Project). Inclusiveness indicators included: lower dropout rates, lower rates of absenteeism, higher academic achievement, lower rate of repetition, greater coverage of pre-school and primary education, the degree of mobility to higher levels of education, the rate of segregation, students’ and parents’ satisfaction with school, intersectoral cooperation at the municipal level. The sample included a representative group of 16 schools, one from each region that received grants and a control group of 16 schools with similar geographic and demographic conditions that did not. For training, all teachers in the 16 schools in the sample were covered. 44      6. The evaluation concludes “Based on the results and opinions of the different stakeholders, it can be concluded that the DILS school grants program has achieved its goal: it has contributed to a more successful implementation of inclusive education, led to higher academic achievements of the students with disabilities and made them feel more satisfied and accepted within the school. Furthermore, it led to cultivating inclusive culture within schools to a greater extent.” Also, the results revealed that schools that benefited from grants achieved “remarkable progress” in relation to the control group. With respect to training, the evaluation found that the greatest effect or range of impact was attributed to the Inclusive Education – Strategies and methods of adapting instruction for children with disabilities and gifted children training, especially in increasing the sensitivity of teachers of students from vulnerable groups. The evaluation also found that the grants awarded under the Education Inclusion of Roma program also met their objective by contributing to better cooperation of different agencies at the local level in providing services and support to children from deprived groups, uniting various stakeholders, enabling good cooperation, etc. According to the evaluation’s findings, stakeholders reported significantly reduced absenteeism of students from deprived communities and their increased school achievements, although dropout rates and coverage of primary education continue to warrant attention. Finally, the evaluation highlighted the importance of synergy for the successful implementation of inclusive education: cooperation with NGOs that provide pedagogical assistants and field work, with municipalities that help meet basic needs of this group (meals, clothing and transportation). Ministry of Health (MOH) 7. The MOH provided grants under four lines: (a) grants for increasing access to healthcare for vulnerable population groups; (b) PHC accreditation grants; (c) quality improvements through investment grants; and (d) grants for introducing MIS software in PHCs. Progress in each of these is described separately below. 8. Grants were awarded in three cycles to LSGs for increasing access to healthcare for vulnerable population groups following a process of working with the LSGs to provide training in proposal writing, project management, and management and administration of grant funding. In the first cycle, 42 LSGs received grants to implement one-year projects. The largest share of these grants were for projects directed at addressing needs of persons over 65, mostly in rural areas (22 grants, covering 15,025 persons), followed by youth (9 projects, covering 7,390 youth), the disabled (7 grants, covering 1,920 disabled) and Roma (4 grants, covering 1,405 Roma). In the second cycle, grants were awarded to 26 LSGs that had shown the best implementation results in the first cycle (one LSG dropped out subsequently). The largest share of the second cycle grants were for projects directed at increasing access to healthcare for persons over 65 living, usually with chronic illnesses and dependent on care and assistance (14 grants), followed by increasing healthcare for the disabled (3 grants for adults and 2 for children with disabilities), for youth to improve their reproductive health and prevent addictions (4 grants) and two grants intended for uninsured, unemployed persons. In addition, during the second cycle, grants were provided to 9 PHCs to increase the availability of dental healthcare and improve the oral health of the population. In the third cycle, 12 grants were provided to LSGs to increase healthcare to vulnerable groups, and 9 grants to increase their access to dental healthcare, covering about 7,500 members of these groups. 9. It is difficult to measure the impact of these grants in such a short period, but unofficial feedback highlights an increase in the number of newly diagnosed cardiovascular diseases, elevated blood sugar, elevated triglycerides and cholesterol, and addition to other non-communicable diseases such as cancer. Nevertheless, better regulation of blood pressures, blood sugar, cholesterol and triglyceride levels should lead to improved health outcomes, and there have been efforts to promote healthy lifestyles and choices, identifying risk factors and required behavior changes. Palliative care provided to the elderly undoubtedly improved the quality of life of the terminally ill and of those that depend on care and assistance. Single elderly were provided with access to nurses, in cooperation with centers for social work, or with volunteers from NGOs. Specialist referrals, transportation, guidance on obtaining health insurance where warranted, 45      were additional outputs that should improve health outcomes of vulnerable groups. PHCs purchased medical supplies and equipment, and in partnership with LSGs provided vehicles, home nurses, food etc. NGOs involved with vulnerable groups provided critical support. 10. An external evaluation, Technical Assistance in DILS Grants Evaluation, found, inter alia, that: (a) PHCs procured the necessary equipment, including dental, that will allow them to continue providing health services to vulnerable population groups; (b) cooperation of PHCs with LSGs and NGOs improved, while the expectation that they would enhance cooperation with educational, cultural and other institutions still needs further work; (c) increased public trust in the availability of health services for vulnerable population groups, and (d) the PHCs’ skills in preparing project proposals, implementation and reporting have improved allowing them to continue competing for and receiving funds for projects. Almost all PHCs have continued their activities in some way, some independently and others with the support of LSGs. 11. PHC Accreditation Grants were awarded to the 42 PHCs participating in the Project, and an additional 20 PHCs based on a request by the MOH. Accreditation standards for PHCs with the process defined in scope, form and duration by the Agency for Accreditation of Health Care Institutions in Serbia, include: (a) standards for patient care in general practice, gynecology, pediatrics, specialist-consultative services and home care; (b) standards of clinical support: pharmaceutical services, laboratory diagnostics and diagnostic radiology; and (c) non-clinical standards for the environment, human resources, information management, governance and management. PHCs were selected according to pre-defined criteria that included: (a) size of the PHC; (b) degree of development of the municipality; (c) percentage of positive responses to questionnaires sent to all PHC on the degree of fulfillment of the criteria to be considered in the accreditation process; and (d) express desire for entry in the accreditation process. Grant funding was provided in fixed but varying amounts, depending on the size of the PHCs (small, medium or large), with the difference provided by the PHC in cooperation with the respective LSG. The grants provided funding for training of the accreditation teams in PHCs, development of the strategic plan for the PHC, and carrying out of self-assessment according to the standards of accreditation. 12. The external evaluation confirms that 83 PHCs, in three cycles, implemented the accreditation process, as well as the Institute for the Health Care of students in Belgrade. An additional 33 institutions have begun the process (but without grant funding). The MOH remarked that accreditation has produced the following benefits: “the development of multidisciplinary teams, review of the institutions’ operational policies, improving data systems, generating local and national prestige, and improved networking between primary health centers in exchange of good practices.” The evaluation found that PHC institutions recognize accreditation as one of the most important external mechanisms for improving the quality of health care. 13. Quality improvements through investment grants were supported for 43 PHCs participating in the Project. This funding had the following objectives: to support local cooperation between healthcare institutions and LSGs, and to serve the practical integration of all activities in the field of quality improvement that were implemented at the local level, ranging from strategic planning, through recommendations made in the process of accreditation to the development of an integrated quality improvement plan and its implementation. Funds were used to finance minor construction works, such as repairs, window replacement, leakage control, purchases of medical equipment and transport vehicles, including ambulances, to respond to patients requiring home care, transport to health facilities, etc. Only 14 PHCs had the financial support of their LSG, which varied form over 50 percent of the investment to only 2 percent. 14. The MOH has found that for the majority of the PHCs this was an opportunity to understand how investments can be used for quality improvements, as they were able to analyze their needs, prioritize, and compete for funding, explaining the rationale with evidence to support their investment proposals. The external evaluation reports that PHCs in general found that the investments the grants supported improved the quality and efficiency with which services are delivered, increased the interest of the respective LSGs in healthcare issues, and enhanced public trust in the system. All have gained competence in how to 46      prioritize investments within constrained resources, how to prepare investment proposals and how to monitor and evaluate progress. 15. Grants for introducing software were provided to all (158) PHCs, as well as to 3 health institutions for healthcare of students and the Republican Gerontology Institute. The amount of each individual grant varied, depending on the availability of software in the PHC, the size of the institution as measured by the number of doctors, the number of clinics, and the economic development of the municipality. LSGs were expected to participate (except for 46 of the poorest municipalities) by contributing up to 30 percent of the cost of introducing software, on an as need basis. In fact, only one-third of the PHCs counted on the financial support of the respective LSG, and most of those that did not highlighted that the grant financing was sufficient to cover their needs. 16. The grant funds were used to: (a) purchase certified software with the accompanying package of service for all PHCs, or reinstallation and customization of existing software for PHCs that have introduced it; (b) training staff to use the software; (c) implementation of software functionality through a 6-month warranty period, and (d) creating conditions for PHCs to be included in the integrated health information system. (The purchase of IT equipment was carried out centrally by the MOH under a separate contract, as was the process for certification of software.) Any unused grant funds upon full installation of the software could be used by the PHC (with prior approval of the MOH) for: (a) purchase of hardware or IT equipment; (b) purchase of new software modules (e.g., laboratory information systems); and (c) maintaining IT systems in the facility. The implementation of the electronic health record and respective user training were implemented in 156 health facilities (96.3%). 17. The evaluation report highlights several positive benefits of this line of grants. These include, among others: (a) it has standardized the information system with respect to all components, hardware, LAN, WAN, software, user training, documentation, maintenance; (b) the relationship between PHC management and health personnel towards the process of computerization has changed considerably in a positive direction; (c) the software is characterized by medical personnel as a very useful tool, which facilitate the work and definitely improves the quality and efficiency of health services and delivery; (d) patients accept positively the existence of their electronic health records as the consider that it improves the quality of services especially where services that visibly facilitate access to medical treatment have been introduced; (e) PHCs have continued to introduce new modules providing own funds after seeing the benefits of the initial software installed; and (f) LSGs have become increasingly involved in the process of computerization, more than before, and at the level of decision-making along with other stakeholders. As to impact, the evaluation notes: “the impact of computerization in the PHC facilities on direct beneficiaries, local and regional environment, as well as the entire health system has been extremely positive and visible in many aspects. Accuracy, uniformity, speed of delivery of health services were all improved which significantly raised the quality. The care procedures related to chronic patients, for screening, prescription drugs, prescriptions, referrals, making appointments, etc. were all simplified. These positive changes have been observed by both doctors and patients. The relationship between the management of the facilities and health personnel towards the process of computerization changed to a large extent compared to the beginning of the project, and in a positive direction. Local government provided the media promotion of the computerization of institutions and is familiar with the problem of sustainability. At the regional level, connection with institutes of public health is being partly achieved. Also, selected physician and nurses (team) generally have an increased awareness of the importance and responsibility of their role in the creation of electronic health information on an individual patient. The process of computerization in primary health care has been intensified, this initiated the founding of the Association of IT Experts in Health Care Facilities, which aims to increase the impact of information technologies on the process of computerization of health care in Serbia.” 47      Ministry of Labor, Employment, Veterans, and Social Policy (MOLEVSP) 18. The MOLEVSP provided grants to 87 projects of Disabled People’s Organizations under the Disability Fund. Previously, the Disability Fund had provided financing on an incremental, yearly basis, with no monitoring of results and impacts on the end users. A main objective of reforming the process of allocating funds was to ensure that funding provided through the Disability Fund were directed towards activities that provided the greatest results for persons with disabilities (PWD). 19. The Disability Fund was reformed in four stages. Using a first draft of the Grant Operations Manual, two projects targeting persons with disabilities were piloted: sign interpreting services and companion services. The Grant Operations Manual was revised to incorporate experience under these two projects. A second call for grant proposals was conducted in 2010, and led to a further revision of the Grant Operations Manual, a third call for proposals in 2011, and further revisions and finally a fourth call for proposals. The final Grant Operations Manual incorporates improved formats for applying for grants and template for reporting on expenditures and activities implemented, and improved scales for assessing project proposals, which are transparent and public. The Manual includes clearly defined modalities of project financing, conditions, criteria and procedures for awarding grants, for monitoring funds usage, etc. The MOLEVSP conducted training, at various intervals, to associations focused on persons with disabilities on the preparation of grant funding proposals, and monitoring and implementation of their projects. Monitoring of grant implementation, reporting and monitoring was carried out be external consultants that periodically verified compliance with the Manual, implementation progress, financial management and issues. Periodic interviews were also held with project coordinators, members of project teams, beneficiaries, and others involved in implementation. 20. Following the granting and implementation of the two pilot grants totaling approximately Euro 50,000, 47 projects of associations (providers of service could be citizens’ associations, NGOs, private groups, etc.) out of a total 350 organizations that submitted proposals were awarded grants in the total amount of approximately Euro 388,313. In 2011, the Disability Fund awarded 36 grants totaling Euro 287,777. In 2012, 4 grants were awarded for approximately Euro 90,000. 21. The Disability Fund grants covered the following main types of activities: • Training activities aimed directly at strengthening the capacity of PWD organizations and providing training to have an active role as the service providers in their local communities—implementation of training, education, workshops, and seminars through which representatives of PWD organization gained knowledge and skills to effectively design and develop program activities and launch services, actively participate in the creation and implementation of programs within the defined policies for the improvement of the position of PWN at the local community level; • The provision of social services and psycho-social support to PWD and their families through the launch of innovative activities and further development of the newly establish social services for PWD piloted through the previously awards grants. Funded services predominantly were classified as advisory-therapeutic and social and educational service, as well as support services for independent living (the funding for which in accordance to the Law on Social Protection is under the jurisdiction of local governments); • Training activities for broader groups of citizens, as well selected categories (teachers, employees of public institutions, children with typical development of pre-school or school age, students, youth) through implemented educational workshops, lectures and specific trainings, and media and public campaigns that contributed to creating a positive environment and conditions for the social integration of PWD, raising public awareness about the presence, needs and abilities of PWD, overcoming negative perceptions of PWD based on a medical approach and acceptance of PWD as equal member of society with equal rights; • Activities aimed at improving the availability and effectiveness of services for PWD—collecting, compiling and disseminating information about existing social services for PWD, connecting service 48      providers with local authorities and relevant public institutions and planning of joint actions, development of a legal framework for the establishment of the center for improving the standards for services provided by the PWD associations, and • Activities of assessment and planning of social services for PWD. 22. Most of the grants included several categories of these activities, adjusted to address local needs of the PWDs in their communities. Informal evaluations, on the basis of grant monitoring, show that from 80 to 100 percent achieved expected results as defined by their measurable indicators, and one-third exceeded initial expectations, primarily in the number and motivation of direct users, as well as the sensibility of the public and institutions to the needs and problems of PWD (i.e., accepting PWD as equal members of society with equal rights). 23. The main target beneficiaries of the Disability Fund grants awarded were employees of state institutions (that by nature of their work are in daily contact with PWD) and organizations and PWD themselves. Targeting employees of state institutions provides for easier integration of PWD into society by providing for their easier use of existing services. PWD attended different trainings, workshops and meetings to acquire new knowledge and skills to make use of existing social services, and systematically influence changes at the national and local levels. Some grant project targeted marginalized groups such as Roma PWD and Roma children. 24. All grant projects were implemented by PWD organization and NGOs. Most were carried out independently, while eight projects were implemented in partnership with local stakeholders (LSGs, Centers for Social Work, other public institutions at the local level), or other related civic associations. In addition to covering part of the direct costs, local level partners assisted with the technical, logistical, organization aspects, in addition to providing technical expertise and professional resources (e.g., psychological counseling, psycho-social and other activities in accredited programs, establishing specific services and educational facilities). The informal support of local institutions, LSGs, NGOs, and the local media was instrumental in providing the conditions for promotion and implementation of activities, especially for those project whose implementation was directly related to the cooperation with these local institutions and the inclusion of their representatives. The establishment of formal and informal partnerships with key local stakeholders reflects the recognition of PWD association of the need for networking and connecting with these groups to overcome discrimination towards PWD, to protect their rights and ensure equal treatment for PWD. Also, one third of all grant projects involved actively volunteers in the implementation of their activities. 49      Annex 3: Economic and Financial Analysis 1. The development objective of the DILS Project was to increase the capacity of institutional actors and beneficiaries to improve access to and the efficiency, equity and quality of local delivery of health, education and social protection services in a decentralizing environment in Serbia. The project had five components to: Make fiscal decentralization work through investments in goods, technical assistance and training for line ministries and Local Self Government (LSG) agencies; Improve outreach and access through development and expansion of innovations in service delivery; Support a new, regulatory oversight and quality-assurance role for State level Ministries; Improve capacity of LSGs and other local public institutions as service providers; and the last component was to provide project implementation support. 2. Although the project generated a number of direct and indirect multiplier impacts, the nature of the activities pursued and the resulting benefits could not be quantified. Hence a traditional economic analysis with rates of return analysis has not been attempted in the Project Appraisal Document (PAD) and in this completion report. Instead this section traces the economic impacts of activities pursued under the project to improve access to and the efficiency and quality of local delivery of health, education and social protection services in a decentralizing environment based on evidence from the literature. Economic impact of decentralization on delivery of health, education and social services 3. The decentralization of social sectors services produced significant impact on quality of local delivery of health, education and social protection services. Estimation of the impacts of decentralization of these services requires major research effort and often sufficient data do not exist to quantify the impacts (Bossert, 2002)4. We do not have sufficient data to analyze and estimate the direct and indirect impacts of activities under this project and hence is not attempted here. Studies by Robalino, Picazo and Voetberg (2011) showed significant impacts of fiscal, administrative and political decentralization on variables used to measure access to health care and health outcomes. Studies using panel data suggest significant association between both fiscal and administrative decentralization and delivery of health services and health outcomes at the local level. The World Development Report 2004 further argues that political decentralization brings about accountability to the system and thus improves health service delivery5. 4. The health sector activities implemented under the DILS project included 6 : streamlining the financing of the primary healthcare; performance payments; training for employees in LSGs and PHCs; training in financing and accounting procedures like step-down method of analyzing costs and PHC financing; assessment of energy efficiency of PHC buildings; support to National Insurance Fund to build capacity for the introduction of diagnosis related groups; analysis of the activities and financing of National Reference Laboratories (NRL), their equipment and staffing, with a view to improve their services and efficiency; and capacity building for staff of PHCs on EU health policy and preparation of project proposals for EU financing. All of these activities produced substantial direct and indirect multiplier impacts in local delivery of health services. Though we do not have data to document those impacts the evidences from the literature on the impacts of these activities cited above and those presented in the PAD provide justification for significant economic impacts. 5. Evidences from the literature also show significant impact of decentralization on education outcomes. Evidence to date suggests activities like increasing school autonomy, delivery of schooling, fiscal decentralization and school governance have significant impacts on educational outcomes (USAID,                                                              4 Bossert, T. (1998). "Analyzing the decentralization of health systems in developing countries: decision space, innovation and performance." Soc Sci Med 47(10): 1513-27. 5 Robalino, David; Picazo, Oscar; Voetberg Albertus. 2011. Does fiscal decentralization improve health outcomes: evidence from a cross country analysisǁ. The World Bank. 6 Detailed list of activities and outputs is presented in Annex 2. 50      2007)7. The education sector activities and outputs undertaken include: piloting central and local per capita funding formulae, training and seminars on the funding formulae, grants to educational institutions without violence and to Roma children, grants to inclusive schools, trainings for staff and teams on inclusive schools, Procurement of Assistive Technologies equipment and software, and distribution to 6 schools, consultant services and technical assistance and analytical services to support education sector. As in the case of health sector interventions, these activities in the education sector generated impacts both direct and indirect. However, we do not have data and information to quantify the direct and multiplier impacts. But evidences from literature cited above and in the PAD suggest possible impacts of these interventions on local delivery of education services. 6. Available evidence further suggests that decentralization of services delivery and involvement of decentralized local governments increases efficiency and impacts of social protection programs. Intergovernmental arrangements and coordination, local government capacity development, fiscal decentralization and accountability and local capacity development were found to increase economic impact of social protection programs (UNDP, 2013)8. 7. Consistent with the above evidence from literature, the activities and outcomes from the DILS project improved access to and efficiency, equity and quality of local delivery of health, education and social protection services by increasing capacity of institutional actors and beneficiaries in a decentralized environment. It also increased accountability and governance of LSGs by better coordination and loops of communication among the different agents. A. Marginal benefits with the project 8. This section presents the marginal benefits with the project. The project had significant impact in improving access to health, education and social protection services. In the education sector, the project improved access and equity in pre-school and primary education through the implementation of sub- projects awarded under two grant programs. The two sub-projects were strengthening schools for inclusive education and education inclusion of Roma. 9. The project also had an influential impact on improving the quality of local delivery of health, education and social protection services. In health the project financed activities at the state level as well as activities with LSGs and project PHCs, especially, three types of sub-project grants aimed at helping the PHCs accomplish the Project’s objectives: (a) PHC Accreditation Grants; (b) Quality Improvements through Investment Grants; and (c) Grants for Introducing Information System Software. In education the project achieved impressive results through a variety of activities like participation in student assessments and technical assistance and training for the implementation of standards for the end of compulsory education (8th grade) and related capacity building. In order to improve social protection at the local level the project developed a centralized management information system and provided with infrastructure and software in all 265 CSWs, and 600 staff in all of the CSWs have been trained in its use. 10. All of these activities generated significant marginal benefits with the project. The Project Appraisal document outlines a number case studies and results from literature that demonstrate positive impacts of different activities in health, education and social protection services. B. Macroeconomic growth and financial sustainability 11. After the recession of 2014, the economy is gradually showing signs of stabilization but faces downside risks. The recent recovery is supported by robust increase in industrial production and exports, lower oil prices and stronger growth in the euro area. GDP growth is expected to remain flat in sort-run.                                                              7 USAID (2007) Identifying the impact of education decentralization on the quality of education. Working Paper. EQUIP 2. http://www.equip123.net/docs/e2-DecentQuality_WP.pdf 8 UNDP (2013) Strengthening the governance of social protection: The role of local government. 51      More robust growth rates of around 2-3% are forecasted over the medium term. Inflation is declining in recent years falling from 11.1 percent in 2011 to 2.1 in 2014 and is expected to be stay around 4 percent in the medium term. But a large public debt accumulated through 2014 will require further efforts in order to control it over the medium-term. Public revenue as a percent of GDP increased from 37.9 percent in 2013 to 40 percent in 2014 and is forecasted to gradually decline over the medium term (ie through 2020) to around 38.2 percent of GDP. Government expenditures as percent of GDP has been relatively high at around 45 percent in the recent past and is expected to gradually decrease falling to 40 percent by 2020. The fiscal balance as percent of GDP also improved from -6.7 percent in 2014 to projected 4.1 in 2015 and is expected to be around 2 percent by 2020. Table A3.1: Selected macroeconomic indicators, 2009-2014 2009 2010 2011 2012 2013 2014 GDP Growth (%) -3.1 0.6 1.4 -1.0 2.6 -1.8 Inflation CPI (average, %) 8.1 6.2 11.1 7.3 7.8 2.1 Exchange rate (Dinar/ US $) 67.5 77.9 73.3 88.1 85.2 88.5 Gross official reserves (Billion 10.6 10.0 12.1 10.9 11.2 9.9 Euro) Revenue (% of GDP) 39.8 39.9 38.2 39.4 37.9 40.0 Govt. Expenditure (% of GDP) 44.2 44.6 43.1 46.6 43.5 46.7 Fiscal balance (% of GDP) -4.3 -4.7 -4.9 -7.2 -5.6 -6.7 Source: Statistics Office; National Bank of Serbia; Ministry of Finance, World Bank staff calculations 12. The Government put in place a fiscal consolidation program in 2014 which has contributed to improved fiscal performance in 2015. The general government deficit over the first nine months of 2015 was 1.3 percent of full-year GDP, down from 3.9 percent in the same period of 2014. The deficit reduction came primarily as a result of increased revenues (up 6.5 percent y/y in nominal terms). VAT and excises pushed total tax revenues up by 2.3 percent. However, the strong revenues were mainly supported a by major increase in non-tax revenues primarily due to one-off measures (i.e., payment of net income from state owned enterprises and proceeds from the sale of 4G licenses) and from the introduction of surcharges on public sector wages. Total nominal government expenditures declined by 1.7 percent as a result of major savings from wage and pension reforms (down by 11.4 and 3.5 percent, respectively). Wage bill reduction comes as a result of introduction of a 10 percent cut in wages across the public sector and continued implementation of a hiring freeze (introduced in January 2014). Savings on spending on pensions comes as a result of the reduction of pensions higher than RSD 25,000 by 22 percent and those above RSD 40,000 were cut by 25 percent. Table A3.2: Public expenditures in social sectors (% of GDP) Average 2006-08 2014 Total expenditures 44.1 46.7 General services 4.5 6.1 Defense 2.4 1.3 Public Order and Safety 2.4 2.7 Economic affairs 6.2 6.0 Environment 0.3 0.3 Housing and Communal services 1.7 1.3 Health 5.8 5.7 Sport, culture, religion 1.0 1.1 Education 3.8 4.2 Social assistance 16.1 17.8 Source: Ministry of Finance, World Bank staff calculations 52      13. As the expenditures in the health, education and social protection sectors have increased and are projected to stay at the similar level in the future we could expect more decentralized delivery of health, education and social services as the government’s commitment to decentralization, which indicates the sustainability of the project and its development objectives. 14. Fiscal weaknesses over the 2012-2014 period led to a significant increase in public debt. General government expenditures averaged 45.6 percent of GDP over 2012 to 2014, about 2.5 percentage points more than in the pre-crisis period. At the same time revenues declined and consequently the fiscal deficit rose, reaching historically high levels (averaging 6.5 percent over 2012-14). As a result, public debt grew rapidly, reaching 72.3 percent of GDP at the end of 2014. 15. Notwithstanding lower net fiscal financing needs, central government debt (including guarantees) has moved up slightly over 2015, to 73.7 percent at end-September 2015 from 72.3 percent at end-2014. As well as the continued, if narrowed deficit, this is partially explained by the US dollar strengthen in early 2015 (with 33 percent of debt dollar-denominated). The stock of guarantees have been gradually declining – from 7.9 at end-2014 to 7.5 percent of GDP in September 2015, as the policy of no new liquidity guarantees is being strictly observed. Most of the public debt relate to external direct debt (38.8 percent of GDP) while the domestic public debt accounts for 26 percent of GDP. C. Sustainability at the local level 16. The decentralization of public revenues to local governments was facilitated by the Law on Self- Government (2002). This law increased the autonomy of local self-governments and improved the transparency of intergovernmental fiscal relations. 17. In 2007 the new law on local government finance was enacted creating four principal sources of revenue, namely: local taxes, shared taxes, non-categorical intergovernmental transfers; and categorical intergovernmental transfers. This new law eliminated the arbitrary and unpredictable distribution of revenues that characterized the prior system of local public finance. It also replaced a system based on varying shares of the sales tax with fixed shares of the payroll tax and formula-driven transfers. The new law improved the predictability and transparency of the intergovernmental finance system. Among individual jurisdictions, the reforms achieved a significant reduction of disparities in per capita revenues. 18. The new law on local self-government reduced disparities in spending in social sectors as the law is being implemented. The DILS project also reduced the variations in spending in the social sectors as the variations in per capita revenues continue to decrease. D. Impact of the project on poverty, income distribution and other distributional impacts 19. The service delivery at the local decentralized level under the DILS project increased access to health, education and social services at the local level, targeting the poor and vulnerable groups. With relatively high levels of poverty among the children and youth, at about 50 percent and about ten times the national average at the Roma level, the project, through targeted delivery of social services at the local level and among vulnerable groups had significant impact on poverty and social outcomes. 20. The decentralization and targeting of local decentralized institutions under the DILS project facilitated targeted support in health, education and social services to households and vulnerable groups at the local level. The decentralization improved efficiency and targeting of poor and low-income groups and regions. Access to public pre-school education increased from 41.5 percent in 2003. Similarly, the project significantly increased access to pre-school education among poor children in the project area from 15.9 percent before the implementation of the project. The DILS project supported reforms to improve access and quality of pre-school and secondary education by: (i) improving the equity in the allocation of funding; 53      (ii) increasing local control over the allocation of resources- by providing block grants on a capitation basis and allowing school directors to allocate funds among different economic categories of expenditure on the basis of local priorities; and (iii) encouraging network rationalization. 21. In conclusion, the DILS project provided improved access to health and health services at the local level. As evidenced in the economics literature improved health and education services and safety nets improved human capital, which in turn had significant impact on poverty in the project area. These impacts will continue through in the long run through better health, education and thus improved productivity of human capital among the youth and adults. The project also accelerated the pace of decentralization in delivery of health, education and social services. 54      Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team Members Responsibility/ Names  Title  Unit  Specialty  Lending  Ivana Aleksic  Human Development Specialist ECSH2   Nicholay Chistyakov  Senior Finance Officer CTRLN   Olav Rex Christensen  Senior Public Finance Specialist  HDNED   William R. Dillinger  Lead Public Sector Management  ECSP4    Dominic S. Haazen  Lead Health Policy Specialist AFTHW   Nikola Kerleta  Procurement Specialist ECSO2   Carmen F. Laurente  Senior Program Assistant ECSHD   Tobias Linden  Lead Education Specialist SASED   Imelda Mueller  Operations Analyst ECSH2   Marina Petrovic  Consultant ECSH3   Gennady Pilch  Senior Counsel  LEGOP   Dena Ringold  Lead Economist  AFTSW   Maria Laura Sanchez Puerta  Senior Economist HDNSP   Pia Helene Schneider  Lead Evaluation Officer IEGPS   Hermina Vukovic Tasic  Program Assistant ECCYU   Supervision/ICR  Ivana Aleksic  Human Development Specialist ECSH2   Juan Diego Alonso  Senior Economist  LCSHE   Aleksandar Crnomarkovic  Sr Financial Management Specialist ECSO3    Michele Gragnolati  Sector Leader  LCSHD   Ana Holt  Health Specialist ECSH1   Marijana Jasarevic  Operations Analyst ECSH3   Nikola Kerleta  Procurement Specialist ECSO2   Tobias Linden  Lead Education Specialist SASED   Imelda Mueller  Operations Analyst ECSH2   Ethan Yeh  Economist  ECSH1    (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only)  Stage of Project Cycle  USD Thousands (including No. of staff weeks  travel and consultant costs) Lending      FY06  16.9 119.78 FY07  90.89 327.97 FY08  51.23 102.20 Total Lending:  159.02 549.95 Supervision/ICR  347.56 1094.89 Total:  506.58 1644.84 55      Annex 5. Beneficiary Survey Results No beneficiary survey has been carried out. 56      Annex 6. Stakeholder Workshop Report and Results No stakeholder workshop has been carried out. 57      Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Borrower’s ICR: The Ministry of Health and Ministry of Education, Science and Technological Development submitted official Completion Reports dated November 15, 2013 and January 2015, respectively. The Ministry of Labor, Employment, Veterans and Social Policy submitted an unofficial version of its respective report. All reports are available upon request.  Borrower’s comments on Draft ICR: Translation of Letter from the Ministry of Finance of November 4, 2015 Republic of Serbia MINISTRY OF FINANCE PUBLIC DEBT ADMINISTRATION Number: 401-2234/2015-001 November 4, 2015 Belgrade WORLD BANK OFFICE, SERBIA BELGRADE Bulevar Kralja Aleksandra 86 Subject: IBRD “Delivery of Improved Local Services” – Final Report on Implementation and Results Dear Sir/Madam, The Loan Agreement (“Delivery of Improved Local Services” Project) between the Republic of Serbia and the International Bank for Reconstruction and Development in the amount of EUR 32,000,000 was signed on April 11, 2008. The National Assembly of the Republic of Serbia passed the Law on Confirming the Loan Agreement (“Delivery of Improved Local Services” Project) between the Republic of Serbia and the International Bank for Reconstruction and Development that was published in the “Official Gazette of the Republic of Serbia – International Agreements”, number 121/08. The project “Delivery of Improved Local Services” (hereinafter referred to as: Project) aimed at increased capacity of institutional stakeholders on the republic level, level of local government, service providers from the non-governmental sector, primary health care centers, schools and social work centers in order to improve access and efficiency, equality and quality of service delivery in the area of health, education and social protection on the local level in the decentralizing environment. The Project was implemented through the Ministry of Health, Ministry of Education, Science and Technological Development and Ministry of Labor, Employment, Veteran and Social Affairs. The total amount of disbursed loan proceeds is EUR 31,838,697 while the amount of EUR 161,303 was cancelled. It was established that in the Final Report on Implementation and Results of the Loan (IBRD 75100) in the amount of EUR 32.0 million (equivalent to USD 46.4 million) for the Republic of Serbia, the project “Delivery of Improved Local Services” (DILS), in Annex 3, Economic and Financial Analysis, B. Macroeconomic Growth and Financial Sustainability, the stated data are not up-to-date. With regards to that, we indicate that it is necessary to correct the stated data along with mentioning the institutions of the Republic of Serbia as relevant sources of these data. Furthermore, we kindly ask you to state the reasons 58      for unavailability of data from Annex 5: The Results of the Survey among the Beneficiaries, Annex 6: The Report and Results from the Workshop for Interested Stakeholders and Annex 8: Comments of Co-funders and Other Partners/Interested Stakeholders. Besides the abovementioned, we particularly emphasize that it is unacceptable to show the territory of the Republic of Serbia on the map at the end of the Report without its integral part –AP Kosovo and Metohija, since it violates the provisions of the Constitution of the Republic of Serbia. Sincerely Yours, Acting Director Branko Drcelic Response from the ICR team: The Ministry of Health, the Ministry of Education, Science and Technological Development and the Ministry of Labor, Employment, Veterans and Social Policy have not provided comments to the Draft ICR. In response to the Ministry of Finance comments and request, received on November 4, 2015, the following changes to the draft ICR have been made: (i) the map has been removed from the Report; and (ii) Section B (Macroeconomic Growth and Financial Sustainability) of Annex 3, Economic and Financial Analysis, has been revised in line with the Ministry of Finance’s comments and the sources of the data have been added. Further, as to the points regarding missing information in Annex 5: The Results of the Survey among the Beneficiaries, Annex 6: The Report and Results from the Workshop for Interested Stakeholders, and Annex 8: Comments of Co-founders and Other Partners/Interested Stakeholders, the ICR Team would like to inform that the Education and Social Protection respective share of the Loan (closed on December 31, 2013) were transferred to the Health sector (which remained active until March 31, 2015) to help mitigate the damage originating from floods in Serbia. Consequently, the Education and Social Protection Project Administration Teams (PATs) ceased to exist starting beginning of January 2014 while the Health PAT was operating under reduced capacity only to ensure successful implementation and monitoring of the remaining activities. In addition to that, the leadership of all three Ministries changed in March 2014. There was no consensus among the respective Ministries and no adequate PAT’s capacity to undertake beneficiary survey nor to organize workshop for stakeholders. 59      Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Not applicable. 60      Annex 9. List of Supporting Documents Center for Education Policy, Evaluation of DILS Trainings and Grant Programs for Inclusive Education, December 2013 Euro Health Group, Republic of Serbia, Delivery of Improved Local Services Project – DILS, Technical Assistance in DILS Grants Evaluation, August 2014. Republic of Serbia, Ministry of Education, Science and Technological Development, Delivery of Improved Local Services-DILS Project, DILS: Summary of the Results, December 2013. Republic of Serbia, Ministry of Health, Implementation Completion Report, Delivery of Improved Local Services-DILS Project, January 2015. Republic of Serbia, Ministry of Labor, Employment and Social Protection, Implementation Completion Report, Delivery of Improved Local Services-DILS Project, January 2014. World Bank Country Partnership Strategy for the Republic of Serbia, Report No. 41210-YF, dated November 13, 2007. World Bank, Country Partnership Strategy for the Republic of Serbia, Report No. 65379-YF, dated November 15, 2011 World Bank, Delivery of Improved Local Services (DILS) Project, Project Appraisal Document, Report No. 38921 – YF, dated February 21, 2008 World Bank, Restructuring Papers on Proposed Project Restructurings of the Delivery of Local Service Project, Loan Number 7510-YF, Report Nos. 67670-YF, 74057-YF,dated June 26, 2012, November 30, 2012, October 30, 2013 and August 7, 2014, respectively. World Bank, Delivery of Improved Local Services (DILS) Project, Implementation Status and Results Reports and Mission Aide Memoires 61      Annex 10. Revisions in Results Framework and Monitoring after Level 2 Project Restructuring of June 2012 PDO  Project Outcome Indicators  Rationale for Revision of Project Outcome Indicators Original  Revised  Original  Revised    Increase the No • Allocation of capacity of change  financing from • Percent of children from Revised to: i) measure institutional State government vulnerable groups in an outcome rather than actors and to local self- project schools an output; ii) be beneficiaries governments for (vulnerable groups measurable; and iii) be in order to health and according to OECD specific to a particular improve education services classification: category sector. First, grants are access to and made according to A – children with instruments and not the the Law on Local disabilities; B – children outcomes. efficiency, Self Government with learning difficulties; Implementation of the equity and financing (Official and C – children from grant does not quality of Gazette of the socio-economically necessarily reflect an local delivery Republic of disadvantaged groups) increase in inclusion. of health, Serbia, 62/06) Second, the original education and indicator was not easily social measurable as protection “achieved success” can services in a be defined differently decentralizing • Share of MOLEVSP for each grant awarded. environment  • Percentage of financing allocated for The indicator also did grants awarded by Disabled Peoples the Project to not have a definition Organizations (DPOs), for “success” in the support allowing for equal access PAD or Loan innovations in and improved inclusion, that Agreement. Third, the transparency, and results- indicator is now have achieved based. specific to the success in increasing education sector, and inclusion of separate indicators will marginalized be used to measure groups (the poor) inclusion in health and to health, social protection. It is education and difficult to measure social protection achievement if the services indicator is applicable to multiple sectors. Rephrased to be more • Number of Roma specific to the actual • Share of children receiving financing reform- MOLEVSP vaccinations through the providing equal access financing for Roma health mediators to public funds for disabled groups program. addressing the needs of that is allocated disabled people according to a new (competitive scheme 62      PDO  Project Outcome Indicators  Rationale for Revision of Project Outcome Indicators Original  Revised  Original  Revised    program-based where funds are model of allotted according to allocation. the quality, relevance, and results-focus of proposals) and transparent use of these funds by DPOs. At the time of the restructuring, the share of financing had surpassed the original target and was 100% following full implementation of the Law on the Associations of Citizens passed in 2009. New indicator • Drop proposed to measure results in improving inclusion in the health • Satisfaction with sector. The Roma the quality of health mediator delivery of program is a substantial services among result of the MoH users of health under the Project, facilities and which is now reflected disability services in the Results in municipalities Framework. benefiting from the Project, as This outcome indicator proxied by has been removed since responses from patient satisfaction is exit surveys of subjective and does not users in the provide as rigorous a municipalities measure of quality of care improvements as accreditation (which has been kept as an outcome indicator). Patient satisfaction • Percent of PHCs more often reflects a receiving at least a 3- doctor’s and/or nurse’s year certificate of communication skills. • Number of local accreditation The MoH also finds the service providers satisfaction measures (PHCs, schools, from 2008 until now CSWs and NGOs) less reliable as a who have gone measure of quality. through a quality accreditation process, designed 63      PDO  Project Outcome Indicators  Rationale for Revision of Project Outcome Indicators Original  Revised  Original  Revised    and administered Revised to be specific by the relevant to health, since quality institution in their accreditation does not sector. take place in education • Drop or social protection, nor does it apply to NGOs. Since the accreditation process by itself is not • Establishment of an outcome, the information indicator is revised management from “gone through a systems linking quality accreditation local service process” to receiving a providers (PHCs, 3-year certificate of schools, CSWs accreditation. and NGOs) with Moved to output the relevant line indicator because the ministry in their establishment of an sector, and where   information system is relevant to the an output, not an successful outcome. In the delivery of original Results services, that Framework, the allow information indicator was included sharing across both as an outcome sectors.  indicator and as an output indicator. In addition, the information systems do not directly link NGOs Intermediate Outcome Indicators  Rational for Revision of Intermediate Outcome Indicator  Original  Revised    Component 1: Transforming the Financing Models • % of treatment • Legislative framework Original indicator was not relevant or municipalities making allows for capitation- applicable. This indicator has been revised financial allocation based on and output-based with two actual financing changes in health: agreed formula in health formula i) the introduction of capitation- and output- sector • Improved allocation based payments to PHC providers, and ii) a new framework for equalization funds. The framework for equalization funds development of the legislative framework developed enables capitation- and output-based payments, which is an outcome indicator. The second indicator refers to developing methodology and criteria for equalization • % of population registered funds, which are distributed to PHCs in the with their chosen doctor in poorest municipalities. Currently, these are treatment municipalities • Drop distributed without criteria. 64      Intermediate Outcome Indicators  Rational for Revision of Intermediate Outcome Indicator  Original  Revised    Indicator has been dropped because the Project does not substantially impact chosen doctor registration rates. Registration rates • Per-capita financing are already high (73.42% in 2011), prior to formulae developed for any activities taking place. A new indicator • Central and local education  formulae developed  (below) has been added to replace this one. Slightly reformulated to be more specific Component 2: Improving Access and Equity in Service Delivery • By end of project, at least • Number of educational Indicator is revised to be specific to the 60% of grants provided for institutions (schools education sector and to be linked to the inclusion activities to and preschools) by type outcome indicator. Access of vulnerable marginalized groups in of grants received children to school (outcome indication) is a project municipalities are (school and municipal) better metric in assessing the impact of the implemented successfully. grants. Accordingly, the number of educational institutions by type of grant • Number of medical received is deemed an appropriate output staff and associates indicator associated with the desired trained to recognize outcome. needs of vulnerable The original output indicator referred to groups grants for inclusion in multiple sectors • Operational structure of the (health and education). This indicator has MOLEVSP Disability Fund been revised to be specific to the health • Drop restructured to improve its sector, and is related to grants providing for effectiveness. trainings to recognize needs of vulnerable • Model for allocating • Drop  groups. MOLEVSP financing for Dropped since similar to outcome indicator disability services, developed and introduced in number of project Dropped since similar to outcome indicator  municipalities.  Component 3: Improving Quality and Accountability • Quality guidelines • Number of PHC Indicator revised to specify the specific developed for improved centers that have quality guidelines developed in health local service delivery adopted clinical clinical pathways. More importantly, the pathways indicator has been improved to specify the number of PHC centers that have adopted pathways, rather than just the development • At least 60% of grants of a generic set of quality guidelines. awarded by project to • Percent of PHCs that Indicator revised since it is unclear how to support implementation of have completed quality measure grants as being “implemented new quality guidelines for accreditation process successfully”. DILS provides PHC centers local service delivery with grants in order to undergo a quality providers are implemented accreditation process. The revised indicator successfully • Number of schools that will measure the percent of PHC centers that have undergone a have completed the accreditation process, school performance which is linked to an outcome indicator. • Results of impact external evaluation. evaluations carried out to Indicator revised because the designed measure the quality and rigorous impact evaluation, with a valid effectiveness of the counterfactual, will not be carried out. The decentralization of services 65      Intermediate Outcome Indicators  Rational for Revision of Intermediate Outcome Indicator  Original  Revised    are disseminated to the • Number of training new indicator reflects an activity related to relevant stakeholders. participants among improvements in the quality of education. education staff that • Increased percentage of staff have undergone Revised to be more specific to trainings in education institutions that development training effectively contributing to quality are undergoing development organized by the improvements in teaching. Project. training in participating municipalities, compared to • Number of inter- other municipalities sectoral committees New indicator to reflect inter-sectoral trained. committees that have been set up to support inclusive education. Dropped since it is similar to an output • Drop • By the end of the Project, at indicator in Component 2. least 60% of schools who receive grants are satisfactorily meeting the objectives in their grant • Percent of PHC centers Revised for clarity and to be specific to each proposals. with fully operational sector. The original indicator was unclear as HMIS platform at the to how progress would be measured. This PHC level has been split into 3 sector-specific • Increased connectivity to indicators (one for health and two for social and usage of national protection). The revised health and SP information management • Centralized MIS fully operational in all social indicators measure the completion of networks, by local service information systems in those sectors. An providers  protection institutions. additional indicator for SP will measure progress in training on the usage of the new • Number of staff in MIS. There is no indicator for education, as social protection the implementation of the MIS will take institutions trained and longer than the Project’s life and it will be certified in the use of financed by the European Investment Bank.  the centralized MIS 66      Annex 11. Revisions to Project Description after Level 2 Project Restructurings 1. The Project had two Level 2 Project Restructurings that modified its Project Description, the first on June 26, 2012 and the second on October 30, 2013. 2. The June 26, 2012 Level 2 Project Restructuring (Section 1.3) modified the Project description to: (a) better align the Project’s structure with its implementation arrangements and outcomes; (b) add the design and implementation of communication campaigns and of measures to improve knowledge as activities and non-consultant services as a disbursement category under Components 1, 2 and 3; (c) add people with disabilities to the excluded groups targeted by Grants for the development of outreach services; and (d) correct minor inaccuracies in references to governance and implementation arrangements. The revised Project description combined Components 3 and 4 into a single component (Table 1) since most activities under these components were being carried out jointly and contributed to the same results. As such, the Government faced difficulties in planning and reporting results by components. The merging of the two components, did not entail any changes to their content, resolved these ambiguities and better aligned the project structure with implementation arrangements and outcomes. The revised Project description is presented in detail in Table 2. Table 1: Original and Revised Project Components after June 26, 2012 Restructuring Original  Revised Component 1: Making Fiscal Decentralization Component 1: Transform Financing Models  Work  Component 2: Improving Outreach and Access Component 2: Improve Access and Equity  through Development and Expansion of Innovations in Service Delivery  Component 3: Supporting a New Regulatory, Component 3: Improve Accountability and Oversight and Quality Assurance Roles for the Quality  Borrower’s Ministries  Component 4: Improving Capacity of LSGs and other Local Public Institutions as Service Providers  Component 5: Support for Project Component 4: Support Project Implementation  Implementation  3. The changes to the Project’s structure were two-fold. First, the revised Project description combined Components 3 and 4 into a single component. The original design assumed that activities defining roles and responsibilities of actors, strengthening capacities, and enhancing accountability to improve service quality would be carried out separately for the central and for local governments and produce independent results. In reality, however, most activities under these components were being carried out jointly and contributed to the same results. As such, the Government faced difficulties in planning and reporting results by components. The merging of the two components, which did not entail any changes to their content, resolved these ambiguities and better aligned the project structure with implementation arrangements and outcomes. Second, the revised description adopted different names for Components 1 to 3 to better capture the expected results. 4. The restructuring also added the design and development of communication campaigns (Components 1 and 2) and of measures to develop knowledge (Component 3) to the Project description. The original description envisage the development of information material to improve the knowledge of citizen’s to safeguard their rights under Component 2. Assessments, however, demonstrated the importance of broader communication efforts under the revised components as being critical to build and sustain reform 67      momentum, prompting the Government to seek the Bank’s co-financing for this purpose. The introduction of broader communication activities to the Project description required in turn adding non-consultant services (defined as printing, reproducing, publishing and disseminating information materials) as an expenditure category. 5. The restructuring also added people with disabilities to the excluded groups targeted by Grants for the development of outreach services. The original description of Component 2 included technical assistance to reform and evaluate the financing and delivery of services for disabled people, including Serbia’s Disability Fund. It also included Grants to develop outreach services to excluded groups, explicitly mentioning Roma, internally displaced persons and refugees---but not people with disabilities. However, the reform of Serbia’s Disability Fund, supported by the Project, increased access and, thus, demand of local organizations for Grant funding. Moreover, assessments demonstrated the results of such local initiatives. Both the increase demand and the demonstrated results prompted the Government to scale up the Fund’s grant scheme, and to seek Bank financing for it under the Project. 6. The October 30, 2013 Level II Project Restructuring introduced changes to Component 3 to reflect the inclusion of Project support to the GoS’ reform on centralized procurement of pharmaceuticals and efficiency improvement. Table 2: Original and Revised Components and Project Description Original Revised Original Description Revised Description after June Components  Components  2012 and October 2013 Restructurings (the latter in bold) Component 1: Component 1: Development of a public Development of a public financing Making Fiscal Transform financing framework that framework that increases efficiency Decentralization Financing increases efficiency while while compensating for inequities Work  Models  compensating for inequities across municipalities and promoting across municipalities and the process of rationalization of the promoting the process of service network at the local level by rationalization of the service means of investments in good, network at the local level, technical assistance, training and non- through the implementation of consultant services as follows: the Law on Local Self Government Finance by means of investments in goods, technical assistance, training and non-consultant services as follows: (a) Development of new “funds- (a) Development of new follow-the-user” financing “funds-follow-the-user” mechanisms and a financing formulae and a financing framework with the Borrower’s line framework with the Borrower’s ministries and LSGs, including line ministries and LSGs, “compensating weights” to correct including “compensating inequalities across municipalities so weights” to correct inequalities funds are allocated to municipalities across municipalities so funds in a transparent, rational and are allocated to municipalities predictable way consistent with their in a transparent, rational and new service delivery responsibilities. predictable way consistent with their new service delivery responsibilities. (b) Provision of training for all (b) Support for the design and sector actors at the State, LSG implementation of communication and service provider levels in campaigns and training for all sector 68      Original Revised Original Description Revised Description after June Components  Components  2012 and October 2013 Restructurings (the latter in bold) application of the new actors at the state, LSG and service financing framework, including provider levels in support of the provision of training to LSGs application of the new financing and service providers in order mechanisms and financial framework, to strengthen their ability to use including provision of training to the money they receive from LSGs and service providers in order various sources to improve the to strengthen their ability to use the quality and efficiency of money they receive from various service delivery. sources to improve the quality and   efficiency of service delivery.   Component 2: Component 2: Provision of goods, technical Provisions of goods, technical Improving Improve access assistance, training and assistance, training, non-consultant outreach and and quality  financing of Grants to services and financing of Grants to access through Beneficiaries to strengthen their Beneficiaries to strengthen their development and ability in identifying new ability in identifying new approaches expansion of approaches and models for and models for delivering services, innovations in delivering services, and and including the following: service delivery  including the following: (a) Provision of Grants to (a) Provision of Grants to Beneficiaries to develop Beneficiaries to develop outreach outreach services to exclude services to excluded groups (such as groups (such as Roma, Roma, internally displaced persons, internally displaced persons and refugees, and people with disabilities. refugees). (b) Reform the financing and (b) Reform the financing and delivery of services for disabled delivery of services for disabled people including: (i) provision people including: (i) provision of of technical assistance to technical assistance to MOLEVSP, MOLEVSP, de-concentrated de-concentrated centers for social centers for social work and work and local authorities in LSGs local authorities in LSGs responsible for social services to responsible for social services disabled people; (ii) provision of to disabled people; (ii) technical assistance to review the provision of technical operational structure of, and reform assistance to review the the Disability Funds with a view to operational structure of, and adopting a competitive project-based reform the Disability Fund with financing mechanism, and financing a view to adopting a Grants for said entities to bridge the competitive project-based transition from incremental to said financing mechanism, and project-based financing mechanism; financing Grants for said and (iii) carrying out evaluations of entities to bridge the transition the Disability Fund’s operation and from incremental to said its activities. project-based financing mechanism; and (iii) carrying out rigorous impact evaluation of activities supported by the Fund. (c) Expand LSG’s capacity to (c) Expand the Borrower’s address the specific needs of ministries’ and the LSGs’ capacity to excluded vulnerable groups, jointly address the specific needs of 69      Original Revised Original Description Revised Description after June Components  Components  2012 and October 2013 Restructurings (the latter in bold) including the capacity to excluded vulnerable groups, identify and implement including the capacity to identify and alternative service delivery implement alternative service arrangements and collaborative delivery arrangements and approaches. collaborative approaches. (d) Improve knowledge and (d) Improve knowledge and establish establish institutions to institutions to safeguard citizens’ safeguard citizens’ rights, rights, including the design and including the development of implementation of communication information material and campaigns and safeguard institutions safeguard institutions and and mechanisms as required mechanisms as required therefore. therefore.     Component 3: Component 3: Support for the institutional Support to the Borrower’s ministries, Supporting a new Improve shift from direct provision to a LSGs and other local service regulatory, accountability new regulatory, oversight and providers to assume their new roles oversight and and quality  quality assurance role for the and responsibilities, including quality assurance Borrower’s ministries, through regulation, oversight, quality roles for the capacity building measures, assurance, management, planning, Borrower’s including staff training, the budgeting and service delivery Ministries  development of new protocols through capacity building measures, and tools for monitoring and including the design and evaluation, encompassing the implementation of protocols, tools provision of training, technical and measures to develop knowledge assistance and goods for the and skills, encompassing training, following: technical assistance, non-consultant services, goods, and the financing of Grants for the following: (a) Development and support of (a) Development and support of the the training required for: (i) the training required for: (i) the Borrower’s ministries to shift to Borrower’s ministries to shift to a a regulatory, oversight and regulatory, oversight and quality quality assurance role, assurance role, development, through development, through consultation with stakeholders, of consultation with stakeholders, appropriate regulatory frameworks of appropriate regulatory across a number of areas (such as frameworks across a number of standards for preschools and training areas (such as standards for providers), and preparation and preschools and training publication of reports of service providers), and preparation and delivery at the State, LSG and service publication of reports of service provider levels, and (ii) for the LSGs delivery at the State, LSG and to develop competencies in the service provider levels. delivery of services being decentralized, with a view to enhancing transparency and good governance at the LSG level. (b) Definition of regulations and (b)Definition of regulations and standards for local service delivery, standards for local service including the specification of the new 70      Original Revised Original Description Revised Description after June Components  Components  2012 and October 2013 Restructurings (the latter in bold) delivery, including the roles and responsibilities of specification of the new roles municipal staff and service providers and responsibilities of in the decentralized system and municipal staff and service creation of mechanisms to foster and providers in a decentralized assure quality, including developing system. systems of accreditation, licensing and accountability to enable line ministries and municipalities to monitor service delivery, to ensure compliance with procedures and standards as well as accountability in management and service delivery. (c) Support for MoE-supported Grant mechanisms aiming at promoting locally-inspired quality improvements, through the financing of Grants to schools, LSGs and education service providers, and the provision of technical assistance to regional school administrations to further enhance their capacity to work with schools on their SDPs. (d)Improvement in the use of information, statistics and Information Communications and Technology (ICT), through the coordination of existing ICT strategies in each of the education, health and social protection sectors, the design and implementation of mechanisms to enable existing data bases to share data in real time and adding new data as future needs arise, including investments needed to improve: (i) connectivity of local service providers (PHCs, schools, decentralized centers for social work, and other social protection institutions) to the internet and national data networks, and (ii) ICT, and provision of technical assistance on the legal and privacy aspects of administrative use of information on (c) Development of citizens with a view to ensure mechanisms to reinforce the compliance with EU information and Borrower’s capacity in ensuring privacy standards. quality improvements to service delivery through: (i) piloting, (e) Development of mechanisms to monitoring, impact evaluation reinforce the Borrower’s capacity in and performance analysis to ensuring quality and efficiency ensure the effectiveness of improvements to service delivery specific Project investments through: (i) piloting, monitoring, and to support evidence-based impact evaluation and performance decision making and analysis to ensure the effectiveness of 71      Original Revised Original Description Revised Description after June Components  Components  2012 and October 2013 Restructurings (the latter in bold) accountability, including the specific project investments and to provision of Grants to local support evidence-based decision service providers undertaking making and accountability, including these monitoring and evaluation the provision of Grants to local activities; (ii) conducting service providers undertaking these impact evaluations and surveys monitoring and evaluation activities; of decentralization in the (ii) conducting impact evaluations education, health and social and surveys of decentralization in the protection sectors; and (iii) education, health and social strengthening the accountability protection sectors; and (iii) system in the education sector strengthening the accountability so as to enable the Borrower to system in the education sector so as participate in international to enable the Borrower to participate assessments of student in international assessments of performance, conduct research student performance and teaching on student and school and learning practices in schools, performance, and carry out conduct research on student and external evaluations of schools. school performance, and carry out external evaluations of schools; and (d) Provision of Grants to carry (iv) strengthening capacity for out minor civil works for the efficient pharmaceutical renovation of PHCs. procurement.   (f) Provision of Grants to carry out minor civil works for the renovation of PHCs and schools.  Component 4:   Carrying out of a program to Incorporated under Component 3 Improving ensure that staff in both the (above).  capacity of LSGs LSGs and other local service and other local providers have the information public institutions and communication technology as service tools, knowledge and skills providers  required to make decisions in management strategic planning, needs assessment and budgeting, service delivery and monitoring and evaluation, and provision of investments needed to improve: (i) connectivity of local service providers (PHCs, schools and decentralized centers for social work) to the internet and national data networks, and (ii) information and communication technology, through the provision of technical assistance, goods, training and Grants to schools, PHCs and municipalities, as follows: (a) Creation of mechanisms to foster and assure quality, including developing systems of accreditation, licensing and 72      Original Revised Original Description Revised Description after June Components  Components  2012 and October 2013 Restructurings (the latter in bold) accountability to enable line ministries and municipalities to monitor service delivery, and to ensure compliance with procedures and standards as well as accountability in management and service delivery. (b)Support to capacity building tailored to LSG needs, consisting of the development and delivery of training programs to build competencies required for delivery of the services being decentralized, with a view to enhancing transparency and good governance at the LSG level. (c)Support for MoE-supported Grant mechanisms aiming at promoting locally-inspired quality improvements, through the financing of Grants to schools, LSGs and education service providers, and the provision of technical assistance to regional school administrations to further enhance their capacity to work with schools on their SDPs. (d)Improvement in the use of information, statistics and ICT through the coordination of existing ICT strategies in each of the education, health and social protection sectors, and the design of a mechanism to enable existing data bases to share data in real time and adding new data as future needs arise, and provision of technical assistance on the legal and privacy aspects of administrative use of information on citizens with a view to ensure compliance with EU information and privacy standards. (e) Provision of Grants to carry out minor civil works for the renovation of schools. 73      Original Revised Original Description Revised Description after June Components  Components  2012 and October 2013 Restructurings (the latter in bold) Component 5: Component 4: Provision of technical No change  Support for Support for assistance, training and goods Project Project to the Fiduciary Services Unit Implementation  Implementation  and to the Borrower’s Project Administration Teams (PATs) responsible for project implementation in the line ministries, including the preparation of the audit. 74      Annex 12. Project Results, as per the Amended Results Framework (June 26, 2012 and October 30, 2013) Project Development Objective: Increase the capacity of institutional actors and beneficiaries in order to improve access to and the efficiency, equity and quality of local delivery of health, education and social protection services in a decentralizing environment. PDO Level Results Indicators  Baseline Target Value Actual Value Year 4 March 2015 (2012)    Indicator One: PHC providers paid Target Achieved: PHC Primary health care financing PHC providers paid according to output- providers paid according to allocated according to only by salary  based formula  output-based formula  capitation- and output-based formula  Indicator Two: Piloting completed in Target Not Achieved: Central and local per capita Non-existent  15 municipalities  Theoretical piloting started funding formulae in the but actual piloting was not education sector piloted  conducted in any of the municipalities  Indicator Three: Percent of children from vulnerable groups in project Target Partially Achieved: schools (vulnerable groups 3.2% - 3.6%  7.5%  6.56% (the actual value according to OECD achieved reached 8.44% in classification: Category A- 2010/11; the final actual children with disabilities; B- value achieved is in children with learning comparison with 4.11% for difficulties; C-children from non-grant schools)  socio-economically disadvantaged groups)  Indicator Four: Percent of MOLEVSP financing allocated for 20%  100%  Target Achieved: 100%  Disabled Peoples Organizations (DOPs), allowing for equal access and improved transparency and based on results  Indicator Five: Target Achieved (and Number of Roma children 0%  18,795  exceeded): 30,018 children receiving vaccinations through vaccinated  the Roma health mediators program  Indicator Six: Target Achieved (and Percent of primary health care 0%  25%  exceeded): 39.62%  centers receiving at least a 3- year certificate of accreditation  Intermediate Results:  Component One: Transform Financing Models Intermediate Indicator One: No legislative Law passed and by- Target Achieved: Law Legislative framework allows framework  law adopted to enable passed and by-aw adopted to for capitation- and output- capitation  enable capitation  based formula  Intermediate Indicator Two: No clear Framework with Target Achieved: Improved allocation framework methodology and methodology and Framework with for equalization funds criteria for criteria for allocation methodology and criteria for developed  75      equalization fund of equalization funds allocation of equalization allocation developed developed  Intermediate Indicator Three: Target Partially Achieved: Central and local formulae Non-existent  Formulae developed  Central formulae nearly developed  completed, local formulae development initiated Component Two: Improve access and quality Intermediate Indicator Four: Target Achieved (and Number of educational 0 EIs for schools 37 EIs for schools exceeded): institutions (schools and without violence without violence preschools) by type of grants 560 schools: o/w received  0 EIs for inclusive 330 EIs for inclusive 37 EIs for schools without education education violence 0 EIs for inclusion 196 EIs for inclusion 330 EIs for inclusive of Roma children  of Roma children  education 193 EIs for inclusion of Roma children  Intermediate Indicator Five: Target Achieved: Number of medical staff and 0  2,000  2000  associates trained to recognize needs of vulnerable groups  Component Three: Improve accountability and quality Intermediate Indicator Six: Target Achieved: Percent of primary health care 0  50%  51%  centers that have completed quality accreditation process  Intermediate Indicator Seven: Target Achieved: Number of PHC centers that 0  50  50  have adopted clinical pathways  Intermediate Indicator Eight: Achieved (and exceeded); Number of schools that have 0  60  100 schools   undergone a school performance external evaluation  Intermediate Indicator Nine: Target Achieved: Number of training participants 0  23,000 participants 23,387  among education staff that (14,000 staff)  have undergone development training organized by the project  Intermediate Indicator Ten: Target Achieved: Number of inter-sectoral 0  150  150  committees trained  Intermediate Indicator Eleven: Target Achieved (and Percent of PHC centers with 0  85%  exceeded): fully operational HMIS 95.7%  platform at the PHC level  Intermediate Indicator Twelve: MIS established and Target Partially Achieved: Central management Non-existent  introduced in all MIS system developed  information system (MIS) fully Social Protection operational in all social institutions  protection institutions  Intermediate Indicator Target Achieved: Thirteen: 0  600 (2 to 3 persons in 600  Number of staff in social across all locations in protection institutions trained the country) 76      and certified in the use of the Centralized MIS  Intermediate Indicator Health Insurance Target achieved: Fourteen: None  Fund awards Health Insurance Fund Centralized procurement of framework awarded framework pharmaceuticals initiated  agreements for at agreements for the 50 most least 50 drugs from frequently dispensed drugs the B-list from the B-list  77      Annex 13. Project Results Framework, as per Project Appraisal Document PDO Level Results Indicators  Baseline Target Value Year 4 (December 31, 2012)  Indicator One: Not available Not available Financing for decentralized health and education from central government to LSGs is allocated according to the Law on Self Government Financing, in municipalities benefitting from the Project.  Indicator Two: Not available Not available Percentage of grants awarded by the Project to support innovation in inclusion, that have achieved success in increasing inclusion of marginalized groups (proxied by poor households) to health, education and social protection services.  Indicator Three: Not available Not available Share of MOLEVSP financing for disabled groups that is allocated according to a new program-based model of allocation.  Indicator Four: Not available Not available Improved satisfaction with quality of service delivery among users of health facilities, in municipalities benefited by the Project (and where parallel interventions similar to the DILS Project have been launched), compared to other municipalities.  Indicator Five: Not available Not available Number of local service providers who have gone through a quality accreditation process, approved by the relevant institution in their sector.  Indicator Six: Not available Not available Establishment of information management systems linking local service providers with the relevant line ministry in their sector, and where relevant to the successful delivery of services, that allow information sharing across sectors.  Intermediate Results: Component One: Making Fiscal Decentralization Work Intermediate Indicator One: % of treatment municipalities making 0%  75%  financial allocations based on agreed formula in health sector  Intermediate Indicator Two: 70% of population in municipalities % of population registered with their None  covered by the Project  chosen doctor, in treatment municipalities  Intermediate Indicator Three: Develop legislation for per capita Per-capita financing formula developed None  financing in education  for education  Component Two: Improving Outreach and Access through Development and Expansion of Innovations in Service Delivery Intermediate Indicator Four: By the end of the Project, at least 60 60% of grants completed in previous 12 percent of grants provided for inclusion None months are successful  78      activities to marginalized groups in project municipalities are implemented successfully  Intermediate Indicator Five: Operational structure of the MOLEVSP 20% of the DF budget allocated 60% of the DF budget allocated for Disability Fund restructure to improve its for results-oriented projects  results-oriented projects  effectiveness  Intermediate Indicator Six: Model for allocating MOLEVSP None  Model put in place  financing for disability services, developed and introduced in # of project municipalities  Component Three: Supporting a New Regulatory, Oversight and Quality-Assurance Role for State Level Ministries Intermediate Indicator Seven Quality guidelines developed for No quality guidelines Quality guidelines for decentralized improved local service delivery  established services formulated (by 2009)    Intermediate Indicator Eight: At least 60 percent of grants awarded by No quality guidelines 60% project to support implementation of new established    quality guidelines for local service delivery providers are implemented successfully  Intermediate Indicator Nine: Results of impact evaluations carried out Limited capacity for impact Pilot results evaluated, and results to measure the quality and effectiveness evaluation  disseminated  of the decentralization of services are disseminated to the relevant stakeholders  Component Four: Improving Capacity of LSGs and other Local Public Institutions as Service Providers Intermediate Indicator Ten: By the end of the Project, at least 60 No grants awarded 60% of grants completed in previous 12 percent of schools who receive grants are   months are successful  satisfactorily meeting the objectives in their grant proposals  Intermediate Indicator Eleven: On average 50% more staff in Increase percentage of staff in education education institutions in institutions that are undergoing participating municipalities development training in participating undergo development training 50%  municipalities, compared to other compared to other municipalities  municipalities Intermediate Indicator Twelve: 90% of schools are using the 100% of schools using the national Increased connectivity to and usage of national Education Information Education Information System  national information management System networks, by local service providers  LSGs using “offline” information management systems 25 CSWs operate local information management systems No national Social Protection information system management in place 79