Document of The World Bank Report No: ICR00001629 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-79480) ON A LOAN IN THE AMOUNT OF US$220 MILLION TO THE UNITED MEXICAN STATES FOR A SCHOOL-BASED MANAGEMENT PROJECT IN SUPPORT OF THE SECOND PHASE OF THE SCHOOL-BASED MANAGEMENT PROGRAM January 29, 2015 Education Global Practice Latin America and the Caribbean Region CURRENCY EQUIVALENTS (Exchange Rate Effective January 26, 2015) Currency Unit MX$1.00 = US$0.07 US$1.00 = MX$14.63 FISCAL YEAR January 1 – December 31 ACRONYMS AEF Federal Education Authority (Autoridad Educativa Federal) AEL Local Education Authority (Autoridad Educativa Local) AGE Support for School Management (Apoyos a la Gestión Escolar) APL Adaptable Program Loan CEMI Evaluation, Monitoring and Information Unit at the Directorate of School Management (Coordinación de Evaluación, Monitoreo e Información de la DGAG) CEPS School Council of Social Participation (Consejos Escolares de Participación Social) CONAFE National Council for Education Development (Consejo Nacional de Fomento Educativo) CONAPASE National Council of Social Participation in Education (Consejo Nacional de Participación Social en la Educación) CONEVAL National Council for the Evaluation of Social Development Policy (Consejo Nacional de la Evaluación de la Política de Desarrollo Social) CPS Country Partnership Strategy CTE Technical School Council (Consejo Técnico Escolar) CTPS Social Participation Technical Council (Consejo Técnico de Participación Social) DGAG Directorate of School Management (Dirección General Adjunta de Gestión) DGDGIE General Directorate of School Management Development and Education Innovation (Dirección General de Desarrollo de la Gestión e Innovación Educativa) DGE General Directorate of Evaluation (Dirección General de la Evaluación) ENLACE National Evaluation of Academic Achievement in Schools (Evaluación Nacional del Logro Académico de Centros Escolares) FEEC State Trust Funds (Fideicomisos Estatales de Escuelas de Calidad) FM Financial Management FNEC National Trust Fund (Fideicomiso Nacional de Escuelas de Calidad) FY Fiscal Year GDP Gross Domestic Product GoM Government of Mexico HDI Human Development Index i IBRD International Bank for Reconstruction and Development IDA International Development Association IERR Internal Economic Rate of Return IMF International Monetary Fund INEE National Institute for the Evaluation of Education (Instituto Nacional para la Evaluación de la Educación) INEGI National Institute of Statistics and Geography (Instituto Nacional de Estadística y Geografía) IPP Indigenous Peoples Plan (Plan de Pueblos Indígenas) LA Loan Agreement LAC Latin America and the Caribbean M&E Monitoring and Evaluation MGEE Strategic Model of School Based Management (Modelo de Gestión Escolar con Enfoque Estratégico) MIS Management Information System NAFIN National Financial Development Bank (Nacional Financiera, S.N.C., I.B.D) NDP National Development Plan (Plan Nacional de Desarrollo) OECD Organization for Economic Co-operation and Development PAD Project Appraisal Document PAT Annual Work Plan (Programa Anual de Trabajo) PDO Project Development Objective PEC Quality Schools Program (Programa Escuelas de Calidad) PETC Full Time Schools Program (Programa Escuelas de Tiempo Completo) PETE Strategic School Transformation Plan (Plan Estratégico de Transformación Escolar) PISA Program for International Student Assessments PRF Project Results Framework PSE Education Sector Program (Programa Sectorial de Educación) SBM School Based Management SEB Subsecretariat of Basic Education (Subsecretaría de Educación Básica) SEP Ministry of Public Education (Secretaría de Educación Pública) SHCP Ministry of Finance and Public Credit (Secretaría de Hacienda y Crédito Público) SIGED Mexico’s Education Management Information System (Sistema de Información y Gestión Educativa) SIPEC Information System for the Quality Schools Program (Sistema de Información del Programa Escuelas de Calidad) SPD Professional Teaching Service (Servicio Profesional Docente) SNTE National Teacher’s Union (Sindicato Nacional de Trabajadores de Educación) Vice President: Jorge Familiar Country Director: Gerardo M. Corrochano Practice Manager: Reema Nayar Project Team Leader: Rafael E. De Hoyos Navarro ICR Team Leader Rafael E. De Hoyos Navarro ii UNITED MEXICAN STATES SCHOOL-BASED MANAGEMENT PROJECT IN SUPPORT OF THE SECOND PHASE OF THE SCHOOL-BASED MANAGEMENT PROGRAM CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design ............................................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................................. 4 3. Assessment of Outcomes ............................................................................................................ 9 4. Assessment of Risk to Development Outcome ......................................................................... 17 5. Assessment of Bank and Borrower Performance ..................................................................... 17 6. Lessons Learned........................................................................................................................ 19 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners........................... 20 Annex 1. Project Costs and Financing .......................................................................................... 21 Annex 2. Outputs by Component.................................................................................................. 22 Annex 3. Economic and Financial Analysis ................................................................................. 25 Annex 4. Bank Lending and Implementation Support/Supervision Processes............................. 27 Annex 5. Beneficiary Survey Results ........................................................................................... 29 Annex 6. Stakeholder Workshop Report and Results ................................................................... 30 Annex 7. Summary of Implementing Agency’s ICR.................................................................... 31 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................................... 41 Annex 9. List of Supporting Documents ...................................................................................... 42 MAP .............................................................................................................................................. 45 iii A. Basic Information School Based Country: Mexico Project Name: Management (APL II) Project ID: P115347 L/C/TF Number(s): IBRD-79480 ICR Date: 12/15/2014 ICR Type: Core ICR GOVERNMENT OF Lending Instrument: APL Borrower: MEXICO Original Total USD 220.00M Disbursed Amount: USD 220.00M Commitment: Revised Amount: USD 220.00M Environmental Category: C Implementing Agencies: Ministry of Public Education (SEP) Cofinanciers and Other External Partners: B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 05/27/2009 Effectiveness: 12/20/2010 12/20/2010 04/25/2013 Appraisal: 02/22/2010 Restructuring(s): 09/20/2013 06/26/2014 Approval: 06/17/2010 Mid-term Review: 05/17/2012 05/17/2012 Closing: 06/30/2013 06/30/2014 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Low or Negligible Bank Performance: Moderately Satisfactory Borrower Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Moderately Quality at Entry: Government: Satisfactory Unsatisfactory Implementing Quality of Supervision: Moderately Satisfactory Moderately Satisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Moderately Satisfactory Performance: Performance: iv C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments (if Indicators Rating Performance any) Potential Problem Project Quality at Entry No None at any time (Yes/No): (QEA): Problem Project at any Quality of Supervision No None time (Yes/No): (QSA): DO rating before Moderately Closing/Inactive status: Satisfactory D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Pre-primary education 30 30 Primary education 60 60 Secondary education 10 10 Theme Code (as % of total Bank financing) Education for all 30 30 Managing for development results 30 30 Participation and civic engagement 40 40 E. Bank Staff Positions At ICR At Approval Vice President: Jorge Familiar Pamela Cox Country Director: Gerardo M. Corrochano Gloria M. Grandolini Practice Manager: Reema Nayar Chingboon Lee Project Team Leader: Rafael E. De Hoyos Navarro Ricardo Rocha Silveira ICR Team Leader: Rafael E. De Hoyos Navarro Abril Ibarra ICR Primary Author: Vicente Garcia F. Results Framework Analysis Project Development Objectives (from the Loan Agreement) The objective of the Project is to strengthen PEC by increasing overall coverage and social participation in Eligible Schools while putting greater emphasis on public marginalized schools and on the indigenous population, as well as a reorientation of the direct support to schools to improve public schools’ internal efficiency and learning outcomes. v Revised Project Development Objectives (as approved by original approving authority) N/A (a) PDO Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Indicator 1 : Percentage of basic and special education schools participating in PEC Value 18 (original) (quantitative 22.00 23.00 23.85 20.30 (revised) or Qualitative) Date achieved 06/30/2010 06/30/2013 06/30/3014 06/30/2014 Target exceeded. Actual value reflects data from 2012-13 school year. Indicator revised in the April 2013 restructuring. The original wording of this indicator was “number of schools participating in PEC as percentage of the total number of basic education Comments schools.” In addition, the original Project Results Framework (PRF) had an indicator (incl. % related to the “total number of basic education schools participating in PEC which was achievement) expected to increase 25 percent by 2013 (from 39,180 in 2009 to around 50,000 in 2013).” This target was met since the total number of PEC schools was 50,624 in the school year 2013-14 (in 2009, the baseline was revised to 40,790). Indicator 2 : Percentage of school community members who participate in the design of the PETE Value (quantitative 74.20 80.00 75.00 72.70 or Qualitative) Date achieved 06/30/2010 06/30/2013 06/30/2014 06/30/2014 Target not met. Indicator revised in the April 2013 restructuring. The original wording of this indicator was “commitment to the goals and activities of the school plan (PETE) Comments as expressed by parental participation in PETE design and adjustment.” The revised (incl. % target aimed to keep parental participation constant (with an aimed increase of 0.8 achievement) percent). Although the target was not met, the 1.5 percent decline from the baseline was not statistically significant, indicating that parental participation was maintained. Actual value reflects data from 2012-13 school year. Indicator 3 : Percentage of PEC schools in highly marginalized and marginalized areas Value 40.00 (original/preliminary) quantitative or 50.00 55.00 51.35 38.4 (revised/final) Qualitative) Date achieved 06/30/2010 06/30/2013 06/30/2014 06/30/2014 Original target exceeded and 93 percent achieved against revised target. Chiapas and Veracruz (states with high percentage of schools located in highly marginalized and marginalized areas) were not able to participate in the last year of the Project because they did not provide their counterpart funding conditioning the delivery of the federal Comments funds. However, the overall number of schools located in highly marginalized areas that (incl. % receive support to implement their school improvement plans in Mexico has increased achievement) with the new targeting and approach of programs such as Programa Escuelas de Tiempo Completo (PETC) which, among others, now provide direct support to schools targeting those in marginalized and highly marginalized areas, following the PEC school based management (SBM) model. Although the disaggregated number of PETC schools located vi Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years in highly marginalized and marginalized areas is not yet available for the school year 2013-14, the overall number of PETC schools expanded from 6,708 in 2012-13 to 15,349 in 2013-14 and up to 23,182 to date. Indicator 4 : Percentage of Indigenous modality schools participating in PEC Value 7.0 (original) 13.5 (original) (quantitative 15.00 19.30 15.40 (revised) 18.00 (revised) or Qualitative) Date achieved 06/30/2010 06/30/2013 06/30/2014 06/30/2014 90 percent of original target met and revised target 93 percent met. Indicator revised in the April 2013 restructuring. The original wording of this indicator was “percentage of PEC schools with indigenous students,” which accounted for schools with one or more indigenous students (actual original reflects data from 2011-12 school year). Chiapas and Veracruz (states with a high percentage of schools located in indigenous communities) were not able to participate in the last year of the Project because they did not provide Comments their counterpart funding conditioning the delivery of the federal funds. However, the (incl. % overall number of schools located in indigenous communities that receive support to achievement) implement their school improvement plans in Mexico has increased with the new targeting and approach of programs such as PETC which, among others, now provide direct support to indigenous schools following the PEC SBM model. Although the disaggregated number of PETC schools located in highly marginalized and marginalized areas is not yet available for the school year 2013-14, the overall number of PETC schools expanded from 6,708 in 2012-13 to 15,349 in 2013-14 and up to 23,182 to date. Indicator 5 : Percentage of school grants spent on education quality inputs Value quantitative or 59.70 N/A 66.86 74.54 Qualitative) Date achieved 06/30/2010 6/30/2014 06/30/2014 Comments (incl. % Target exceeded. Indicator added in the April 2013 restructuring. achievement) (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised Target approval Completion or Values documents) Target Years Indicator 1 : Percentage of community members that are familiar with PETE Value (quantitative 76.80 82.00 84.00 83.50 or Qualitative) Date achieved 06/30/2010 06/30/2013 06/30/2014 06/30/2014 Comments Original target exceeded and formally revised target 99 percent achieved. This indicator (incl. % was revised in the April 2013 restructuring. The original wording for this indicator was achievement) vii Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised Target approval Completion or Values documents) Target Years “Percentage of community members that know the PETE.” Actual value reflects data from 2012-13 school year. Percentage of community members that observe that parents are informed about student Indicator 2 : performance. Value (quantitative 99.00 99.00 99.80 or Qualitative) Date achieved 06/30/2010 06/30/2013 06/30/2014 Comments (incl. % Target exceeded. Actual value reflects data from 2012-13 school year. achievement) Percentage of community members that observe participatory decision-making between Indicator 3 : parents, teachers and principal about school affairs Value (quantitative 90.20 93.00 91.00 93.50 or Qualitative) Date achieved 06/30/2010 06/30/2013 06/30/2014 06/30/2014 Comments (incl. % Targets exceeded. Actual value reflects data from 2012-13 school year. achievement) Percentage of community members that observe teachers encouraging and supporting Indicator 4 : student performance Value (quantitative 72.10 77.00 78.00 75.20 or Qualitative) Date achieved 06/30/2010 06/30/2013 06/30/2014 06/30/2014 Comments Original target 98 percent achieved and revised target 96 percent achieved. Actual value (incl. % reflects data from 2012-13 school year. achievement) Percentage of community members that observe teachers encouraging the active Indicator 5 : participation of the students Value (quantitative 66.90 72.00 74.00 91.50 or Qualitative) Date achieved 06/30/2010 06/30/2013 06/30/2014 06/30/2014 Comments (incl. % Targets exceeded. Actual value reflects data from 2012-13 school year. achievement) Indicator 6 : Total hours of technical assistance provided by the CNEPEC to the CGEPECs per year Value (quantitative 732 970 3,748 or Qualitative) Date achieved 06/30/2010 06/30/2014 06/30/2014 viii Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised Target approval Completion or Values documents) Target Years Comments Target exceeded. Indicator added in the April 2013 restructuring to measure technical (incl. % assistance given to States in the form of capacity building to follow PEC Operating Rules achievement) and prioritize schools in indigenous and marginalized areas. Percentage of financed PEC schools that report financial information on time to PEC's Indicator 7: National Management Information System Value 24 states (original) 31 (original) (quantitative 31 states 94.40 88.40 (revised) 74.90 (revised) or Qualitative) Date achieved 06/30/2010 06/30/2013 06/30/2014 06/30/2014 Original target achieved. Revised target 85 percent met. Actual value reflects preliminary Comments data for the 2012-13 school year. Revised in the April 2013 restructuring. The original (incl. % wording of this indicator was “Percent of states that report on time the information of the achievement) Program through a certified MIS.” Indicator 8 : Dissemination of the simplified school planning instrument Value Unsimplified school (quantitative Dissemination Dissemination planning instrument or Qualitative) Date achieved 06/30/2010 06/30/2014 06/30/2014 Target met. Indicator added in the April 2013 restructuring. The dissemination of the Comments simplified school planning instrument was implemented through different strategies, (incl. % including the distribution of the new format to schools and the capacity building of achievement) relevant authorities at the state level on the use of the tool to enable them to train their schools. Indicator 9: Dissemination of PEC's objectives, activities and results Publication of EDUCARE Continued Continued Value magazine; design and implementation implementation of (quantitative launch of PEC volumes, of communications or Qualitative) bulletins and informative communication strategy notes s strategy Date achieved 06/30/2010 06/30/2014 06/30/2014 Target met. Indicator added in the April 2013 restructuring. The Ministry of Public Comments Education (Secretaría de Educación Pública, SEP) implemented a comprehensive (incl. % communications campaign that included, among others, the constant updating and achievement) upgrading of the program’s webpage and the production and transmission of several TV programs related to PEC. ix G. Ratings of Project Performance in ISRs Date ISR Actual Disbursements No. DO IP Archived (USD millions) 1 07/30/2010 Satisfactory Satisfactory 0.00 2 02/23/2011 Satisfactory Satisfactory 44.00 3 08/15/2011 Satisfactory Satisfactory 61.27 4 03/25/2012 Satisfactory Satisfactory 114.31 5 10/15/2012 Satisfactory Satisfactory 114.31 6 06/30/2013 Satisfactory Satisfactory 187.96 7 01/08/2014 Satisfactory Satisfactory 218.73 8 07/10/2014 Moderately Satisfactory Moderately Satisfactory 219.45 H. Restructuring (if any) ISR Ratings at Amount Board Restructuring Restructuring Disbursed at Reason for Restructuring & Key Approved PDO Date(s) Restructuring Changes Made Change DO IP in USD millions Dropped Component 3 because most activities were not critical to the achievement of the PDO and were supported outside the Project. Revised Component 2 to include the impact evaluation previously under Component 3, the provision of technical assistance to states to improve their prioritization of marginalized and indigenous schools, and the carrying out of an M&E survey to collect data from 04/25/2013 S S 187.96 community members on the SBM model in PEC schools, among others. The restructuring also adjusted the results framework to ensure full measurement of the achievement of the PDO. A reallocation of part of the loan proceeds to increase the number of beneficiary schools and an extension of the closing date by one year were also part of this restructuring. The only change was a reallocation of all funds (US$500,000.00) from 09/20/2013 S S 187.96 Category 3 (Consultant Services) to Category 1 (Goods, Works and Consultant's Services supported x ISR Ratings at Amount Board Restructuring Restructuring Disbursed at Reason for Restructuring & Key Approved PDO Date(s) Restructuring Changes Made Change DO IP in USD millions under direct support to schools). This change was needed because SEP decided to finance the impact evaluation of PEC with its own resources. The Bank continued to oversee the quality of this impact evaluation. This reallocation also allowed assigning more direct support to schools. The only change was a reallocation of US$93,331.55 from Category 2 (Incremental Project Costs) to Category 1 (Goods, Works and Consultants' Services supported under the direct support to schools). 06/26/2014 S S 219.45 The change was necessary, given SEP's decision to finance the remaining operating costs with its own resources. The reallocation also allowed assigning more direct support to schools. I. Disbursement Profile xi 1. Project Context, Development Objectives and Design A. Country and sector issues 1.1 Context at Appraisal 1. Improving basic educational attainment and learning outcomes for all was, and still is, central to improving Mexico’s competitiveness in the global economy and to decrease inequality. At the time of the Project’s Appraisal (May 2010), Mexico was the world’s fourteenth largest economy, yet it was ranked 60th in terms of global competitiveness, one of the lowest for an OECD country and below several other countries in the region (Global Competitiveness Report 2009-10). Mexico’s labor productivity was low compared to other OECD countries and in 2011 was even below its level in 1995. Therefore, Mexican labor was not able to compete with low cost producers in Asia and Central America. Mexico thus needed to focus more on moving from a low- cost labor-based economy to a knowledge-based economy. In addition, Mexico ranked 56th in the Human Development Index (HDI), the second lowest score among OECD countries, and had a significant percentage of its population living in moderate and extreme poverty in 2010: 51 and 19 percent, respectively. 2. The quality of education faced important challenges. Mexico’s performance in the Program for International Student Assessment (PISA) was the lowest among OECD countries. In PISA 2009 (with an average of 500 points and a standard deviation of 100), Mexico achieved a greater total of 425 points in reading, 419 points in math, and 416 points in science. By comparison, the United States, one of Mexico’s main commercial competitors, performed 69 points higher in math. Learning outcomes, as measured by the national standardized test ENLACE (Evaluación Nacional del Logro Académico en Centros Escolares), were also low, especially among the poor. Although test scores have generally increased over time for all students (including indigenous and disadvantaged students), consistent disparities remain. Students from public schools performed lower than those from private schools by 0.6 standard deviations (60 points) in ENLACE 2010 (with an average of 500 points and a standard deviation of 100), with this gap increasing for indigenous and disadvantaged students. In addition, while educational enrollments were increasing, those that remained outside of the system were mainly poor or indigenous. For instance, the percentage of net enrollment in 9th grade in 2010 was 87 percent; however, the net enrollment for the same grade was only 75.30 percent for students living in highly disadvantaged localities and 85 percent for indigenous students. 3. To address these challenges, the Alliance for Education Quality (La Alianza para la Calidad de la Educación) was created through a formal agreement between the Federal Government and the largest National Teacher’s Union (Sindicato Nacional de Trabajadores de Educación, SNTE). The Alliance sought to improve student learning in basic education via the mobilization of civil society and state governments, improved participation of all stakeholders, and the transformation of management and service delivery in education. In this context, the Quality Schools Program (Programa Escuelas de Calidad, PEC), created in 2001 and supported by the Bank since 2006, that focused mainly on the peri-urban poor became even more relevant to advance the country’s education agenda by increasing school autonomy and social participation. 1 1.2 Original Project Development Objectives (PDO) and Key Indicators 4. The Loan Agreement (LA) states that the objective of the Project is to “strengthen PEC by increasing overall coverage and social participation in Eligible Schools while putting greater emphasis on public marginalized schools and on the indigenous population, as well as a reorientation of the direct support to schools1 to improve public schools’ internal efficiency and learning outcomes.” The objective on the LA differs from the Project Appraisal Document (PAD). The main inconsistency is that the PAD makes it clear that the long term development objective of the Adaptable Program Loan (APL) is “to improve the quality of education as measured by coverage, social participation, and educational outcomes” and that educational outcomes were not Project specific objectives as implied by the PDO in the Loan Agreement. 5. The Key Outcome Indicators of the Project as stated on page 33 of the PAD were:  Number of schools participating in PEC as percentage of the total number of basic education schools (from 18 in 2009 to 22 percent in 2013);  Total number of basic education schools participating in PEC (from 39,180 in 2009 to 50,000 in 2013);  Commitment to the goals and activities of the Strategic School Transformation Plan (Plan Estratégico de Transformación Escolar, PETE) (as expressed by parental participation in PETE design or adjustment (as percentage of total parents) (from 74.2 percent in 2009 to 80 percent in 2013);  Percentage of PEC schools in highly marginalized and marginalized areas (from 40 percent in 2009 to 50 percent in 2013); and  Percentage of PEC schools with indigenous students (from 7 percent in 2009 to 15 percent in 2013). 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 6. The PDO was not revised. In the April 2013 restructuring, three out of five key indicators were revised, one dropped (due to repetition), and one added to ensure the full measurement of the PDO achievement. The following are the revised/added key indicators:  Percent of basic and special education schools participating in PEC (from 20.3 in 2009 to 23 percent in 2013). Revised to clarify its meaning, includes an explicit reference to special education schools (also covered by PEC).  Percentage of school community members who participate in the design of the PETE (from 74.2 in 2009 to 75 percent in 2013). Revised because baseline data reflected the participation of community members, not parents.  Percentage of Indigenous modality schools participating in PEC (from 15.4 in 2009 to 19.35 percent in 2013). Revised to better reflect the Project’s impact on PEC’s reach to the 1 School Grants, originally used in the PDO, has been replaced with the term “direct support to schools” at the request of the Borrower 2 indigenous population. Under the original indicator, a PEC school was counted if it had only one indigenous student.  Percentage of direct support to schools spent on education quality inputs (from 59.7 in 2009 to 66.86 percent in 2013). Added to measure the reorientation of the direct support to schools. 1.4 Main Beneficiaries 7. The Project benefited public basic education (pre-primary, primary and lower secondary) schools participating in PEC in all 31 states and the Federal District while putting greater emphasis on marginalized areas and on the indigenous population. Principals, parents, teachers, and students in approximately 49,000 PEC schools and 7,669,759 students (annual average) benefited through the provision of direct support to schools and technical assistance to implement school improvement plans. Indirect Project beneficiaries were education authorities at the state level that received technical assistance to oversee the Project and improve their prioritization of indigenous schools and schools in marginalized areas. 1.5 Original Components (as approved) 8. Component 1: Direct Support to Schools. Provision of support to PEC, through the provision of direct support to Eligible Schools to implement school improvement plans. 9. Component 2: Monitoring and Oversight. Provision of support for: (i) the continuous operation, maintenance and upgrading of the PEC national management information system (MIS); and (ii) the accreditation and strengthening of state programs to manage information systems, to ensure that the information obtained through the referred state systems is compatible with the national MIS and provides all the technical and fiduciary information required for the satisfactory implementation of the Project in the relevant State. Provision of technical assistance to monitor and oversee the Project through: (i) the carrying out of supervision visits; (ii) the organization and carrying out of capacity building activities at the state level; and (iii) the coordination of regional and national meetings attended by PEC’s federal and state authorities. Design and implementation of an information campaign to disseminate Project objectives, activities and results. 10. Component 3: Carrying out of evaluations and assessments to create a strong analytical basis for development and improvement of PEC, including: (i) an external evaluation of PEC’s performance and preparation of an annual report; (ii) a qualitative evaluation to determine the non- quantifiable benefits accruing from PEC; (iii) Ministry of Public Education (Secretaría de Educación Pública, SEP) student learning achievement assessments using national education standards for each school year; and (iv) one impact evaluation of the Project. Carrying out of Project-related policy studies including: (i) studies focusing on improving the coordination of PEC with other SEP programs, including a study to improve the coordination of PEC with the program administered by the National Council for Educational Development (Consejo Nacional de Fomento Educativo¸ CONAFE), and a study to improve the coordination of PEC with the Oportunidades Program; and (ii) studies focusing on improving the equity and efficiency of PEC. 3 1.6 Revised Components 11. Component 2 was amended in the April 2013 restructuring to: (i) specify the provision of technical assistance to the states to improve their prioritization of indigenous schools and schools in marginalized areas and monitor their adherence to the PEC Operating Rules; (ii) add the dissemination of the simplified school planning instrument; (iii) add the carrying out of a monitoring and evaluation (M&E) survey to collect data from community members on the strategic school based management (SBM) model in PEC schools; and (iv) carry out an impact evaluation and a qualitative evaluation of the Project. 12. Component 3 was deleted. 1.7 Other significant changes 13. The April 2013 restructuring also extended the closing date of the Project by one year, from June 30, 2013 to June 30, 2014, to allow time for the completion of all Project activities. In addition, all Project restructurings (April 2013, September 2013, and June 2014) reallocated part of the Loan Proceeds from Categories 2 (Incremental Project Costs) and/or 3 (Consultant Services) to Category 1 (Goods, Works and Consultants’ Services under direct support to schools) in order to increase the number of beneficiary schools and to reflect that the Government was using its own funds to support the incremental Project costs and consultant services originally planned under the Project. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry Strengths  Project preparation took place from May 2009 to June 2010. The Project’s overall focus on PEC, which aims to increase school autonomy and social participation, as well as the Project’s PDO, which placed greater emphasis on public marginalized schools and on the indigenous population, were well aligned with the Country Partnership Strategy (CPS) FY2008-14 (Report No. 42846-MX) objectives of providing comprehensive assistance to promote social inclusion and reduce poverty through human development programs.  Based on the lessons learned from the first phase of the series (Mexico Education Quality Project, P088728, Ln. 7347-MX), the new design simplified the school improvement plans to increase schools’ and parents’ participation.  The Project design took into account most of the potential risks in achieving the PDO. The main risks considered were: inertia to school management and social participation, lack of information on what happens at the school level, complexity in the supervision of funds as they flow into schools, and the risk that schools with indigenous children or in low income households would not benefit from PEC. The necessary actions to mitigate these risks were also considered. The Project design incorporated: (i) specific dissemination and capacity building activities to reach the most disadvantaged schools; (ii) technical assistance to states so that they could help schools gradually change their culture and increase parental participation; (iii) prioritization of indigenous and marginalized schools in PEC Operating 4 Rules; and (iv) smaller/rural schools benefited from a less complex school planning instrument, among others.  The Project design focused on increasing the active role of the school community by, among others, providing direct support to schools only when the school improvement plans were developed by a school community that received capacity building in this area. Weaknesses  The PDO was poorly articulated. It was hard to understand what the Project strove to achieve. Increasing PEC’s coverage meant expanding SBM to more schools but was not explicitly mentioned. Likewise, social participation meant community participation in different aspects of the school life (participation in the school council, in the decision process to use the direct support to schools, and in developing and implementing projects).  The PDO in the LA was inconsistent with the PAD, including long term development objectives such as improvements in internal efficiency and learning outcomes indicating insufficient quality control. The PAD documentation indicated these objectives were for the end of the APL series, not for the Project.  Some lessons regarding the challenges of undertaking rigorous impact evaluations of PEC due to the self-selection bias of schools were overseen in the Project’s design. The Project could have embedded an Impact Evaluation Strategy, including technical support to develop the design and the rest of the impact evaluation stages.  The challenges associated with the procurement of technical assistance under the first phase of the APL, mainly related to the client’s preference to follow local procedures were not taken into account. In this regard, better specifications of the procurement processes and guidelines, as well as allowing more time to carry out these processes would have been helpful.  The PAD’s economic analysis did not explore the Internal Economic Rate of Return (IERR) of the Project or any other cost-effectiveness methodology. 2.2 Implementation 14. Component 1 was disbursed in a timely manner and was, by far, the largest in terms of Bank support, with 99.3 percent of the loan allocated to it by the end of the Project. Overall, the Project supported approximately 75 percent of the total direct support to schools, since schools received additional funds from states with a federal to state contribution of 3 to 1. Even with increased federal budget in the first years of the Project, SEP was able to transfer resources to the National Trust Fund (Fideicomiso Nacional de Escuelas de Calidad, FNEC) efficiently and from the FNEC to the State Trust Funds (Fideicomisos Estatales de Escuelas de Calidad, FEEC) as soon as the states deposited their counterpart funding. All 31 states in Mexico and the Federal District participated in PEC, with the exception of Sonora and Veracruz in the 2012-13 school year and Chiapas and Veracruz in 2013-14, due to their financial circumstances. 15. The main challenge faced under this Component was that in some cases, the direct support to schools arrived late in the school year, preventing them from implementing their Annual Work Plan (Programa Anual de Trabajo, PAT) as planned. The delays were due to some states’ inability to make their deposits to the FEEC on time. To try to ensure that schools received the direct support at the beginning of the school year, SEP, the implementing agency: (i) shortened the deadlines for 5 states’ deposits in the Operating Rules (as early as June 30 when the school year starts in mid- August); (ii) built a system of incentives where states that made their deposits on time were able to get additional federal resources; and (iii) sent official letters (“oficios”) and continuous reminders to prompt states to deposit in a timely manner. In 2012, 15 states did not disburse direct support to schools’ funds on time (by September); in 2013, only 10 states exceeded the time on the disbursement of these funds. 16. Component 2 (Monitoring and Oversight). Despite the variety of ways in which the states reported their information to SEP, the Federal authority had enough capacity to aggregate the data to generate a dataset with sufficient information to monitor the performance of the Project and PEC. However, the continuous operation, maintenance and upgrading of the Information System for the Quality Schools Program (Sistema de Información del Programa Escuelas de Calidad, SIPEC) and compatible state systems faced important challenges, mainly due to the disparity among the states’ institutional capacity, technical glitches, and the server’s low capacity. San Luis Potosí, Michoacán, Estado de México, and Guanajuato were able to smoothly run their information systems from the beginning of the Project, ensuring that schools reported information on-line to feed the SIPEC. The rest of the states continued to use outdated methods of data collection and analysis, such as pen and paper instruments or basic excel sheets. In an effort to help states with M&E shortcomings, SEP customized and targeted its support to weaker states, resulting in overall strengthening of these states’ M&E capacity. However, the large degree of heterogeneity in terms of the reliability and frequency at which states report data persists. In 2014, the Bank provided technical assistance to review the PEC national databases and recommended harmonizing all historical information to develop one single data set of PEC schools over time.2 17. After the April 2013 restructuring, SEP began targeting technical assistance to the states to particularly improve the prioritization of indigenous schools and marginalized areas. For instance, the last PEC Annual Meetings focused, among others, on targeting criteria and states’ adherence to the PEC Operating Rules. In addition, SEP started formally reviewing the states’ calls for proposals for PEC and the explicit inclusion of prioritization criteria for schools located in indigenous and marginalized communities. 18. The impact evaluation was delayed due to the difficulty in reconciling certain administrative requirements with the requirements of the Bank’s procurement procedures; nevertheless, this activity was not crucial to the achievement of the PDO. However, SEP was successful in the implementation of an information campaign to disseminate the Project’s objectives, activities and results. The dissemination of the simplified school planning instrument was carried out through the Project and reinforced in the last year and was critical to boosting participation of more disadvantaged schools in PEC and to increasing social participation among school stakeholders. 19. Component 3 (Policy Development and Evaluation). The implementation of the activities under this Component proved challenging because the Government was undertaking most of the 2 The Bank also recommended distinguishing PEC schools receiving only technical assistance from those receiving technical assistance and direct support. Aside from these efforts, the information on the roster of schools participating in PEC continues to suffer from the lack of information from a few schools. 6 activities with their own budgetary funds outside of the Project. For instance, the external evaluation of PEC’s performance and preparation of an annual report was developed by the National Council for the Evaluation of Social Development Policy (Consejo Nacional de la Evaluación de la Política de Desarrollo Social, CONEVAL) through annual reports in different areas including design, objectives, performance and consistency and results. In the April 2013 restructuring, this Component was eliminated because none of the activities (apart from the impact evaluation of the program, which was moved to Component 2) were critical to the achievement of the PDO. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 20. Design: The Project’s Results Framework (PRF) was not fully developed. The key indicators did not measure all aspects of the PDO achievement. In other cases, indicators were repetitive or lacked clarity. The PAD also had a lack of consistency regarding key indicators in different sections of the document (page 7 and page 33). It integrated an indicator to measure learning outcomes, one of the long term objectives after Project closing, but not an indicator on internal efficiency, the other long term goal. The PRF included outcome indicators that were more aligned with the Project’s PDO and activities than others and some that were broad and difficult to measure. Not all aspects of the PDO were covered by the original key indicators; in particular, the reorientation of direct support to schools. The original data collection methods explanations lacked clarity and were in some instances absent from the PAD or other Project documents, and several baselines were estimated with preliminary data. 21. Implementation: Data was collected and reported periodically throughout the Project’s implementation. However, M&E remained a weakness throughout most of the Project until the April 2013 restructuring, where indicators were revised and the necessary arrangements and clarifications were made to start collecting data to gather information for some indicators. With the April 2013 restructuring, many of the shortcomings at design were addressed. In particular, the restructuring added an indicator to fully measure the PDO and clarified methodologies and resources for collecting data and tracking Project indicators. For example, an M&E survey to collect data from community members on the strategic SBM model in PEC schools was explicitly included in Component 2 to ensure that it was implemented. The M&E survey was carried out once and could be compared to a previous survey conducted by SEP; however, the second round of the survey has not been conducted because SEP was unable to contract the firm to undertake this activity due to the long administrative process it entailed, including the reconciliation of the Bank’s procurement guidelines with internal administrative procedures. Additionally, the lack of timely information from a few states to the Federal Government prevented SEP from updating the PRF on some occasions. In any case, the biannual Project Implementation Reports and implementation support missions were always useful and provided enough information to understand the status of the Project. 22. Utilization: The implementing agency and the Bank constantly assessed and used Project data to inform decision-making and resource allocation. For instance, information was used to constantly revise the PEC Operating Rules, which experienced various changes such as: (i) changes in the timing in which states needed to deposit their counterpart funds in the FNEC; (ii) changes to further stress marginalized and indigenous schools prioritization; and (iii) changes in 7 the percentages of direct supports to schools to be spent on education quality inputs, to name a few. In addition, the M&E information was also useful to provide just in time support and information to states and indigenous and marginalized schools in order to ensure that they applied for entry into the program and benefited from it. 2.4 Safeguard and Fiduciary Compliance 23. Environmental and social safeguards: The Project had an environmental category C. In some cases, PEC schools conducted minor rehabilitation works that did not entail a negative environmental impact. The Project also triggered OP/BP 4.10 (Indigenous Peoples). The Indigenous Peoples Plan (IPP) Plan Indígena Programa Escuelas de Calidad 2010 (published on the Bank’s and SEP’s webpages) was based on an evaluation of PEC execution in indigenous schools and consultations with stakeholders. The IPP was implemented adequately, ensuring that indigenous schools participated and benefited from the Project and reaching 7 percent of indigenous schools as a proportion of the total number of schools in the program in the school year 2013-2014, when the target set in the IPP was of only 4 percent. 24. Financial Management (FM): The Project’s FM arrangements were adequate to provide reasonable assurance that Project funds were used for the purposes intended. Minor shortcomings in FM existed, but did not prevent the timely and reliable provision of information required to manage and monitor Project implementation. Project audit reports were delivered on time throughout the life of the Project and were considered acceptable to the Bank. 25. Procurement: Ex-post reviews of PEC schools’ procurement were carried out in Guanajuato, Michoacán, Puebla, State of Mexico, Tlaxcala and Aguascalientes. Minor procurement weaknesses were identified, mainly in the integration of records, requests for quotations, and schools investments. These shortcomings did not have material impacts on compliance with procurement policy requirements. 2.5 Post-completion Operation/Next Phase 26. PEC is a top priority for SEP, as it is seen as the program for spearheading the implementation of an ambitious Reform launched in 2013 to improve the quality of education through, among others, strengthening SBM. The SBM policy is currently implemented through different national programs such as PEC, PETC and Support for School Management (Apoyos a la Gestión Escolar, AGE – managed by the National Council for Education Development, Consejo Nacional de Fomento Educativo, CONAFE). In 2007-08, SEP introduced PETC in basic education to increase the number of school-day hours and promote SBM following the PEC scheme. The PETC’s SBM component is exactly the same as PEC’s. These programs share the goal of contributing to students’ learning within a framework of more school autonomy and social participation. In addition, based on the lessons learned from this Project, the use of evaluations to improve pedagogical strategies and managerial capacities at the school level are being introduced. The Bank is supporting the Government with a new School Based Management Project (P147185, Ln. 8446-MX, approved by the Board on October 24, 2014) which will provide support to PEC and PETC. The development objective of this operation is to improve schools’ managerial capacity and parental participation to reduce dropout, repetition and failure rates among schools 8 participating in PEC and PETC. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation Objectives Rating: Pre April 2013 Restructuring – High / Post April 2013 Restructuring High 27. The relevance of the PDO is deemed to be High pre and post the April 2013 restructuring. The Project objectives are relevant to the World Bank Group’s Country Partnership Strategy (CPS) for FY14-FY19 (Report No. 83496) and fit under the CPS theme II “Increasing Social Prosperity,” under the fifth area of engagement “Promoting Labor Markets for Inclusive Growth.” One of the main priorities for Mexico’s growth and development, emphasized both in the reform process and in the National Development Plan (NDP) for 2013-18, is improving the quality of education, which is consistent with the Project’s long term objective of improving learning outcomes. The NDP also focuses on reducing the achievement gaps between rich and poor to increase productivity and long- term growth while reducing social inequalities, which is consistent with the PDO’s focus on indigenous and marginalized populations. In addition, the 2013 constitutional mandate and the Education Sector Program 2013-18 (Plan Sectorial de Educación, PSE) put schools at the center of the education system by fostering national strategies to, among others: introduce mechanisms to allocate direct support to schools and increase schools’ management capacity; strengthen the relationship between the school, parents, and the broader community to collaborate with the school; and increase transparency and accountability, all of which are consistent with the Project’s objectives of expanding the PEC SBM model and increasing social participation to eventually improve public schools’ internal efficiency and learning outcomes. Design Rating: Pre April 2013 Restructuring – Substantial / Post April 2013 Restructuring – Substantial 28. The Project design is considered Substantial before and after the April 2013 restructuring. The Project mainly focused on providing direct support to schools to implement their school improvement plans and providing technical assistance to states, crucial activities to expand the PEC SBM model to more schools. Project activities such as the dissemination of the simplified school planning instrument were highly relevant to increase participation of schools located in indigenous communities and marginalized areas and increase social participation. The provision of technical assistance and guidance to develop school improvement plans in a collaborative way (by the school director, teachers, parents, the community and the students) was crucial to build the schools capacity to spend the direct support in education quality inputs while increasing community participation in different aspects of the school life. Technical assistance for an M&E survey to collect data on social participation and an impact evaluation was included. While not critical to the achievement of the PDO, this technical assistance was instrumental in measuring PDO achievement and in providing stronger evidence of impact that could inform future interventions. The design had minor shortcomings, for example, it could have included technical assistance to design an evaluation strategy and to explicitly train principals at the school level. 9 29. There is emerging literature that SBM can lead to improved learning outcomes and internal efficiency.3 Three impact evaluations show significant reductions in dropout and repetition rates and increments in pass rates in PEC schools (Skoufias and Shapiro 2006, Murnane, Willet and Cardenas 2006, C230 Consultores, 2012). 3. 2 Achievement of Project Development Objectives. 30. Table 1 summarizes the achievement of results by objective, relying on evidence from the PDO Indicators and on supplementary indicators introduced at the April 2013 Restructuring and at the Implementation Completion and Results Report (ICR) stage. Table 1: Summary of Project Outcomes Baseline End Target June End of Project Indicator 2009 2014 June 2014 Strengthen PEC by increasing overall coverage Percentage of basic and special education schools 20.30 22.00 pre-restructuring 23.85 participating in PEC a and 23.00 post- (2012-13) restructuring Number of Schools in PEC b 40,790 50,000 50,624 (2013-14) Strengthen PEC by increasing overall social participation Percentage of school community members who 74.20 80.00 pre-restructuring 72.70 participate in the design of the PETE a/d (2008-9) and 75 post- (2012-13) restructuring Percentage of community members that observe 90.20 93.00 pre restructuring 93.50 participatory decision-making between parents, teachers and 91.00 post (2012-13) and principal about school affairs c/f restructuring Greater emphasis on marginalized schools Percentage of PEC schools in marginalized and highly 38.40 50.00 pre restructuring 51.35 marginalized areas b and 55.00 post (2013-14) restructuring Percentage of PEC schools in marginalized, highly 65.30 No target 78 marginalized and medium level of marginalization areas (2012-13) f/g Greater emphasis on the indigenous population Percentage of indigenous modality schools participating 15.40 19.30 18.00 in PECa (2012-13) 3 Evidence of links between SBM and improved learning outcomes is mixed. Evidence from Kenya shows that parent- teacher associations lead to better learning outcomes (Duflo et al 2014) when parents are empowered through specific capacity building. In The Gambia, SBM was shown to reduce student and teacher absenteeism; SBM only had a positive effect on learning outcomes in communities with high literacy, otherwise the effect could be potentially negative in communities with low literacy (Blimpo, Evans 2011). In Indonesia, SBM was found to have links to improved learning outcomes only when school based management is complemented by linkages between the school committee (Pradhan et al. 2014). Evidence also shows that SBM has improved social participation, governance, transparency, and accountability, leading to lower dropout and repetition rates (Gertler et al 2012). A recent observational study suggests that school autonomy may lead to improved learning outcomes when strong central institutions are in place, but otherwise possibly detrimental (Hanushek et al. 2012). Finally, another study argues that SBM programs need to be in place for at least 5 years to achieve significant results and at least 8 years to “transform” schools (Borman et al. 2003). 10 Baseline End Target June End of Project Indicator 2009 2014 June 2014 Percentage of PEC schools with indigenous students b 7 15 13.5 (2011-12) Simplification of the school planning instrument f NO YES YES Reorientation of the direct support to schools Percentage of direct support to schools spent on education 59.70 66.86 74.54 quality inputs a (2012-13) PEC public schools learning outcomes Average ENLACE test score in Math in PEC primary 503 No target 560 schools e (2012-13) Average ENLACE test score in Spanish in PEC primary 498 No target 534 schools e (2012-13) Average ENLACE test score in Math in PEC secondary 494 No target 537 schools e (2012-13) Average ENLACE test score in Spanish in PEC 485 No target 481 secondary schools e (2012-13) PEC public schools internal efficiency Average pass rate in PEC primary schools e 96% No target 99% (2012-13) Average pass rate in PEC secondary schools e 89% No target 92% (2012-13) Average Dropout rate for PEC primary schools f 0.041 No target 0.038 (2012-13) Average Dropout rate for PEC secondary schools f 0.054 No target 0.052 (2012-13) a revised outcome indicator; b original outcome indicator; c the percentage of schools where community members confirmed that they observed participatory decision-making between parents, teachers and principal about school affairs; d the percentage of schools where community members confirmed that they participated in the design of the PETE; e indicator added at restructuring ; f indicator added at ICR stage; g Page 6 of the PAD states as a trigger from Phase II to Phase III that the number of PEC schools located in areas of medium, high, and very high disadvantaged (marginalization levels) will increase by 30 percent. Strengthen PEC by increasing its overall coverage Rating: Pre April 2013 Restructuring – High / Post April 2013 Restructuring – High 31. The number of schools operating under the PEC SBM model increased from 20.30 percent of the total of basic and special education schools to 23.85 percent in the 2013-14 school year, exceeding both the original target of 22 percent and the revised target of 23 percent. Almost 10,000 additional schools participated in the program, from 40,790 in 2008-9 to 50,624 in 2013-14, also exceeding the 50,000 PEC schools target set at Appraisal. 32. The Project contributed to increasing the coverage of eligible schools by offering direct support to all new schools entering the program. Having a school improvement plan developed by the school community was mandatory to access the direct school support that was then used to support activities related the school improvement plan at each school (which vary considerably depending on specific schools’ needs). Aggregate numbers for 2012-13 show that most of the direct support to schools (39.4 percent) was spent on schools’ technical equipment and furniture and on teaching materials (18.5 percent). Schools also used direct support for rehabilitation (15.3), school materials (8.8 percent), minor constructions (5 percent), capacity building activities (1.5 percent), and others (11.5 percent). Direct support to schools average around US$4,000 per year 11 and, while small compared to the total costs of running schools, are not insignificant when compared to the budget managed directly by the schools. 33. The Project also helped ensure the participation of a higher number of schools through the simplification of the schools’ application process and the school planning instrument (PETE). In addition, the design and implementation of the information campaign to disseminate PEC’s objectives, activities and results contributed by increasing awareness. Finally, the Project supported technical assistance to the states, which strengthened their capacity to reach out to a greater number of schools and encouraged them to apply to and stay in the Program. Strengthen PEC by increasing overall social participation Rating: Pre April 2013 Restructuring – Modest / Post April 2013 Restructuring – Modest 34. The percentage of surveyed school community members who participated in the design of the school planning instrument (PETE) remained at around 73 percent throughout the Project and the pre and post restructuring targets of 80 and 75 percent, respectively, were not achieved. On the other hand, the percentage of surveyed school community members that observe participatory decision-making between parents, teachers and principals increased from an already high 90.2 in 2008-9 to 93.5 percent in 2012-13, exceeding the original target of 93 and the restructured target of 91 percent. 35. The Project, through PEC, helped improve parental participation by providing supplemental direct support to schools that mobilized additional local resources to implement their school plans. The supplemental supports matched the amount of local resources raised by the school. This worked as an incentive for parents and the community to work together to raise funds for the school and become more involved in the development of school plans and their implementation. In addition, the Project engaged parents and community members to actively participate in the school self-evaluation, planning, and implementation processes by enforcing the collaborative development of school improvement plans and building the capacity of school communities on collaborative SBM. The dissemination of the simplified school planning instrument (consisting of a manual with a series of recommendations intended to make the school planning process clearer, easier, and more efficient for schools while easily explaining the role of different stakeholders in collaborative SBM) was also critical to encouraging greater participation of parents and the school community in school affairs since it allowed poorly educated parents to better understand the school planning processes and thus, be better able to contribute to them. Greater emphasis on marginalized schools Rating: Pre April 2013 Restructuring – Substantial / Post April 2013 Restructuring – Substantial 36. The percentage of PEC schools in marginalized and highly marginalized areas increased from 38.4 percent in 2009-10 to 51.35 percent in the 201-14 school year; exceeding the original target and almost meeting the revised target, which was 93 percent achieved. The percentage of PEC schools located in areas with a medium level of marginalization (that still face important socioeconomic challenges) and in highly and very highly marginalized areas reached 78 percent in 2012-13, compared to only 65.3 percent in the school year 2009-10. One of the major factors affecting the full achievement of the target was that Chiapas and Veracruz, two of the states with 12 a higher percentage of schools located in marginalized and indigenous communities, were not able to participate in the last year of the Project due mainly to their financial circumstances, preventing participation of an important percentage of marginalized schools. 37. The Project contributed to increasing the participation of marginalized schools by constantly revising the PEC Operating Rules to stress prioritization of schools located in indigenous and marginalized communities and by providing technical assistance to the states to improve their prioritization of these schools. Specific activities supported under the Project were: (i) regular supervision visits to all states to ensure that prioritization of schools in indigenous and marginalized areas was made adequately; (ii) the organization of capacity building activities at the state level to build the states’ capacity to manage the program and better select schools; and (iii) regional and national meetings by federal and state authorities. In addition, the Project contributed to increasing the number of PEC schools located in marginalized communities by supporting the dissemination of the simplified school planning instrument, which helped school directors, parents, teachers, and students be more aware of and better understand the simplified instrument, put together their school plans even with low capacity and resources, apply to PEC, and increase their chances of being selected. Greater emphasis on the Indigenous Population Rating: Pre April 2013 Restructuring – Substantial / Post April 2013 Restructuring – Substantial 38. The number of PEC schools that reported to have one or more indigenous students increased from 7 percent in the school year 2009-10 to 13.5 percent in the school year 2011-12, meeting 90 percent of the original target. As for the percentage of indigenous modality schools participating in PEC (indicator introduced at restructuring), 18 percent of indigenous modality schools participated in the last year of the Project, meeting 93 percent of the revised target. However, one year earlier, 20.8 percent of indigenous schools participated in PEC, surpassing this target. 39. The Project contributed in the same way to achieving this objective as to the objective of greater emphasis on marginalized schools. Reorientation of the Direct Support to Schools Rating: Pre April 2013 Restructuring – High / Post April 2013 Restructuring – High 40. The percentage of direct support to schools spent on education quality inputs increased from 59.70 percent in the school year 2009-10 to 74.54 percent in the 2012-13 school year, widely exceeding the 66.86 percent target set during the April 2013 restructuring. Education quality inputs included furniture and technical equipment, teaching materials, school materials, and capacity building. The Project helped reorient direct school support to education quality inputs by providing the necessary technical assistance to the states and schools to make adequate and informed choices when preparing and implementing their school plans with PEC resources. For instance, the Project strengthened states’ capacity building on the consolidation of the Modelo de Gestión Escolar con Enfoque Estratégico (MGEE), a SBM model that helps the school community use local information in a systematic way to use PEC support to prioritize needs that are more correlated with student learning. In addition, the Project contributed to reorienting direct support 13 to schools to education quality inputs by incorporating specific percentages of the direct school support to be allocated by schools to certain categories. For instance, through the Project’s length, newly incorporated schools should spend 30 percent of their direct school support on the improvement of students’ learning processes, which includes teachers’, school directors’, and parents’ capacity building activities and the acquisition of school materials; schools with more years in the program should spend 50 percent in the same category. Improved public schools’ internal efficiency Rating: Pre April 2013 Restructuring – Substantial / Post April 2013 Restructuring – Substantial 41. This ICR defines the schools’ internal efficiency as the schools’ ability to retain students and ensure that they advance through the education system as planned. Average dropout rates for PEC primary schools decreased from 0.041 to 0.038 percent from 2008-9 to 2012-13 and from 0.054 to 0.052 percent for secondary schools. Average pass rates in PEC primary schools increased from 96 percent in 2008-9 to 99 percent in 2012-13 and from 89 to 92 percent in PEC secondary schools. Both rates are better for PEC than for non-PEC schools. 42. To contribute to the improvement of the internal efficiency of PEC schools, the Project helped increase school autonomy in managing the direct support to schools, as well as in spending the direct support on education quality inputs. The Project’s efforts to increase community members’ participation in the school improvement plans led to greater engagement in school life, and there is evidence that these efforts have also led to greater teacher accountability. For example, the percentage of community members that observed that parents were informed about student performance increased from 99 to 99.80 percent. In addition, the percentage of parents that observe: (i) teachers encouraging and supporting students’ performance increased from 90.20 to 93.50; and (ii) teacher encouraging the active participation of students increased from 66.90 to 91.50. Improved public schools’ learning outcomes Rating: Pre April 2013 Restructuring – Modest / Post April 2013 Restructuring – Modest 43. PEC schools improved their ENLACE results with greater gains in Mathematics. ENLACE results in math went from 503 and 494 in the 2008-09 school year for primary and secondary, respectively, to 506 and 537 in 2012-13. In the same time period, Spanish ENLACE results for primary improved from 498 to 534 and remained stable at the secondary level. 44. The Project contributed in the same way to achieving this objective as it did to meeting the improved public schools’ internal efficiency. 3.3 Efficiency Rating: Pre-2013 restructuring – Modest / Post-2013 restructuring – Modest 45. According to the economic analysis in the PAD, the increase in the participation of indigenous and marginalized schools in PEC should help to better target the disadvantaged population in Mexico and thus improve the equity of basic education. This was confirmed during the 2010-13 period, when PEC indigenous schools decreased their failure rate by 4.5 percent, 14 producing an increase in indigenous students’ years of schooling of 0.045 for each PEC cohort during the same period. When these individuals enter the labor market, the increased years of education will result in an increase of 0.225 percent in their monthly salary. During the same period, failure rates decreased by 3.4 for disadvantaged students in PEC, producing an increase in these students’ years of schooling of 0.034. When these individuals enter the labor market, the increased years of education will result in an increment of 0.187 percent in their monthly salary (see Annex 3 for further details). 46. In addition, secondary schools increased math achievement by 0.30 standard deviation in 3 years (during the Project), from an average PISA score of 355 to 385. Using the estimates and methodology elaborated by De Hoyos (2010) to measure the economic benefits of increased learning outcomes, the result could be translated into an increase of 16 percent of the annual income of Mexican families in 2010, which translates to a decrease in poverty of 3.88 million inhabitants in Mexico. While the result is for all the secondary schools in Mexico, PEC schools have contributed to this result (PEC schools outperformed other schools in ENLACE in the same period of time). 47. Although no IERR was calculated at Appraisal, the Project’s economic efficiency is also illustrated by the results from recent economic analysis of the program, which yields a 6.5 percent IERR. The costs of the Project were US$366.66 million for the four years of the Project, or an average of US$91.66 million per year. The average federal annual unit cost per school decreased over the life of the Project because each year more schools were accepted into PEC (from 40,790 in 2009 to 50,624 in 2014, exceeding the original target of 50,000) while the budget increased only slightly from MX$1,477,418,501 in 2009 to MX$1,515,911,529 in 2014. The annual cost per school and student at the end of the Project were lower than at Appraisal (US$1,873 now versus US$2,247 at Appraisal and US$14 now versus US$16 at Appraisal, respectively). Therefore, efficiency increased through the Project’s life. In addition, a comparison of PEC’s student unit cost to other education interventions shows that PEC is relatively inexpensive and renders important gains in terms of improved public schools’ internal efficiency. This cost is lower than the unit cost per student of PETC, where the cost per student in 2012 was US$167. 48. Other aspects of efficiency that have to be taken into account are the one year extension of the Project’s closing date and the states’ delays in depositing their contributions for the direct support to schools. The former illustrates modest inefficiencies in completing the Project in a timely manner, while the latter reflects some inefficiency (around 10 out of 32 states delayed their funding in 2013, an improvement from the 15 that had delayed in 2012) in ensuring that the direct support to schools reached them in time to implement their school plans in an effective and efficient way. 3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 49. Relevance ratings for objectives are High for both the pre- and post-2013 restructuring periods. The relevance rating for design pre and post restructuring is Substantial. Efficacy ratings are Substantial for both periods. Finally, the Efficiency rating is Modest for both periods. The combination of these ratings yields an overall Moderately Satisfactory rating for both periods 15 (Table 2). In turn, the overall ratings for the two periods are weighted by the disbursement percentages prior to and following the April 2013 restructuring, yielding an overall weighted rating of 4.0–equivalent to Moderately Satisfactory. Table 2. Weighted Project Rating Pre- Post- Overall restructuring restructuring Pre- and Post-restructuring outcome rating Relevance Objectives High High Design Substantial Substantial Substantial Efficacy Strengthen PEC by increasing its overall High High coverage Strengthen PEC by increasing overall social participation Modest Modest Greater emphasis on schools in marginalized Substantial Substantial areas Substantial Greater emphasis on the indigenous Substantial Substantial population Reorientation of the Direct Support to school High High to education quality inputs Improved public schools’ internal efficiency Substantial Substantial Improved public schools’ learning Modest Modest outcomes* Efficiency Modest Modest Modest Rating Moderately Moderately Moderately Satisfactory Satisfactory Satisfactory Overall outcome rating Rating value 4 4 Weight 86% 14% Weighted 3.44 0.56 4 value Final Rating Moderately Satisfactory *This is a long term objective, beyond the scope of the Project. As such, the ICR does report on and rate progress, but the overall rating puts less weight on this objective. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 50. The Project contributes to the Bank’s twin goals of eliminating extreme poverty and boosting shared prosperity by contributing to the long-term objectives of improving learning outcomes and retention rates among students in public schools, particularly those from marginalized areas, through strengthened SBM. Recent evidence shows that SBM programs can be an effective strategy to increase parental participation in school decisions; reduce dropout, repetition and failure rates and, under certain conditions, increase learning outcomes. 51. At the macro level, a society with more human capital–measured, for instance, by years of schooling or learning outcomes (long term objectives to which the Project contributed to) in a standardized test—enjoys higher rates of long term economic growth. Conversely, from the micro perspective, more human capital translates into higher personal or household incomes, less poverty 16 and, if the increases in human capital take place among disadvantaged households (which is the case in this Project), less income inequality. (b) Institutional Change/Strengthening N/A (c) Other Unintended Outcomes and Impacts (positive or negative) 52. The transformation of the PEC program into a nationwide school autonomy policy was an unintended outcome of this Project. This policy decision has also contributed to the expansion of PETC. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops N/A 4. Assessment of Risk to Development Outcome Rating: Negligible to Low 53. The 2013 education reform has reinforced the policy to strengthen school autonomy, allowing SEP to expand the number of public schools that receive support following PEC’s design in all public programs in basic education that have a SBM and autonomy component. The overall number of basic education schools with support to increase school autonomy is expected to expand considerably with the implementation of the reform. In consequence, social participation, the reorientation of direct support to schools to education quality inputs, schools’ internal efficiency and learning outcomes are expected to continue to improve in all schools benefiting from the PEC SBM model expanded by the new SBM policy. In addition, the new School Based Management Project (P147185, Ln. 8446-MX) will help foster the autonomy policy through the provision of support to PEC and PETC in order to ensure that the development outcomes of PEC (Phase II) are maintained. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Unsatisfactory 54. The Bank provided assistance to the Government to prepare a Project that was relevant and strategic for the Mexican context. The Project design also took into account poverty and social development aspects, putting greater emphasis on public marginalized schools and on the indigenous population. The major shortcomings were the poor articulation of the PDO and the weaknesses of the PRF. In addition, the Bank did not foresee the technical, administrative and procurement challenges related to undertaking a rigorous impact evaluation of PEC and other technical assistance, which had also emerged in the first phase of the APL. Hence, many of the technical assistance activities designed at Appraisal, including the impact evaluation, could not be finalized as part of the Project, though others were carried out as they were part of the activities 17 that the Government undertakes regularly through their own budget resources. These activities and the impact evaluation were not critical to the achievement of the PDO. In terms of FM, the Project was designed in a way that made the flow of funds quite complex for the Government and the Bank. The economic analysis did not present an IERR or any methodology to measure the cost- effectiveness of the Project. (c) Quality of Supervision Rating: Moderately Satisfactory 55. The Bank conducted regular implementation support missions that included meetings with the Ministry of Finance and Public Credit (Secretaría de Hacienda y Crédito Público, SHCP), National Financial Development Bank (Nacional Financiera, S.N.C., I.B.D, NAFIN) and SEP, as well as field visits to PEC schools in different states and in the Federal District. It also sustained a close and continuous dialogue with the Subsecretariat of Basic Education (Subsecretaría de Educación Básica, SEB), which facilitated the close monitoring of Project implementation. Bank implementation support helped identify the challenges being faced at various stages of Project implementation in a timely manner. For instance, it was aware of the delays in the states’ payments and helped the implementing agency find, implement and follow up on alternatives to solve this challenge. The Bank also provided key technical assistance when needed. For instance, it provided feedback to the impact evaluation design and organized a conference on quality of education and SBM to reflect on the achievements of the PEC model and to learn from international experiences to improve Project implementation and the new SBM reform. The Bank processed a useful Level 2 restructuring that helped ensure that the PDO was well measured and targets on indicators achieved. The safeguards and fiduciary teams continuously supervised and developed follow-up reports regarding the implementation of the IPP and followed up on detailed aspects of disbursements of direct support to schools, audits, etc. while building staff capacity in the Directorate of School Management (Dirección General Adjunta de Gestión, DGAG) in both fiduciary and technical aspects. There were moderate shortcomings. To address design and weaknesses in M&E, the restructuring should have been conducted much earlier and the PDO should have been revised to make it clear that learning outcomes and internal efficiency were objectives beyond the scope of the Project. Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 56. Given that Bank performance was rated Moderately Unsatisfactory at Ensuring Quality at Entry and Moderately Satisfactory at Quality of Supervision, the Bank’s overall performance is considered Moderately Satisfactory, taking into account the outcome rating of Moderately Satisfactory. 5.2 Borrower Performance (a) Government Performance Rating: Satisfactory 18 57. The Government showed ownership of the Project and commitment to achieving its development objectives. Throughout the duration of the Project, SHCP provided enough budget to SEP, and PEC in particular, and ensured that the loan was fully disbursed by its revised closing date, with most of the funds reaching the schools directly through the direct supports to schools. SHCP continuously followed up with the Bank, the implementing agency, and NAFIN to ensure that the Project worked as planned, including overseeing the Project implementation arrangements, the implementing agency capacity, and addressing challenges to the successful achievement of the Project’s objectives. Despite a change in the administration (and political parties) in December 2012, the Government ensured the continuity of PEC and requested Bank support for a new project in order to develop activities to foster and expand SBM and autonomy through PEC and PETC. NAFIN also performed well in its role of fiduciary agent of the Government for the Loan, ensuring that FM, governance, provision of counterpart funding, procurement, reimbursements, and compliance with covenants were followed. (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory 58. SEP, through the General Directorate of School Management and Education Innovation (Dirección General de Desarrollo de la Gestión e Innovación Educativa, DGDGIE), and in particular the DGAG, executed PEC’s budget in a timely fashion and ensured that most of the Project activities were implemented as planned. In close collaboration with the states, SEP ensured the expansion of PEC in general and of PEC schools located in indigenous and marginalized areas in particular. The adequate use of direct support to schools and the enhancement of social participation at the school level were also a top priority for the implementing agency. SEP provided technical assistance to the states so that they could run the program as specified in the Operating Rules, including better prioritization of vulnerable schools. There were moderate shortcomings. The main challenge that SEP faced was to ensure that M&E arrangements were in place at the state level. In addition, although SEP made an effort to execute the studies envisioned in the Project, it was not always able to expedite administrative and procurement procedures to ensure that these activities took place on time and with Project funds. In particular, greater attention to the implementation of the impact evaluation would have been desirable. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory 59. The overall rating for Borrower Performance is Moderately Satisfactory. 6. Lessons Learned  Early adjustment of PDO and PRF is desirable.  A rigorous impact evaluation of PEC is crucial to understand the effect of the program on learning outcomes. Undertaking rigorous impact evaluations of programs with a voluntary enrollment nature, such as PEC, has been challenging, mainly due to the schools’ self- selection biases introduced to conventional non-randomized evaluation approaches. In the case of PEC, an impact evaluation should only be supported in future Projects if the 19 necessary technical assistance is provided to develop the evaluation’s design and if states commit to operate PEC in a different manner to comply with randomization criteria.  Local Education Authorities (Autoridades Educativas Locales, AELs) play an important role in planning and implementing federal programs on the ground. It is important to find compelling ways to ensure that all states provide their PEC counterpart financing in a timely manner and comply with MIS that are compatible with the federal MIS. It is also crucial to ensure that states comply with the rules and criteria to guide the program while taking into account the disparities regarding the institutional capacity among states to distribute federal resources and technical assistance. It is important to provide each state with a suite of services to implement school autonomy and accountability initiatives, according to their particular needs and shortcomings.  In future operations, Project procurement and FM should be simplified in regards to the direct support to schools to avoid burdening schools with excessive administrative requirements and delays in the reporting of disbursements and proof of expenditures from the Government to the Bank.  Moving into country systems in countries with a solid fiduciary structure (such as Mexico) could help expedite technical assistance procurement processes and ensure that countries benefit from the technical assistance built into the project designs without necessarily compromising healthy procurement procedures.  The simplification and dissemination of school planning instruments help disadvantaged schools put together initial school improvement plans and therefore allow them to benefit more frequently from programs such as PEC. It is crucial, however, to provide schools with the resources and technical assistance to develop their plans to ensure that they design and implement plans that actually improve student learning. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 60. At the Borrower’s request, all references to “grants” were replaced by the term “direct support.” (b) Cofinanciers N/A (c) Other partners and stakeholders N/A 20 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Appraisal Estimate Actual/Latest Percentage of Components (USD) Estimate (USD) Appraisal Component 1 211,720,000.00 218,543,100.55 103% Component 2 4,300,000.00 906,899.45 21% Component 3 3,430,000.00 0.00 Total Baseline Cost 219,450,000.00 219,450,000.00 100% Physical Contingencies 0.00 0.00 Price Contingencies 0.00 0.00 Total Project Costs 219,450,000.00 219,450,000.00 100% Front-end fee PPF 0.00 0.00 Front-end fee IBRD 550,000.00 550,000.00 100% Total Financing Required 220,000,000.00 220,000,000.00 100% (b) Financing Appraisal Actual/Latest Type of Estimate Estimate Percentage Source of Funds Cofinancing (USD (USD of Appraisal millions) millions) Borrower 146.66 0.00 40% International Bank for Reconstruction 220.00 220.00 60% and Development 21 Annex 2. Outputs by Component Component/Subcomponent Project Outputs Component 1. Direct Support to Schools Provision of support to PEC,  The Project supported around 60.3 percent of the Federal Government’s budget for PEC direct through the provision of direct support to schools. support to Eligible Schools to  Technical assistance provided by procurement and FM specialists to adequately manage the Project implement school improvement and PEC’s budget from the Federal Government. plans.  Constant modification of PEC Operating Rules: (i) changes in the timing in which states needed to deposit their counterpart funds; (ii) changes to further stress marginalized and indigenous schools prioritization; (iii) established clear percentages of direct supports to schools to be spent on education quality inputs. Component 2. Monitoring and Oversight Provision of Support for the  The electronic module for capturing the strategic plan for school transformation entitled “Plan continuous operation, maintenance Estratégico de Transformación Escolar” was developed. and upgrading of the PEC National  The system of management of PEC was concluded including four modules: Plan Estratégico de MIS and the accreditation and Transformación Escolar, Presupuesto, Seguimiento de Indicadores, Mejores Prácticas Docentes strengthening of State programs to and Geo-básica to geographically locate PEC schools. manage information systems, to  Since April 2013, SIPEC has been under a review process to work properly. ensure that the information  The DGDGIE and DGAG in particular have been working actively with SEB representatives in obtained through the referred State charge of the design of new Management and Information System for Education (Sistema de systems is compatible with the Información y Gestión Educativa) that will work for all aspects of elementary schools including national MIS and provides all the PEC. technical and fiduciary information  Four states haveelectronic information systems to manage program resources: San Luis Potosí, required for the satisfactory Michoacán, Guanajuato and Mexico State (1st Semester 2011). implementation of the Project the  Development of the electronic modules of SIPEC for entering the PETE, schools budget, follow up relevant State. of key program indicators, best teaching practice, and a geo-referencing module to locate all schools participating in PEC. Provision of Technical Supervisory visits assistance to the States to  Advice and support, as well as guidance for the consolidation of the school management model with improve their prioritization of Strategic Approach (MGEE) was provided to the 32 States’ Managing Units of PEC. Indigenous Schools and 22 Marginalized Area Schools  Visits were carried out in various states in order to strengthen the actions between CNPEC, and monitor their adherence to CGEPEC and to comply with the goals and PEC Operation Rules. the PEC operating rules through (i)  The hours of supervision visits were tracked and can be accessed in the Implementation Status the carrying out of supervision Reports Results Framework. visits; (ii) the organization and Capacity Building Activities carrying out of capacity building  Participation in various forums such as that of "Participation of unions and teachers' organizations activities at state level; and (iii) the in the processes of educational reform and improvement." coordination of regional and  International Seminar on Integral Education, National Congress of Researchers in Education and national meetings attended by the Family, among others. PEC’s federal and state authorities.  Workshop for the construction of the Standard: "Coordinación de Proceso de Gestión Educativa Estratégica en Cetros Escolares de Educación Básica” 2011.  Participation in capacity building activities with various institutions such as CONAPASE, Crefal, World Bank and various programs such as the PNL, the PNIEB, PMLE, among others.  Workshop on good practices in school autonomy and SBM (Seminario de Buenas Prácticas en Autonomía y Gestión Escolar) delivered in June 2013 put together all states, the Federal Government, the Bank and experts from Brazil, Chile, Argentina and Colombia to, among others, learn about States’ experiences with PEC and its Operating Rules and to analyze international good practices to gradually provide more autonomy to schools, increase social participation and eventually improve learning outcomes. National and regional meetings.  Working meetings to define the objectives of the pilot innovation "Strengthening School Governance and Oversight" project.  Regional meetings in the West, Central, North and East regions with the participation of the State Coordinators of PEC.  Meetings to track, "Innovation project for the improvement of management and teaching practice in the classroom."  Capacity Building activity to disseminate the applicable guidelines and provide the necessary guidance and arrangements for the follow-up of the Project for Institutional Strengthening for Management and School Supervision.  Participation in the Second Meeting of School Directors of PEC Schools: "The Model of Strategic Educational Management, from Theory to Practice" Nuevo León. 23  Regional Certification Workshop of Directors that comprise the National Team of Expert Evaluators.  National Evaluation Meeting for PEC 2011.  Monitoring and support to the state strategy for the Consejos Técnicos de Zona y Consejos Técnicos. Escolares, both for the Intensive (August 2013) and the Ordinary Phase (2013-2014 school year).  National capacity building program with Supervisors of Basic Education School.  Capacity building activities for Consejos Técnicos Escolares.  National Meetings of PEC and Escuela Segura. Design and implementation of an Dissemination of PEC activities: Information campaigns information campaign to  Redesign of the website of the program and updating of its content. disseminate Project objectives,  Release of news that refer to the activities of the Program on the PEC’s and DGDGIE websites. activities and results and the  Production of two TV show called: “A new perspective on the PEC” and “PEC: at my school we dissemination of the simplified all express.” school planning instrument  Production and transmission of television programs to mark the 10th anniversary of PEC. (PETE).  Transmission of short stories at EDUSAT Network, Channel 22 and two open Television Programs: I. A new perspective of PEC and II. ‘At my school we all express” PEC.  Production of the first TV series for “Shared Visions” entitled “Initiatives to Strengthen Educational Quality in Basic Education.” Carry out of an M&E survey to  2012 Survey to measure indicators related to social participation in PEC schools collect data from community members on the strategic school based management (SBM) model in PEC schools Carry out an impact evaluation and  Impact evaluation design 2014 a qualitative evaluation of the Project. 24 Annex 3. Economic and Financial Analysis 1. De Hoyos (2010) analyzes the consequences of the lack of education reforms that could have increased education quality during the 1970s in Mexico. The author argues that if an educational reform had happened during the 1970s, it would have increased PISA educational achievement by half of a standard deviation in 10 years from an average PISA score of 406 to 456. The result would have been an increase of 28% in the annual income of Mexican families by 2010, which translates to a decrease in poverty of 6.8 million Mexicans. 2. In addition, secondary schools increased math achievement by 0.30 standard deviation in 3 years (during the Project) from an average PISA score of 355 to 385. Using the estimates and methodology elaborated by De Hoyos (2010) to measure the economic benefits of increased learning outcomes, the result could be translated to an increase of 16 percent of the annual income of Mexican families in 2010, which translates to a decrease in poverty of 3.88 million inhabitants in Mexico. While the result is for all the secondary schools in Mexico, PEC schools have contributed to this result (PEC schools outperformed other schools in ENLACE in the same period of time). 3. The incremental participation of indigenous and marginalized schools in PEC should help to better target the disadvantaged population in Mexico and thus improve the equity of basic education. During the 2010 to 2013 period, PEC indigenous schools decreased their failure rate by 4.5 percent, producing an increase in indigenous students’ years of schooling of 0.045 for each PEC cohort during the same period. 4. With the expansion of PEC and its potential effects on indigenous students, the average worker will have an additional 0.045 years of schooling in 2037, reaching 7.5 years of schooling, according to our calculations. Although this seems to be a small change, it is far from trivial. For instance, according to the ENIGH 2012, the average annual household per capita income in urban, semi-urban and semi-rural areas in Mexico is MX$144,525.7 pesos of 2012. Assuming a return to the additional year of schooling in these localities of 5 percent, every generation of project beneficiaries would experience an average increase of MX$325.18 (=MX$144,525*0.045*0.05) per year vis-à-vis the labor remuneration that the same generation would have obtained in the absence of the Project. The monetary effects for the affected fraction of the workforce of the increase years of schooling brought about by the expansion of the Project is defined by the following expression: B Y ∗ β ∗ ∆S ∗ P 5. Where B is the monetary benefits of the Project at time “t”, P is the workforce, ∆ is a change operator capturing the effects of the Project. Using the expression and assumptions above, plus the workforce of 59.0 million, in 2022 around 2.48 million will be affected by the expansion of PEC in indigenous schools. Additionally, the monetary benefits of the Project after one year of completion are equal to MX$806 thousands pesos (=MX$144,525*0.045*0.002*2.48 million) at 2012 prices. The sum of the flows of economic benefits up to 2037 amount to MX$12.1 million or US$896 thousand dollars at the official exchange rate of MX$13.5 per dollar (May 22, 2014). 25 6. During the same period, failure rate decreased by 3.4 percent for disadvantaged students in PEC, producing an increase in these students’ years of schooling of 0.034. When these individuals enter the labor market, the increased years of education will result in an increment of 0.187 percent in their monthly salary. 7. Finally, the costs of the Project were US$366.66 million (including counterpart funding) for the four years of the Project, or an average of US$91.66 million per year. The average federal annual unity cost per school decreased from Appraisal to Project closing because each year more schools were accepted into PEC (from 40,790 in 2009 to 50,624 in 2014, exceeding the original target of 50,000) while the budget increased only slightly from 1,477,418,501 in 2010 to 1,515,911,529 in 2014. The cost per school and student at the end of the Project were lower than at Appraisal (US$1,873 now versus US$2,247 at Appraisal and US$14 now versus US$16 at Appraisal, respectively). Therefore, efficiency increased through the Project’s life. In addition, comparing PEC’s student unit cost to other similar education interventions shows that PEC is relatively inexpensive and renders important gains in terms of improved public schools’ internal efficiency. This cost is considerably lower than the unit cost per student of PETC (which includes PEC’s SBM component and an extended school day, among others) where the cost per student in 2012 was US$167. 26 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Names Title Unit Specialty Lending Erik A. Bloom Senior Economist LCSHE Task Team Leader Mariangeles Sabella Senior Counsel LEGLA Senior Counsel Jose C. Janeiro Senior Finance Officer CTRFC Senior Finance Officer Social Safeguards Maria E. Castro-Munoz Sr Social Scientist LCSSO Specialist Maria E. Colchao Senior Program Assistant LCSHE Senior Program Assistant Financial Management Dmitri Gourfinkel Financial Management Analyst LCSFM Specialist Gabriel Penaloza Procurement Analyst LCSPT Procurement Specialist Ricardo Rocha Silveira Senior Operations Officer HDNGA Task Team Leader Isy Faingold ET Consultant LCSHE Economist Christoph Kurowski Sector Leader LCSHD Sector Leader Chingboon Lee Sector Manager LCSHE Sector Manager Tatiana Proskuryakova Senior Operations Officer LCSHE Senior Operations Officer Senior Procurement Tomas Socias Senior Procurement Specialist LCSPT Specialist Juan Carlos Serrano Senior Financial Management Senior Financial LCSFM Machorro Specialist Management Specialist Antonella Novali ET Temporary LCSHE Program Assistant Supervision/ICR Rafael E. de Hoyos Senior Economist GEDDR Task Team Leader Navarro Raja Bentaouet Kattan Program Leader AFCF1 Task Team Leader Wendy Cunningham Program Leader LCC1C Program Leader Reema Nayar Practice Manager GEDDR Practice Manager Janet K. Entwistle Senior Operations Officer GEDDR Senior Operations Officer Ciro Avitabile Economist GEDDR Economist Abril Alicia Ibarra Education Specialist/ICR ET Consultant GEDDR Castaneda Author Vicente Garcia Moreno Consultant GEDDR ICR Co-Author M. Yaa Pokua Afriyie Senior Social Development Senior Social GSURR Oppong Specialist Development Specialist Juan Carlos Serrano Senior Financial Management Senior Financial GGODR Machorro Specialist Management Specialist Gabriel Penaloza Procurement Specialist GGODR Procurement Specialist Fabiola Altimari Senior Counsel LEGLE Senior Counsel Anna Popova ET Consultant LCSHE ET Consultant 27 Christoph Kurowski Sector Leader LCSHD Sector Leader Erik A. Bloom Senior Economist LCSHE Task Team Leader Isy Faingold ET Consultant LCSHE Economist Maria E. Colchao Senior Program Assistant LCSHE Senior Program Assistant (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY09 2.92 14.03 FY10 39.71 214.07 Total: 42.63 228.10 Supervision/ICR FY11 25.91 112.67 FY12 23.16 115.49 FY13 33.54 129.27 FY14 24.89 79.40 FY15 4.50 18.67 Total: 112.00 455.50 28 Annex 5. Beneficiary Survey Results Not Applicable 29 Annex 6. Stakeholder Workshop Report and Results Not Applicable 30 Annex 7. Summary of Implementing Agency’s ICR This is a summary of the Borrower’s “Informe de Cierre de 2008-2013” translated into English and redacted by the Bank. Any mistakes are the sole responsibility of the Bank. A. ASSESSMENT OF THE PROJECT DEVELOPMENT OBJECTIVE This section describes the Project Development Objective and its components. B. ACHIEVEMENT OF OBJECTIVES The long term development objective specified that the Project should improve quality of education as measured by three indicators: Coverage of PEC, Social Participation, and Educational Outcomes. Coverage: As shown in the Figure 1, the target of the percentage of schools participating in PEC was exceeded in all years. The Project’s restructuring set a goal for 2013, which follows the trend of steady growth projected in previous years. In 2009, when the baseline was established, the number of schools in PEC was preliminary with a figure of 39,180; however, this baseline was adjusted in the April 2013 restructuring to 40,790 so the percentage of participating schools was 20.3 percent, rather than 18 percent as initially calculated. Figure 1: Percentage of Basic and Special Education Schools Participating in PEC Restructuring Target Schools with technical support and financing ‐ CIA Database 26.7 Schools 22 23.8 22.3 23 20.1 21 20.3 2009 2010 2011 2012 2013 Social Participation: Data was collected in 2009 (baseline) and in 2012 (follow up). The data was collected through a survey administered to a representative sample of randomly selected schools nationwide. The main results are presented in Table 1. Table 1: Main Indicators to Measure Social Participation Baseline Indicator 2012 ∆ 2009 1. Percentage of community members that know the PETE 76.8% 83.5% + 6.7%*** 2. Percentage of community members that participate in the design and adjustment of 74.2% 72.7% --- PETE 3. Percentage of community members that observe that decisions related to school aspects 90.2% 93.5% + 3.3%*** are taken in a participatory manner among school directors, teachers and parents. The column ∆ (change) refers to the differences between the indicators in 2008 and 2012 and if these are statistically significant. --- not significant, *significant at 10%, ** significant at 5% and *** significant at 1%. 31 Educational Outcomes: Overall educational outcomes of PEC schools improved, in terms of average test scores in math and Spanish in ENLACE. Figures 2 and 3 show ENLACE test scores for PEC schools throughout the length of the Project. Figure 2: Average ENLACE Percentages for PEC schools, primary Average Percentage in Spanish in Primary DGAG Database Average Percentage in Math in Primary DGAG Database 559.8 524.2 533.2 513.2 531.4 499.2 513.1 493.8 503.2 2008 2009 2010 2011 2012 Figure 3: Average ENLACE Percentages for PEC schools, secondary Average Percentage in Spanish in Secondary DGAG Database Average Percentage in Math in Secondary DGAG Database 507.4 537.6 493.7 517.7 505.0 485.2 488.5 471.0 476.0 481.2 2008 2009 2010 2011 2012 The following table illustrates some of the achievements of the Project and its PDO. Achievements Status Met. The program had 48,938 in the school year 2012-13 and has as a current 1. Number of schools in PEC increases figure of 50,624 beneficiary schools for the 2013-14 school year. The 25% (from 39,180 in 2009 to Program met its objective of increasing from 39,180 in 2009 to 50,624 in approximately 50,000 in 2013) 2013. Not Met: As mentioned in the social participation section, the percentage of community members that participate in the design and adjustment of the PETE reported in the first data collection in 2009 was of 74.2% and it was 2. Percentage of parents that expected that by 2013 the percentage would be of 80%. In the second data participate in the design of the PETE collection exercise, the percentage was of 72.7%, with a slight decrease of increases from 74.2% in 2009 to 1.5%; however, this figure was not statistically significant. The Program 80% in 2013. aimed for a third round of data collection to better inform these results, but, the administrative process to obtain the required budget for this exercise has not been concluded. 3. The percentage of PEC schools Exceeded: The Program met the specified percentage by obtaining a located in highly marginalized and percentage of schools in highly marginalized and marginalized areas marginalized areas increases from participating in the Program of 51.35% for the school year 2013-14.* 40% in 2009 to 50% in 2013. 4. The percentage of schools with Met. PEC has encouraged the inclusion of schools with indigenous students. students with indigenous students For the school year 2010-11, the percentage of schools with indigenous 32 Achievements Status increases from 7% in 2009 to 15% in students that participated in PEC was 17% and for the school year 2013-14 2013. that percentage was of 19.3%. * These figures were obtained with the M-U-R-CIA (Marginación Urbana Rural establecida por la Coordinación de Información y Análisis) method, which uses the urban level of marginalization determined by AGEBs (Áreas Geoestadísticas Básicas) and the level of marginalization by locality according to information from the national census. The method prioritized the urban level of marginalization in those schools where there is no marginalization level and put in second place the level of marginalization by locality. (This section also mentions the triggers for a third phase of the APL and their status, not included in this translation.) Achievements by Component Component 1: Direct Support to Schools PEC Phase X, for the 2010-11 school year. This phase involved 32 states that made their contributions according to PEC’s Operating Rules in an amount of MX$386,642,367.00. The Federal Government contributions were of MX$1,159,927,101.00. Some states also made additional contributions in the amount of MX$130,811,497.30, receiving a federal match of MX$154, 357,566.00 at a rate of 1.18 federal Mexican Pesos per each state’s Mexican Peso. PEC Phase XI, for the 2011-12 school year. This phase involved 32 states that made their contributions according to PEC’s Operating Rules in the amount of MX$416,795,291.00, with a federal match of MX$1,250,385,873.00. In this period, states also made additional contributions of MX$84,247,146.00, receiving a federal match of MX$151,644,862.80, at a rate of MX$1.80 for every state Peso contributed. PEC Phase XII, for the 2012-13 school year. This phase only involved 30 states that made their contributions according to PEC’s Operating Rules in the amount of MX$362,425,019.00, receiving a federal match of MX$1,087,275,056.89. In this period, states contributed additional resources in the amount of MX$95,936,053.00, receiving a federal counterpart of MX$95,936,053.00, at a rate of MX$1.00 for every state Peso contributed. Note that in this phase, Sonora and Veracruz did not participate in PEC. PEC Phase XIII, for the 2013-2014 school year. This phase only involved 30 states that made their contributions according to PEC’s Operating Rules in the amount of MX$459,406,529.00, receiving a federal match of MX$1,378,219,586.80. In this period, states made additional contributions in the amount of MX$31,808,411.04, receiving a federal counterpart amount of MX$31,808,411.04, at a rate of MX$1.00 for every state Peso contributed. Note that in this phase, Chiapas and Veracruz did not participate in PEC. Component 2: Monitoring and Oversight The Project supported the delivery of technical assistance to the 32 states that included: 1. Advisory Services. Provided relevant technical assistance for the effective implementation of the different operational processes of PEC, such as Operating Rules, calls for proposals, review process, incorporation, follow-up, assessment, transparency and accountability. 2. Accompanying Services. In states that required assistance from PEC’s national coordination unit, CNPEC worked closely with the PEC state coordination to ensure the adequate 33 implementation of the Project. 3. Supervision Visits. In states where areas of improvement were identified, mainly due to constant changes in state personnel, the Federal Government made several visits to provide support to the states on administrative, technical, and operational shortcomings. 4. Capacity Building Events. Initiatives at both the national and state level to foster stakeholder competencies in educational management and the leadership of supervisors and school directors of participating schools. 5. National and regional meetings. With the goal of following up, providing feedback, and sharing good practices, national and regional meetings were held with state teams and, in some cases, school directors and supervisors, parents, and education authorities. Other activities under this Component were financed with Government resources. Component 3: Policy Development and Evaluation This component was eliminated in the restructuring approved by the Bank on May 2, 2013. However, many of the activities envisioned were undertaken using non-Project resources. Among others:  Annual External Evaluation Reports were conducted by CONEVAL.  The Centro de Estudios Educativos, A. C. performed the Feasibility Analysis for PEC’s Impact Assessment and concluded that the program has the necessary and sufficient characteristics to measure its impact on student learning.  A meeting with staff of the General Directorate of Indigenous Education (Dirección General de Educación Indígena, DGEI) was conducted to present the origins, history, evolution, and main guidelines of PEC and its Strategic Model of School Based Management (MGEE), the simplified PETE, to implement the Indigenous Peoples Plan and promote increased participation of indigenous schools in PEC.  In 2012, the Centro de Estudios Educativos, A. C. conducted a study to integrate a State of the Art of the Strategic and Participatory Management Process of the PEC and Escuela Segura programs.  A Beneficiary Satisfaction Survey was conducted in 2012 by the DGAG to explore the degree of satisfaction of school directors, in terms of the goods and services that PEC provided to schools.  PEC was assessed as a federal program with a “high potential of performance” by the Performance Index of Federal Public Programs (Índice de Desempeño de los Programas Públicos Federales, INDEP 2012), meaning that it has a sound design and that it has adequately achieved its goals, but might lack enough resources to attend all its potential beneficiaries. C. MAIN FACTORS AFFECTING THE IMPLEMENTATION OF THE PROJECT Factors outside the control of the Government and/or PEC. Although the educational federalism and autonomy exercised by the states has strengthened the program, it has also resulted in a number of drawbacks in its implementation and monitoring: (i) Budget planning in the states has caused delays in the definition of participating schools, and even non-participation in the program, at certain stages of its implementation. 34 (ii) In some cases, the extensions granted for delivery of the matching resources from states have delayed the access of schools to federal resources and PEC resources in some states. (iii) The lack of uniformity in the implementation and dispersion of resources subject to state matching complicates the periodic reconciliation of the amounts at the central level and the timely delivery of resources to schools, often causing panic buying and poor strategy implementation and reporting at the school level. (iv) The resistance of some agencies involved in implementing the program, such as teachers’ unions, which have resisted, in some states, the application of standardized assessments and consolidation of the Program. (v) The Execution System for the Procurement Plan (Sistema de Ejecución para el Plan de Adquisiciones, SEPA) should improve efficiency in procurement and serve as a tool to promote transparency and strengthen accountability instead of an obstacle. Factors generally subject to the Government. (i) The rotation of mid-level managers and operational staff caused by the instability of their working conditions, both at the federal and state level, have caused a lack of continuity and adequate follow-up and monitoring processes and integration of program information. (ii) The lack of personnel in the state and the national coordination units to support the implementation of PEC in schools in remote areas. Factors generally subject to PEC. The Information System, SIPEC, was not as effective and efficient as expected in recording, organizing, and exploiting information from the participating schools due to technical failures, server capacity, compatibility, operation and maintenance. Costs and Project Financing (i) Specify the date of the last disbursement of the Bank loan. Letter ICL-413-2014 dated June 18, 2014 issued by NAFIN notified that the Bank completed Disbursement Request No. 8, thereby disbursing the full amount of Loan 7948-MX. (ii) Comments on the loan disbursement schedule. Although SEB sought to comply with the disbursement projections established in the PAD, they were not met mainly due to a delay in signing of the Loan Agreement. This did not, however, present any difficulty to SEB in executing the entire loan and only involved a change in the amounts to be disbursed for the fiscal year, as per the table below. Years PAD Estimates Actual Disbursement 2010 0.00 44,550,000.00 2011 116,000,000.00 70,305,704.90 2012 72,000,000.00 73,657,750.47 2013 3,200,000.00 30,763,060.48 2014 0.00 723,484.15 (iii) Comments regarding the availability of funds. National counterpart funding was considered exclusively for Category 1 (Goods, Works and Consultant’s Services supported under direct support to schools). This represented 10% of the total eligible resources for this Category. (iv) Summarize the results of the Project audits and the level of compliance with the audit recommendations. The Borrower’s Informe de Cierre provides a detailed explanation of 35 audits: 2/2012 for the execution of 2011 funds; 5/2013 for the execution of 2012 funds; 2/2014 for the execution of 2013 funds; 234 for the Fideicomiso Fondo Nacional para Escuelas de Calidad; and Auditoría Externa for the Fideicomiso Fondo Nacional para Escuelas de Calidad. (v) Evaluate the system of accounting and financial control of the Project (Coordination of Financial Management). The Coordination of Financial Management (CGF) at all times complied with the goals and commitments outlined in the Loan Agreement and its Annexes, the Operational Manual, and the Contrato de Mandato, in coordination with each of the participants in the Project implementation areas. This unit (CGF) was responsible for implementing transfers to the states, requesting disbursements, and compiling quarterly and semiannual monitoring reports while ensuring effective and appropriate management of resources and information systems. Some systems implemented during project execution were:  Planning system: based on the Expenditure Budget of the Federation (PEF) for each fiscal year, as well as financial planning as established in the PAD.  Cash Flow System: Using defined processes for transfers to states that allowed the monitoring and recording of financial transactions.  Information system: The CGF was responsible for compiling the quarterly, semi-annual, and monitoring reports describing the financial and operating performance of the Project in a given period in relation to the components.  Control System: physical and electronic records were generated of all documents related to the Loan for internal control and audits carried out for the Project. In regards to contractual and statutory obligations of the Loan, the SEB complied every time and always under the accompaniment of NAFIN. (vi) Assess the procurement system. The SEPA web-based system was used for the reporting of the Procurement Plan. The information was uploaded into the system according to DGDGIE’s planning for transfers and procurement. No consulting services were carried out under Bank procurement processes. SEB, therefore, requested a reallocation of Loan proceeds to increase the amount allocated to Category 1 (Goods, Works and Consultant’s Services supported under direct support to schools) in order to increase the number of beneficiary schools. This also reflected SEB’s inability to exercise all of the resources for Category 2 (Incremental Project Costs) and Category 3 (Consultant Services). It is important to note that consulting services were carried out under national processes. The inability to perform these services was due mainly to issues that did not depend directly on the implementing agency. Original Category Reallocated Amount Modified Amount Amount 1. Goods, Works and Consultant’s Services 211,720,000.00 6,729,769.00 218,449,769.00 supported under direct support to schools 2. Incremental Project Costs 3,300,000.00 -2,299,769.00 1,000,231.00 3. Consultant Services 4,430,000.00 -4,430,000.00 0.00 4. Front-end Fee 550,000.00 0.00 550,000.00 36 D. PROJECT SUSTAINABILITY The Project is financially sustainable, derived from the interest of the SEB to continue receiving the Bank’s support and technical assistance. PEC is a SEP initiative aimed at strengthening basic education and SBM as part of the provisions of Articles 3. of the Constitution; 77 of the Federal Budget and Fiscal Responsibility; 176 of the Rules; 30, 31 and 40 and Annex 24 of the Expenditure Budget of the Federation for Fiscal Year 2015. The Government has the obligation of improving educational processes and outcomes. It is necessary to provide an inclusive education that respects and values diversity sustained in intercultural relations, which successfully combines fairness with quality in the pursuit of greater equality of opportunity for all Mexicans. In the Decree of Articles 3 and 73 of the Constitution posted February 26, 2013 in their Fifth Transitory, Section III, paragraph a) is amended, establishing the need to "strengthen the autonomy of school management in order to improve its infrastructure, purchase educational materials, solve problems and promote basic operating conditions of participation for students, teachers and parents, under the leadership of the director, be involved in resolving the challenges that each school faces.” In this context, Constitutional Article 3 adds, among other things: "The State shall guarantee the quality of compulsory education so that the educational materials and methods, school organization, educational infrastructure and the adequacy of teachers and managers ensure crowning achievement of student learning." The General Education Law, Rules 12, Section V and XII Bis; 25 and 28 Bis establishes guidelines for the Program to reorient their strategies and actions to contribute to the fulfillment of the constitutional mandate for autonomy in school management. The National Development Plan 2013-18 establishes as a national goal “Mexico with Quality Education.” (A comprehensive account of the Plan and its relation to school autonomy and SBM is also provided in this section – not included in this translation). In this context, the Program compiles the experience and lessons learned from recent years that has driven the transformation of the management of public elementary schools. The goal is to reorient their strategies and actions to help achieve the constitutional mandate for autonomy in school management, as well as to strengthen actions leading to the formation of the Basic Education System and the provision of technical and financial support to public elementary schools in the creation and consolidation of conditions that enable them to strengthen their organizational culture and performance through the development of school management and autonomy capacities. E. PERFORMANCE EVALUATION OF THE PROJECT This section summarizes the available evidence and information from external evaluations and studies of PEC since the beginning of the Second Phase of the School Based Management Project. Program Strengths  Coverage and Targeting. There is great progress in terms of coverage (schools in 90% of municipalities nationwide participate in the program) (PEC, 2012). In recent years, targeting efforts increased participation of indigenous schools and schools in areas of high and very high marginalization. (World Bank, 2012)  Training and Continuous Professional Development. School directors and other project beneficiaries received training for the development of the PETE and the PAT. (Mayaink, 2008) 37  Achievement and School Performance. The Program reduced dropout rates and decreased student failure and repetition rates (C230 AC, 2012).  Communication and Dissemination. The Program offers vast public information on the regulations, beneficiaries, and main results, which can be found on PEC’s website. (Bracho González, 2008)  Operation of the Program. The Program has documented procedures and mechanisms for its operation, such as receipt, recording and processing of applications; the selection of beneficiaries; and the granting of school support at every stage. (White 2011-2012). SEP had steady and constant support from the Bank throughout Project implementation. Monitoring and support was provided by each of the three Task Team Leaders and their teams. The Bank also provided technical assistance to solve problems arising from Project implementation. The Bank played an active role in detecting, reporting and working in those areas that needed improvement. This observation applies equally to the financial agent, NAFIN, which was key to Project performance through the provision of strategic support when needed. Program Weaknesses  Coverage and Targeting. Targeting criteria was limited by the selection criteria, which included all public schools located in areas of "Very High, High, Medium, Low, and Very Low marginalization” (Briefing Note, Target Population. CNPEC 2011 and White, 2011-2012). There is no strategic planning related to the Program’s coverage, at least in the short and medium term (White 2011-2012).  School Management and Strategic Planning. The Program can create an administrative and academic overload to the schools (Miranda Lopez, 2008 and Centre for Educational Studies, 2012).  Training and Continuous Professional Development. Insufficient training and technical, academic, and institutional support for developing the necessary theoretical and practical skills directly related to school management. (Velázquez, 2010). Lack of knowledge regarding the adequacy and usefulness of the training received by school directors in the development of PETE and PAT. (White 2011-2012)  Achievement and School Performance. The impact of the Program on educational achievement and teacher performance does not grow in relation to the number of participating schools and school directors trained in strategic management. (CONEVAL, 2008)  Perception of the Value and Utility of PEC. Information related to the perceptions about PEC of General Directors in each state are collected, but not about the perception of direct beneficiaries. (White 2011-2012)  Program Operation. There is a lack of effective engagement of Supervisors and Technical Educational Consultants (ATP) to provide advice and support in the Program’s operation at the school level (Centre for Educational Studies, 2012). The positive experiences of the participating schools are not exploited. The schools’ technical and pedagogical reports are not used adequately to better understand the problems they face (in particular for the urban and marginalized schools) (Bracho González, 2008). Lack of stability of PEC staff and high uncertainty in job permanency and job benefits. Particularly in the case of ATPs. (Bracho Gonzalez, 2008)  Accountability. Failure in measuring activity level indicators for the components in the MIR (White 2011-2012). Need of end of Project indicators to measure the improvement in learning outcomes comparing PEC schools vs non-PEC schools (White 2011-2012). There are problems 38 of consistency in the Operating Rules indicators. Formulas are not sufficiently clear or relevant, or have no relevance for measuring institutional program objectives (CONEVAL, 2008).  Information systems and evaluation. The Program needs a comprehensive set of indicators that cover various aspects of school planning and can be measured through surveys administered to representative samples of schools (Bracho González, 2003) (White, 2011-2012). There are limitations to reach valid conclusions about the education system since most data comes from the first years of operation of PEC. (White 2011-2012) There are not sufficient monitoring actions to update the Program’s indicator matrix and insufficient monitoring of the institutional objectives of PEC at the federal and state levels. (Bracho González, 2008). SIPEC fails to provide timely and relevant information. The SIPEC has limitations in monitoring the operation and results of the program. ((Bracho González, 2008 and Santibanez, 2008). The user of SIPEC or the general public cannot verify the source of information with which the application is built and no information on updating databases is provided (White 2011-2012). The databases are disjointed and decentralized, causing confusion and inaccuracies in the data at the national level, entity and program phase (White 2011-2012). Few qualitative studies (different impact assessment) on some of the crucial processes of the program: training of school personnel, processes and formulation of PETE and PAT and their implementation in the schools, social participation and pedagogical changes (White 2011-2012). Lack of and impact evaluation to identify the causal effects of the program with data covering all years of operation (Santibanez, 2008). The Project could not use the funds under the Category 3 of the Bank Loan (Consultant Services) for activities under Component 3 of the Project (Development and Policy Evaluation) and needed to restructure the Project to reallocate from Categories 3 and 2 to 1. This was due to internal administrative obstacles to proceed with the Bank’s procurement guidelines. Program Opportunities  In Relation to the Education System. The contribution of the Program to improve quality of basic education is complemented with other federal programs that address various issues and efforts to improve educational conditions for students such as Escuela siempre abierta and Oportunidades (White 2011-2012). PEC has the potential to articulate and coordinate all educational actions at the school level from the local to the federal levels. (CONEVAL, 2010) Risks to the Program  School Management and Strategic Planning. Schools have limited autonomy in terms of personnel management (World Bank, 2012)  Education System Facts that Affect PEC’s functioning. The education system does not have a comprehensive database of schools, the population served, and the communities where they are located (Centre for Educational Studies, 2012). At the federal level, there is a lack of articulation between the programs that aim to reach schools and the state teams have limitations for training and providing support to PEC schools (Centre for Educational Studies, 2012). ENLACE shows limitations to measure the specific impact of PEC in students’ educational attainment over time (Bracho González, 2008 and Center for Educational Studies, 2012).  School-level Facts that Affect PEC’s functioning. There are important challenges to build a common vision that gives identity and helps school stakeholders commit to school-based management (Miranda Lopez, 2008). Schools have weak capacities to build a shared vision and act collectively, identify and prioritize school problems, articulate effective and collective 39 interventions to solve those problems, and lack feedback and evaluation mechanisms. (Miranda Lopez, 2008).  State-level Facts that Affect PEC’s functioning. In some cases, noncompliance of the national selection and targeting criteria by the state education authorities result in a large variation in the understanding of the program and its institutional objectives (Abreu, 2010). F. LESSONS LEARNED The 4 years of operation of Phase II of PEC have produced important lessons learned that have made the Program and education policies around it evolve, including:  PEC X contributed to the improvement of educational attainment through fostering strategic educational management;  PEC XI established strategic educational management to strengthen the organizational culture and functioning of schools, focusing on students’ needs in order to contribute to improved learning outcomes;  PEC XII helped improve the academic achievement of students in public elementary schools, sharpening targeting criteria (equity) and establishing a regional vision; and,  PEC XIII strengthened autonomy and SBM and encouraged collaboration of the school community in solving the challenges faced by each school for the improvement of learning outcomes. PEC established a model of SBM based on democratic principles that identify the school as the center of all educational initiatives and drives on the school community to strengthen autonomy, as well as the principles of equity, relevance, effectiveness, and efficiency to promote quality of education through various elements. Among these are shared leadership, collaborative work, strategic planning, monitoring and evaluation for continuous improvement, responsible social participation, shared decision making, flexible practices and relationships, project financing, and transparency and accountability. The development of these elements in PEC participating schools have created a more effective use of resources and better identification of school needs. Ultimately, PEC has helped improve the quality of education as a result of a more efficient and transparent use of resources to generate more relevant and contextualized actions at the school level. PEC has contributed to a more open and welcoming school environment because the community is involved in its administration. Increased participation of all local stakeholders in decision-making processes also encourages more participatory, collaborative decision-making; greater satisfaction for all students; better student performance as a result of lower repetition and dropout rates, and eventually better learning outcomes. Finally, it should be noted that these lessons are players in the current educational reform to strengthen autonomy and school management. In addition, basic education schools have a school organization that ensures maximum learning achievement for all students by ensuring the effectiveness and efficiency of educational services provided in schools. 40 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Not applicable 41 Annex 9. List of Supporting Documents Abreu, R. C.-M. (2010). Evaluación de impacto del proyecto piloto PEC-FIDE. Fundación IDEA A.C. Recuperado de SEP-PDF. Blanco, E. (2011-2012). Evaluación de Consistencia y Resultados 2011-2012. Blimpo, M. and D. Evans. 2011. School-Based Management and Educational Outcomes: Lessons from a Randomized Field Experiment. World Bank's Africa Program for Education Impact Evaluation. Available at: http://siteresources.worldbank.org/EDUCATION/Resources/Blimpo-Evans_WSD-2012-01- 12.pdf Borman. G. G.M. Hewes, L.T. Overman and S. Brown. 2003. Comprehensive School Reform and Achievement: A Meta-Analysis. Review of Educational Research. American Education Research Association. Summer. Vol. 73. No. 2. pp. 125–230. Bracho González, T. (2003). Programa Escuelas de Calidad PEC 2002. Segunda evaluación externa. CIDE. Bracho González, T. (2008). Informe Final de la evaluación de consistencia y resultados del Programa Escuelas de Calidad PEC VII. Available at: http://basica.sep.gob.mx/pec/pdf/evaluaciones/informe2007.pdf. C230 A.C. (2012). Evaluación no experimental del PEC. México: C230. Centro de Estudios Educativos. (2012). Análisis de los materiales encontrados e identificación de fortalezas y debilidades de los programas. CONEVAL. (2008). Informe de la Evaluación Específica de Desempeño . CONEVAL. (2010). Informe de la evaluación específica de desempeño 2009-2010. De Hoyos. (2010) Dinero y Crisis Educativa. Revista Nexos. Available at: http://www.nexos.com.mx/?p=14061 Duflo, E. P. Dupas and M. Kremer. Revised September 2014. School Governance, Teacher Incentives, and Pupil-Teacher Ratios: Experimental Evidence from Kenyan Primary Schools. Working Paper available at: http://web.stanford.edu/~pdupas/ETP_paper.pdf Gertler, P.J., H.A. Patrinos and M. Rubio-Codina. September 2012. Empowering parents to improve education: Evidence from rural Mexico. Journal of Development Economics. Volume 99. Issue 1. pp. 68–79. Government of Mexico. 2013. Plan Nacional de Desarrollo, 2013-2018. Poder Ejecutivo Federal (Presidencia de la República). Mexico. 42 Hanushek E.A, E. Linkd, and S.Woessmann. L. September 2013. Does school autonomy make sense everywhere? Panel estimates from PISA. Journal of Development Economics. Volume 104. pp. 212–232. Mayaink, G. (2008). Levantamiento de encuestas de impacto entre los actores escolares (padres de familia, alumnos, directores de planteles, y supervisores escolares) de escuelas incorporadas al Programa Escuelas de Calidad (PEC). Miranda López, F. S. (2008). Programa Escuelas de Calidad.Evaluación Externa 2008. Recuperado de FLACSO-PDF. Murnane R.J., S.J. Willet and S. Cardenas 2006 Did the Participation of Schools in Programa Escuelas de Calidad (PEC) influence students’ outcomes. Working Paper, Harvard University, Graduate School of Education, Cambridge, MA. PEC. (2012). Lorgros yResultados del PEC 2001-2012. México: PEC Pradhan, M.and J. de Ree. June 2014. District Governance and Student Learning in Indonesia. Asian Development Bank. Economics Working Paper Series. 397. Pradhan, M. et al. September 2011. Improving Educational Quality through Enhancing Community Participation Results from a Randomized Field Experiment in Indonesia. Policy Research Working Paper 5795. The World Bank. East Asia and Pacific Region. Human Development Sector Department Santibáñez, L. A. (2008). Evaluación específica de desempeño. c230. Secretaría de Educación Pública (Diario Oficial de la Federación). December 28, 2013. ACUERDO número 703 por el que se emiten las Reglas de Operación del Programa Escuelas de Calidad. Secretaría de Educación Pública (Diario Oficial de la Federación). December 28, 2013. ACUERDO número 704 por el que se emiten las Reglas de Operación del Programa Escuelas de Tiempo Completo. Secretaría de Educación Pública. 2013. Programa Sectorial de Educación 2013-2018. Mexico. Secretaría de Hacienda y Crédito Público, Secretaría de Educación Pública y Nacional Financiera, S.N.C., I.B.D. Informes Semestrales de Avance. Proyecto Escuelas de Calidad, Fase II. Préstamo 7948MX. Skoufias, E. and J. Shapiro. 2006. Evaluating the Impact of Mexico's Quality Schools Program: The Pitfalls of Using Nonexperimental Data. World Bank, Washington, DC. Available at: Velázquez, C. (2010). Evaluación Específica de desempeño 2010-201 43 World Bank. (2012). Good to Great through School Autonomy and Accountability in Mexico. World Bank. May 20, 2010. Project Appraisal Document on a Proposed Loan in the amount of US$220 Million to the United Mexican States for a School-Based Management Project in support of the Second Phase of the School-based Management Program. Report No. 52955-MX. Washington, DC. World Bank. November 9, 2010. Loan Agreement. LOAN NUMBER 7948-MX. Washington, DC. World Bank. April 19, 2013. Restructuring Paper on a Proposed Restructuring of the School based Management Project – Phase II. Loan 7948-mx Approved by the Board on June 17, 2010 to the United Mexican States. Report No. 73325-MX. Washington, DC. World Bank. May 2, 2013. Amendment to the Loan Agreement (73325-MX). Washington, DC. World Bank. Aide Memoires and Back-to-Office reports for Preparation and Supervision missions. World Bank. Implementation Status Reports (ISRs) for the School-Based Management Project in support of the Second Phase of the School-based Management Program. 44 MAP 45