Technical Assistance for Local Governments to Improve Education Outcomes AN IMPLEMENTATION GUIDE INSPIRED BY THE CASE OF CEARÁ, BRAZIL Technical Assistance for Local Governments to Improve Education Outcomes: An Implementation Guide Inspired by the Case of Ceará, Brazil This document presents a detailed implementation guide for a Technical Assistance (TA) system for improving education outcomes, inspired by the successful case of the state of Ceará in Brazil, which achieved universal literacy. The document provides concrete steps for governments that are willing to establish a TA system for subnational governments to improve education outcomes with a focus on learning. This document was prepared by André Loureiro, Fátima Alves, Louisee Cruz, Matheus Assunção, and Thiago Cardoso in the context of the REACH Trust Fund: Supporting Education Systems to Replicate the Brazil - Ceará Results-Based Financing Model to Reduce Learning Poverty. acknowledgements The authors would like to thank the invaluable exchange and collaboration with the state secretary of education of Ceara and the municipal governments that make the Ceará model of education a success, particularly the municipalities of Sobral, Quixeramobim, Ararendá, Milhã, Coreaú e Dep. Irapuan Pinheiro. A special thanks also to Lemann Foundation, Bem Comum Association, Natura Institute and Movimento Colabora, as well as to several World Bank colleagues, including Reema Nayar, Victoria Levin, Julia Liberman, Michael Crawford, Ezequiel Molina, Diego Bazaldua and Halsey Rogers for comments and suggestions. This work is sponsored by the Results in Education for All Children (REACH) Trust Fund. Table of Content A successful example of technical assistance to local governments 1 —  to improve education quality: The case of Ceará in Brazil 6 The technical assistance framework for municipal 2 —  secretariats of education in Ceará 10 3 — Laying the groundwork  15 4 — Implementation Roadmap 19 STAGE 1  19 Establish a learning assessment to 1.1 —  guide the intervention 20 Conduct a broad evidence-based 1.2 —  diagnostic of the education system 20 1.3 — Define and communicate targets 23 STAGE 2  24 Design structured materials using the 2.1 —  literacy assessment results  24 Develop a teacher and school management 2.2 —  training program 25 Develop an incentive mechanism with high and 2.3 —  low performance schools working together 26 2.4 — Create initiatives to foster reading 27 2.5 — Organize an ECE Expansion Plan 28 STAGE 3  29 3.1 — Distribute the structured materials 29 Train regional and local teams to conduct 3.2 —  teacher and school management training 30 3.3 — Conduct the classroom assessment 32 3.4 — Implement the monitoring tools 32 3.5 — Foster innovation at local levels 33 STAGE 4  34 4.1 — Conduct a standardized learning assessment 35 Reward best performers and support 4.2 —  those lagging behind 36 4.3 — Analyze literacy outcomes 36 5 — Implementing the model in different contexts 39 What if the student learning assessment system is not 5.1.  consolidated or does not exist? 39 What if there is no monitoring and 5.2.  evaluation system in place? 39 5.3.  What if the curriculum is not updated or well-established?  40 What if the supply chain for the printing and 5.4.  distribution of learning materials is limited? 40 5.5.  What if there is diversity in language in the region? 41 What if there is limited decentralization or little financial 5.6.  autonomy at local level? 41 6 — Fundamental aspects of policy continuity 43 01 –––––––––– 02 • 03 • 04 • 05 • 06 A SUCCESSFUL EXAMPLE OF TECHNICAL ASSISTANCE TO LOCAL GOVERNMENTS TO IMPROVE EDUCATION QUALITY: THE CASE OF CEARÁ IN BRAZIL 1 — A successful example of technical assistance to local governments to improve education quality: The case of Ceará in Brazil The state of Ceará, Brazil, transformed its education system and is currently among the top performers in the country, despite the state’s low socioeconomic status. Cea- rá is a poor state with around 9 million people, roughly the size of Austria or Papua New Guinea. It is the fifth poorest of Brazil’s 26 states. Yet, Ceará has seen the largest increa- se in its national education quality index in both primary and lower secondary education since 2005, with 10 municipalities among the top 20 in the national ranking. A set of education reforms initiated in 2007 led the state to occupy fourth place among all Brazi- lian states in the national ranking of education quality in lower secondary education and sixth place in primary education. A key driver of success in Ceará was technical assistance (TA) to the state’s munici- palities. In 2007, the Ceará government established the ‘Achieving Literacy at the Right Age Program’ (Programa de Alfabetização na Idade Certa, PAIC), which set out a clear goal: all students in public schools should read by the end of grade 2 (age 7). Unlike most states in Brazil at that time, the large majority of primary students in Ceará were already enrolled in public schools run by their municipal government1. To achieve PAIC’s goal, the state govern- ment provided substantial technical support to the 184 municipalities of Ceará. Municipal education secretariats receive structured learning materials that provide a clear routine for classes and prioritize basic skills, especially literacy, in the early grades. Teachers receive re- gular training on the use of those materials, including classroom observation and feedback. Low-performing municipalities get additional support. The state also provides budget and curriculum support for early childhood education (ECE) to ensure that children get started right. The state provides training and materials to help municipal education secretariats in- crease teaching time in classes, adopt meritocratic criteria for the selection of school princi- pals, and to provide incentives—financial and non-financial—for teachers whose students are meeting literacy targets. Ceará put learning at the center of its education strategy, with early literacy as the foundation. The Ceará government adopted a supervisory role in basic education—a very proactive approach for a state government in Brazil. In addition to TA, it established incentives for municipalities to achieve education outcomes, accelerated the decentra- lization of the management of lower secondary schools to municipal governments, and established regular monitoring of learning along with accountability actions. At the heart 1  In the following years, many state governments followed that trend and accelerated the process of devolving public primary schools to municipal administrations. As of 2018, all states have the majority of primary education students enrolled in municipal schools, and in many states this percentage is close to 100%. 7 __ of this series of reforms was sustained political leadership and focus.2 Sustained Political Leadership Figure 1 — The conditions for success Fiscal Technical in Ceará incentives for assistance to municpalities to achieve Achieve municipal school education outcomes literacy for all networks in early grades Municipalities with Regular autonomy and monitoring of accountability learning followed to achieve learning by action Learning is the main driver of improvements in education quality observed in the Ceará municipalities. The top education quality performers in Ceará, according to the national edu- cation quality index (IDEB),3 are also among those with the greatest improvements in the lear- ning component of the IDEB between 2005 and 2017. Improvements in education quality in Ce- ará can also be observed in the learning poverty index, as there are decreases in the proportion of 10-year-olds who cannot read and interpret a simple text or who are out of school (figure 2). Figure 2 — Learning poverty for Brazilian municipalities, 2017 (82,98] (67, 82] (51, 67] (35,51] (20,35] (4,20] Source: World Bank with INEP/MEC data. 2 For more details on the implementation of a results-based financing mechanism in education, see the companion guide: Holanda, M.; Barbosa, M., Cruz, L.; Loureiro, A. Implementing a Results-Based Financing Mechanism for Subnational Governments to Improve Education Outcomes: an Implementation Guide Inspired by the Case of Ceará, Brazil, World Bank, 2020. http://documents1.worldbank.org/curated/en/561471606111232725/pdf/ Implementing-a-Results-Based-Financing-Mechanism-for-Subnational-Governments-to-Improve-Education- Outcomes-An-Implementation-Guide-Inspired-by-the-Case-of-Ceara-Brazil.pdf 3 Índice de Desenvolvimento da Educação Básica (IDEB) is composed of student learning scores and school progres- sion rates in Portuguese and mathematics. 8 __ Education outcomes for schools in Ceará are substantially higher than expected when considering their socioeconomic context. The distributions of IDEB scores for municipal schools with different socioeconomic conditions, measured by the national socioeconomic in- dex (INSE), are presented in Figure 3, highlighting the schools in Ceará municipalities that have the highest levels of IDEB in both primary and lower secondary education. All schools in Ceará have low levels of socioeconomic development, and most are in the upper half of the distribu- tion of IDEB scores. A deeper look at the top 20 municipalities in Brazil with regard to IDEB in primary education reveals that all 10 Ceará municipalities in the list are lower on the school socioeconomic index than all other municipalities in the southern states of Brazil.4 Figure 3 — 10 Education quality measured 8 by IDEB (2017) Primary Education vs. socioeconomic 6 status (INSE, 2015) - municipal schools - primary 4 and lower secondary 2 education 0 0 20 30 40 50 60 70 Socioeconomic Level Sobral Ceará Brasil 8 Lower secondary education 6 4 2 0 20 30 40 50 60 70 Socioeconomic Level  Sobral Ceará Brasil Source: World Bank with INEP/MEC data 4 With the exception of Coruripe in Alagoas. For more information on how Ceará improved its education outcomes, see: Loureiro, Cruz, Lautharte, and Evans. The State of Ceará in Brazil is a Role Model for Reducing Learning Po- verty, World Bank, 2020. https://openknowledge.worldbank.org/handle/10986/34156 9 __ 01 • 02 –––––––––– 03 • 04 • 05 • 06 THE TECHNICAL ASSISTANCE FRAMEWORK FOR MUNICIPAL SECRETARIATS OF EDUCATION IN CEARÁ 2 — The technical assistance framework for municipal secretariats of education in Ceará A key pillar of the PAIC in Ceará is technical assistance for the municipal secretariats of edu- cation. The technical assistance (TA) in Ceará for municipal secretariats of education takes place in a context in which the Brazilian constitution establishes responsibility for different levels of education for each of the three levels of government. The provision of public early childhood education, primary education (grades 1 to 5), and lower secondary education (grades 6 to 9) are assigned to be the respon- sibility of municipal governments, while public upper-secondary education (grades 10 to 12) is provided by state governments. The federal government focuses on tertiary education but provides supplemen- tary support to pre-university education and establishes the national norms for education at all levels. State governments have a role in the coordination of education policies for their municipalities. The technical assistance provided by the Ceará state government to the municipal se- cretariats of education has three main components: 1. Providing literacy support to the municipalities; 2. Strengthening the governance of municipal secretariats of education and providing incentives to promote the exchange of best pedagogical practices; 3. Promoting the pedagogical use of student assessment results. Each of the components of the technical assistance for municipal secretariats of education in Ceará is described in Figure 4 and Table 1. LITERACY SUPPORT TA1 TA2 STRENGTHENING GOVERNANCE AND CREATING INCENTIVES Designing scripted 1.1 –  learning materials 1.2 – P roviding direct Creating a central 2.1 –  support to governance to establish teacher support and train training on literacy local teams 1.3 – Creating activities TA3 2.2 – G enerating to foster reading incentives for 1.4 –  Giving support exchanging best to expand and practices among improve the quality school systems of early childhood education (ECE) PEDAGOGICAL USE OF STUDENT ASSESSMENT RESULTS Figure 4 — Components and Subcomponents of the Providing financial and technical support for the 3.1 –  Technical Assistance for implementation of local learning assessments Municipal Secretariats Training at local and school levels to allow a 3.2 –  of Education in Ceará systematic use of assessment results 11 __ Table 1 — Components and Subcomponents of the Technical Assistance for Municipal Secretariats of Education in Ceará 1. Providing literacy support to the municipalities Scripted The state government provides municipalities with structured learning literacy booklets, especially for resource-constrained materials municipalities. These materials present concrete guidance for the use of time and the indicated activities, with a clear pedagogical routine proposed for each class. The materials are aligned with the state curriculum, which has a clear learning sequence prioritizing foundational skills, particularly literacy at the right age. Support for Municipal secretariats of education receive training to improve municipal the use of resources. State tutors conduct regular training for a teacher training group of technical personnel from municipalities to strengthen their focused on teacher training activities. In municipalities with critically low literacy learning levels, the state trains the teachers directly. Teacher training is focused materials and on the use of the structured material and the classroom routine and classroom practices, rather than theoretical knowledge. These actions also practice involve classroom observations by teacher trainers and are based on the booklets that teachers use as part of their daily routine. Activities to The state government encourages municipalities to carry foster reading out activities to develop reading skills. These activities include the establishment of a small library for students and teachers in each classroom and a magazine to share best practices among teachers to foster reading outside of school. Support to The state provides guidance for municipalities to develop a expand and curriculum focused on school readiness and age-appropriate improve early development. The state team offers training for teachers and childhood administrative staff to develop an appropriate pedagogical curriculum education (ECE) for ECE and co-finances the construction of ECE centers. 12 __  trengthening the governance of municipal secretariats of education and 2. S providing incentives to promote the exchange of best pedagogical practices Implementing The state government implements a collaborative governance central arrangement with solid monitoring and evaluation of municipal governance to education policies. This governance is structured in three levels: 1) support and State level coordination, responsible for engaging political leaders in each train municipal municipality and monitoring and supporting regional teams. 2) Regional teams teams that support and monitor municipal education departments in management, pedagogical, and training aspects. 3) Municipal teams, responsible for training teachers and principals in their schools, monitoring learning targets, and supporting the implementation of ECE. Strengthening The municipal secretaries of education gradually gain expertise municipal in learning monitoring and pedagogical management. TA aims to pedagogical strengthen municipal capacity to implement and monitor a structured management literacy policy with solid pedagogical management. The state government designs monitoring indicators and instruments to support schools and municipalities. School leaders regularly monitor student performance, especially in reading; attendance of students, teachers, and school staff; the number of books read by students; the use of structured material; and compliance with the school days established by law. At the municipal education department, TA promotes the structuring of processes in the following topics: elimination of multi-grade classes; effective use of the pedagogical time and the school year; strengthening school autonomy; incentives for teachers of literacy classes; the adoption of meritocratic criteria for selecting school principals. Providing The state government establishes incentives for exchanging incentives for good practices in pedagogical management and fosters healthy exchanging best competition. At the end of every year, the Escola Nota 10 Prize is awarded practices among to the best-performing teachers and their schools receive financial and school networks social rewards. To fully receive the prize, high-performing schools need to assist low-performing schools over the course of one year. If the low- performing school improves its performance in the following year, it also receives a prize. This strategy fosters the exchange of practices between teachers and school leaders and stimulates all players at the school level. 13 __ 3. Promoting the pedagogical use of the results of student assessments Providing Municipalities receive assistance to implement a learning diagnostic for financial and each student at the beginning of the school year. The state prepares a technical diagnostic exam in Language and Mathematics and protocols for applying support for the it, in addition to providing a digital platform to input exam answers. implementation Municipalities are responsible for exam printing, implementation, and data of municipal entry. The platform displays a detailed learning diagnostic of students’ learning skills, including their previous knowledge in Mathematics and Portuguese. assessments Results are disaggregated by class and student and guide teacher training. Establishing a The state government established a yearly standardized literacy standardized exam (called SPAECE-Alfa) that assesses all students in grade 2. literacy exam At the end of the school year, all grade 2 students take a literacy exam for grade 2 that provides municipalities with information about each student’s learning. This standardized assessment is also key for incentive mechanisms established in Ceará (the distribution of ICMS revenues among municipalities and the school prize Escola Nota 10). Training Conducting learning evaluations regularly is an effective way to municipalities monitor learning and a powerful pedagogical instrument to guide and schools teachers’ practice. The TA provides municipalities with the instruments to on the apply learning evaluations regularly and the training to make pedagogical systematic use use of these assessments. Systematic training aims to improve data of assessment analysis skills among educational teams and promote the intense and results growing involvement of education actors in monitoring student learning. BOX 1: A BRIEF HISTORY OF PAIC An assessment of literacy skills in some municipalities in Ceará revealed the need for a systematic approach to achieve universal literacy. In 2001, the city of Sobral conducted a learning evaluation that showed that most of its students, even at the end of primary education, were unable to read. This triggered a series of education reforms in the city, including structured pe- dagogy, teacher professional development, and monitoring of student lear- ning. The experience of Sobral encouraged the State Legislative Assembly to create the Ceará Committee for the Elimination of School Illiteracy in 2004. The Committee conducted a literacy assessment in other municipalities, whi- ch found results similar to Sobral. This led to the establishment of a pilot TA program in 56 municipalities between 2005 and 2006. The program was implemented in partnership with UNICEF, the state section of the National Union of Municipal Secretaries of Education (UNDIME) and the association of mayors in Ceará (APRECE). In 2007, the new state administration took over the implementation of the program, expanding it to the whole state. 14 __ 01 • 02 • 03 –––––––––– 04 • 05 • 06 LAYING THE GROUNDWORK 3 — Laying the groundwork As mentioned in section 1, the experience of Ceará in fostering learning outcomes through TA and incentives is supported by three key conditions for success: monitoring and evaluation (M&E), local autonomy, and political leadership. In this section, we explain how and where poli- tical leadership engages with local governments to lay the groundwork for the implementation of the TA initiatives. Head of Head of the central Full alignment the central government is between Central and government commited to the central regional engages the education government Shared-pact between governance with local reform and and her/his central and local levels governments to willing to play secretaries around universal literacy to present implement a decisive of education the policy the project leadership role and finance and get in this effort for the reform support Figure 5 — Key aspects to lay the groundwork of the technical assistance program The first step is ensuring that the head of the government (i.e., the president or gover- nor) is committed to the education reform and willing to pay a decisive leadership role in this effort. Most often, education reforms are led by the education ministers or education secretariats of local governments. To implement the kind of reform seen in Ceará, it is critical that the head of government leads the process. A key reason for this is that the incentive mechanism involves linking non-educational revenue to education performance, which requires the support of Treasury.5 The head of government has greater ability and legitimacy to place the reform at the top of both education and finance agendas. The reform also impacts local budgets, which is a matter of great concern to local heads of government, like mayors, and requires their buy-in. The dialogue between local and central governments should happen between those who can influence both education and treasury decisions. In addition, support at the leadership level sends an im- portant message to society about the importance of education to the administration. Therefore, the head of the central government must be aware and committed to his/her leadership role. A successful education reform requires solid coordination between the head of government and the ministers of education and finance. The head of the government has a key coordination role in the successful implementation of an education reform of the kind seen in Ceará, since the 5  For details on how to operationalize a results-based financing framework, see Holanda, Barbosa, Cruz, Loureiro. Im- plementing a Results-Based Financing Mechanism for Subnational Governments to Improve Education Outcomes: An Implementation Guide Inspired by the Case of Ceará, Brazil, World Bank, 2020. 16 __ impacts of the education reform are greater when technical assistance is combined with incen- tives.6 As such, both the ministers of education and finance should be involved. The ministry of finance can lead the design and implementation of the incentive mechanism, while the ministry of education is responsible for the process of setting learning goals and supporting local gover- nments to raise education quality. The head of government is the most suitable person to liaise between secretariats, assuring that both are committed to fostering education, dedicating time to implement the reform, and assisting local governments. It is also critical that the head of government engage with local governments to introdu- ce the education reform and get their support. A precondition of Ceará’s literacy policy was the autonomy of local governments in education provision. Therefore, it is key to obtain political support from local governments. Given the dual characteristic of the reform – educational and financial – the head of government is best positioned to lead this engagement and to establish a partnership between central and local governments. Two aspects should be at the center of this dialogue: 1) the importance of education and the role of literacy in setting the basis for learning and 2) the establishment of a cooperation framework between state and municipalities with the goal of improving education rather than auditing or controlling local governments. The partnership between central and local governments to foster education is materia- lized through a shared pact. After getting the support of the heads of local governments, it is important to widely communicate this partnership. Public statements about the partnership promote the engagement of the local secretariats of education and finance, as well as teachers, principals, and the school community. It helps to garner civil society support, which raises local governments’ accountability for results. The sustainability of the shared pact can be maintained by a structured governance. The head of government is responsible for creating a political governance structure between central and local levels,7 which is operationalized by an executive secretariat and is the basis for the implementation of the program. The executive secretariat supports local political leaders and stimulates their involvement in the program. The operation of the TA is structured in a cascade model, with a central implemen- ting unit (CIU), regional teams, and local teams. Both CIU and regional units have a similar structure, organized in four areas: 1) literacy, 2) student assessment, 3) early childhood education (ECE), and 4) teacher training.8 The CIU designs the instruction material, learning assessments, and operation guidelines to support the regional and local levels. Regional teams train local staff and support and monitor local implementation. Local teams can be organized under the same areas of the central and regional units, but this decision depends on each government, since the literacy policy is a cooperation framework and local governments have autonomy to organize themselves in the way they consider most appropriate. The central government finances a local coordinator for each local government to support program implementation and raise the technical capacity of local education secretariats. This operational structure is responsible for supporting teachers and schools to meet the objectives of the program, implementing the literacy strategy in each class. The structure also fosters accountability of each educational agent involved in the process 6  Lautharte, I., Oliveira, V. H., Loureiro, A. Incentives for Mayors to Improve Learning: Evidence from state reforms in Ceará, Brazil. Banco Mundial. Policy Research Working Papers (forthcoming)., 2020. 7 For details on how to operationalize a results-based financing framework, see Holanda, Barbosa, Cruz, Loureiro. Implementing a Results-Based Financing Mechanism for Subnational Governments to Improve Education Outco- mes: An Implementation Guide Inspired by the Case of Ceará, Brazil, World Bank, 2020. 8  In Ceará, as the education policy evolved, the program expanded its coverage from literacy to all primary and lower secondary education subjects. In this process, it included two new organizational areas, one for the final years of primary education (after the student is literate) and another for lower secondary education. In both cases, the focus is on Language and Mathematics. 17 __ GOVERNANCE POLITICAL TECHNICAL RESPONSABILITIES LEVELS GUIDANCE TEAMS AND TEAM PROFILE Central Executive Central Implementation Units Desing secretariat instruction material, assessments and guidelines + support Student ECE Teacher and monitor Literacy regional teams. assessment training The unit profile includes technical specialists from government, universities and civil society Regional Regional Implementation Units Represent the central level in a delimitated geographic area. Dialogue, monitor and train local Student ECE Teacher levels. Literacy assessment training The unit profile consists of teachers and civil servants responsible for training local teams and liaising between central and local governments. Local Local Local Local teams coordinator Local teams political implement leader the literacy policy at their jurisdiction + train teachers Student ECE Teacher and school Literacy principals. assessment training Count with the support of the local coordinator, which responds to the regional unit. School Implement the literacy activities + engage school staff Figure 6 — Operational governance and implementation levels 18 __ 01 • 02 • 03 • 04 –––––––––– 05 • 06 IMPLEMENTATION ROADMAP 4 — Implementation Roadmap This section presents the main steps for the planning and implementing an educational model for achieving universal literacy at the right age, inspired by the successful case of Ceará. The implemen- tation roadmap is aligned with the school year calendar and its structure is divided into four stages: 1) From diagnosis to a target; 2) Structure and planning; 3) Policy implementation; and 4) Check and Adjust. The roadmap highlights the synergy between the three components of technical assistance, as well as the importance of constant updates to the plan and adjustments of the actions based on each region’s context and the results of the monitoring learning protocols. FROM STRUCTURE GETTING CHECK AND 1 DIAGNOSIS 2 AND 3 STARTED 4 ADJUST TO A TARGET PLANNING Establish a learning Design structured Distribute Conduct a assessment materials materiais standardized learning assessment Conduct a system Develop a teacher Train regional diagnostic and school and local teams management Reward best training program performers and Define and Conduct the support those communicate classroom lagging behind targets assessment Establish incentives Implement the Analyze outcomes Create initiatives monitoring tools to foster reading Foster innovation Organize an ECE Expansion Plan Figure 7 — Implementation Roadmap Stages STAGE 1 FROM DIAGNOSIS TO A TARGET 1 Define a reasonable, challenging, and achievable learning target based on a census-based diagnostic of student learning, focusing on literacy The first step in establishing a TA system is to conduct, analyze, and communicate a diag- nostic evaluation of the network to identify student literacy levels and define a reasonable, challenging, and achievable target. Good communication with the regional and local levels is key. This three-step stage is a crucial element in establishing a robust shared pact between central and local governments to promote literacy with well-establish targets. It is impossible to create the necessary sense of urgency to boost TA activities without evidence of the level of students’ literacy and learning. 20 __ ESTABLISH A LEARNING ASSESSMENT TO GUIDE THE INTERVENTION 1.1 —  FROM DIAGNOSIS TO A TARGET 1 Establishaalearning Establish learning Conduct a system Define and communicate assessment assessment diagnostic targets Gather information on the strengths and weaknesses of the local education systems. The CIU must define an assessment of literacy levels to understand the level and inequalities in learning within the local education systems, the regions, and the schools. The student learning assessment design must be able to assess the level of proficiency of each student in literacy. The use of item response theory (IRT) allows for a comparison of the literacy scores over time. The learning assessment provides results for the education system, municipal, and school levels to support the definition of the literacy goals for the TA activities and shared pact. Some of the key questions to be answered by the assessment are: What is the current level of literacy for each student, including at the end of 2nd grade? • • Which local systems/schools have the highest and lowest learning levels? • Which local systems/schools have the highest and lowest rates of learning inequality among their students? The evaluation’s organization and application by an external entity for all local systems and schools generates greater reliability in the evaluation results and also mitigates the risks of the leakage of tests or manipulation of the results. 1.2 — CONDUCT A BROAD EVIDENCE-BASED DIAGNOSTIC OF THE EDUCATION SYSTEM FROM DIAGNOSIS TO A TARGET 1 Establish a learning Conduct a system Conduct system Define and communicate assessment diagnostic diagnostic targets Engage teams from different areas to develop and apply a comprehensive diagnosis of the local systems, and to organize and tabulate the information collected. Representati- ves of the teams responsible for pedagogy, teacher training, evaluation, and dialogue with local networks should participate in the elaboration of the diagnosis, in addition to the team respon- sible for information management and data collection at the central level. The CIU should prepare a first version of the diagnosis, defining the main questions to be answered, considering the project’s different action fronts and the instruments to be used. The following table presents a suggestion of key questions and proposed instruments for the diag- 21 __ nostic, focusing on three areas: literacy instruction, pedagogical support, and governance practices. It is advisable to pilot the instrument on a small sample to calibrate and refine the questions. BOX 4: KEY QUESTIONS AND INSTRUMENTS FOR THE EVIDENCE-BASED DIAGNOSTIC 1. Literacy instruction Key Questions • What percentage of teachers with adequate training? • What is the main area of training of teachers? • Does initial teacher training cover classroom management? • Is initial training in line with the development of the knowledge proposed in the basic education curriculum? • Does initial training address teaching-learning methodologies for literacy at the right age? • What actions are underway in local networks and schools regarding literacy instruction? • What are the best practices in literacy instruction? • Are results of the best practices being monitored at the local level? Instruments • Identify and organize administrative data on this topic • Survey local networks and schools to map ongoing actions • Undertake a qualitative analysis of university teaching curricula 22 __  trengthening the governance of municipal secretariats of education and 2. S providing incentives to promote the exchange of best pedagogical practices Key Questions • Which networks/schools have a specific policy on literacy at the right age in place? • Which networks/schools have staff and the structure for training focused on literacy at the right age? • Which networks/schools offer educational support material for teachers and students with a focus on literacy at the right age? • Which networks have the financial capacity to structure a training and monitoring team for the program? • What is the availability of a library with educational books and space for teacher training in schools? • What are the organizational characteristics of each network (total schools, total students per school in the 1st and 2nd year) • What is the organizational structure of the schools in the network? Do all schools have a staff member, in addition to the school principal, whose responsibility is to manage the school’s pedagogical aspects (e.g., a pedagogical coordinator)? • What are the training processes for teachers and school managers in networks and schools? • What are the best practices focused on technical aspects and what are the main obstacles in the current selection processes? Instruments • Questionnaire application and interviews with local networks and schools to map ongoing actions • Search information already gathered about the infrastructure and enrollment of schools and networks 3. Promoting the pedagogical use of the results of student assessments Key Questions • Are there student learning assessment policies in the networks? Are they carried out by the networks themselves or third parties? • Are there procedural or diagnostic assessments carried out periodically throughout the year at the local level? • To what extent are the pedagogical results of the evaluations used in planning pedagogical actions? Instruments • Questionnaire application and interviews with local networks and schools to map ongoing actions 23 __ Finding Talent. The diagnostic process can serve not only to better understand the characteris- tics of the education networks, but also to identify the networks, schools, and teachers that can support the implementation of the policy itself. In this initial mapping phase, people capable of contributing to the structuring of support actions for literacy will be identified, such as professio- nals who can prepare teaching materials and implement training policies, structure the evaluation and monitoring policy, and develop governance models that can serve as a basis for central policy. 1.3 — DEFINE AND COMMUNICATE TARGETS FROM DIAGNOSIS TO A TARGET 1 Establish a learning Conduct a system Define Define and and communicate communicate assessment diagnostic targets targets Once the diagnostic is completed, define clear learning goals. With the structure and capaci- ty identified in the evidence-based diagnostic and the student literacy levels achieved in the first assessment, the CIU should move to define goals for the improvement of the latter. Considering the goal of literacy at the right age, the main goal should be the literacy of all students up to the age of 8. The State of Ceará showed that it is possible to increase the percentage of literate grade two students from 40% to about 90%. To evaluate literacy level, it is necessary to develop a learning indicator for each school. This in- dicator should consider the average student’s performance and measure learning inequality and participation in the external evaluation. The box below presents a proposed indicator. BOX 5: SCHOOL PERFORMANCE INDEX An indicator for monitoring school literacy results, known as the “School Performance Index,” is calculated based on three factors: proficiency of a school’s students in student learning assessment, the participation rate of a school’s students in student learning assessments, and a factor adjustment that considers the level of learning inequality among students from the same school. This indicator is converted into a scale from 0 to 10 to better inform the network and civil society about the relative performance of each school. School Performance Index = (Average school proficiency)/(maximum profi- ciency in the evaluation) x 10 x Participation rate x Adjustment factor The inclusion of a participation rate encourages schools to mobilize all their students to participate in the student learning assessment, not just those who perform well. The adjustment factor encourages the school to take actions directed at all students, since schools with greater learning inequality will be penalized. 24 __ The setting of literacy goals throughout each year should be proposed considering the diag- nosis and the complexities of the network. Also, the central level must define minimum lear- ning expectations for the student throughout the 1st and 2nd year cycles. These guidelines, which can serve as a basis for the elaboration of structured materials, the development of trai- ning activities, and the elaboration of student learning assessment and diagnostic evaluation, which can be widely shared and studied by all pedagogical actors involved in the program. The CIU define the goals and work actively to communicate them to all levels and actors involved in the policy. The goals should be presented for each local level and made public to enable access to anyone interested in them. The publication of the goals is important, both as an orientation and as an incentive for managers, teachers, and students. In addition, it is important to involve the leaders of local and regional teams in the analysis of the results of the diagnosis and present a summary of the information to all schools. STAGE 2 STRUCTURE AND PLANNING 2 Design and plan activities and guidelines based on the diagnostic results, with the central level working on each area A clear and stable governance among actors is decisive to an effective implementation of an edu- cational policy to achieve literacy at the right age. Teams at the central, regional, and local levels must have clear roles and good communication. This five-step stage establishes the technical framework of all TA activities. Each area works to set a high-level and detailed action plan to support the regional and local levels. Each team includes technical specialists from the central government and experts from universities or civil society. 2.1 — DESIGN STRUCTURED MATERIALS USING THE LITERACY ASSESSMENT RESULTS STRUCTURE AND PLANNING 2 Develop a teacher Create Organize Design Design and school Establish initiatives an ECE structured structured management incentives to foster Expansion materials materials training reading Plan program Design structured material to support literacy learning, especially for resource-constrained municipalities. The structured material based on theoretical-practical knowledge of literacy, 25 __ as well as results of diagnostic learning, is composed of four documents (guidelines for regio- nal and local trainers, teacher manual, student book, and activity booklet). These materials balance time-on-task with the number and complexity of activities and proposes a clear struc- tured pedagogical route to prioritize foundational literacy skills acquisition. The central level also develops a schedule of pedagogical activities, with orientations for each week. The structured material activities are organized within a daily routine of 90 minu- tes, divided into three pedagogical periods: time to enjoy reading, time for reading, and time for speaking and writing acquisition. This material provides guidance on actions to carry out in each classroom and school. It also allows for training actions and follow-up meetings between mana- gement teams to maintain an objective focus and effectively discuss actions aimed at learning. Engaging school and local levels in the use of structured materials: The central and regional teams are responsible for engaging the key stakeholders to mitigate the risks of misinterpretation and/or resistance to the structured material. In the first year of implementation, a fast and reliable solution is the procurement of structured materials from private companies. In this case, it is essential that the central level designs the material methodology and defines the learning goals that must be met at the end of every year. Also, it is important to allow local governments and schools to choose from at least three sets of materials aligned with central level guidelines. Including schools and local governments in the decision process promotes the effective use of the structured materials in the classroom. After capacity is built and technical teams at all levels are strong, it is possible to designate a unit to produce structured materials at the central level. It is important to involve actors from different governance levels in this material production. 2.2 — DEVELOP A TEACHER AND SCHOOL MANAGEMENT TRAINING PROGRAM STRUCTURE AND PLANNING 2 Develop Develop teacher aateacher Create Organize Design structured and and school school Establish initiatives an ECE materials management management incentives to foster Expansion training training reading Plan program program The central level should design an in-service professional development program for tea- chers and school management teams. The program’s key aspects are: 26 __ Central tutors conduct regular training sessions for a group of technical personnel among the regional trainers, and the regional trainers conduct the training for local trainers. In municipalities with critically low literacy levels, the regional level trainers can also train teachers directly. Each trainer becomes responsible for the training of a limited set of teachers. The local team and school principals 1) Cascade set the number of teachers assigned to each trainer. Successful programs point model to the ideal workload for regional and local trainer training (64 hours distributed in 4 bi-monthly modules of 16 hours each) and for teacher training (64 hours distributed in 8 monthly 8-hour meetings, preferably subdivided into two days per month). Regional trainings should happen before the beginning school year. After that, the local training takes place at the beginning of the first school term. Teacher training can also take place starting in the first month of classes. 2) Guidelines to Regional teams must offer technical assistance during the selection of local select and train trainers. Although the local level is autonomous in managing the local education the trainers system, the financial resources to hire the trainers (both regional and local) will (regional and local). be provided by the central level. 3) Practical T o promote the use of active learning strategies, practices, and structured materials and tailored in classrooms. These actions also involve classroom observations by trainers and are meetings based on the structured material that teachers use in their daily routines. 4) School To promote the use of the monitoring protocols. To define the activities management calendar, the school management team will need to work with the local team to training determine the days of each month when the training will take place. program 2.3 — DEVELOP AN INCENTIVE MECHANISM WITH HIGH AND LOW PERFORMANCE SCHOOLS WORKING TOGETHER STRUCTURE AND PLANNING 2 Develop a teacher Create Organize Design structured and school Establish Establish initiatives an ECE materials management incentives incentives to foster Expansion training reading Plan program Develop incentives for good pedagogical management, defined by student learning as- sessment results, and enforce a healthy competitive environment. At the end of every year, the best-performing teachers and their schools are given financial and social rewards. Also, the best performing municipalities are rewarded with more funding from the central level. 27 __ Low performing municipalities and schools receive extra technical support and funding from the central level. At the beginning of the school year, local level managers, school principals, and teachers must know about the incentive mechanisms in place. For this, it is essential that mechanis- ms are easy to understand and that they are well communicated by the central level. Incentives for best performing schools supporting low performing schools. The central le- vel can provide pecuniary and reputational rewards to best performing schools to partner with support low performing schools, with both schools receiving a prize for improvements in edu- cation outcomes. The total score for each school should account not only for learning, but also for the percentage of students that participated in the student learning assessment. This is a strategy to encourage the participation of all students in the assessment, not just the best-per- forming students. Part of the monetary award can be given when the school presents a plan for the use of the resources; the remaining funds can disbursed at the end of the next school year if the school is able to maintain its good educational results and support a low performing school to improve its results.9 2.4 — CREATE INITIATIVES TO FOSTER READING STRUCTURE AND PLANNING 2 Develop a teacher Create Organize Design and school Establish Create initiatives an ECE structured management incentives to foster to initiatives Expansion materials training foster reading reading Plan program Develop initiatives to promote reading culture based on the assumption that literacy and reading are essential to reducing learning poverty. Learning to read represents the acquisition of a new language to access knowledge and information and to develop critical thinking. Thus, it is necessary to practice reading to train the reader and create the joy of reading. It is important that students have constant and easy access to age-appropriate and stimula- ting materials connected to intentional pedagogical actions mediated by the teacher. 9 For the corresponding program in Ceará, see: Bacalhau, Priscilla; Lautharte, Ildo; Goldemberg, Diana. 2020. Can Peer Mentoring Coupled with Incentives Affect School Turnaround? Evidence from Ceará State in Brazil (English). Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/996321595484841150/Can- Peer-Mentoring-Coupled-with-Incentives-Affect-School-Turnaround-Evidence-from-Ceara-State-in-Brazil 28 __ The main goal is to create a small library for students and teachers in each ECE classroom and in the 1st and 2nd grade classrooms of primary Classroom education. The creation of literature collections is driven by two processes: libraries the purchasing of titles from publishers and the organization of a national contest to award and publish new literature. The national contest allows the central level to invest in the local production of books that reflect regional values, themes, and cultures. The central level Literature is responsible for the layout, illustration, and publishing of the books. The Contest winning books are printed and distributed at the local level. The Ceará case shows that investing in regional literature can be cheaper than purchasing books from publishers. The central level also develops activities with teachers to develop reading habits outside of school. There is a relationship between student reading habits and teachers’ reading culture. For this reason, the central level is responsible for editing a magazine for teachers to share classroom Reading culture experiences through interviews, articles, and essays on literacy and reading. The central level supports the local levels to organize workshops to stimulate the use of the literature collection with storytelling and the creation of reading clubs as spaces to meet and integrate teachers and the school community in shared reading experiences. 2.5 — ORGANIZE AN ECE EXPANSION PLAN STRUCTURE AND PLANNING 2 Develop a teacher Create Organize Design and school Establish initiatives an ECEan Organize structured management incentives to foster ECE Expansion Expansion materials training reading Plan Plan program Offer technical and financial support to expand and enhance ECE provision. ECE sets the basis for a successful literacy strategy, since it stimulates children and foster their nutritional and socioemotional development. The way that central level can support local governments in this matter can go from financial support to build ECE facilities and purchase necessary mate- rials and equipment, as well as technical support in providing pedagogical guidance and teacher training focused on ECE. 29 __ STAGE 3 GETTING STARTED 3 Implement the program activities and establish a solid and dynamic monitoring environment between central, regional, and local levels With the planning and management structure in place, it is time to implement the pro- gram activities. The implementation happens with materials and training to improve pedagogical practices and is supported by local M&E and innovation. This section describes the five-step stage that begins with the distribution of the structured materials and the training of regional and local teams to conduct teacher and school manage- ment training, followed by a classroom-level assessment and the implementation of monitoring tools, and ending with the fostering of innovation at the local level. 3.1 — DISTRIBUTE THE STRUCTURED MATERIALS GETTING STARTED 3 Train Conduct the Implement the Distribute Distribute Foster materials regional and classroom monitoring materials innovation local teams assessment tools Coordinate logistics to ensure that the structured materials arrive before the school year starts. The distribution of the materials involves not only logistics, but also an important round of engagement of local teams and schools. The materials are the first physical embodi- ment of the new model. In addition, they serve as the first contribution from the central and regional governments to the local level, signaling that collaboration is being enhanced. The regional team is responsible for distribution. This is an opportunity for the team to engage with local and school teams and to talk about the TA, its governance, the targets set in the first stage of the process, and their link with the material being distributed. 30 __ 3.2 — TRAIN REGIONAL AND LOCAL TEAMS TO CONDUCT TEACHER AND SCHOOL MANAGEMENT TRAINING GETTING STARTED 3 Train Train Conduct the Implement the Distribute Foster regional regional and and classroom monitoring materials local teams innovation local teams assessment tools Promote in-service professional development program for teacher and school principals throu- ghout the year based on the planed methodology and guidelines. For an effective implementation of a local and school training routine, there must be clear and agile communication between the different management levels. Thus, it is fundamental that each actor acknowledges the goals to be Training achieved and its individual responsibility in the process. One way to ensure seminars alignment is to hold periodic training seminars. A suggestion of the meetings arrangement and discussion guidelines is presented in table 3, which can serve as a basis for the preparation of the annual plan for training and monitoring. An increasing level of autonomy for local teams in defining the meeting agendas is expected. In general, the first meetings start from guidelines proposed by the central team and from which the regional teams were formed. The initial guidelines tend to be focused on practical aspects of the classroom, such as the Increasing application and use of assessment instruments, the use of structured materials, school and local and strategies for promoting reading culture. As local networks gain expertise level autonomy and autonomy, there is a demand for training focused on the theoretical aspects that support the foundations of the proposed methodology. Autonomy and innovation at the local levels, focused on improving learning indicators, are highly desired and encouraged. The central level should maintain a steady content production and a large repository of resources related to management and teacher training. The central level provides local teams and schools with proposed guidelines for the Structured preparatory pedagogical week, which happens at the beginning of the school materials for year, and booklets emphasizing fundamental aspects of school management management for school principals and local team staff. Materials with literacy strategies and and teacher training teams assessment instruments are also distributed to the local level and schools. A well-organized central team is fundamental in energizing the whole system with a culture of intense growth and learning, directed to the accomplishment of learning goals. 31 __ Central with Regional with Local with   Regional levels Local levels School levels Recommended Meetings with all Regional teams The frequency of Periodicity regional teams must must visit each meetings between be held monthly. local network local teams and their Meetings or at least once a schools depends discussions between month to monitor on the size of each the central team ongoing actions local network, but and the regional and their results. at least once every teams must take two months. place daily. Recommended Monitoring of Monitoring of Monitoring of meeting agenda meetings held meetings held attendance between regional between the indicators and local teams local team and and student each school performance Discussion of in summative planned actions Discussion of and formative based on visits by planned actions assessments regional teams based on visits carried out by made by local the schools Discussion of teams in each ongoing actions by school Planning of actions local teams that to support students most need support Planning of with the greatest training actions difficulties at the local level Discussion of the Discussion of training guidelines specific actions to carried out by the be undertaken in schools in their schools that most training sessions need support and of the class observations made by the pedagogical coordinator(s) Monitoring the number of classes held effectively in each school Monitoring the use of structured materials and strategies for promoting Table 2 — reading culture Recommended periodicity and agenda for the training meetings 32 __ 3.3 — CONDUCT THE CLASSROOM ASSESSMENT GETTING STARTED 3 Train Conductthe Conduct the Implement the Distribute Foster regional and classroom classroom monitoring materials assessment innovation local teams assessment tools With the structure implemented and the materials and training put in place, the CIU must take assessment to the classroom level. and weaknesses. The standardized learning evaluation is important to track progress along the years, but is not the most appropriate tool to support teachers. Not only because of the nature of the exam, but also because it is conduc- ted at the end of the school year. In this sense, it is advisable to have a pedagogical assessment to help teachers understand the learning level of its students at the beginning of the school year. It serves as a baseline and can be reapplied in the mid-term to track progress and guide the pedagogical action. This gives teachers a way to determine how their students are advancing and to help them based on their individual strengths The classroom assessment involves the action of all levels, from the top to the school. The central level promotes the application of the classroom assessment protocol at the begin- ning of the school year (diagnosis phase). The regional team gives technical support to the local level during the implementation process. It is responsible for organizing the discussion with the local teams and adapting the structured material according to the results. Local teams are responsible for organizing the implementation and consolidating the results. Lastly, schools are responsible for ensuring student engagement and participation. 3.4 — IMPLEMENT THE MONITORING TOOLS GETTING STARTED 3 Train Conduct the Implement the Implement the Distribute Foster regional and classroom monitoring monitoring materials tools innovation local teams assessment tools Monitoring of the literacy policy must occur on a daily basis, guided by protocols established by the central level and specific and measurable indicators. Some of the tools that support the TA monitoring system and the indicators defined by the central level are described below. SMART To achieve the program target, as described in section 1.3, the central level monitoring defines a set of monitoring indicators for each TA governance area. TA indicators monitoring indicators should follow SMART (Specific, Measurable, Attainable, defined by the Relevant and Time-Bound) methodology, which should serve as a guide for the central level development of new indicators or the adaptation of TA indicators in each region. 33 __ The central level provides regional and local levels with clear and complete monitoring protocols. These protocols instruct teams to 1) elaborate a diagnosis Clear of the local level educational context; 2) based on the diagnosis, elaborate the monitoring local work plan; and 3) keep a regular record of the program indicators. The protocols protocols are guided by the established program indicators. For each indicator, there is an instrument to be used by the regional and local teams. The central level should produce evaluation rubrics for the monitoring of each established indicator. Schools might also have specified instruments to monitor each literacy level and reading routine, and to monitor teacher Monitoring tools attendance and their use of booklets. The instruments are filed at the end of each bimester. It is recommended that the central level provide a digital platform for regional, local, and school teams to regularly enter the monitoring results into an organized framework. The monitoring protocol instructs teams at every level to keep an updated list of the contacts of its counterparts at the level below. Regional level leadership might maintain a contact list and a messaging group with the local level Strong secretaries of education. Regional level technical teams keep contact with local communication level technical teams. Local teams keep contact with school principals. This communication occurs on a daily basis, focused on the monitoring of specific actions, sharing of materials, and support in the preparation of work plans and in the execution of specific actions. 3.5 — FOSTER INNOVATION AT LOCAL LEVELS GETTING STARTED 3 Train Conduct the Implement the Distribute Foster Foster regional and classroom monitoring innovation materials innovation local teams assessment tools The central and regional teams should foster a collaborative environment that promotes innovative actions and sharing of best practices within the local levels of each region. The municipalities that stand out the most have the common characteristic of going beyond the instruments proposed by the central level. Some of the most common innovations esta- blished by the local management teams are highlighted below: 34 __ The central level provide an annual external assessment to be carried out at the Expansion and end of the school year, and an initial diagnostic assessment to be carried out at improvement of the beginning of the school year. Local levels can deepen this dynamic, creating the evaluation their own assessment system with periodic formative assessments, carried out process throughout the school year, and they can expand assessments for series and curriculum components not covered by the central level. The involvement of families in the student learning process is highlighted by local leaders as a fundamental aspect for the effectiveness of the policy. It is up Family to the local teams to foster a welcoming and communicative environment, with engagement periodic meetings and individualized conversations to explain student potential and challenges. Engaged parents tend to demand good results not only from students but also from school management teams. Bearing in mind that the learning indicators proposed by the central level consider not only the average results but also the level of educational inequality present in schools and at local levels, local levels can carry out extra class Extra classes activities for students who are most in need. These students should be selected for those who based on the results of the diagnostic and formative assessments. In general, need it most extra classes should occur after school regular time, with tutoring and specific activities for each student to overcome their learning gaps. These actions are carried out with local level resources. STAGE 4 CHECK AND ADJUST 4 Assess the literacy level of the students at the end of the school year, reward the best performers, and plan the next pedagogical cycle with extra support for those lagging behind. The establishment of a solid and reliable M&E system that continuously measures key education outcomes—including student learning—is an important element for the TA in assessing student achievement. Results provide a diagnostic of literacy levels and support the establishment of lear- ning goals, which is transmitted to teachers and schools through training and monitoring activities. This subsection describes the three-step stage that begins with the implementation of a literacy summative assessment, followed by the awarding of a prize for the best schools, and ending with the analysis of the literacy outcomes that will be used to plan the next school year, closing the cycle. 35 __ 4.1 — CONDUCT A STANDARDIZED LEARNING ASSESSMENT CHECK AND ADJUST 4 Conduct Reward best performers Conduct aa standardized standardized learning and support those Analyze outcomes learning assessment assessment lagging behind At the end of each school year, a student learning assessment must be carried out to diag- nose the literacy levels of students from all networks. The evaluation serves as the basis for defining the learning indicators of each school and for directing the work of the management teams for the next school year. Its results should also serve to reward the best performing scho- ols and to allocate additional support for the worst performing schools. The summative assessment closes the cycle started with the initial literacy assessment and followed by classroom assessments in the beginning and middle of the school year. It serves to evaluate the progress of the students, schools, and networks towards the established goals and to define goals for the next year. It is also the main tool used by the central govern- ment to evaluate policy implementation outcomes. Figure X shows the assessments throughout the school year. BEGINING OF SCHOOL YEAR Learning drivers, classroom observation and school management Summative Assessment protocols Item Response Theory Data disaggregated by External evaluation student, teacher and school Data disaggregated by classroom Classroom Assessment I Classroom Assessment II Period: first month of school Period: one month before the Short term feedback Summative Assessment Pencil & paper test/oral reading test Short term feedback Data disaggregated by student Pencil & paper test/oral reading test Data disaggregated by student Figure 8 — Assessments throughout the school year 36 __ 4.2 — REWARD BEST PERFORMERS AND SUPPORT THOSE LAGGING BEHIND CHECK AND ADJUST 4 Reward Reward best best performers performers and Conduct a standardized and support those support those lagging behind Analyze outcomes learning assessment lagging behind With the results of the student learning assessment, it is time to reward schools that are performing well and to define support policies for schools that still face challenges. The presence of teachers, school managers, and local, regional, and central level An annual event leaders, helps to give social and financial recognition to all who are responsible for to reward best good educational performance. The central level should be responsible for organizing performers the event and ensuring the necessary resources for the awards. The analysis of the student learning assessment results and the re-planning of the next school year is a fundamental pillar of the policy, as described in the next step. Along with the evaluation results, the central level publishes a map with the Performance learning indicator results for each local level. This map indicates clearly for civil map society which local levels are prioritizing and achieving good educational results. The transparency and communication of these results have the effect of creating a healthy competitive environment between schools and municipalities to achieve better educational results. 4.3 — ANALYZE LITERACY OUTCOMES CHECK AND ADJUST 4 Reward best performers Conduct a standardized Analyze and support those Analyzeoutcomes outcomes learning assessment lagging behind The analysis of the literacy outcomes subsidizes the planning of the following year. The results of the summative assessment should be compared to the targets set in step 1.3., and the expected outcomes for schools and students should be compared to the results achieved. Students’ individual results should be used at the school. The CIU, with the support of the regional level, should produce and communicate reports to the schools with the results achie- ved by their students. Literacy assessment results for each student should be the main topic of discussion and work in the planning meetings at the beginning of the next school year. Students 37 __ who do not reach the expected minimum literacy proficiency should be accompanied and recei- ve extra support in the following school cycle. The results should also be analyzed at the school level. The CIU must produce reports at the school level to share with regional and local levels so they can evaluate school progress in policy implementation and student learning. Schools that have not achieved their goals should also be monitored and defined as priority schools for the next school year. Finally, the CIU must measure and analyze the results achieved by each local and regional level. The regional and local networks with poorer results can receive special attention in the next year to help identify the reasons for their difficulty and to determine the necessary support needed to overcome them. The results of the summative assessment serve as an initial assessment for the next year, thus closing the plan-do-check-adjust cycle. The work does not end by finishing the first year of implementation and must be continuously adjusted so the education management and the student learning can be continuously improved. The results analyzed at this step are equivalent to those in step 1.1 and are the basis for the new targets to be set as the previous targets in step 1.3. 38 __ 01 • 02 • 03 • 04 • 05 –––––––––– 06 IMPLEMENTING THE MODEL IN 5 — Implementing the model in different contexts The structure presented in Chapter 3 and the capacity supposed in Chapter 4 are a benchmark for the implementation of the TA. There are not, however, absolute fixed structures and capacities needed for TA to succeed. Each region has its own reality and pre-existing conditions. This section proposes ways forward for contexts in which the necessary factors are not in place prior to the implementation of the path presented in this guide. WHAT IF THE STUDENT LEARNING ASSESSMENT SYSTEM 5.1.  IS NOT CONSOLIDATED OR DOES NOT EXIST? In this case, it is recommended that the government hire a specialized institution to organize and apply, at least, a literacy assessment for all students in the 2nd grade, including the diagnosis (step 1.1) and the summative (step 4.1). This creates the initial conditions to start a one-year cycle of the TA. This experience will build technical capacity at the central level and consolidate the student learning assessment system among the local levels and main stakeholders. Once that happens, the central level can gradually expand the assessment to all grades, including other subjects. As mentioned in the implementation roadmap, IRT is recommended for use in the assessment de- sign. However, in the case of budget constraints, it is possible to start the process with the more straightforward assessment process outlined in step 1.1. Sobral’s education system in Ceará orga- nized the diagnosis with a simple reading fluency test applied to 2nd grade students.10 This activity needs a significant amount of effort and resources, and the central level must take part in the budget to allow the learning assessment to continue every year. However, literacy level improve- ments will prove the importance of this investment.11 WHAT IF THERE IS NO MONITORING AND 5.2.  EVALUATION SYSTEM IN PLACE? The TA implementation roadmap creates conditions to establish or review an M&E system. Step 10 Cruz, Louisee; Loureiro, Andre, 2020. Achieving World-Class Education in Adverse Socioeconomic Conditions: The Case of Sobral in Brazil. World Bank, Washington, DC. https://openknowledge.worldbank.org/handle/10986/34150 11 See the World Bank guides for establishing and managing Learning assessment systems: https://openknowledge. worldbank.org/handle/10986/32461 40 __ 1.2 shows the key questions and instruments for the evidence-based diagnostic. Those questions are an excellent guide to start designing an M&E system at the central level. The central level’s TA operational governance has a technical team (student assessment area) responsible for designing monitoring protocols and boosting indicators and data use at the local level. The communication channel between the local, regional, and central levels will allow access to updated information. For the TA first-year cycle, the data monitoring could be done on electronic sheets or even on paper, con- solidating the data at the regional level. However, the central level needs to provide a digital platform to establish a robust M&E system, improving the TA’s efficiency. WHAT IF THE CURRICULUM IS NOT 5.3.  UPDATED OR WELL-ESTABLISHED? In this case, the recommendation is to establish clear learning goals focused on literacy and re- ading skills. The learning goals are defined for each bimester to support formative assessment systems throughout the school years. The learning goals defined by the central level must apply to all local levels and schools to avoid asymmetries and learning inequality. Sometimes, a cur- riculum review is a long-term process, and it could delay the TA implementation. An excellent way to avoid this problem is by starting a discussion about which skills should be expected for all students at the end of the 2nd year of elementary school (8 years old). After a few TA year- s-cycles, the central level can conduct a broader review of the curriculum using the cumulative knowledge and data to support the process. WHAT IF THE SUPPLY CHAIN FOR THE PRINTING AND 5.4.  DISTRIBUTION OF LEARNING MATERIALS IS LIMITED? During Step 2.1, if the central government does not have the capacity to print and distribute the structured materials, the CIU must research the local printing and distribution sector to identi- fy its capacity. Where there is no structured local supply chain for the printing and distribution of the learning materials, the execution of Step 3.1 may then be require hiring internationally or, in the case of existing capacity that can be expanded, subsidized by the central government. The existence of a national program of structured materials for schools should foster the indus- try and reinforce capacity. With the annual process of design, printing, and distribution of the materials, it is expected that the local printing and distribution sector would be strengthened. The importing of materials or subsidies will ultimately be reduced and ended as the local capa- city increases to be meet the demand of the whole program. 41 __ WHAT IF THERE IS LANGUAGE DIVERSITY? 5.5.  If there is a well-established language-of-instruction policy, it is possible to include this aspect in the TA implementation roadmap. One possibility is to create an additional technical team on operational governance to coordinate language diversity issues with the other TA teams. In the case of the discussion on language-of-instruction not being consolidated in the country and/or specific region, it would be possible to start the TA implementation with the official languages and add additional support at the local level to gradually include the other languages. It would be also important that the central level include indicators in the monitoring protocols related to the diversity of language and institutional capacity at the local level. A successful TA implemen- tation will create an operational structure at the local level and build the capacity to implement evidence-based programs on teacher training and learning materials in other languages. WHAT IF THERE IS LIMITED DECENTRALIZATION OR 5.6.  LITTLE FINANCIAL AUTONOMY AT LOCAL LEVEL? In cases where the educational budget is highly centralized, it would be important that the cen- tral government provide resources for the local units to implement the TA. The central level can make available for local levels the student learning assessment system, the purchase of structu- red booklets and reading materials, resources for carrying out training seminars, as well as the structuring of regional and local teams. For a higher level of effectiveness of the education policy, it would be important to allow some level of managerial decentralization. The central level can focus its responsibility on defining the gui- delines and coordinating the program’s actions. In turn, each local and school level should ideally have some autonomy to carry out its own actions and innovations, considering their local context and the learning goals established by the central level. Some level of discretion of the local levels to implement the program and pursue the targets set is fundamental for policy success. In Brazil, there is a national program for direct resource transfer from the federal government (central level) to schools across the country. Based on the guidelines established by the Ministry of Education, schools have the autonomy to use these resources to solve problems related to their daily tasks and implement additional actions directed to improve student learning and reading skills. 42 __ 01 • 02 • 03 • 04 • 05 • 06 –––––––––– FUNDAMENTAL ASPECTS OF POLICY CONTINUITY 6 — Fundamental aspects of policy continuity A fundamental aspect of Ceará’s successful educational model is that the program was main- tained and continued to improve over different political terms. Some of the actions addressed in the implementation path have been catalysts to guarantee this continuity. In this section, we emphasize some of the aspects considered fundamental for policy continuity. An effective implementation of these aspects is especially important. Having a concrete learning target is required to communicate and engage. If society is aware and supports the policy, politicians will Ensure that certainly communicate the results, which, in turn, will increase civil society to public support. This creates a virtuous cycle between policy and supports the society, making politicians (both those in the current administration learning goal and newcomers) more likely to focus on education. In so doing, politicians will achieve political gains and civil society will continue to demand this policy direction and accountability for its outcomes. Building capacity and fostering ownership among technical teams at the central, regional, and local levels favors policy continuity. When knowledge is spread across the implementation chain, more Strengthen people know about how the program operates, reducing the impacts technical of personnel changes on teams. Strengthening local teams also teams contributes to tailoring the policy to local contexts, fostering greater impact. Empowered and well-trained technical teams are less susceptible to political rather than technical shifts in the program. Encouraging local governments to select school principals based on merit is critical for education systems with schools with a high level of autonomy. Strengthened school management allows governments to increase school autonomy, which has positive effects on schools Increase school and the secretary of education. Schools feel more empowered and autonomy have more possibilities to design pedagogical interventions based on their students’ needs. The government has more time to focus on the pedagogical aspects of the education policy, rather than administrative tasks. Principals who feel more responsible for student learning are also less susceptible to big shifts in policy direction. 44 __ A well-designed incentive mechanism for local governments and the award to top schools have short- and long-term incentives. The Strong incentives are critical for a full engagement of local governments incentives and principals with the technical assistance, as well as incentivizes them to maintain a good level performance over time. Ceará’s educational model was very successful in enforcing a healthy Promote competitive environment between municipalities and schools. When the a healthy goal is to improve student learning, competition between schools and competitive municipalities to perform is highly desirable. When political leadership environment changes, changing the direction of action risks leaving the municipality behind in the race for leadership in educational performance. The incentive mechanisms and the technical assistance program in Ceará were enshrined in state laws. The discussion of the program in an Legal framework assembly of representatives, resulting in a broad debate with civil society and politicians from the different political parties, is a relevant factor to guarantee the robustness of the policy and the continuity of its actions. 45 __ About the Authors ANDRE LOUREIRO is a senior economist at the World Bank’s Educa- tion Global Practice. He holds a Ph.D. in Economics from the University of Edinburgh, United Kingdom, and a Bachelor’s and Master’s degree in Economics from the Federal University of Ceará. FÁTIMA ALVES is an Education Specialist Consultant for international organi- zations (World Bank, GPE, and UNESCO). She holds a Ph.D. in Education from the Pontifical Catholic University of Rio de Janeiro (PUC-Rio). From 2008 to 2014, Fatima was a professor in the Department of Education at PUC-Rio. LOUISEE CRUZ is a consultant at the World Bank’s Education Global Practice. She holds a double MSc degree in Public Policy and Human Development from Maastricht University and the United Nations Uni- versity, and a BA degree in International Relations from the University of São Paulo. MATHEUS ASSUNÇÃO is a consultant at the World Bank’s Education Global Practice. He holds a MSc in Economic Theory from the University of São Paulo and a BA in Economics from the University of Brasília. THIAGO CARDOSO is a consultant at the World Bank’s Education Global Practice. He holds a Master’s degree in Economic Theory from the University of São Paulo. Technical As- sistance for Local Govern ments to Im- prove Educa- tion Outcome