Document of The World Bank Report No. 24433 PROJECT PERFORMANCE ASSESSMENT REPORT BRAZIL Innovations in Basic Education Project (Loan 3375-BR) Second Northeast Basic Education Project (Loan 3604-BR) Third Northeast Basic Education Project (Loan 3663-BR) School Improvement Project-FUNDESCOLA I (Loan 4311-BR) June 20, 2002 Sector and Themnatic Evaluation Group Operations Evaluation Departinent Currency Equivalents (annual averages) Currency Unit = Cruzeiro (Cz) and Real (R$A)a Appraisal Completion IEB US$1.00 Cz$230.34 R$1.18 NEBE II US$1.00 Cz$3789.9 R$1.71 NEBE III US$1.00 Cz$128.06 R$1.71 FUNDESCOLA I US$1.00 R$2.71 a. The Cruzado (CZS) was in use at the time of Project appraisal. The Real (R$) was introduced in 1994. Abbreviations and Acronyms GDP Gross domestic product ICR Implementation Completion Report IADB Interamerican Development Bank INEP Instituto Nacional de Estudos e Pesquisas Educacionais LIL Learning and Innovation Lending MIS Management Information System NEBE North East Basic Education Project(s) NGO Non-governmental Organization OED Operations Evaluation Department PDE Plano de Desenvolvimento da Escola PME Projeto de Melhoria da Escola PPAR Project Performance Assessment Report* PCR Project Completion Report PIU Project Implementation Unit SAEB Sistema Nacional de Avaliacao da Educacao Basica (National System for Basic Education Evaluation) SAR Staff Appraisal Report SARESP Sistema de Avaliacao do Rendimento Escolar de Sao Paulo TVET Technical and vocational education and training UNESCO United Nations Educational, Scientific, and Cultural Organization * As of July 1, 2001, Project Performance Audits have been renamed Project Performance Assessments Fiscal Year Government: January I-December 31 Director-General, Operations Evaluation : Mr. Robert Picciotto Director, Operations Evaluation Department Mr. Gregory K. Ingram Manager, Sector and Thematic Evaluation : Mr. Alain Barbu Task Manager Ms. Helen Abadzi The World Bank Washington, D.C. 20433 U.S.A. Office of the Director-General Operations Evaluation June 20, 2002 MEMORANDUM TO THE EXECUTIVE DIRECTORS AND THE PRESIDENT SUBJECT: Project Performance Assessment Repbrt on Brazil Innovations in Basic Education Project (Loan 3375-BR) Second Northeast Basic Education Project (Loan 3604-BR) Third Northeast Basic Education Project (Loan 3663-BR) School Improvement Project-FUNDESCOLA I (Loan 3375-BR) This is a Performance Assessment Report (PAR) on four education projects in Brazil: * Innovations in Basic Educationt Project (Loan 33 75-BR) for US$245 million, which was approved in FY9 1. The loan was extended by one year and closed on April 3, 1999. * Second Northteast Basic Education Project (Loan 3604-BR) for US$212 million, which was approved in FY93. The loan was extended by one year and closed on June 30, 1999. * Third Northeast Basic Education Project (Loan 3663-BR) for US$206.6 million, which was approved in FY93. The loan was extended by one year and closed on December 31, 1999. * School Improvement Project-FUNDESCOLA I (Loan 4311-BR) for US$62.5 million, which was approved in FY99. The loan closed as scheduled-on June 30, 2001 and disbursed completely. The Bank's human resource development strategy for Brazil was relevant to its economic needs, as perceived in each period, and the objectives of the projects have been relevant to the economic needs of the country. But though the strategy (focusing on management improvement at various levels) was necessary, it was not sufficient for alleviating poverty through education. The Bank missed opportunities to enunciate classroom efficiency standards when it developed the minimum operational standards for schools and to help state secretaries enforce them. The relevance and efficiency of the Innovations and Northeast Brazil projects were modest; that of the Fundescola project, which focused more on schools, is rated substantial, but this project served few rural areas, and its poverty alleviation effect was limited. The outcome of the Fundescola project is rated satisfactory. The outcomes of the Innovations of Basic Education and Northeast Basic Education (NEBE) projects are rated moderately satisfactory because project objectives were only partly achieved. The projects exceeded or surpassed most targets for physical inputs and training but it is not clear that the (vaguely worded) learning goals were met. The objectives of the two NEBE projects were focused towards central management improvements, and these had little or no impact on quality improvement. Due to design shortcomings, Bank performance for these two projects is rated unsatisfactory. Bank performance for the Fundescola project, which was designed to target schools, was satisfactory. Borrower performance is rated satisfactory for all but the Innovations in Education project. 2 Overall, the Bank has promoted efficiency in various management levels, from state secretariats to school-based management, but has not yet promoted effective actions to make classroom instruction more efficient. Specific efficiency goals (e.g. spending 75 percent of class time on actual learning activities) could become a part of the minimum operational standards. Lessons Experience with the projects confirms a number of OED lessons from education projects: * Attention to state, municipal, or even school management is necessary but is not sufficient for improving classroom instruction effectiveness and efficiency. Attention must also be paid to how the precious classroom time is used. * States and municipalities that are better off are more likely to take advantage of Bank- financed resources. The challenge is to encourage the poorer ones to benefit as well. * Decentralizing school management to the municipal level is not a panacea. Autonomy permits the poorest or least knowledgeable municipalities to fall behind, without much possibility for helping their residents. * In addition to achievement tests, classroom observations may be used as predictors of achievement. Unless there is after-hours coaching, children's knowledge is likely to reflect information processing in classrooms. Classroom observations can potentially provide feedback and promote improvements sooner than achievement tests, which must first be developed, administered, and analyzed. * Students' health needs and problems must be addressed to optimize cognitive development and classroom learning, requiring closer inter-agency coordination. Attachment OED Mission: Enhancing development effectiveness through excellence and independence in evaluation. About this Report The Operations Evaluation Department assesses the programs and activities of the World Bank for two purposes: first, to ensure the integrity of the Bank's self-evaluation process and to verify that the Bank's work is producing the expected results, and second, to help develop improved directions, policies, and procedures through the dissemination of lessons drawn from experience. As part of this work, OED annually assesses about 25 percent of the Bank's lending operations. In selecting operations for assessment, preference is given to those that are innovative, large, or complex; those that are relevant to upcoming studies or country evaluations; those for which Executive Directors or Bank management have requested assessments; and those that are likely to generate important lessons. The projects, topics, and analytical approaches selected for assessment support larger evaluation studies. A Project Performance Assessment Report (PPAR) is based on a review of the Implementation Completion Report (a self-evaluation by the responsible Bank department) and fieldwork conducted by OED. To prepare PPARs, OED staff examine project files and other documents, interview operational staff, and in most cases visit the borrowing country for onsite discussions with project staff and beneficiaries. The PPAR thereby seeks to validate and augment the information provided in the ICR, as well as examine issues of special interest to broader OED studies. Each PPAR is subject to a peer review process and OED management approval. Once cleared intemally, the PPAR is reviewed by the responsible Bank department and amended as necessary. The completed PPAR is then sent to the borrower for review; the borrowers' comments are attached to the document that is sent to the Bank's Board of Executive Directors. After an assessment report has been sent to the Board, it is disclosed to the public. About the OED Rating System The time-tested evaluation methods used by OED are suited to the broad range of the World Bank's work. The methods offer both rigor and a necessary level of flexibility to adapt to lending instrument, project design, or sectoral approach. OED evaluators all apply the same basic method to arrive at their project ratings. Following is the definition and rating scale used for each evaluation criterion (more information is available on the OED website: http:l/worldbank.org/oedleta-mainpage.html). Relevance of Objectives: The extent to which the project's objectives are consistent with the country's current development priorities and with current Bank country and sectoral assistance strategies and corporate goals (expressed in Poverty Reduction Strategy Papers, Country Assistance Strategies, Sector Strategy Papers, Operational Policies). Possible ratings: High, Substantial, Modest, Negligible. Efficacy: The extent to which the project's objectives were achieved, or expected to be achieved, taking into account their relative importance. Possible ratings: High, Substantial, Modest, Negligible. Efficiency: The extent to which the project achieved, or is expected to achieve, a return higher than the opportunity cost of capital and benefits at least cost compared to alternatives. Possible ratings: High, Substantial, Modest, Negligible. This rating is not generally applied to adjustment operations. Sustainability: The resilience to risk of net benefits flows over time. Possible ratings: Highly Likely, Likely, Unlikely, Highly Unlikely, Not Evaluable. Institutional Development Impact: The extent to which a project improves the ability of a country or region to make more efficient, equitable and sustainable use of its human, financial, and natural resources through: (a) better definition, stability, transparency, enforceability, and predictability of institutional arrangements and/or (b) better alignment of the mission and capacity of an organization with its mandate, which derives from these institutional arrangements. Institutional Development Impact includes both intended and unintended effects of a project. Possible ratings: High, Substantial, Modest, Negligible. Outcome: The extent to which the project's major relevant objectives were achieved, or are expected to be achieved, efficiently. Possible ratings: Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory. Bank Performance: The extent to which services provided by the Bank ensured quality at entry and supported implementation through appropriate supervision (including ensuring adequate transition arrangements for regular operation of the project). Possible ratings: Highly Satisfactory, Satisfactory, Unsatisfactory, Highly Unsatisfactory. Borrower Performance: The extent to which the borrower assumed ownership and responsibility to ensure quality of preparation and implementation, and complied with covenants and agreements, towards the achievement of development objectives and sustainability. Possible ratings: Highly Satisfactory, Satisfactory, Unsatisfactory, Highly Unsatisfactory. Contents Principal Ratings .................................................................... v Key Staff Responsible .................................................................... vi Preface ....... ............................................................. vii 1. Background .....................................................................1 Other Donor Activities .................................................................. 3 2. Objectives and Implementation of the Assessed Projects .................................................... 4 Innovations in Basic Education Project (Loan 3375-BR) ..................................................5 Northeast Brazil II and Northeast Brazil III Projects (Loans 3604-BR and 3663-BR ) ..... 5 Observations of Classroom Events in Project Areas .........I.............................................. 10 Note: some schools visited did not have classes at the time ...................... ........................................ 13 3. Ratings .................................................................... 14 Institutional Development Impact .................................................................. 15 Sustainability .................................................................. 16 Bank Performance .................................................................. 16 Borrower Performance .................................................................. 17 Bank Strategy Contributions Toward the Education of the Poor ..................................... 17 4. Issues for Future Consideration ...................................................... 18 Project Designs Built on Untested Theories .................................................................. 18 Much Classroom Time is Spent on Ineffective Tasks or Inactivity ................................... 19 Improve Poor Students' Chances for Catching up and Keeping up ................................. 20 Increase Preschool and School Health Interventions ....................................................... 21 Autonomy Enables Disinterested Municipalities to Do Little ........................................... 21 Lessons .................................................................. 22 Annex A. Project Activities .................................................................... 25 Table 1. Innovations in Basic Education ............................................................ 25 Table 2. Northeast Brazil II and III ............................................................ 26 Table 3. FUNDESCOLA I ............................................................ 28 This report was prepared by Helen Abadzi, who assessed the project in January 2002. The report was edited by William B. Hurlbut, and Pilar Barquero provided adininistrative support. l iv Annex B. Assessment Methodology .................................................... 29 Annex C. Specific Instructionally Oriented Issues ........................... ......................... 31 Table 2. World Bank Lending in the Education Sector of Brazil ..................................... 33 Table 2. Local Staff Opinions Regarding Project Aspects ....................... ........................ 34 Annex C. Basic Data .................................................... 41 INNOVA TIONS IN BASIC EDUCA TION PROJECT (LOAN 33 75-BR) .................... .................... 41 SECOND NORTHF,ASTBASICEDUCATIONPROJECT (LOAN 3604-BR) ............... ................... 43 THIRD NORTHEASTBASIC EDUCA TION PROJECT (LOAN 3663-BR) ...................................... 45 SCHOOL IMPROVEMENT PROJECT-FUNDESCOLA I (LOAN 4311-BR) ................................. 46 v Principal Ratings ICR* ES' PPAR Innovations in Basic Education Project (Loan 3375-BR) Outcome Satisfactory Moderately satisfactory Moderately satisfactory Sustainability Uncertain Uncertain Likely Institutional Substantial Modest Modest Development Borrower Satisfactory Unsatisfactory Unsatisfactory Performance Bank Performance Satisfactory Unsatisfactory Satisfactory Second Northeast Basic Education Project (Loan 3604-BR) Outcome Satisfactory Moderately unsatisfactory Moderately satisfactory Sustainability Likely Likely Likely Institutional Substantial Modest Modest Development Borrower Satisfactory Satisfactory Satisfactory Performance Bank Performance Satisfactory Unsatisfactory Unsatisfactory Third Northeast Basic Education Project (Loan 3663-BR) Outcome Satisfactory Moderately unsatisfactory Moderately Satisfactory Sustainability Likely Likely Likely Institutional Substantial Modest Modest Development Borrower Satisfactory Satisfactory Satisfactory Performance Bank Performance Satisfactory Unsatisfactory Unsatisfactory FUNDESCOLA I (Loan 4311-BR) Outcome Satisfactory Satisfactory Satisfactory Sustainability Likely Likely Likely Institutional Substantial Substantial Substantial Development Borrower Satisfactory Satisfactory Satisfactory Performance Bank Performance Satisfactory Satisfactory Satisfactory * The Implementation Completion Report (ICR) is a self-evaluation by the responsible operational division of the Bank. The Evaluation Summary (ES) is an intermediate OED product that seeks to independently verify the findings of the ICR. vi Key Staff Responsible Project Task Manager/Leader Division Chiefl Country Director Sector Director Innovatfons in Basic Education Project (Loan 3375-BR) Appraisal Lawrence Wolff K. Y. Amoako Armeane Choksi Completion Robin Horn Jamil Salmi Gobind Nankani Second Northeast Basic Education Project (Loan 3604-BR) Appraisal Robin Hom K. Y. Amoako Armeane Choksi Completion Robin Hom Jamil Salmi Gobind Nankani Third Northeast Basic Education Project (Loan 3663-BR) Appraisal Robin Horn K. Y. Amoako Armeane Choksi Completion Robin Hom Jamil Salmi Gobind Nankani FUNDESCOLA l(Loan 4311-BR) Appraisal Robin Horn Julian Schweitzer Gobind Nankani Completion Robin Horni Xavier Coll Vinod Thomas vi' Preface This Project Performance Assessment Report (PPAR) covers the four most recently completed education projects in Brazil: * Innovations in Basic Education Project (Loan 33 75-BR) for US$245 million, which was approved in FY91. The loan was extended by one year and closed on April 3, 1999. * Second Northteast Basic Education Project (Loan 3604-BR) for US$212 million, which was approved in FY93. The loan was extended by one year and closed on June 30, 1999. * Third Northeast Basic Education Project (Loan 3663-BR) for US$206.6 million, which was approved in FY93. The loan was extended by one year and closed on December 31, 1999. * School Improvement Project-FUNDESCOLA I(Loan 43 11-BR) for US$62.5 million, which was approved in FY99. The loan closed as scheduled on June 30, 2001 and disbursed completely. The assessments were conducted to study the effectiveness of Bank strategy in some extremely poor areas of the country with complex cultural and economic problems. The PPAR is based on the following sources: Project or Implementation Completion Reports (ICRs), Staff Appraisal Reports (SARs), Credit Agreements for the projects, and project files, particularly the supervision reports. An OED mission visited Brazil in February-March 2002 to collect other pertinent information. The author thanks the government officials who received the mission for their extensive cooperation. Thanks are also given to the country office staff who helped the mission. Following standard OED procedures, copies of the draft PPAR were sent to the relevant government officials and agencies for their review and comments, but none were received. 1. Background 1.1 Brazil is working hard to equalize opportunities for its citizens, because historical factors have created very inequitable distributions of wealth.' The income of the top fifth of the population is 25.5 times higher than that of the bottom fifth. Inevitably, education has a similar distribution. Public spending on primary education was only US$250 per capita in 1995, low for a country with Brazil's per capita income.2 Inequity is particularly acute in the northeastern states, where public spending on education amounted only to US$30 per capita up to 1995. As a result, middle- class families send their children to private schools, until secondary or higher education, when these with the highest scores attend public universities. 1.2 The inequitable distribution of govemment resources to schools has created problems, which Bank projects of the 1980s and 1990s attempted to remedy. Primary education is almost universal in Brazil,3 but only 51 percent of the students finishing 4th and 8th grade in 1995 could do math wvork at their grade levels.4 Low quality of education results in high repetition rate in poorer areas, dropout and subsequent low earnings as a result of limited schooling.5 Instructional effectiveness and resources are at least partly to blame. Of the 69,300 primary school teachers who had not finished even primary school, 47,000 or 67.8 percent were in the northeast in 1995, 96 percent of them in villages. Salaries as low as US$30 (20 percent of the national average)6 meant that villages were unable to hire more qualified teachers. Textbooks were rare, and classes covered only 60 to 70 percent of the curriculum.7 Teacher training (often through lectures) or mere salary increases had no impact on student achievement.8 Despite known benefits, preschool attendance was low.9 Due to the complexities of state and municipal autonomy regulations, only 12 percent of the earmarked funds could be transferred to beneficiaries. As discussed below, the efforts were met with partial success, and the Bank has searched for different intervention designs. 1. Gordon, Lincoln. 2001. Brazil's Second Chance. Washington, DC: Brookings Institution. p 123. 2. Brazil: Structural Reform and Sector Policies. Report no. 13913-BR, June 30, 1995. In higher education public spending is excessive, US$6000 per student, almost twice the cost of the best private universities (p. 52, 53). 3. Overall gross enrollment rate was 120% and net enrollment rate was 90% in 1998; repetition was 38.2% and dropout 21.2% in the Northeast in 1995. (School census data produced by the Fundescola project implementation unit, March 2002). 4. Chamada a Aqao. Ministry of Education. Brasilia 1997, p. 25; for all of Brazil the percentages were 65% and 74% respectively. 5. Lam, David and Suzanne Duryea. 1999. Effects of Schooling on Fertility, Labor Supply, and Investments in Children with Evidence from Brazil. The Journal of Human resources, 34, 160-192. Schooling reduces fertility over the first 8 years and has strong effects on children's survival and schooling. 6. Ibid p. 33, 45. 7. Ibid p.41. 8. Brazil: Teacher's Development and Incentives. Washington: World Bank, LAC Region. Report No. 20408 BR. 2001. 9. Brazil: Early childhood Development: A Focus on the Impact of Preschools 2000. World Bank. Human Development Department. Preschool attendance had a positive and significant impact on the average years of schooling ultimately attained and reduces repetition by 3-5%. One additional year of preschool increases schooling attained by about half a year. Possibly the gain is higher for children of illiterate parents. Though malnutrition is the most deleterious during the early years, only 13% of the school feeding budget was targeted to children 0-6 years. 2 1.3 In 1995, Joao Santos, a desperate fifth grader from a rural school of the northeastern state of Ceara, wrote a letter to the regional director complaining that his teacher was teaching very little during class and expressing his fears that he would fail another year. In that year, a new government renewed its commitment to education and brought about wide-ranging administrative changes to increase education financing in the Northeast. Joao's letter has been the basis for 13 diagnostic studies and a sector study, "Call to Action" ("Chamada a Acao") to improve the schools for the Brazilian poor.'" Education of the disadvantaged became a comerstone of the poverty alleviation strategy of Brazil," and the studies formed the basis for a new generation of projects. 1.4 The studies offered specific recommendations for use in designing future interventions. One of them, involving classroom observations in 94 schools of Ceara and Bahia, pinpointed the instructional problems with considerable specificity.'2 Use of time ("time on task"), classroom organization, and opportunities to learn were closely related to early reading perfornance. Teachers rarely used textbooks in class. They mainly gave writing exercises, such as copying from the blackboard, dictation, and mimeographed sheets. These routine tasks also became more frequent in the third and fourth hour of school, as teachers spent class time grading papers. The classes that did more copying and other routine work had lower reading scores than classes doing more reading and student-generated materials. School duration was actually three hours rather than four.'3 Time on task was more important for raising the achievement of boys, who often end up less educated and vulnerable to violent deaths.'4 Class size did not make a difference, but the teacher's general education did. So did preschool attendance. The research also showed that the average educational level of the community mattered more than the teachers' educational level, but that the home environment accounted for only 10 percent of the reading achievement variance; poor instruction mattered more. By grade 2, achievement levels were higher in state than in municipal schools. Clearly, the important variables that determine achievement were found to be in principle under government control; the challenge was and is how to overcome institutional obstacles and complexities and teach the poor information that they can retain and use. 1.5 The World Bank's involvement in Brazilian education has been extensive, both before and after the "Call to Action." Since 1971, the Bank has financed 19 education projects, totaling US$1.941 billion, of which 13 have been completed (Annex B Table 1). The main focus has been primary education, but also on science and technology. In February-March 2002, an Operations Evaluation Department mission assessed the outcomes of four primary education projects that were implemented during the 1 990s. The review was based on the Staff Appraisal Reports,'5 Project Completion Reports (for older projects), and Implementation Completion Reports (ICRs), 10. Ibid p. 7. I1. Brazil: Attacking Brazil's Poverty. World Bank: Report No. 20475-BR, March 31, 2001. 12. Fuller, B. Dellagnelo, L, Strath, A, et al 1999. How to raise children's early literacy? The influence of family, teacher, and classroom in Northeast Brazil. Comparative Education Review, 43, 1-35. Based on research reports, such as: Portela, A. L, et al., Conhecendo o Universo da Sala de Aula. Brasilia: Ministry of Education, 1998; Areias Prado, 1. G. et al., Educacao, Escola e Comunidade. Brasilia: Ministry of Education, 1997. 13. Ibid. Though a school should function for about 4 hours of classes (229 minutes), the average functioning time was 184 minutes (three hours), including lunch and recess. 14. Brazil Gender Review: Issues and Recommendations. Brazil Country Management Unit, January 2000. t 5. Innovations in Basic Education (SAR #7686, June 1991, ICR #20238, May 2000); Northeast Brazil 11 (SAR #11298, April 1993, ICR #20238, May 2000); Northeast Brazil Ill (SAR #11954, October 1993, ICR #20239, May 2000); Fundescola I (PAD #17402, March 1998, ICR #23296, November 2001). 3 archives for all completed projects, extensive staff interviews and school observations. (Details on methodology are presented in Annex B). Other Donor Activities 1.6 The World Bank is the main donor in Brazil for primary education. The Interamerican Development Bank (IADB) invests in secondary, higher, vocational, and early childhood education and spans various social agencies aside from the Ministry of Education. The World Bank and IADB complement each other in their distribution of sectoral activities. However, according to Ministry of Finance officials, there has been little coordination between the two donors. Officials describe the procedures of LADB as more flexible, requiring fewer covenants and conditions, with procurement standards closer to those of the government. On the other hand, IADB task managers are reportedly present less often in the field, and the World Bank is seen as offering more valuable sectoral advice. 4 2. Objectives and Implementation of the Assessed Projects 2.1 During the 1990s, the World Bank financed six primary education projects, of which four have been completed and are the subject of this report. The primary focus of these projects was quality improvement, rather than expansion of the system; primary education has in principle been universalized, and the birth rate has dropped steadily in Brazil. The projects sought to improve leaming conditions in existing schools, decrease the dropout and repetition rates, and keep students in school on grade level. The guiding principle was that if management and financing improvements were made in the system, leaming improvements would follow. (Table 1). Table 1. Objectives, and Components of the Assessed Projects'6 Overall goals or Quality Physical Management General Objectives improvements- (where applicable) Financing Innovations in Develop a new curricular approach Expand low cost pre- Strengthen state and municipal Primary Education and extend school day for the first primary education education authorities. Main objective: two years of primary school provided by private through in-service teacher training, non-profit organizations Conduct evaluation studies and Improve primary school new educational materials, more and municipalities in dissemination activities. leaming and retention efficient school feeding programs, poor neighborhoods; among poor and and school construction to allow Reform the school health migrant children in primary schools in poor program to emphasize cost- Greater Sao Paulo; neighborhoods to operate two effective screening, referral, and possibly adapt to other daytime five-hour shifts. health education. states. Northeast Brazil 11 Increase state and municipal Finance a Ume slice of Streamline state education (Ceara, Maranhao, availability of educational materials the states' school bureaucracies to help achieve Pemambuco, Sergipe) Improve state and municipal rehabilitabon and more efficient use of human Goal: Increase student teachers' skills and classroom construction programs, resources and spending learning, graduation effectiveness and the leadership conditional upon states' allocabon. rates, reduce repetition skills of school directors. progress in Strengthen the Ministry of and dropout, through saend implementing agreed Education's capacity to oversee improved quality of Support promising state and management refoms. key and support functions (e.g. state and municipal mughimatching grants from a national testing; planning, policy primary education and through matin s from and norm-setting, financial more efficient . transfers to equalize educaton education management spending across states). Reform the centralized national textbook and school lunch distributon systems Northeast Brazil III Deliver a package of essential Support a program of Provide the tools and incentives (Paraiba, Alagoas, Rio educational inputs and services to school renovation and to improve the efficiency of Grande - Norte, all public school children in the first expansion for urban education system management Paraiba, Bahia) four grades in the project states and rural schools and administration Goal: increase student leaming, graduation rates, reduce repetition and dropout Fundescola I Increase the participation, Strengthen primary schools and promotion and graduation rates, the public institutions that are and achievement levels of children responsible for them within a in the North and Center-West coordinated management capital microregions of Brazil framework 16. There is some confusion regarding the goals and objectives of the NEBE projects. The ICRs describe the learning outcomes as objectives, and these were clearly not met. However, the SARs describe them as goals (pp. 12 and 13 respectively), to be met through improvements in management, educational materials, and training. This report follows the latter definition. 5 2.2 All four projects initially faced procurement difficulties and implementation delays but were eventually completed, and most activities were carried out Innovations in Basic Education Project (Loan 3375-BR) 2.3 It took about four years to prepare this project and over seven years (1992-1999) to implement this project, which focused on the state of Sao Paulo. The delays were due to counterpart fund timing difficulties, which plagued all Bank projects until 1995. Donor fund disbursements required congress approval; this was often given in October and funds had to be spent by December. By 1995, only 15 percent of the loan had disbursed, and the Bank considered canceling it. However, in 1995 a new State Secretary took over, who was keenly interested in seeing the project through and was chiefly responsible for its completion. (See Annex A, Table I for details). 2.4 The project introduced interventions that were subsequently widely used in Brazil. About 60 percent of the loan proceeds were spent on civil works to refurbish schools in the poorer areas of the state and bring them up to minimum operational standards (which included textbooks, instructional materials, and computers for grades 5-8). Over 100,000 teachers of state schools were trained on various curricular issues, and many received university education. Also, schools carried out needs assessments and deliberations similar to those later used in the Fundescola projects. By expanding planning activities and computerization, the Secretaria became able to follow individual students and cleaned its rolls of multiple enrollments that overestimated school population by 200,000. The project also helped develop the SARESP (Sistema de Avaliacao do Rendimento Escolar de Sao Paulo), a criterion-referenced test that is regularly administered in all schools to monitor school perfomnance. Acceleration and recuperation classes were set up in many schools to enable overage and weak students to catch up. Students in lower grades were separated from students in higher grades. A longer school day (of five hours) was established, an intervention that was shown to raise student scores according to a 1996 impact evaluation.'7 A number of schools were handed over to municipalities (a process that continues), where parents and local officials may have closer quality control. 2.5 Other activities were not as successful. Emphasis was on providing diverse classroom materials, not necessarily on reading them. Preschool education was not expanded as expected, and the NGOs that would have been partners to this effort were found to have limited interest and capacity. Prefects who were to receive new buildings for preschool education often made requests for high-cost structures, which the implementation unit rejected. Very few of the contemplated health interventions took place, partly because collaboration with the health secretariat proved difficult. Northeast Brazil II and Northeast Brazil III Projects (Loans 3604-BR and 3663-BR) 2.6 The two projects aimed at increasing student learning and graduation rates, while reducing repetition and dropout (Table 1). Each project targeted a group of northeastern states with the same components. The govemment implemented them as one project, often referred to 17. The evaluation was held before the project closed. Schools with good standards (that were, however, better schools) and a longer school day had higher scores. (Neubauer, Rose, Claudia Davis, and Yara Lucia Esposito. "Avalia9,ao dos processos de inova,ces no ciclo basico e seu impacto sobre a situa,ao de ensino-aprendisajem na regiao metropolitana de Sao Paulo." Estudos em Avaliacao Educacional. Fundacao Carlos Chagas. January-June 1996, p. 35-91.) 6 as NEBE (Northeast Basic Education Project). Learning was to increase through improving the management of state secretariats of education and the physical environment of schools. The Bank hypothesized that if state-level management became more financially efficient and if educational materials and facilities were provided, the quality of education would improve.'8 Thus the projects mainly financed construction and repairs for about 26,000 classrooms (about 49 percent of project costs), textbooks and materials (about 30 percent of project costs), training (7 percent of project costs.) and management studies (5 percent). A federal-level component (9 percent of costs) created a sample-based achievement testing system (Sistema Nacional de Avaliacao da Educacao Basica - SAEB), pilot-tested a multigrade methodology (called escola ativa), mass-produced textbooks, and developed a management information system. (Annex A, Table I for details). 2.7 Similar objectives had been pursued earlier in the first Northeast Brazil project, with ambiguous results.'9 Lessons were not sufficiently taken into account, either from the first northeast Brazil or the Innovations project. The design of the NEBE II and III projects proved very complex, as they tried to address simultaneously many of the education problems of the NE with a broad coverage. They involved federal, state, and municipal levels of govemments and nine states, each with its own political priorities and constraints. Initially, the projects' financing system ignored the fact that municipalities were autonomous, and put states in charge of implementing in areas where they had no responsibility. In some respects, the projects focused on paperwork rather than on students. The states were required to do complicated management studies and calculate annual management efficiency ratios and targets for staff, materials and textbooks per student as a condition for getting funds for school construction. The Ministry of Education unit in charge of bringing this work about was situated at a relatively low administrative level and had limited authority. At the same time, the project had no clear academic indicators or monitoring mechanisms beyond management efficiency targets. 2.8 As might be expected, quality at entry was unsatisfactory, and the NEBE project faced severe delays for the first two years. (Total implementation took eight years.) The archives show much correspondence inquiring about planning exercises in various states, hiring of consultants and computer firms to calculate the management efficiency targets, transfer of raw records to Brasilia to carry out this work. Municipalities were often unresponsive to the proposed leadership of states, sometimes because of political differences. Due to financing rules,20 counterpart funds for school construction in the poorer northeast became higher than those required for other states. The states of Maranhao, Piaui, Alagoas, and Pernambuco had particular difficulty implementing their share of the project. NEBE was simplified in 1995, when a new govemment renewed interest in education, and municipalities were able to get funds directly from the federal government. A semi-autonomous project implementation unit was formed, with full- 18. Inappropriate teaching strategies are discussed in para. 1.31 (p. 9), but the teacher training component does not prescribe specific remedies (p. 26; Staff Appraisal Report. Brazil: Second Northeast Basic Education Project, Report no. 11298-BR, April 20, 1993). Similarly, see project description, p. 13-16 (Staff Appraisal Report. Brazil: Third Northeast Basic Education Project, Report no. 11959-BR, October 29, 1993). 19. The Northeast Brazil project (Ln. 1867BR; 1980-88) built schools in 400 municipal units and attempted to improve learning outcomes for 800,000 students. The audit report rated the project outcomes satisfactory but found that instruction did not improve, and teacher salaries remained low; financing was also complicated. (Brazil. Northeast Basic Education Project (Ln. 1867-BR) Performance Audit Report. No. 13242. World Bank 1994, p. 4-6; Brazil. Northeast Basic Education Project (Ln. 1867-BR. Project Completion Report. No. 8266. World Bank 1989). An impact evaluation corroborated findings (Educational performance of the poor: Lessons from rural northeast Brazil, March 1992, Report No. 10881). 20. Beyond the scope of this document. See "The Northeast Basic Education Projects: Implementation Problems." July 30, 1995, in Project files, p. 6. 7 time staff for the work. Eventually, numerical output targets were met in terms of school construction, large-scale textbook distribution, and large-scale teacher training (Annex A, table 2). A school census and management information system were developed and adopted by most states, which made it-possible to get some student-level statistics. Large-scale teacher training took place in a standardized system of lectures, but its impact was unknown. Project outputs were scattered through a large number of schools, with some getting just paint, others teacher training. Often funds went to those agencies that best knew how to take advantage of them; the poorest municipalities had little understanding, or perhaps, interest in applying for project benefits, so the poorest of the poor students probably got few if any benefits. 2.9 Due to increased government attention to the education of the poor, impressive changes took place in the Northeast between 1996 and 2000; net enrollment rates in grades 1-8 increased from 77.3 percent to 92.8 percent, the highest increment within the country. During the same period, the promotion rate in the Northeast increased from 61.8 percent to 68 percent, while dropout rates decreased from 21.2 percent to 18 percent.2' Project-level data are limited, but improvement in dropout and repetition approximated that of state averages (See state averages in Table 2). A target of 5 percent in dropout rates was met by six of the nine states, but a 10 percent reduction in repetition was achieved only by one state (Annex A Table 2). A four-year study cycle now gives students four years to learn basic reading without repeating years. Also, some municipalities and states (e.g. Ceara) instituted automatic promotion policies, so children may have attained higher grades due to these policies rather than increased learning. Thus, students who earlier were seen as failing are now considered as learning within allotted time limits. Nevertheless, it is unknown how the poor areas reached by the projects would have performed without project inputs. 21. School census data produced by the Fundescola project implementation unit, March 2002. 8 Table 1. Some Educational Statistics by State, 1996-2000 Basic Education Promotion Rates Repetiiion Dropout State Total Total Total 1996 2000 96/2000 1996 2000 9612000 1996 2000 96/2000 Brasil 71.8 77.0 5.2 13.9 10.7 -3.2 14.3 12.3 -2.0 Norte 60.9 68.6 7.7 18.4 13.6 -4.8 20.7 17.8 -2.9 Rond6nia 68.5 73.3 4.8 15.5 11.1 -4.4 16.0 15.5 -0.5 Acre 63.1 70.7 7.6 15.9 12.8 -3.1 21.0 16.5 -4.5 Amazonas 63.9 71.4 7.5 16.6 13.9 -2.7 19.5 14.8 -4.7 Roraima 73.6 77.3 3.7 15.1 7.5 -7.6 11.3 15.3 4.0 Para 56.6 64.3 7.7 21.4 15.5 -5.9 22.0 20.2 -1.8 AmapA 66.1 77.7 11.6 22.2 14.2 -8.0 11.8 8.1 -3.7 Tocantins 61.8 72.8 11.0 12.6 8.0 -4.6 25.5 19.2 -6.3 Nordeste 61.8 68.0 6.2 17.0 14.0 -3.0 21.2 18.0 -3.2 Maranhao 64.4 72.2 7.8 16.5 13.5 -3.0 19.2 14.3 -4.9 Piaui 56.5 68.5 12.0 23.8 16.9 -6.9 19.8 14.6 -5.2 Ceara 67.2 79.8 12.6 11.8 8.1 -3.7 21.0 12.1 -8.9 R. G. Do Norte 64.2 70.1 5.9 17.0 14.8 -2.2 18.7 15.0 -3.7 Paralba 62.4 68.4 6.0 16.6 13.3 -3.3 21.0 18.4 -2.6 Pernambuco 60.6 67.9 7.3 17.6 15.7 -1.9 21.8 16A -5.4 Hala aos 53.0 64.2 11.2 23.6 16.8 -6.8 23.4 19.0 -4.4 Sergipe 53.4 61.1 7.7 24.7 21.1 -3.6 21.8 17.9 -3.9 Bahia 61.7 61.4 -0.3 14.4 -1.9 17.5 22.1 24.2 2.1 Sudeste 80.9 86.5 5.6 10.0 6.5 -3.5 9.1 7.0 -2.1 Minas Gerais 80.2 83.4 3.2 10.5 7.1 -3.4 9.3 9.5 0.2 Espinto Santo 72.3 82.3 10.0 12.7 7.6 -5.1 15.0 10.1 -4.9 Rio de Janeiro 75.8 81.0 5.2 12.7 10.2 -2.5 11.5 8.7 -2.8 Sao Paulo 83.7 90.9 7.2 8.5 4.7 -3.8 7.8 4.5 -3.3 _ Sul 76.8 83.1 6.3 14.7 10.8 -3.9 8.5 6.1 -2.4 Parana 74.4 84.3 9.9 14.1 9.5 -4.6 11.4 6.2 -5.2 Santa Catarina 79.4 84.9 5.5 13.4 8.2 -5.2 7.2 6.9 -0.3 R. G. Do Sul 77.8 81.0 3.2 16.1 13.6 -2.5 6.1 5.5 -0.6 Centro-Oeste 69.2 73.8 4.6 14.5 11.2 -3.3 16.4 15.0 -1.4 M.G. Do Sul 70.5 75.4 4.9 18.4 15.5 -2.9 11.1 9.1 -2.0 Mato Grosso 68.6 72.9 4.3 13.4 8.0 -5.4 18.0 19.1 1.1 Goias 65.7 72.2 6.5 12.9 9.2 -3.7 21.4 18.6 -2.8 Distrito Federal 78.3 78.1 -0.2 15.9 17.4 1.5 5.8 4.6 -1.2 2.10 Project impact on learning not be substantiated, because there were no baseline data on project beneficiaries. 22 However, testing issues also made comparisons difficult. SAEB test content was altered, and many states showed small improvements or even decreases in performance of all students between 1995 and 1997.23 Overall, project effects are confounded with the important political developments in the country. In 1995, Brazil mandated training of unqualified teachers, financing for municipalities, and school lunches for the poor. Since then, the country has also enjoyed political and financial stability. The federal-level components of the 22. Implementation Completion Report. Brazil: Third Northeast Basic Education Project, May 31, 2000, Report No. 20239, p. 7. 23. Overall scores in the northeast dropped from 179 to 168 between 1995 to 1999 whereas those of all Brazil dropped from 190 to 181. (SAEB. Estudos Comparativos 1995/1997/1999. November 2000). For enrollments see INEP (Instituto Nacional de Estudos e Pesquisas Educacionais). Sinopse Estatistica da Educa,ao Basica, Brazilia, April 21, 2001, p. 33-50. INEP. 9 project, which supported these policies (large-scale textbook delivery and development of a national testing service) were successfully implemented. As a result, textbooks now reach almost all schools, and basic-education students use them free of charge. School Improvement Project-FUNDESCOLA I (Loan 4311-BR) 2.11 Fundescola was designed to deal with some of the difficulties faced by the NEBE projects as well as with the recommendations of the "Call to Action." The design evolved in two important aspects: (a) integrating areas of state and municipal authorities and (b) focusing on school-based management and strengthening. To reduce the complexities of state vs. municipal financing, the management information system data available through the NEBE projects was to establish microregions of intervention. These consist of densely populated urban areas (ultimately established through legislation) along with associated periurban and rural areas, and include state as well as municipal schools served by the agencies that operate in the microregion. Thus, it was possible to concentrate multiple inputs and resources to a small number of schools rather than disperse resources in a large number of schools and serve municipal schools directly. Given the expense of reaching the sparsely populated rural areas, it was considered the best possible design. The project was the first phase of program that will last 10 years. When 50 percent of the funds are disbursed and 75 percent committed, the second project starts, which operates on a different group of schools, municipalities and states. Fundescola I implemented in the north and central states, while the ongoing Fundescola II includes the Northeast (See Annex A, Table 3 for details). In some respects, it expects to show the states how much school performance can improve if multiple inputs are concentrated on a school that has strengthened school-based management. The expectation is that states and municipalities may adopt this school management model. 2.12 The schools chosen for resource transfer in the selected microregions are larger, with over 200 and possibly better. (The follow-on project targets schools of over 100 students). The project was designed to improve efficiency of resource delivery and decrease inequity within a microregion. Its purpose is not to decrease inequity overall; in effect, it increases inequity between project and non-project schools. (Equity may increase in the future if the local governments adopt and expand the project design.) Part of the improved resources to schools consists of repairs to attain minimum operating standards for schools (which also apply to all schools in the country). Funds were also distributed to schools, and their use was audited on a sample basis. Parental involvement is a key ingredient of the program. Small rural schools with multigrade classrooms are attended to through the escola ativa program, which involves teacher training, special instructional materials to enable students to form groups and partly study on their own. Fundescola I operated 178 such rural schools, the follow-on project has thus far expanded the number to a total of about 1225 schools in 19 states, ultimately benefiting 42,719 students and 1,838 teachers. 2.13 The Fundescola I schools are asked to carry out an educational development program (Plano de Desenvolvimento da Escola -PDE), during which they assess their needs and the deficiencies of the school and students through an extensive and complex questionnaire. To improve school performance, they must request materials and resources through a Projeto de Melhoria da Escola (PME). The development process has been very popular, serving 1,724 schools rather than the planned 401 and cost US$11.5 million rather than the planned US$5.2 million. Five states (Acre, Mato Grosso, Goias, Tocantins, and Ceara) have adopted this plan as a model for all schools. 2.14 Schools with and without PDE were compared through SAEB scores in mathematics and Portuguese, and small but significant performance increases were found for schools with PDE. 10 Perhaps the PDE effects are partly due to parental tendency to send their children to these schools, since the schools participating in Fundescola tend to be larger, better, and more likely to have good results. Nevertheless, the magnitude of the effect was small. A more extensive evaluation of 400 schools is underway. 24 Observations of Classroom Events in Project Areas 2.15 To assess project effects empirically, the OED mission visited state secretariat offices and schools: (a) For the Innovations in Basic Education Project, eight schools in the state of Sao Paulo, two of which were municipal; four were in outlying and poor municipalities largely populated by migrants from the northeast. (b) For the NEBE project, 21 periurban and rural (albeit not remote) schools in Ceara, Rio Grande de Norte, and Alagoas. (c) For Fundescola I, five schools in the state of Goias, a state ranking in the top 10 states of math SAEB scores in 1999. The mission also visited eight schools in the northeast that participated in the Fundescola II project, whose design is the same as that of Fundescola I. Almost all the schools visited had received some benefit from the projects, either in terms of physical inputs or in teacher training. Schools were chosen randomly from those located at distance that the mission could visit from a state capital. (See Annex B for more detail for methodology and conclusions.) Observations were (Annex C for more details): 2.16 Physically, most schools were in very good condition. Most were attractive and clean, with the necessary space for materials and gatherings. So, the prerequisite attractive and physical environment that is conducive to learning has been fulfilled. 2.17 Textbooks and instructional materials were present in most schools. The larger periurban schools in the Northeast had books as well as audiovisual equipment, but rural and semi-rural schools did not have enough textbooks for each student. Audiovisual equipment was minimal and/or broken down. Most Sao Paulo schools reported having received instructional materials, including mathematical games and art history panels. However, textbooks were sometimes unavailable to the students or not used. Some of the more populous Sao Paulo schools reported having a smaller number of textbooks than students, and thus kept the textbooks in class rather than hand them out to students for the duration of the year. For various reasons, some schools had still not handed textbooks out, one month after classes had started. 2.18 All teachers were reportedly qualified to teach. According to teacher and principal reports, all teachers had at least secondary-level education. There is continuous training, and teachers state that they find it useful. There was evidence of lesson planning, and material written on the blackboard. 2.19 Some innovative measures have been implemented to improve learning. These include accelerated classes for overaged children, adult literacy for parents and recuperation classes for students falling behind (in Sao Paulo). However, these measures do not exist in many Northeast schools. 2.20 However, in the schools observed, the classroom problems cited in earlier research and "Call to Action" were evident. The teachers still do not use class time efficiently because of: 24. Initial draft of degree of implementation report. Paper presented to the American Educational Research Association conference in New Orleans on March 11, 2002. I1 2.21 Unoccupied students in class. Often a teacher worked with one or two students while the remaining students did nothing. Sometimes class work started late or ended earlier, with students unoccupied during the first and last 5-10 minutes. Several classes were found without teachers, with students either unattended or being partly attended by substitute teachers. On other occasions the students had been sent home. The unattended classes were noisy and possibly disturbing other classes. Principals reported that absenteeism is frequent, since teachers have a right to a certain number of absences.: Also, teachers leave classes unattended to take training. Reportedly, the frequent absence of teachers is one reason why curricula are not being completed. Several schools in the Northeast said they cannot hire substitute teachers. Though the schools of Sao Paulo can hire one, often there are more than one teacher absent. 2.22 Copying. Students in all states and all levels were observed copying extensively from the blackboard. The older students copied because they had no textbooks. The younger ones often copied a series of questions requiring brief answers (an activity that teachers described as 'text production'). Acceleration classes to help overage students to catch up also included extensive copying. When teachers were asked why such work was not assigned as homework (assuming it fulfilled specific objectives), they often responded that the parents were too poor to monitor students' homework, and there was no point in assigning any. The copying and laborious production of short answers also seemed to be boring and unchallenging work. 2.23 Spending much class time in art and play. Particularly in Sao Paulo, classes were repeatedly observed cutting up and pasting materials to help learn reading, and the mission was shown workbooks with elaborate drawings and collages. However, these activities seemed to take up much of class time that would have been devoted to actual reading practice and attainment of basic skills. For example, students in remedial classes spend many hours of cutting up and pasting articles in workbooks. The rationale given by teachers was that children presumably need to construct the world from their own experience, so they must learn to read from regular print and through primitive writing. Instead of using textbooks for grade 1, teachers in periurban Sao Paulo schools taught reading through children's story books and magazines. Though such activities may be pleasant and beneficial in the long run, they consume much class time of students who are at a risk of early dropout. 2.24 Many students could not read. Overall, copying and art were observed in 34-47 percent of classes visited (Table 3). Limited time on task may be one of the reasons why most students in the Northeast who were asked to read could not read aloud fluently until grade 5 and could not calculate well. Students in multigrade classrooms (escolas ativas) of the Northeast were particularly weak and could not read well enough to carry out the independent work required of them. Some students in grade 2 were observed playing with letters while unable to read the beginning of the grade I textbook. Though most students who were asked to read in downtown Sao Paulo schools did so, the children in migrant periurban municipalities also could not read fluently until grade 4. The low performing classes had many reading materials, but it was unlikely that students would benefit from them if they did not know how to read. Though it was not possible to attribute the skills level to the project or to compare NEBE with non-NEBE schools in this aspect, SAEB scores and student flow rates bear witness of the overall low student performance of the Northeast and point to the need for direct learning-oriented interventions. 2.25 Repeatedly, teachers attributed the students' low reading performance to family poverty. Overall, principals and teachers reported satisfaction with the quality of work they were doing, though several said that they did not have enough time in a school year to complete the curriculum. Two Sao Paulo state schools reported that their SARESP test scores were above average, although in both cases the scores reflected that students were achieving on the average only half the objectives. In periurban Sao Paulo, however, teachers declared themselves unable to 12 deal with all the problems of the students, who live in very poor conditions and study in overcrowded classes. Yet, the schools were not able to get extra help (such as tutoring) for students who are falling behind in the early grades. 2.26 Some children were small and listless. Visibly, many students in the rural northeast were smaller than children seen in Sao Paulo and Goias, and seemed less active. Perhaps some lack basic vitamins and micronutrients that may help maintain attention in class, or have had infections and violence incidents that have been shown to damage cognitive processing.25 The effects of these factors on school performance are receiving increasing attention in the educational literature. 2.27 Goias classrooms seemed more efficient. The Goias schools that had implemented Fundescola I were well organized, and nearly all classes were observed engaged in learning work (Annex table 3). All students who were asked to read, even in grade 1, did so relatively fluently. Students in the one escola ativa visited were carrying out the instructional tasks as expected and were articulate about the work they were doing. The state of Goias has devoted a lot of resources towards achieving the minimum operational standards for schools and carrying out school development projects. A large number of supervisors and trainers is available to follow up on trainees; for example, there are part-time trainers, each responsible for following up on project actions with five schools. The mission did not observe much time wastage in class but, according to a local director the problem had been endemic. About four years earlier, the state made a study and found out that four hours were reduced into two and that teachers spent most of the time copying from blackboard. Actions were taken to improve time on task through inservice teacher training, supervision, and follow-up. 25. Berkman , Douglas S., Andres G. Lescano, Robert H. Gilman, Sonia L. Lopez, and Maureen M. Black. 2000. Effects of stunting, diarrhoeal disease, and parasitic infection during infancy on cognition in late childhood: a follow up study. The Lancet, February 16, 2002. Violence exposure linked to lower IQ in children. Archives of Pediatrics and Adolescent Medicine 2002;156:280-285. reutershealth.com, March 12, 2002. 13 Table 3. Class activity observed at moment of mission entry State Ceara Rio Grande de Norte Alagoas Goias Sao Paulo Copying 6 6 2 8 (2 secondary) 40% 33% 22% 17% Text production 1 3 3 4 2 7% 17% 33% 28% 4% Reading lesson in class 2 1 1 1 13% 6% 11% 2% No activity 1 5 (2 waiting for bell) 1 8 7% 28% 11% 17% (some teachers .___________ _ _ absent) Teacher busy with 2-3 1 1 4 students only 7% 11% 8% Teacher teaching whole 2 1 6 10 cass 13% 11% 43% 22% Group practice 1 2 3 4 7% 11% 21% 8% Art and Play 1 (scheduled activity) 1 (play with letters) 1 (play with 1 (drawing) 8 (durng subject 7% 6% letters) 7% matter) 11% 17% No. classes 15 18 9 14 45 No schools 8 6 7 5 8 Apparently instructional activities (some 34% 45% 48% 92% 40% suboptimal) Note: some schools visited did not have classes at the time 2.28 All teachers and principals in the Northeast who were asked agreed that Fundescola's school development process (PDE) was an excellent opportunity, and that their school was better off as a result. Some found out that they needed training in mathematics, methodology, or in computers. They had decided they needed audiovisual equipment (all eight schools) and computers (four schools). However, the classes of these schools had limited instructional activities, and most students who were asked to read could not do so. Arguably, PDE has been in operation in the northeast only since 2000, and benefits might take longer to appear in schools. However, it was unclear how basic reading and math skills would improve through audiovisual equipment. One might have expected the schools requesting tutors for students who have fallen behind, such as secondary school students who would help for a nominal amount. But only one school in Fortaleza expressed an interest in such a program. (At any rate, tutors might not be an eligible expenditure item under the project.) It is possible that the school development process questionnaire unwittingly guides teachers to consider systemic school problems rather than their own wasteful classroom practices. In a Fundescola training session of 120 principals, some raised the issue of the questionnaire's complexity and vagueness, although recognized its relevance. 2.29 State staff had positive opinions regarding project effectiveness. The OED mission held meetings with project implementation unit staff as well as state teams in charge of implementing the projects and asked a series of questions regarding benefits, challenges, and outcomes. Overall, staff expressed favorable opinions regarding project effects (Annex B, Table 2). Though the management studies of the NEBE projects were deemed onerous, staff believe that they forced some accountability, transparency, learning about computers, and awareness of the availability of educational information and data. Microplanning data were seen as a useful tool to reduce politically motivated decisions. School buildings were viewed as very positive 14 contributions, which children cherish. Difficulties and problems included implementation of instructional components, lack of local-resources, complexity of the tasks, and difficulty in including the poorer municipalities. Overall, staff thought that the states visited are much better off having executed the project than if they had not. 3. Ratings 3.1 Strictly speaking, the Bank's human resource development strategy for Brazil was relevant to its economic needs, as perceived in each period, and the objectives of the projects have been relevant to the economic needs of the country. But it was not sufficient to provide the skills that might alleviate poverty through education. The Bank missed opportunities to enunciate classroom efficiency standards when it developed the minimum operational standards for schools and to help state secretaries enforce them. It also led the govemment to spend scarce resources in trying to bring about bureaucratic changes. The projects' relevance and efficiency were modest. (See discussion at the end of this section.) 3.2 The Innovations in Basic Education project established some innovations that have proved effective and were institutionalized. It attained most of its student access and teaching objectives, but the important health and preschool education objectives were not attained, partly due to a weak coordination among government and non-government agencies. Secretariat staff considered that activities such as remedial classes improved learning, and the evaluation, which was carried out only on certain activities, showed that a longer school day increased student scores. However, the extent of overall learning improvement and retention of students as a result of the project is uncertain. For these reasons, outcome is rated moderately satisfactory. 3.3 The NEBE projects met and sometimes surpassed their physical targets (Annex A, Table 2) in terms of school construction, materials, and training. Bureaucracies were not streamlined as a result of management studies, but computerized management information and school census became a reality as a result of the projects. The sustained testing work carried out through SAEB and the continuing ability to deliver large numbers of free textbooks throughout the country show that the Ministry of Education acquired the capacity to carry out such complex tasks. Teacher training did take place, though its effectiveness is unknown and dubious. States carried out very few of project-supported innovations. The projects were to increase students' learning, but it is unclear whether the goal has been achieved, since no baseline achievement data were collected.26 Observations by the mission and opinions of local staff suggest that learning achievements were limited, but it is unknown how students would perform without the project. 3.4 Also it is difficult to separate the effects of the project from the effects of better government policies. The general reform of 1995 made it possible to increase financing for the poorer municipalities and for teacher salaries. The project coincided with a period of political and financial stability as well as staff continuity, both at the federal and at the state levels. Although project funds accounted for about I percent of the education budget, the government 26. The states visited by the mission had evaluation departments, which were publishing reports regarding performance of various areas in the SAEB, institutional evaluation, and other issues (Eg. Avaliacao das Escolas Publicas. Relatorio Final de Lengua Portuguesa. Rio Grande do Norte, Secretaria de Estado da Educacao, March 1999. Avaliagao das Escolas Publicas. Relatorio Final de Matematica. Rio Grande do Norte, Secretaria de Estado da Educacao, March 1999. Estudos das PrAticas Avaliativas de Professores das Series Iniciais. Secretaria de Educacao Basica do Estado do Ceara, 1997.) However none of the states visited had evaluated the effectiveness of the NEBE and Fundescola projects at the state level. 15 found the projects useful to implement. The outcomes of the NEBE projects are rated moderately satisfactory. 3.5 Like NEBE, Fundescola I met and surpassed its physical targets. Its interventions have been enthusiastically received, and teachers state that their schools are stronger as a result of the interventions. Many staff who were interviewed stated that the repaired schools, sports facilities, clean places, and existence of materials nearby make students and teachers willing to come to school and stay. However it is unknown how much school appearance and availability of audiovisual materials contributes towards performance improvement. The needs assessment instrument does not help teachers focus on how they spend class time, a major problem identified by previous research and mission observations. However, a small but significant difference in learning outcomes was shown in favor of Fundescola schools. Given physical accomplishments and staff opinions, outcome is rated as satisfactory. 3.6 Thought the Innovations and the Fundescola projects had evaluation designs, lack of longitudinal data collection makes it hard to attribute student achievement improvements to the project. As discussed earlier, SAEB test scores dropped significantly all over Brazil between 1997 and 1999 for grade 4. (Decreases at grade 8 were small.) Though the reduction is attributed partly to test changes, it may also be due partly to a larger number of poorer students taking the test, some of whom had gotten to grade 4 through automatic promotion. The projects in each state reached only a small proportion of students. Thus, the overall effects of Bank projects cannot be substantiated.27 Institutional Development Impact 3.7 To reach the hundreds of thousands of schools in Brazil, overlapping and partly redundant structures have evolved, creating federal, state, and municipal school systems. In some parts of the country they make bloated bureaucracies, with teachers leaving difficult classrooms for desk jobs. All projects attempted to improve the effectiveness of these structures. 3.8 The Sao Paulo "Innovations" project attempted to establish a sustainable testing system, shrink the number of staff involved in managing the system, and promote private-public partnerships in preschool education. Though the testing system was established, the partnerships for preschool education were not. Given a policy of gradually assigning primary schools from the state to municipalities, it is uncertain whether the project helped reduce the number of administrative staff. Overall, institutional development is rated modest. 3.9 The NEBE projects operated from the untested hypothesis that if staff were reduced in the nine state secretariats of education and funds were better used, learning in schools would increase. At the same time, the projects initially ignored the fact that municipalities were autonomous and did not have to follow directives from states. Though Bank financing ushered the computer age in the state education offices of the Northeast, the institutional development effect at that level was negligible. As a secondary activity, however, the projects also supported important national efforts to produce and deliver large quantities of textbooks as well as develop a national achievement test. Overall, institutional development is rated modest. 27. Ideally, achievement and time on task for the effects of NEBE 11 and III on the states of Bahia and Ceara could be evaluated by repeating the literacy achievement test used in 1996 in the same schools and comparing the data with the baseline established at that time. 16 3.10 The Fundescola I project attempted to deal with the overlapping municipal and state administrations by creating notional microregions that contain both types of schools and focusing on school-based management therein. The goal of the projects is to convince local authorities that the microregions are an effective management tool that is worth "multiplying" in other areas. The concept may succeed; the state of Goias announced its intention to create school development projects in all schools. Because the project significantly promoted school-based management, institutional development is rated substantial. Sustainability 3.11 The sustainability of all four projects is rated likely. Recurrent expenditure budgets are available to maintain the buildings, training initiatives, and textbook production and distribution. The testing instruments developed through the Innovations and the NEBE projects have gained wide use and acceptance, and their continued use is all but assured. The sustainability of the Fundescola I project depends on this ability to be replicated throughout the various states where it has operated. The state of Goias is in the process of generalizing the school-centered methodology in all schools. Bank Performance 3.12 The Bank has been faced with the challenge of improving educational effectiveness in a huge country that has overlapping autonomous administrative structures. Bank financing is a minuscule amount of the federal and state budgets, and project effects on the system are small. (For example, Bank projects refurbished buildings in only 96 of the 1200 schools of Alagoas.) The effectiveness of Bank projects depends on a multiplicative effect, i.e. on whether states and municipalities will think that the Bank promotes designs likely to bring about visible improvement in quality and management of about 276,000 schools. Since the federal government does not own any of the schools, Bank efforts must be concentrated in conceiving of effective means to bring about change and persuading various authorities that these are effective. In the past decade, the Bank has been trying hard to find the way. it focused first on physical outputs and expansion of the system, and it is now looking towards improving quality. 3.13 The Innovations project, which dealt with a single state of mainly urban schools, paid attention to instructional issues, learning outcomes, and impact evaluation. But the NEBE projects expected to improve educational quality only indirectly, through encouraging state secretariats to spend their funds more effectively. The SARs of the projects show much detail about organizational effectiveness of central offices, and the PIU and government gave centralized and clear standards for buildings and equipment. However, there were no details on teacher training content or the instructional methods in the textbooks to be printed. It was left up to staff of states and municipalities to figure out what to do on these issues. With hindsight, it is hard to understand why these interventions would bring about classroom improvements or why the Bank would think so. It may be that Bank staff sought remedies in areas where the Bank has more expertise, such as management and economics. This focus led the state governments to spend considerable time and money in calculating annual management efficiency targets. In the meantime, changes in schools as a result of the NEBE projects were minimal. 3.14 Like NEBE, the Fundescola projects have also emphasized management efficiency, but focused on lower levels, municipality and school. Based on the "Call to Action" research, the Bank formed the hypotheses that: (a) multiple concentrated actions on schools would improve learning; (b) operating in microregions was more effective than spreading partial benefits throughout a state; and (c) when school staff discovered their weaknesses and received the 17 resources they thought they needed, classroom instruction would improve. Thus, Fundescola also had a hands-off stance on classroom efficiency, despite specific guidance from research regarding measures to take (para. 1.4). The Project Appraisal Document of Fundescola I (PAD) includes annexes of economic analyses with calculated rates of return, financial and procurement issues, and a poverty analysis, but no specifics on reducing classroom inefficiencies. 3.15 The search for more effective project designs is continuing. The Bank has gradually discovered that by focusing on the school management may not necessarily bring about instructional improvement in the classroom. Therefore the Fundescola II project has more specific teaching and leaming components, and the Fundescola III (currently being appraised) will include even more specific actions, such as video-based integrated teacher training. Because emphasis on microregions and schools has received much acclaim, it forms the basis of the newly effective Bahia and Ceara projects. It is hoped that ongoing and future projects will be adjusted to include greater emphasis on classroom efficiency. 3.16 Overall, Bank perfornance is rated satisfactory for the Innovations project and unsatisfactory for the two NEBE projects. On balance, performance for the Fundescola I project is also rated satisfactory because of the research that went into its design as well as the responsiveness to borrower needs. Borrower Performance 3.17 The Innovations project faced many financial difficulties, and crucial health-related components were not carried out due to a lack of coordination among various government agencies; performance is therefore rated unsatisfactory. Borrower performance for the other three projects is rated satisfactory. Since 1995, the govemment has consistently implemented pro-poor education policies that made it possible to increase teacher salaries and transfer federal funds directly to schools on a per-student basis, thus bypassing state bureaucracies. Also, in the course of the years, the govemment adjusted its regulations and laws to make it possible for donor-financed projects to receive counterpart funds as necessary. In addition, competent staff were appointed to project implementation units, who contributed to the eventual implementation, despite earlier delays. They helped carry out components, particularly for several secretariats in the Northeast that faced implementation difficulties. 3.18 The NEBE projects showed qualified financial audits, particularly for the years 1994 to 1998. There are cases of double billing and ineligible expenditures. The govemment changed the software used to better describe and screen disbursements. The Fundescola I project audits did not indicate problems. Bank Strategy Contributions Toward the Education of the Poor 3.19 Since 1995, the Bank's education strategy documents have focused on reaching the poor and increasing equity in education. What was the long-term effect of the letter that brought forth "Chamada a Acao"? If Joao Santos were to return to his rural Ceari primary school seven years later, what would be different? 3.20 The Bank's country assistance objectives focus on inequity of resource distribution as the major problem in Brazilian education.' The "Call to Action" included 13 studies (at a cost of about US$400,000), the distribution of a consciousness-raising video - featuring a prominent northem singer - to state and municipal secretary training sessions, and the design of the 1 0-year Fundescola program on the basis of findings. However, Fundescola, the Bank's response to "Call 18 to Action", did not attempt to increase equity or improve overall quality of education in the states it operated. It focused on the easier-to-reach urban areas and nearby rural areas (where multigrade escola ativa classes were instituted). The residents of the more populous areas received more inputs; it was possible for a periurban school to receive multiple inputs from NEBE and Fundescola projects. However few if any investments were made in rural areas of the sparsely populated Northeast. Unless Joao's teacher received training from the NEBE project and changed her ineffective behaviors or the municipal secretariat showed unusual initiative and obtained help directly from federal funds, Joao's school may be in about the same condition as in 1995. 3.21 The Bank has had important constraints and limited options in designing projects that would effectively reach Joao's school. Because the federal government does not own any schools, the Bank must somehow work through secretariats, and prefects who often change. The project implementation unit cannot dictate rules but must convince and provide incentives. Some government staff reported to the mission that public interest in the education of the poor is rather limited, because many middle class children go to private schools, and the poor may elect local authorities on the basis of infrastructure accomplishments. Because Bank projects can only reach a finite number of schools, the strategy has been to get the biggest effect possible for the coverage. As a result, Bank financing in the last decade has not focused substantially on the poorest schools or on increasing equity. To the contrary, it mainly financed the larger schools most likely to show results, and in effect may have unwittingly contributed to increased inequity between the urban and rural poor.28 Some actions specifically aimed at inequity are being undertaken with the new Bahia and Ceara basic education projects (Ln. 4591-BR and 4592-BR), which focus on the lowest-performing municipalities. 3.22 Nevertheless, Joao and his younger siblings may have to wait a decade before instruction in their school becomes more efficient, if it does. Perhaps state governments will eventually adopt Fundescola as a state-wide model, and after several years and deliberations, the teachers in Joao's school may realize that they are wasting students' time. The pace of progress towards effective learning in public schools serving the poorest students appears glacial. 4. Issues for Future Consideration Project Designs Built on Untested Theories 4.1 Three of the four projects (i.e., NEBE and Innovations projects) were designed on the untested hypothesis that more efficient management at the central or school level would improve learning outcomes and enable students to progress in school. Two of them were not implemented in a coherent way to demonstrate it, and only the third one has developed concrete strategies and actions to test the hypothesis at the school level. However, there is considerable research in the educational, social and economic organizational theories that might point towards different hypotheses.29 But there is no evidence it was brought to bear by the Bank in designing and appraising the projects. 28. Fundescola 11 includes schools with populations of more than 100 students and may therefore include poorer schools. 29. E.g. Pferrer, Jeffrey. Understanding organizations: Concepts and controversies. In the Handbook of Social Psychology, vol. 2, p. 749-752. New York: McGraw-Hill, 1998. 19 Much Classroom Time is Spent on Ineffective Tasks or Inactivity 4.2 The importance of class time devoted to learning tasks ("time on task") cannot be overemphasized. The international literature is replete with references and indications of its close relation to achievement in various contexts."0 Efficient class time use must be emphasized a great deal at every level of govemment contacts and various training levels. State and municipal secretaries must be 'sold' on the idea; in turn, they need to sell the idea down the line to teachers, supervisors, and school principals. The Fundescola project implementation team has much management expertise; they can identify opportunities to promote specific actions down the line to increase time on task in classes. 4.3 Certain teaching practices and philosophies reduce the valuable classroom time available in class for actual acquisition and processing of useful information. The most important are extensive copying from the blackboard, early laborious writing, extensive art, and doing in class work that could be assigned as homework. Legal teacher absenteeism also is a major problem in areas such as Sao Paulo. (See Annex B for details.) The limited instructional time is related to low achievement in basic skills and possibly to the high repetition and age-distortion rates. Some Brazilian teacher training manuals attribute students' low performance to the inability of poorer families to help them with school work." With many students falling behind, classes are boring and repetitive.,, With the inefficient use of time, the poor are often left out of secondary or higher education. Eventually, few of these siudents are likely to pass the "vestibular," entrance examination for the university. The best students go to the public universities and fill their quotas, so few poor who pass the examination may be unable to afford the private universities that accept them. 4.4 Practices that waste class time ultimately deterrnine how effective the Bank's loans are, and thus cost real money to governments. Increasing the efficiency of educational expenditures is a priority of state economic memoranda of the Northeast.3" Thus, classroom efficiency must become a priority of Bank and government strategy and a primary focus for untapped savings and efficiency gains. To help states improve class efficiency, time on task could be included in the operational standards applicable to all schools. For example, a goal may be that 75 percent of class time will be spent on learning activities. Principals and supervisors may use existing classroom observation instruments to monitor how time is spent. 30. Time on task refers to the amount of time a student is actually engaged in tearning or teacher directed lessons and activities. it is a gross measure of learning because it does not differentiate between levels, such as memorization and analysis (e.g. Prater, M. 1992. Increasing time-on-task in the classroom. Intervention in School and Clinic, 28, 22-27). 31. The teacher training manuals examined by the mission attributed the limited literacy achievement to poverty (Ministerio da EducacAo. Programa de FormacAo de Professores Alfabetizadores. Apresenta9Ao. Brazilia, January 2001. p. 9, 14). Other works, which discussed philosophy, activities, and attitudes to help bring about literacy were: Ministerio da Educacao. Programa de Formacao de Professores Alfabetizadores. Guia de Orentaqoes Metodol6gical Gerais. January 2001; Ministerio da Educacao. Secretaria da Educaao Fundamental. Parametros en AqAo. Textos de Fundamentacao. December 2001. 32. Remedial programs that limit students to repetitive basic skills activities actually "prompt students' lack of engagement in their schoolwork and frequently result in limited achievement. ( Policy Studies Associates (1995). Raising the educational achievement of secondary school students: An idea book. Volume I, summary of promising practices. Washington, DC: U.S. Department of Education. (Retrieved October 4, 2000, from the World Wide Web: htt)://www.ed.cov/pubs/Raising/volI). Also: Dev, P.C. (1997). Intrinsic motivation and academic achievement: What does their relationship imply for the classroom teacher? Remedial and Special Education, 18(1), 12-19. 33. Brazil: Poverty Reduction, Growth, and Fiscal Stability in the State of Ceara. Report no. 19217, August 21, 2000, p. 12. Brazil: Broadening the Base for Growth: A Report on the State of Bahia. Report no. 21377, October 21, 2001, p. 11. 20 4.5 Improvements in how teachers use class time can be brought about through (a) behaviorally oriented inservice training and (b) supervision and follow-up by school principals and supervisors. Time usage must be emphasized during school development projects (PDEs). Since municipal and state officials are the decisionmakers, their training, notably through the Progestao program could include videos and suggestions for specific behavioral change. Ultimately, local media and parents must be sensitized to effective time use in class. The training network is successful in carrying messages to educational staff. If effective messages and behavioral reinforcement are provided to staff, classroom time use may increase. 4.6 To bring about classroom-level instructional improvements and develop monitoring skills at all levels of management leading up to the classroom (including Secretaries of education), a small Learning Innovations Loan (LIL) could be considered by the Bank. Through it, up-to-date cognitive research could be applied to various classroom situations, particularly those of the rural poor. Effects could be studied for maximal learning outcomes. The project would include piloting of innovative teacher training methods that are most likely to bring about teacher behavior changes. Improve Poor Students' Chances for Catching up and Keeping up 4.7 Early acquisition of automatic reading and calculation is a prerequisite for processing a volume of text and carrying out complex calculations.34 Students in multigrade classes (escolas ativas) in particular need to acquiring reading skills early to follow instructions for further study. Brazil still has children who are at high risk for early dropout, and early emphasis on reading skills will increase the probability that they will remain literate after dropping out.35 Research on this should be done. 4.8 Despite some spelling peculiarities in Portuguese, Brazilian students who receive sufficient instruction typically achieve reading fluency by the end of the first grade. However, in the schools that serve the poor, many do not achieve fluency until grade 4 or 5. By then, they are in effect four years behind and may never catch up.36 The challenge is to deal sufficiently with the difficulties of reading Portuguese37 and offer students intensive instruction and practice in the early grades, before they fall behind. 4.9 As a result of delayed reading, cohorts of students have fallen behind and need to catch up. Several states, such as Sao Paulo, have strategies for doing so, such as acceleration and recuperation classes. Given the limited time on task, these classes may not be as effective as 34. Much research exists on this issue; E.g. Adams, M. J. 1990. Beginning to Read: Thinking and Learning about Print. Cambridge, Mass.: MIT Press; Feitelson, D. 1988. Facts and Fads in Beginning Reading. Norwood, N.J.: Ablex. 35. Hartley, M. J. and E. V. Swanson. 1986. Retention of Basic Skills Among Dropouts from Egyptian Primary Schools. Washington, DC: World Bank. 36. Brazil: Teacher's Development and Incentives. Washington: World Bank, LAC Region. Report No. 20408 BR. 2001,p. 11-12. 37. The research on Brazilian Portuguese is somewhat mixed in regards to the regularity of the orthography. Pinheiro (1995) claims it is highly regular for reading but little less so for spelling. There are only nine cases of unique mapping between letters and phonemes, while Rego (1999) points out many more variant patterns and more asymmetry for reading and spelling. (Pinheiro, Angela Maria Vieira. 1995. Reading and spelling development in Brazilian Portuguese. Reading and Writing: An Interdisciplinary Joumal 7: 111-138. Rego, Lucia Lins Brown. 1999. Phonological awareness, syntactic awareness and learning to read and spell in Brazilian Portuguese. In M. Harris and G. Hatano, eds. Leaming to read and write: A cross-linguistic perspective (71-88). Cambridge: Cambridge University Press. 21 expected. In addition to improving their effectiveness, other means must be sought among the many innovations to enable poor students to get special attention. Some are: - Enable schools to pay for students' tutoring more easily. Schools use their PME money to buy audiovisual equipment and teacher training. However, the equipment usually cannot help students improve their basic reading and arithmetic skills. Schools should be spending the money to hire inexpensive tutors for the students who fall behind. These could be secondary or even middle-school 'students. However, there is a concern that teacher trade unions may be against this measure and that employment regulations may make such work difficult. Means might be found to do so. - Support homework and longer student exposure to learning. Several factors, including parental illiteracy, may make it harder for students to focus on homework. Schools may consider an 'integrated' hour for homework and a longer school day through the help of part- time tutors to help students get more reading practice. - Expand bolsa escola. This program, which pays students' families a small amount of money to send their children to school, is functional where the mission visited, but fewer students than the eligible population were being attended. The program could expand. As mentioned earlier, however, classes must spend more of the available time delivering instruction, if the students' time is to be usefully spent. Increase Preschool and School Health Interventions 4.10 It is known that preschool programs have positive long-term effects for children from disadvantaged families."8 These children often come with developmental delays and limited language and psychomotor skills, which preschool programs can improve. However, these have not been extensively financed by the Bank projects. The Innovations in Basic Education project was unable to mobilize NGOs and municipalities to provide a sufficient number of places for the poor. Coordination between different Ministries as well as between state and municipal levels proved problematic, and few health services were given to children. Offering free preschool to the poor should be a high policy priority. 4.11 However, the need for health-related interventions continues, particularly in the Northeast. Some children suffer from malnutrition, lack of vitamins, parasitic infections, and family abuse, which may decrease cognitive capacity later on. Clearly, there is a need for early interventions, such as early childhood education. The Bank may consider including preschool and early childhood education in its lending pipeline. Autonomy Enables Disinterested Municipalities to Do Little 4.12 There is a countrywide process to "municipalize" primary education and pass state schools on to the various municipalities. However, municipalities are not equal. Some have officials who are more interested and motivated than others, so they may get more resources. Also, the poorer municipalities may pay teachers less, and have difficulty finding good teachers. Not surprisingly, children in municipal schools often have lower scores than children in state schools by grade 2. Municipalization may in fact create a lottery for the poorer students. Those in 38. Barnett, Steven. 1992. Benefits of Compensatory Preschool Education. Joumal of Human Resources, 27:2, 279- 312. 22 better municipalities will do better, and those in poorer ones may do worse. Higher-income countries have extensive parental involvement to monitor quality, but many areas in Brazil have not yet gotten to this point. 4.13 In principle, states have a responsibility to help municipalities, though in practice political obstacles may get in the way. Because of decentralization and autonomy, states and municipalities have separate training and supervisory networks that often operate in the same area (e.g. Sao Paulo) and duplicate the same processes, thus resulting in inefficiencies. 4.14 The problem is not easy to solve, since municipalities have the freedom to do as they wish. Perhaps the states might take the leadership and create training program for all teachers, principals, and local officials to increase the time spent on useful learning tasks. Municipalities might be asked if they want to accept it or create their own. Some states have created voluntary partnerships, but these are of little help when municipalities have little interest in promoting the education of the poor. Lessons 4.15 Experience with the projects confirms a number of OED lessons from education projects: * Attention to state, municipal, or even school management is necessary but is not sufficient for improving classroom instruction effectiveness and efficiency. Attention must also be paid to how the precious classroom time is used. * States and municipalities that are better off are more likely to take advantage of Bank- financed resources. The challenge is to encourage the poorer ones to benefit as well. * Political stability, stable project teams, and policies to help the poor receive quality education contribute a great deal to project success. * Decentralizing school management to the municipal level is not a panacea. Autonomy permits the poorest or least knowledgeable municipalities to fall behind, without much possibility for helping their residents. * In addition to achievement tests, classroom observations may be used as predictors of achievement. Unless there is after-hours coaching, children's knowledge is likely to reflect information processing in classrooms. Classroom observations can potentially provide feedback and promote improvements sooner than achievement tests, which must first be developed, administered, and analyzed. * Students' health needs and problems must be addressed to optimize cognitive development and classroom learning. However, agency coordination problems may make this task difficult, and means must be sought early on to eliminate obstacles. * Research financed by projects must be disseminated and put to action. Findings must be communicated clearly and succinctly, through means that are likely to change behaviors and attitudes. It may be cost-effective to spend money for translation (where needed) and develop a follow-up action plan. 23 * The theories and hypotheses that Bank staff develop about improving educational outcomes should be validated and examined in light of pertinent research before being integrated into project designs. * The Bank should acquire more expertise with respect to improving basic skills, particularly in operations that target the poorest children. 25 Annex A. Project Activities Table 1. Innovations in Basic Education Componentsl Activities Targets to be Outputs Outcomes subcomponents achieved A new cunicular in-service teacher distance training Research shows better approach and training, educated teachers extended school have higher student day for the first leaming outcomes two years of primary school provision of new Almost all schools Materials presumably educational materials, received a package used more efficient school Less processed Effects unknown feeding programs, foods school construction to 350 new schools, 3387 classrooms Effect of school allow primary schools about 3000 built environment on in poor neighborhoods classrooms children is considered to operate two refurbish-expand 340 positive daytime five-hour schools, 1500 697 schools shifts classrooms Accelerated classes program in 940 Accelerated classes for overage students schools are effective in placing 133,000 students students at higher participated grades Expansion of low In 1991 enrollments Increase enrollments 30,000 new NGOs and many cost pre-prmary were about 70,000 by 120,000 enrollments mayors not sufficiently education interested Inservice training for 4000 to be trained Number unclear Preschool had limited preschool teachers expansion provided by private Only 53 new Most NGOs did not There is limited non-profit preschools built have the capacity to provision of public or organizations and undertake preschool free preschool municipalities In poor education education in Sao Paulo neighborhoods Technical assistance Some early work was grants to develop a done, limited effect six-year expansion plan Reform of the to emphasize cost- Scaled down to oral Some children sIll at school health effective screening health, visual and risk of various diseases program referral, and health auditory screening and malnutrition- education h and treatment, study related problems. on morbidity; screening eventually stopped Evaluaton studies Impact evaluaton to Study carried out in Limited utility because and disseminahion be carried out on 2 1995, published the program being activities. cohorts of 1000 each evaluated was delayed of 4- and 6- year olds entering school in 1992 Strengthening of Institutional Shrink the Management Outcome unclear state and development and educational structure reorganized municipal reform management but not shrunk education structure authorities Municipalization Carry out action Many primary state The effectiveness of plans and make schools became municipalities depends decisions on municipal on mayors' initiative 26 Componentsl Activities Targets to be Outputs Outcomes subcomponents achieved children's learning municipal on mayors' initiative needs Strengthen planning Train planners Training reportedly State better able to Computerize system took place track students and determine enrollment needs Strengthen Feasibility study to SARESP carried out Assessment capacity assessment capacity establish a system- was an important wide assessment achievement Strengthen public- A study of proposals Not carried out private partnership in to expand the social social services network School lunches Disbursements were minimal Program decentralized, receiving federal funds Table 2. Northeast Brazil II and III Componentsl Activities Targets to be Outputs Outcomes subcomponents achieved General targets: Dropout reducton 5% annually 6 of 9 states met goals (not Ceara, Maranhao, Sergipe) Repetition 10% annually Only Maranhao met Piaui increased reducUon the goal completion rates by 30%; overall rates improved by 15% in the NE Educational Limited government ICR rated Management ownership unsatisfactory Strengthening Increase spending Annual management Any increase was planning capacity efficiency efficiency targets temporary, but created ._______________ ____________________ transparency Human resources Reduce overstaffing Achieve staffing of An undetermined Exodus from classes management 20:1 number of staff has continued, and Review database returned to the dass state secretariats or retired remain overstaffed School quality Monitor student flows monitoring Integrated Each state develops Management Partly operational management an integrated information system of Finally no outcome information investment program 27 modules targets or baseline data completed, slowly getting set up at state level. Support unit Educational materials Textbooks For grades 1-8 62 million distributed; Used sparsely in lower 542,000 kits grades Reading books For grades 1-8 100-130 for each 9.2 million Used sparsely ciass supplementary books distributed Teaching 2 curricular guides, 3 542,000 pedagogical Often locked up for fear materials textbooks for teachers kits distributed of damage or theft 27 Componentsl Activities Targets to be Outputs Outcomes subcomponents achieved Teacher training Untrained staff 213,000 people It is uncertain what delivered 2-week (40 trained in 116 events teachers leamed, if hour) courses through of mainly lecturing anything; no evaluation cascade training to piloted distance Pilot went to scale teachers and directors leaming ___ Time slice financing Civil works About 25,000 classes 60% of them in About 26,000 built or built or renovated municipal schools renovated, 40% were in municipal schools Equipment and Delivered as Some teaching Bookshelves expected equipment underused Innovation fund Grants to support No criteria were Only Maranhao innovative primary established received funding for education programs at There was little accelerated dasses the state and municipal leves ainterest by states municipal levels National components Education Evaluate performance Sample-based Highly sat; tests and evaluation examination (SAEB) scores produced, follow-up possible at school level System policy Support databank for Not done No effect analysis basic education standards Education Conduct policy- 11 studies conducted Book 'Chamada a management oriented studies and disseminated to Acao", video produced, states studies helped design Fundescola project, but no effect on instruction Escola ativa pilot Initiated in 105 Results reported as schools and 6000 encouraging, but in students classes visited reading achievement low Textbook and Study of altemative Lunch program not Alternative Component helped lunch program mechanisms for carried out mechanisms were produce and distribute reform textbooks supported, and millions of textbooks Evaluate cost- textbooks were effectiveness of produced school lunch program 28 Table 3. FUNDESCOLA I Componentsi Actvities Targets to be Outputs Outcomes subcomponents achieved Raise schools to 50% of students in 41% of students in Effect of school minimum schools equipped schools equipped environment on operational with minimum with minimum children is considered standards standards (10% standards positive before the project) Educating and 25,000 teachers Program piloted and Research shows better certifying teachers certified opened space for educated teachers 27,232 teachers to have higher student be certified leaming outcomes supplying basic 5800 classrooms 12,616 classrooms, Effect of school fumiture and benefiting 900,000 environment on equipment students children is presumed positive financing school 401 schools 1513 schools, Effect unknown as yet, improvement attended by 1.2 presumed positive. investments million students or Evaluation pending 54% of students in targeted areas financing school- 5800 classrooms 10,975 classrooms Reportedly, putting managed schools in charge rehabilitation of improved construction physical facilities. quality Establish a school Design and support 401 schools 1724 schools Evaluation drafted for development school development April 2002 process plans Finance school subprojects Plan and provide school microplanning 100% 50% urban zones No effect additional school developing and testing 100%, digital maps places standard architectural and software plans for additional 10 states remained incomplete school places. Strengthen provide specialized Escola ativa 210 943 classes Evaluation being management teaching and leaming classes planned capacity in improvement schools, programs municipalities, and states; foster community Population was to be Process underway Effect and extent participabon informed about unknown Fundescola Strengthen national Testing program Implemented Both initiaives education informaton (SAEB), school continue to be systems and census operational programs 29 Annex B. Assessment Methodology Through field visits, OED collects qualitative information, which is integrated with quantitative data where available. The limited time to assess projects (approximately one week per project) means that missions have limited time in the field, and that travel to remote areas is often not possible. State staff opinions The mission interviewed staff who had helped implement the projects in various states (10 employees in Ceari, six in Rio Grande de Norte, and eight in Alagoas). The questions posed to state and federal implementing teamns were: What were the benefits of the project in your state? What problems did you face in implementing the project? what help did you get? Which components worked best in bringing about results, which did not? How effective were the various consultants who worked on the project? What training did you receive through the project? Did it teach you what it was supposed to? The project had some qualified audits. What financial irregularities were there in your state, if any? What would be different in education sector of your state if the project had not existed? What can you do to reach the poorer municipalities, for which you are not really responsible? A summary of answers is provided in Annex B, Table 2. School visits. The schools were sampled for visits on an incidental basis. Upon arrival at a state, the mission asked state official to indicate sites of project implementation that would be accessible within 1.5 hour of travel from the capital. Efforts were made to visit rural and low- income schools to the extent possible. Also, the mission stopped by schools along the road to various destinations when time permitted. Two or three staff members from the state Secretariat, who knew the road to various schools and the work that had been carried out in them, accompanied the mission. Thus, the schools that could be visited did not constitute a representative sample of schools in a state or a project; it was impossible to visit the more remote municipalities and isolated rural schools (in grot6es), where instructional delivery might be poorer. Also, the time of visits was not random; they took place in late morning and during the afternoon shift. However, visits in the schools were unannounced, and it was possible to see everyday operation. Some class events were videotaped for discussion with officials. School visits involved interviews with the school principal if s/he was available and any teachers who were free. The visitors entered classrooms, observed the tasks in process, and asked the teacher what she was doing and why. Some students in a class were chosen on an incidental basis and asked to read or answer basic arithmetic questions. A questionnaire was prepared for teachers' meetings, but it was applied only in three cases because it was not practical to meet teachers in groups without interrupting classes. To study the effects of the NEBE projects in the Northeast, the mission spent two days visiting six primary schools around Natal, three days visiting eight primary schools around Fortaleza, two days visiting seven schools around Macei6. For Fundescola I, the mission visited schools around Goiania and Ner6polis. For the Innovations in Basic Education, the schools visited 30 were in and around Sao Paulo. The latter schools were larger, and it was possible to observe many classes in each. The schools visited were: in Ceara, Paul Ayrton, Paulino Rocha (Fortaleza municipal), Ruiz Barbosa, Luiz Gozaga dos Santos, (Maracanau), Valdemar Pereiro di Querois (Eusebio), Natalia Candido de Suarez, Prof. Nevens Amando (Aquiraz), Fernando Augusto and one state school (Horizonte). In Rio Grande de Norte, Escuela Amerinda Bezerres Furtado, (municipal) and Francisca de Castro, (state) in Macaiba; Escola Antonio de Castro (state), Francesca Fernandez (municipal) in Parnamirim; Maria Aurineide (state) and Francesca Fernandes (municipal) in Nova Cruz. In Macei6, Marechal Deodoro primary school, Major Alfredo Tenorio Lins (Marechal Deodoro), Escola Oliveira Silva, Eusebio Serafim (Pilar), unidad escolar Valido Maya de Omena, escola Pedro Nnfiez (Craibas), municipal school of Coite de N6ia (secondary grades). In Goias, rural escola ativa and urban escola estadual Dr. Negreitos (Ner6polis), escola Jose Alvos de Assisi, Cecilia Meierlis (periphery of Goiania), rural escola ativa Vincente Guimaraes (Bella Vista). In Sao Paulo, State schools Enio Voss, Cesar Martinez, EEMMDC in Mooca (various areas inside Sao Paulo); Azevedo Soares, Padre Egydio in Caieiras, municipalized school Tania Fernandez in Francisco Morato, Lydia Kitts Moreira in Guarulhos, and Alberto Feijeira Rezende in Jundiai. At the completion of each state visit, meetings were held with state officials, and suggestions were given for improvement. Suggestions given to the officials included means to deal with the issues presented in the main body of the text. (See Annex C). 31 Annex C. Specific Instructionally Oriented Issues A number of issues emerged during school visits: - Extensive copying from the blackboard. Though some copying is necessary in any class, Brazil has a tradition of extensive copying, partly due to an earlier lack of textbooks.39 The students of primary schools copied, because teachers asked them to, those of middle schools because books had not yet been given out, and those in secondary schools because they had no textbooks. Linked to this issue was the limited use of textbooks. - "Text production" for students who do not yet read well. Their laborious writing consumes valuable time that could have been, used to leam better reading. - Limited or no homework for children considered too poor; reading or writing work that could be homework is assigned in class, thus reducing the time available to learn new material. - Basic skills drowned in art. Though some art is motivating and develops psychomotor skills, too much may waste the precious time of poor children and drive them to the point of no catch-up.40 The minimum amount to motivate students and yet not detract them from learning tasks has not been explored. Because art work and secondary activities are also part of remedial classes, they may reduce students' chances for catching up. Discussions with some education specialists responsible for schools revealed a preoccupation with sophisticated tasks (such as detailed diagnostics for children who obviously lack basic skills) and a lack of understanding for the gross deficiencies and the time constraints of poor children. - Legal teacher absenteeism. In various states, teachers' unions have won the right to be absent from class 1-3 days a month. In addition, they leave the classroom to take training, since many have a second and third job and they really do not have the time to devote to training. Rural schools have no substitute teachers, and classes of absent teachers often go home. Urban schools may only have access to one, while there may be six teachers absent at the same time. This is cited as one important reason why teachers send their own children to private schools. Possible ways to address these issues were discussed with state and federal officials during the mission, as follows: (a) Including time on task in the operational standards applicable to all schools. For example, a goal may be that 75 percent of class time will be spent on learning 39. Copying is still recommended as stimulating and pleasant, though boring sometimes. (Ministerio de Educa,ao, Secretaria de Educacao Fundamental. Programa,de desenvolvimento Profissional Continuado - Alfabetiza,ao. Modulo - Alfabetizar com textos. Brazilia 2000, p. 97.) 40. There has been a big push to combine reading and writing in the early grades, with a renewed emphasis on spelling deriving from work on early decoding. This may be overused. The idea of constructivismo is to allow the children to dictate their own stories, which the teachers then write down and read to the children and then allow them to read. It works well when the homes do all of the teaching of reading because few children who need to be taught to read in school can learn from that method. It also requires extremely knowledgeable and dedicated teachers (Richard Venezky, reading expert, Professor of Education at the University of Delaware; personal communication March 16, 2002). 32 activities. Principals and supervisors may use existing classroom observation instruments to monitor how time is spent. (b) Behaviorally oriented inservice training. All Bank projects of the 1 990s invested heavily in teacher training, and the network functions effectively; throughout the country, teachers regularly attend training sessions on various topics. However, lectures and books are usually not very effective in modifying behaviors; role modeling coupled with specific feedback are more effective. Training videos could demonstrate desirable and undesirable instructional behaviors. These can be easily created with a camcorder by visiting classes, or asking trainers to demonstrate (to make sure teachers with negative behaviors are not offended). During training, teachers may be asked to identify desirable and undesirable practices, understand the rationale, reflect on what they do, and receive specific suggestions on what they should do instead, such as: Assigning reading instead of copying and early text production, a list of activities for idle time while waiting for a bell, group activities for better classroom management, and giving doable homework (c) Supervision and follow-up of how class time is used. Usually teacher classes are not systematically supervised. However, Brazil has a human resource structure in place that could do so easily. Many states and municipalities have supervisors (fewer in the Northeast), who visit schools regularly. Also, most schools have full-time principals. Though many are political appointees and not necessarily qualified, they could learn to monitor teachers' use of time. They have a position of authority and can easily hear the noise which usually signals that students are not engaged in learning tasks. Rather than tolerate the situation as "normal", they can suggest suitable activities to teachers. (d) Emphasizing time on task during school development projects (PDEs). The PDE is the Bank's main vehicle for school-level reform. The main process consists of a lengthy questionnaire, which school staffs contemplate over several weeks. A part of the questionnaire asks about the effectiveness of teaching-learning process, but it is not sufficiently specific. For example, it asks whether teachers start and end classes promptly and whether administrative issues are carried out with a minimum of interruption.4' These questions could be rephrased to focus teachers on their use of class time for effective learning. (e) Sensitizing state and municipal officials. Training for state and municipal administrators is carried out often, notably through the progestao program. Videos and suggestions for specific behavioral change could be presented to them. (f) Sensitizing local media and parents though the various meetings that schools are expected to carry out. Better understand of constructive time usage may create community pressures on schools to increase the time devoted to learning. 41. Fundescola. 1999. Como Elaborar o Piano de Desenvolvimento da Escola, p. 67. Table 2. World Bank Lending in the Education Sector of Brazil Project NanieLoan Fiscal Loan Cnelto PrjcOuom Sutiait Institutional Bank Borrower Ratings Project Name LNo Year Amount Cancellation Project Outcome SutainabiIitY Development Perfornance Performance Through US$M US$M Completed Projects Education I 755 1971 8.4 0 Satisfactory Not Rated Not Rated Not Rated Not Rated PAR Educaton II 1067 1975 23.5 2.85 Unsatisfactory Not Rated Not Rated Not Rated Not Rated PAR Education III 1452 1977 32 1.08 Satisfactory Likely Negligible Not Rated Not Rated PCR Education IV 1867 1980 32 0 Satisfactory Likely Substantial Not Rated Not Rated -PAR Educaton V-Agric & Voc 2366 1984 20 6.45 Satisfactory Likely Modest Not Rated Not Rated PCR Education VI-Center-West and North 2412 1984 40 9.42 Satisfactory Uncertain Modest Not Rated Not Rated PCR Science & Technology Educabon 2489 1985 72 0 Satisfactory Likely Substantial Satisfactory Satisfactory PAR Skills Formation 2810 1987 74.5 59 Unsatisfactory Unlikely Negligible Not Rated Not Rated PCR Science Research &Training 3269 1991 150 10 Satisfactory Uncertain Substantial Satisfactory Satisfactory ES Innovations in Basic Moderately Educaton 3375 1991 245 1.89 Satisfactory Uncertain Modest Satisfactory Satisfactory PPAR Moderately NE Basic Education II 3604 1993 212 11.15 Satisfactory Likely Modest Unsatisfactory Satisfactory PPAR Moderately NE Basic Education III 3663 1994 206.6 6.56 Satisfactory Likely Modest Unsatisfactory Satisfactory PPAR Fundescola I 4311 1998 62.5 Satisfactory Likely Modest Satisfactory Satisfactory PPAR Total 1,118.0 Ongoing Projects Minas Gerais Basic Educaton) 3733 1994 150 Parana Basic Educaton 3766 1994 96 Science & Technology III 4266 1998 155 Fundescola II 4487 1999 202 Bahia Basic Education Project 4592 2001 69.6 Ceara Basic Education 4591 2001 90 Total 762.5 Note: Earlier audits and compleiion reports had limited ratings. 34 Table 2. Local Staff Opinions Regarding Project Aspects Innovations in Basic NEBE NEBE NEBE Fund-escola I Ed. Ceara Rio Grande Alagoas Goias Sao Paulo de Norte 10 staff 6 staff 8 staff 8 staff 3 staff What were the most important benefits of the project in your state? Improved condition of school buildings, furmiture, quality control of work X X X X X Refurbished schools are an oasis for students X Management became more transparent through management studies X Management components modemized administration, helped decentralization X Established some quality control for teachers, push to acquire at least the magisterio level X x X Project offered constructive work and sanity at a time of strikes and limited financial capability X Microplanning and school census were later used for Fundescola X X Qualified computer specialists were employed in state secretarats of education X The information system for states facilitates decisionmaking X The school census system operated well Staff worked with knowledgeable local consultants X The secretariat has a new vision X Computer use became common among staff (or teachers) X X Project trained school directors on a large scale X Project develops of school-based management (fundescola) x Textbooks were a well implemented component, many materials to schools X X X The school development project is introduced to all schools of the state X Acceleraton classes worked well, are effective X Innovations in Basic NEBE NEBE NEBE Fund-escola I Ed. Ceara Rio Grande Alagoas Goias Sao Paulo de Norte 10 staff 6 staff 8 staff 8 staff 3 staff What problems did you face in implementing the project? What were its disadvantages? Hard to equip schools with unsuitable infrastructure; they had to repair classes first X Repair work is unfinished; through Fundescola they cannot repair 'wet areas', only classes x Initially, much complicated bureaucracy was required, a 1997 change in PIU administration improved conditons x x The govemment gave clear building and equipment standards, but no guidance on teacher training (Alagoas) X Fundescola reached again some schools refurbished during the earlier projects (Natal) X -Schools that were not refurbished could not sustain and get equipment X In Fundescola, Bank contribution is symbolic, 1% of expenditures, covers travel expenses, some paint X Lack of counterpart funds, stoppage of work X X X Implementation team changed when government changed X No relation between components, objectives, results Management improvements at state level - no ownership, unsatistactory State management studies were useless Project benefits were scattered in multiple schools X There was a cultural barrier, project was too high-level initially X States did not clearly particlpate in the appraisal, were not consulted sufficlently Partnerships with municipalities did not work out well x X The very poor were left behind; if no land were readily available, a school would not be built X Many municipalities and their assorted distant schools have fallen behind x Earlier workmanship of school repairs was poor, many cost more to repair than rebuild X X Lack of resources - small team had to build too many schools X No guidance on how to improve teacher training X No computers in the more remote schools X X Getting teachers to use the materials kits X Initial difficulties in Fundescola were due to its being a separate rather than an integrated program X 36 Innovations in Basic NEBE NEBE NEBE Fund-escola I Ed. Ceara Rio Grande Alagoas Goias Sao Paulo de Norte 10 staff 6 staff 8 staff 8 staff 3 staff The instrucUonal delivery improvements were the most difficult to implement X Getting the supplemental financing from the state to do the work x Schools had difficulties in managing money, lack of experience X Location of escolas ativas was done based on the political will of municipalities X The inputs of the NEBE projects were scattered among the schools, did not have synergies Which components worked best in bringing about results, which did not? Video equipment and telensino are not effective in improving leaming outcomes X Accelerated dasses do not teach advanced skills, need revisions X Management indices calculated showed the magnitude of some problems X Some non-teaching staff went back to dass, others retired, leaving recruitment open for new ideas X The exodus of non-teaching staff had limited effectiveness; Secretaria overstaffed again X Microplanning showed exactly where new schools were needed X With microplanning, team could argue that schools should not be where politicians wanted them X Schools are refurbished based on clear criteria rather than political desire for public relations X Distance teacher training pilot worked well and went to scale during Fundescola X Teacher training was not effective X Equipment given is kept locked in schools to prevent theft or destruction X School development project greatly changed teachers' minds X X How effective were the various Bank staff or consultants who worked on the project? A Bank staff came in to deal with educational work but was transferred and work remained unfinished X Bank staff came repeatedly, were helpful and interested X X X X Bank staff or consultants practically never went there X Innovations In Basic NEBE NEBE NEBE Fund-escola I Ed. Ceara Rio Grande Alagoas Goias Sao Paulo de Norte 10 staff 6 staff 8 staff 8 staff 3 staff Bank advice was appreciated more than the money X Some consultants had rigid requirements, staff leamed to think on the same terms and confront them x What training did you receive through the project? Did it teach you what it was supposed to? Project management training, procurement, bids handling X X X X X Staff leamed to use computers, earlier some were afraid of them X Planning seminars were useful x x Leamed to deal with complicated Bank requirements, such as terms of reference X X Overall, staff learned a lot they did not know X x A lot of training took place x X What financial irregularities or qualified audits were there in your state, if any? Initially civil works quality was low X Team did not correctly follow procedures on a few occasions X Nepotism was encountered in municipalities, some less qualified people were given jobs X Decentralizaton of money to schools created some accountng irregularities due to ignorance X Giving school construction funds to schools reduced overcharging and deficient constructon X No qualified audits X Significant abuse of school lunches, but this did not affect the project X What would be different In education sector of your state if the project had not existed? The team would start from a lower level, no decentralized management X There would not be as much transparency in govemment statistics X There would be no minimum operational standards or implementation capacity X X 38 Innovations in Basic NEBE NEBE NEBE Fund-escola I Ed. Ceara Rio Grande Alagoas Golas Sao Paulo de Norte 10 staff 6 staff 8 staff 8 staff 3 staff Staff would not have been familiarized with computers X X States would not have the capacity to do microplanning Structured teams of staff might not exist in the states SAEB, school census, other federal programs would be further behind Quality of education would probably be the same X Quality of education is improving due to school development projects X Management would be more centralized without it and less transparent X The quality of materials and educational interventions would be lower x What can you do to spread project effects to the less capable municipalities? The state cannot intervene with municipal education, even if the quality of education is low X Nothing, they are impotent; they see bad performance and cannot fix it X X Some municipalities are better at getting resources for themselves x Made agreement with some municipalities to repair their schools X Doing so gradually as all schools of the state do school development projects X X Did a census of poorer municipalities for later inciusion X What evaluations have you carried out of your programs? No evaluation X X Qualitative evaluation, none based on achievement X There were no specific outcome targets or baseline data No baseline data were collected on schools before various interventions X x X Various evaluative data can be produced from computer records, but not on achievement Evaluation was only done for 'concrete materials", not for school performance NEBE project helped create the SAEB for sample-based achievement data SAEB scores were available for various towns, but no specific performance feedback X 39 Innovations in Basic NEBE NEBE NEBE Fund-escola I Ed. Ceara Rio Grande Alagoas Goias Sao Paulo de Norte 10 staff 6 staff 8 staff 8 staff 3 staff They make no effort to show improvement in SAEB scores X Published a book on teacher testing X Have not made specific evaluations, no evaluation for escola ativa X A large quantitative evaluabon, but it was done before most of the project inputs X Other issues and observations A lot depends on the good will of the individual teacher X Students like the repaired schools, take care of them, vandalism reduced X Though 3-5 years have passed, some schools look as if they were repaired yesterday X Dropout repetition were reduced, but due to policies rather than improved leaming x X Directors do only administrative work, quality of education is not in their responsibilitief X X Fundescola rules permit equipment purchase but no hiring of tutors for weak students X Municipal students are in poorer schools X Esola ativa textbooks need to be closer to national curricular standards, currently weaker in math X Dropout repetition were reduced, but due to policies rather than improved leaming X Lack of counterpart funds due to legislative calendar issues greatiy inhibited the project X Bank projects are beneficial because staff must think outside the routine on goals to fulfill expressed overall favorable impression of the project x X X 41 Annex C. Basic Data INNOVATIONS IN BASIC EDUCATION PROJECT (LOAN 3375-BR) Key Project Data (amounts in US$ million) Appraisal Actual or Actual as % of estimate current estimate appraisal estimate Total project costs 600.0 601.3 100.2 Loan amount 245.0 Cancellation 1.9 Cumulative Estimated and Actual Disbursements Fiscal Year" FY92 FY93 FY94 FY95 FY96 FY97 FY98 FY99c Appraisal estimate (US$M)" 56.8 127.8 180.4 214.1 230.1 245.0 -- - Actual (US$M) 20.0 31.5 38.0 48.4 62.3 182.9 220.8 243.1 Actual as % of appraisal 35.2 24.6 21.1 22.6 27.1 74.6 90.1 99.2 Date of final disbursement: May 14, 1999 a. July-June 30 b. Including US$20.25 million deposited in the Special Accounts c. US$1.9 million of the original Loan balance was cancelled at the request of the borrower Project Dates Original Actual Negotiations 03/30/90 05/01/91 Board approval 06/30/91 Signing 10/04/91 10/04/91 Effectiveness 01/07/92 01/07/92 Closing date 06/30/98 04/30/99 Staff Inputs (staff weeks) Stage of Project Cycle Planned Revised Actual Staff Weeks US$ Staff Weeks US$ Staff Weeks US$ Through Appraisal 0.0 -- - -- 197.5 N/a Appraisal-Board 0.0 - - -- 63.2 N/a Board-Effectiveness 0.0 -- - - 71.2 N/a Supervision .309.8 -- - -- 587.6 N/a Completion 15.0 - - - 19.8 N/a Total 324.8 - - -- 939.3 N/a 42 Mission Data Date No. of Staff days Specializations Performance Rating (month/year) persons in field represented rating trend Through Appraisal 12/1987 4 11 Op, Ec, Ph n/a n/a 03/1988 4 15 Ec, Ph n/a n/a 07/1988 6 16 Ed, Ec, Ph, Op n/a n/a 08/1988 2 N/a n/a n/a Appraisal through 02/1989 2 effectiveness 10/1989 2 1 Op. Ed 02/1990 2 5 Op, A 05/1990 2 7 Ed 02/1991 5 1 Op., A 05/1991 4 Op, Ec, Ed, A 05/1991 Initial Summary 05/1991 Portfolio Status Update 11/1991 4 3 Op. A, Ph, Ec Supervision 12/1991 7 4 Ed, Op, Ds, A, 1 Au, Lg. Pr 04/1992 2 10 Op, A 1 07/1992 2 4 Op, Ev 2 08/1992 3 5 Ed n/a 10/1992 2 4 Op, A 2 02/1993 1 3 Op 2 06/1993 Portfohlio Status Update 2 10/1993 6 8 Pr, Tm, A, Tt, Ph 2 02/1994 6 8 Pr, Tm, Ec, A, Tt 2 05/1994 2 1 Ed,Ec n/a 06/1994 7 12 Pr, Op,/Tm, Ec, S S A, Ph, Tt, Nu 09/1994 Portfolio Status Update U S 05/1995 Portfoho Status Update U S 06/1995 Portfolio Status Update U U 01/1996 Portfolio Status Update U U 03/1996 1 n/a Ed U U 12/1996 Portfoho Status Update U U 06/1997 Portfolio Status Update S S 01/1998 3 Ed/Tm, Pr, Ed S S a. A = Architect, Au = Audits, Ds = Disbursements, Ec = Economist, Ed = Education, Ev = Evaluation, Lg= Legal, Nu = Nutrition, Op = Operations, Ph = Public Health; Pr = Procurement, Tm = Task Manager, Tt = Teacher Training 43 SECOND NORTHEAST BASIC EDUCATION PROJECT (LOAN 3604-BR) Key Project Data (amounts in US$ million) Appraisal Actual or Actual as % of estimate current estimate appraisal estimate Total project costs 369.7 348.5 94.3 Loan amount 212.0 Cofinancing Cancellation 11.2 Project Dates Original Actual Negotiations Board approval 05/13/13/1993 05/13/1993 Signing 08/25/1993 Effectiveness 11/24/1993 02/08/1994 Closing date 06/30/1998 06/30/1999 Staff Inputs (staff weeks) Stage of Project Cycle Actual Latest Estimate Staff Weeks US$ Appraisal/Preparation 248.7 578,700.00 Appraisal/Negotiations 95.3 234,301.80 Supervision 134.2 350,565.32 ICR 5 4,750.00 Total 483.20 1,168,317.12 44 Mission Data Date No. of Specializations represented Performance Rating (month/year) persons rating trend Identification/ Preparation 05/27/85 1 Tm, C n/a n/a 05/19/86 1 Tm n/a n/a 09/30/87 2 Ed, C n/a n/a 01/28188 3 Ed, Fms n/a n/a 02/01/88 3 Ed, Ec/Fms n/a n/a 05/20/88 1 Ed,Ed, Padm, C n/a n/a 07/18/00 3 Tm, Ed, C/Ed n/a n/a 10/31/89 3 Tm/ Ed, C/Ed n/a n/a 04/17/90 5 Tm, Pav, C/Ed, C n/a n/a 12/03/90 3 Tm, C/Ed, C n/a n/a 02/11/90 3 Tm, C/Ed n/a n/a 04/04/91 1 Tm n/a n/a 02/11/92 5 Sdir, Tm, Ec(YP), C/A, C/Ed n/a n/a 07/29/92 2 Tm, Ed/Ec, Ec n/a n/a 08/20/92 4 Tm n/a n/a 10/25/92 4 C/Ed, C n/a n/a 02/111/93 6 Tm, Ed/Ec, C/A, C n/a n/a 09/14/93 1 Ed/Ec n/a n/a Appraisal 09/01/93 1 Tm S HS 11/18/93 1 Tm S HS 12/16/93 1 Tm S HS Supervision 01/19/94 4 Tm, Fms, Fms/Cp, C/Pro S HS 05/03/94 4 Rm, Pec, Pms, C S S 09/08194 1 Tm U U 05/05195 1 Tm U U 06/29/95 1 Tm U U 04/04/96 1 Tm S U 02/14/97 1 Tm S S 06/27/97 1 Tm S S 04/28/98 3 Tm, Proa, Ed S S 03/10/99 1 Tm S S ICR 10/04/99 1 C n/a n/a 02/08/00 1 C n/a n/a a. C = Consultant, C/A = Consultant/Architect, C/Ed = Consultant/Educ. Specialist, C/Pro = Consultant/Procurement, Ec = Economist, Ec/Fms = Economist/Financial Management Specialist, Ed = Educaton Specialist, Ed/Ec = Education Economist, Fmr/Cp = Financial Management Specialist/Computer MIS, Fms = Financial Management Specialist, Padm = Public Administration, Pec = Principal Economist, Proa = Procurement Analyst, Sdir = Sector Director, Tm = Task Manager Note. At Appraisal for NEBE II it was decided to split the project into NEBE II and IlIl; as such nearly all of the Identification/Preparation missions are charged to NEBE II. 45 THIRD NORTHEAST BASIC EDUCATION PROJECT (LOAN 3663-BR) Key Project Data (amounts in US$ million) Appraisal Actual or Actual as % of estimate current estimate appraisal estimate Total project costs 366.90 339.77 92.6 Loan amount 206.6 Cancellation 6.6 Project Dates. Original Actual Negotiations Board approval 11/23/93 11/23/93 Signing 02/10/94 Effectiveness 05/11/94 05/27/94 Closing date 12/31/98 12/31/99 Staff Inputs (staff weeks) Stage of Project Cycle Actual Latest Estimate Stage of Project Cycle ~~Staff Weeks UJS$ Identification/Preparation 48.5 97.80 AppraisaVNegotiations 268.0 52.80 Supervision 160.9 453.47 ICR 5.0 4.75 Total 482.4 608.82 Note: These costs may be under-estimated as the NEBE II Project became two projects at appraisal; as such preparation costs for both projects were charged to NEBE IIl Mission Data Date No. of Specializations represented Performance Rating trend (monthlyear) persons rating Identification/ Preparation 08/28/92 2 C/Ed, Ed/Ec, C n/a n/a 10/25/92 4 C/Ed, C n/a n/a 11/10/92 6 Tm, Ed/Ec, C/A n/a n/a 12/27/92 3 Tm, C n/a n/a Appraisal 09/01/93 1 Tm S S 12/16/93 1 Tm S S Supervision 03/17/94 1 Tm S S 09/08/94 1 Tm U U 05/05/95 1 Tm U U 08/29/95 1 Tm U U 04/04/96 1 Ed/Ec S S 02/17/97 1 Tm S S 06/27/97 1 Tm S S 04/28/98 3 Tm. Proa, Ed S S 01/11/99 3 Tm. Proa, Ed S S 12/20/99 .1 Tm. Proa, Ed S S ICR 10/04/99 1 C/Ed n/a n/a 02/08/00 1 C/Ed n/a n/a C = Consultant, C/Ed = Consultan/Educ. Specialist, Proa = Procurement Analyst. Note: At appraisal for NEBE II it was decided to split the project into NEBE II and l1l; as such nearly all of the Identification/Preparation missions are charged to NEBE II. 46 SCHOOL IMPROVEMENT PROJECT-FUNDESCOLA I (LOAN 4311-BR) Key Project Data (amounts in US$ Inillion) Appraisal Actual or Actual as % of estUmate current estimate appraisal estimate Total project costs 125 138.72 111 Loan amount 62.5 Cancellation 0.0 Project Dates Original Actual Negobabons Board approval Signing 06/07/98 Effectiveness 08/30/98 08/30/98 Closing date 06/30/01 06/30/01 Staff Inputs (staff weeks) Actual Latest Estimate Stage of Project Cycle Staff Weeks US$ Identificabon/Preparabon n/a n/a Appraisal/Negotiations n/a 197,145 Supervision 35.7 204,213 Total 38.2 412,003 47 Mission Data Date No. of Specializations represented Performnance Rating (month/year) persons rating trend Identification/ Preparation 04/1997 1 Sr. Education Economist 2 Sr. Education Specialist 1 Education and Health Coordinator 1 School Improvement Specialist 09/1997 1 Sr. Education Economist 1 Sr. Education Specialist I Education and Health Coordinator 1 !School Improvement Specialist Appraisal/NegoUation 11/1997 1 ;Sr. Education Economist 1 Sr. Education Specialist 1 'Education and Health Coordinator 12/1997 1 Sr. Education Economist 2 Sr. Education Specialist 1 Educaton/Health Specialist 1 School Improvement Specialist 1 Operation Specialist Supervision 05/1999 1 Sr. Education Economist S S 1 Operations Analyst 2 rinancial Magmt. Analyst 12/1999 1 Sr. Education Economist S S 06/2000 1 Operations Specialist S S 1 Education Specialist S S 10/2000 1 Sr. Education Economist S S Education Specialist Sr. Education Specialist IMAGING Report No.: 24433 Type: PPAR