measuring the quality of november 2020 MoRA’s education SERVICES this report was prepared by: NOAH YARROW - RYTHIA AFKAR - EEMA MASOOD - Bernard GauthieR This work is a product of the staff of The World Bank, supported by funding from the Australian Government. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions © 2020 The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for non-commercial purposes as long as full attribution to this work is given. All queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Attribution Please cite the work as follows: Yarrow, Noah; Afkar, Rythia; Masood, Eema; Gauthier, Bernard. 2020. Measuring the Quality of MoRA’s Education Services, World Bank, Jakarta. © World Bank. Contact Information The authors can be contacted at nyarrow@worldbank.org, rafkar@worldbank.org Photo Credits Akhmad Dody/World Bank 114 Service Delivery Indicator Survey Indonesia 2020 ABSTRACT We visited 350 primary schools and collected data from 1,838 teachers and 3,368 Grade 4 students to assess the quality of education service delivery in a nationally representative sample of schools of the Ministry of Religious Affairs and a smaller sample of schools of the Ministry of Education and Culture. We found that students were on average 1.5 years behind the learning level expected for 4th grade, representing a learning crisis. These low levels of learning were associated with high levels of teacher absence, low levels of availability of textbooks, and low levels of teacher subject knowledge and pedagogy skills but high levels of student satisfaction. We find higher levels of learning for students who have ever attended early childhood education, those who have eaten breakfast on the day of the assessment, as well as for female students. We offer some recommendations for addressing these challenges, drawing from schools with higher levels of student learning in the sample. Service Delivery Indicator Survey Indonesia 2020 1 CHAPTER 1. 2. CHAPTER measuring the quality of moRA’S education services 2 Service Delivery Indicator Survey Indonesia 2020 Service Delivery Indicator Survey Indonesia 2020 contents 3. CHAPTER CHAPTER 4. 1 Abstract 4 Abbreviations and Acronyms 5 List of Tables 6 List of Figures 7 List of Boxes 5. 8 Foreword 10 Foreword from MoRA Minister CHAPTER 12 Acknowledgement 13 Executive Summary 22 1. Introduction 26 2. Context 32 3. Methodology and Implementation 40 4. Results A. Assesment of Student Learning B. Teacher Effort C. Teacher Competence 6. D. School Inputs 74 5. Correlates of Students Learning Outcomes CHAPTER 88 6. Comparing Indonesia with Other SDI Countries 92 7. Conclusions and Recommendations 98 Appendix A: Sampling Strategy 100 Appendix B: Definition of Indicators 101 Appendix C: Detailed Teacher Assesment 103 Appendix D: Regression Results 109 References 7. CHAPTER Service Delivery Indicator Survey Indonesia 2020 3 Abbreviations &Acronyms BOS School Operational Assistance (Bantuan Operasional Sekolah) BPS Central Bureau of Statistics (Badan Pusat Statistik) DFAT Australia’s Department of Foreign Affairs and Trade DWMS Development World Management Survey ECED Early Childhood Education and Development EGRA Early Grade Reading Assessment EMIS Education management information system GDP Gross Domestic Product GoI Government of Indonesia HCI Human Capital Index IQ Intelligence Quotient MoEC Ministry of Education and Culture MoF Ministry of Finance MoRA Ministry of Religious Affairs NER Net Enrollment Rate NVR Non-verbal Reasoning OECD Organisation for Economic Co-operation and Development PIP Indonesia Education Grant Programme (Program Indonesia Pintar) PIRLS Progress in International Reading Literacy Study PISA Programme for International Student Assessment PNS Civil servant (Pegawai Negeri Sipil) PPG Teacher Professionalism Training (Pendidikan Profesi Guru) SABER Systems Approach for Better Education Results SDI Service Delivery Indicators TVET Technical and Vocational Education and Training UN National Exam (Ujian Nasional) US$ United State Dollar USAID United States Agency for International Development WASH Water, Sanitation, and Hygiene WDR World Development Report WHO World Health Organization 4 Service Delivery Indicator Survey Indonesia 2020 LISTOFTABLES TABLE ES.1. Service Delivery Indicators at a Glance 19 TABLE ES.2. Comparison between Indonesia and other SDI Countries 21 TABLE 3.1. Education SDI Survey Instruments 34 TABLE 3.2. Survey Sample 38 TABLE 4.1. Student Assessment: Detailed Results 47 TABLE 4.2. Teacher Effort 49 TABLE 4.3. Distribution of Absence Rates 51 TABLE 4.4. Absence Rates by Gender, Functions and Origin 53 TABLE 4.5. Teacher Assessment 61 Teacher Assessment Results by Gender, TABLE 4.6. 66 Employment Status and Level of Education (Percentage) TABLE 4.7. Minimum Equipment Availability 71 TABLE 4.8. Minimum Infrastructure Availability 72 TABLE 5.1. Student Gender, Pre-Primary, Breakfast and Scholarship, by School Type (Percentage) 76 TABLE 5.2. Results of Student Assessment, by Gender, Pre-Primary, Breakfast and Scholarship 76 TABLE 5.3. Student Results According to the Gender, by Type of School and Subject 79 TABLE 6.1. Comparison between Indonesia and other SDI countries 90 TABLE A1. Sampling Strata 98 TABLE A2. Number of Schools by Type of School 98 TABLE C1. Teacher Mathematics Assessment 101 TABLE C2. Teacher Indonesian Language Assessment 101 TABLE C3. Teacher Arabic Assessment 102 TABLE C4. Teacher Pedagogical Assessment 102 TABLE D1. Correlates of Teacher Efforts: Regression Results 103 TABLE D2. Correlates of Teacher Evaluations: Regression Results 104 TABLE D3. Correlations Between The Service Delivery Indicators And Student Test Scores 105 TABLE D4. Correlates of Student Performance: Regression Results 106 TABLE D5. Correlates of Student Performance 107 Service Delivery Indicator Survey Indonesia 2020 5 LISTOFfigures FIGURE ES.1. Service Delivery Indicators in The Education Results Chain 18 FIGURE 2.1. Hci For Middle-Income Countries In East Asia Pacific, 2020 28 FIGURE 2.2. Proportion of Students In MoEC And MoRA Public and Private Schools 30 FIGURE 2.3. Most Provincial National Exam Scores Miss The National Passing Grade 31 BOX Relationships of Accountability Between Citizens, Service Providers, 35 FIGURE 3.1.1. And Policy-Makers FIGURE 4.1. Curriculum - Adjusted Years Of Education 42 FIGURE 4.1..A Distribution of Students By Curriculum-Adjusted Years Of Education In Literacy 44 FIGURE 4.1.B. Distribution of students by Curriculum-Adjusted Years of Education in Math 45 FIGURE 4.2. Student Results by Sector, Status And Location 46 FIGURE 4.3. Absence Rates, by Sectors, Status And Location 50 FIGURE 4.4. Location and Activity of The Teachers During Unannounced Visits 52 FIGURE 4.5. Reasons for Teacher Absence 54 FIGURE 4.6. Reasons for Principals’ Absence 54 FIGURE 4.7. Teachers’ Absence Rates According To The Presence of The Principal 55 FIGURE 4.8. From Official to Actual Teaching Time 57 FIGURE 4.9. Correlates of Teacher Absence From School And Classroom 58 FIGURE 4.10. Average Teacher Scores in Math, Indonesian Language, Arabic And Pedagogy 61 FIGURE 4.11. Sensitivity of The Minimal Knowledge Indicator To Passing Grade 65 FIGURE 4.12. Correlates of Teachers Evaluations in Languages and Mathematics 67 BOX Student Perceptions of Aligning Instruction with Learning Ability of MoRA Teachers 68 FIGURE 4.5.1. BOX Student Perceptions of The Ability of MoRA Teachers to Check in with Students 69 FIGURE 4.5.2. 6 Service Delivery Indicator Survey Indonesia 2020 FIGURE 5.1. Student Results by Attendance of A Preschool 77 BOX Parents’ Reasons For Sending Their Children to MoRA Schools 78 FIGURE 5.1.1. (Madrasah and Other Religious Schools) FIGURE 5.2. Student Results According to The Gender, by Status and Subjects 79 FIGURE 5.3. Student Results in Mathematics According To Eating Breakfast, by Type of School 79 Student Results In Mathematics According to Receiving A Pip Scholarship, FIGURE 5.4. 81 by Type of School Characteristics of The 5 Percent Highest Performing Schools FIGURE 5.5. 82 Compared With The Others FIGURE 5.6. Correlates of student Performance: Indonesian language 85 FIGURE 5.7. Correlates of Student Performance: Mathematics 86 FIGURE D1. Correlates of Student Performance: Bahasa Household Sample 103 FIGURE D2. Correlates of Student Performance: Mathematics Household Sample 104 LISTOFBOXES BOX 3.1. Analytical Underpinnings of The Survey 35 BOX 3.2. MoRA Non - Islamic Religious School 39 BOX 4.3. Assessment of Teachers’ Knowledge 60 BOX 4.4. Good Teaching Practices 63 BOX 4.5. Student Perceptions of Teachers 68 BOX 5.1. Parental Choices: Which Schools Do Parents Send Their Children to and Why? 78 BOX 5.2. Development World Management Survey 83 Service Delivery Indicator Survey Indonesia 2020 7 foreword Toby Linden, Education Practice Manager, East Asia and Pacific, World Bank “ In other country contexts, the SDI survey has served as an evidence base for governments to initiate or deepen education reform programs to improve. “ This study grew out of a series of meetings in late such as student literacy levels and mathematics 2017 and early 2018 between the leadership of the competencies. This has also been the motivation in Ministry of Religious Affairs (MoRA), the World Bank Indonesia. and students, teachers, and parents. There was a thirst for systematic, reliable information about teaching The SDI survey adds to work that the World Bank and learning in MoRA schools in order to better direct has been privileged to undertake as MoRA’s partner resources and support, as well as broader priorities in trying to understand and address constraints to for reform. The Ministry of Education and Culture improving student learning over the past three years. (MoEC) joined the effort, and the basic Service This work includes other studies on the education Delivery Indicators (SDI) survey was expanded to information management system, on gender and encompass issues of school management, which social inclusion, as well as on the tertiary education were of particular interest to both MoRA and MoEC system. This work has led to a deep partnership policy-makers, as well as student perceptions, giving supporting reform, where initial results helped MoRA rise to an Indonesian version of the SDI survey. to formulate the focus of the Realizing Education’s Promise project, which aims to improve the In other country contexts, the SDI survey has served management of primary and secondary education as an evidence base for governments to initiate or services under MoRA (MoRA project link). deepen education reform programs to improve both the input side, such as teacher attendance and availability of textbooks, as well as outputs 8 Service Delivery Indicator Survey Indonesia 2020 Data collectors went to schools from West Sumatra This study is the most recent in a tradition of research to Maluku, and from large towns to remote villages. supported by the World Bank and partners to help In all, we collected data from 263 MoRA schools, in a measure progress and identify challenges in the nationally representative sample of the MoRA Islamic Indonesian education system. The data collection school system, so that a rich picture of these schools instruments, as well as the data, will be made could be built up. In addition, 10 MoRA non-Islamic publicly available while protecting the anonymity religious schools and 87 MoEC schools were visited of the respondents, making it possible to use the for comparison, though the small sample size of SDI methodology again in several years’ time to these schools means the results are not statistically benchmark system improvements and change toward representative for all MoEC and MoRA non-Islamic the long-term goal of improving human capital. schools. The data collected is also important and relevant to the reopening and recovery process triggered by the COVID-19 pandemic. The data collected on water, sanitation and health facilities and student/teacher ratios, as well as internet access, teacher attitudes, school financing and management capacity, are all directly relevant to recovery planning and building future resilience (see Estimates of COVID-19 Impacts on Learning and Earning in Indonesia: How to Turn the Tide in English and Indonesian). Service Delivery Indicator Survey Indonesia 2020 9 foreword by the Minister of Religious Affairs H.E. Fachrul Razi, “ Minister of Religious Affairs, Republic of Indonesia This report examines service delivery “ in education, looking at what education providers do, know, and have access to, and how it relates to student learning outcomes. It is with great pleasure I welcome the publication of the program (Program Wajib Belajar 12 Tahun). With over Measuring the Quality of MoRA’s Education Services 64 percent of students in MoRA schools enrolled in report. This report is a result of close collaboration the Indonesia Pintar Program (PIP), MoRA educates a between the Ministry of Religious Affairs (MoRA), significant number of students from underprivileged the Ministry of Education and Culture (MoEC) and backgrounds. MoRA has also committed to improving the World Bank, and draws on consultations with its systems for performance-based planning and teachers, parents and students. It comes out of a series budgeting, teacher training and student assessment. of meetings which identified a need for systematic and reliable information about teaching and learning This report examines service delivery in education, in Indonesian schools. This report looks at the quality looking at what education providers do, know, and of education service delivery across Indonesia with a have access to, and how it relates to student learning focus on schools under the jurisdiction of the Ministry outcomes. The insights from the report can help us of Religious Affairs. to better direct resources and spend them more effectively and efficiently. Many of the report’s The Ministry of Religious Affairs is responsible for recommendations are aligned with MoRA’s key goals. over 50 thousand public and private madrasah For example, the report identifies the need to improve schools serving over 8 million students. Over the teacher subject knowledge and classroom pedagogy. past five years we have seen significant increases This is in line with our current work on teacher in MoRA enrollment, which has contributed to training and will help to achieve our goal to improve the success of the national 12 Years of Education educational productivity and competitiveness. 10 Service Delivery Indicator Survey Indonesia 2020 Other recommendations include improving system guide for teachers to continue to conduct their classes resilience and accountability for performance, which effectively online during emergency circumstances are part of MoRA’s current reform program. such as this pandemic. Resilience is especially important today as we As we continue to find solutions to COVID-19 experience school closures and other impacts of challenges and address existing ones, we are working the COVID-19 pandemic on the education sector. closely with development partners, students, parents, Since March 2020, MoRA has put in place several and teachers. We need to continue to work together measures to support learning from home through to increase the capacity, equity, and accountability our e-Learning Madrasah platform. The platform for learning, to help all Indonesian children live up to allows teachers to conduct online classes through their potential and strengthen our human resources. integrated video conferencing, upload learning I would like to express my gratitude to the World materials and assignments, administer online quizzes Bank team that contributed to this report, and to the and tests, and students to submit their work and Government of Australia for their generosity that has access textbooks online. We have also partnered with supported the production of this report. telecommunications operators to provide subsidies for teachers and students to access internet on their phones for distance learning and provided training and other support for teachers using online learning platforms. This includes an emergency curriculum Service Delivery Indicator Survey Indonesia 2020 11 acknowledgement This report was prepared by Noah Yarrow, Rythia Afkar, Eema Masood and Bernard Gauthier. We recognize and thank the Ministry of Religious Affairs and the Ministry of Education and Culture for their support in undertaking this work. Data collection was managed by SurveyMETER along with Dedi Junaedi and Eema Masood. We would like to thank Toby Linden for his guidance, Citra Kumala and Alexander Michael Tjahjadi for research assistance, Yahya Abou Ly for analytical assistance and Andres Yi Chang and Deepali Gupta for their comments and support. The report has benefited from useful comments by peer reviewers: Dina Abu-Ghaida (Lead Economist), Dewi Susanti (Senior Social Development Specialist), and Pedro Cedran Infantes (Senior Economist). The report was prepared under the overall guidance of Satu Kristiina Kahkonen (Country Director, Indonesia and Timor Leste). Peter Milne edited the report text. A team at Box Office designed and laid out the report, guided by Maureen Rustandi. Lestari Boediono, GB Surya, Jerry Kurniawan, and Maulyati Nuraini provided support on the website and launch event. Funding from the Australian Government, Department of Foreign Affairs and Trade supported the report’s research and development. 12 Service Delivery Indicator Survey Indonesia 2020 executive summary Service Delivery Indicator Survey Indonesia 2020 13 The Service Delivery Indicators (SDI) surveys This analytical report presents the results of the generate data on the quality of education services Education SDI survey in Indonesia. The Indonesian and provide a set of measures to assess the SDI survey was conducted from February to March performance of service delivery in a country. These 2019. The objective of the survey was to support data help identify the source of constraints for the system improvements at the Government of provision of quality primary education services and Indonesia’s (GoI) request for both Ministry of Religious track progress over time and between countries. Affairs (MoRA) and Ministry of Education and Culture Ultimately, the SDI aim to enlighten the various (MoEC) schools, with a focus on MoRA’s system. stakeholders in their decision-making, help them design policies and interventions based on evidence, This survey expands on the World Bank’s SDI and improve the provision of education services and survey by including additional modules such as student learning outcomes. the Development World Management Survey and a Parent Motivation Survey. Information was collected Since their inception in 2011, SDI have been used from 350 public and private primary schools in in more than dozen countries in Africa, as well as Indonesia, as well as from a sample of 3,169 teachers, in Asia. These SDI surveys have identified a range to measure absence rates, 1,838 teachers to assess of constraints in the provision of education services their knowledge, 3,368 Grade 4 students to assess and provider behaviors. The survey conducted in their learning results, and 2,107 households to assess Indonesia in 2019 was implemented by SurveyMETER parental school choice decisions. in close coordination with a team from the World Bank. 14 Service Delivery Indicator Survey Indonesia 2020 Information was collected from public and private primary 350 schools in Indonesia a sample of 3,169 teachers to measure absence rates 1,838 teachers to assess their learning results The focus of the survey was on religious schools Student learning outcomes managed by MoRA, which tend to be Islamic schools In the Indonesian SDI survey, Grade 4 students commonly called madrasahs, but include other performed poorly in mathematics. On average, religions as well, together with schools under MoEC in Grade 4 students in our survey managed to achieve both urban and rural areas. Results presented in this only 42.3 percent of the total possible score in report are representative of MoRA Islamic schools at mathematics. For instance, students close to the mean the national level but are not representative of MoEC of 42.3 percent in mathematics answered correctly and MoRA non-Islamic schools. While the sample size most single-digit addition and subtraction questions was too small in these two categories to be nationally but sometimes got wrong double-digit subtraction, representative, they are included here for indicative as well as single-digit and double-digit multiplication comparison purposes. Future reports will explore questions. In languages, the average scores are 86.6 additional topics based on the data collected. percent in Indonesian language and 59 percent in Arabic. In general, students close to the mean for The results present a portrait of the quality of Indonesian language (i.e., answered correctly more services and the physical environment in which than 90 percent of questions) identifying basic words, primary education services are provided. The survey and reading sentences and paragraphs. Likewise, provides information on: (i) the effort of teachers; for Arabic, most students answered correctly when (ii) the knowledge and skills of teachers; and (iiii) asked to identify letters and words. the availability of basic inputs, such as textbooks, teaching materials and infrastructure (such as sanitation and lighting). It also measures the learning level of Grade 4 students in language (Indonesian language and Arabic), mathematics and non-verbal reasoning, enabling analysis of student performance with school inputs and teacher characteristics. Service Delivery Indicator Survey Indonesia 2020 15 There are high levels of heterogeneity between in Indonesian language than younger students, while groups of students, depending on school location 11-year-olds and older students, who are likely to have and status. These average scores hide significant repeated a grade, obtained the lowest scores, in both levels of heterogeneity between groups of students, language and mathematics, even after controlling for depending on the location of the school (urban, rural) student, teacher and school characteristics. and its status (public, private). Rural schools under both ministries have significantly lower average Students’ socioeconomic characteristics are results in all subjects. The differences are particularly strongly associated with their test scores. In line marked in mathematics, whereas a Grade 4 student with research from other countries and earlier attending a rural school obtains on average a score research in Indonesia, a student’s socioeconomic 5 percentage points lower than a student in an urban characteristics are strongly associated with their test school (40.6 versus 45.1 percent, respectively). The scores. In particular, students receiving the need- gaps are also marked in Indonesian language, with based scholarship (PIP) have significantly lower students in rural schools obtaining 4 percentage scores, with a 6-percentage-point gap in mathematics points less on average than students in urban schools. compared with those without the scholarship, after In Arabic, the gap is less pronounced with about a controlling for various student, teacher and school 2-percentage-point difference. characteristics. Students who reported not having eaten breakfast before coming to school on the day Various student, teacher and school characteristics of the observation also had lower test results. are associated with student performance on the SDI student evaluations. Attending any early childhood Teacher competence is associated with higher education, in particular, shows a strong, positive student outcomes in Indonesian language. Teacher association with student learning in Grade 4 in both competence in the subject they teach, as measured by language and mathematics across all school types. their own subject knowledge evaluation, is associated These trends hold even after controlling for student, with small higher but statistically significant student teacher and school characteristics. Children who learning results in Indonesian language for all students, have attended early childhood education achieve and especially for boys in MoRA Islamic schools. significantly higher test scores in both Indonesian language and mathematics. In particular, boys Higher qualified teachers show higher student appear to benefit from pre-primary education, with a outcomes in mathematics. 7.7-percentage-point higher test score in Indonesian language and a 6-percentage-point higher score Teachers with higher education level, in mathematics compared with those not having especially D4/S1 level compared with attended any pre-primary education. Among female only secondary level and below, show students, those who attended pre-primary are associated with a 6-percentage-point higher score positive association with student in both subjects after controlling for student, teacher achievement in mathematics in both and school characteristics. MoRA Islamic schools and MoEC schools. Gender differences are striking, with females generally outperforming males. In all subjects except However, teachers’ years of experience in education non-verbal reasoning, female students outperform are inversely related with student learning results in male students, especially in mathematics, with at least language and especially mathematics. This inverse a 5-percentage-point gap across all school types. association between teacher experience and student The age of a student is also associated with learning learning in mathematics holds for students of both results. Among the Grade 4 students evaluated, genders, but is larger for male students. 10-year-old students performed significantly better 16 Service Delivery Indicator Survey Indonesia 2020 What teachers do In schools where the principal is Teacher absence for any reason is high, with almost absent for any reason, teachers are one in four teachers not present in class. The absence rate of teachers is in line with data from more than twice as more likely to also other studies: about one in five teachers on average be absent, with teachers’ absence (18.7 percent) was not present at school during the rate rising from 18.7 to 45 percent unannounced visit for reasons other than scheduling when the principal is not there. or non-teaching days (Table ES.1). About one-third of schools in Indonesia have an absence rate from school “Training” is given as a reason for teacher absence of above 20 percent, and private and rural schools only 6 percent of the time in our sample, and only 3 present the highest absence rates of teachers. Even percent of the time when the principal is also absent. if they are in school, teachers in the survey were not It therefore appears that this increased-teacher- necessarily providing classroom instruction. absence-linked-to-principal-absence is not being driven by official events where both teachers and the absence rate of teachers from principals are required to be present, such as off-site class is on average 23.5 percent, or training. just under one in four teachers is As a result, actual daily teaching time is low. High absent from their classroom. levels of absence and short time spent teaching even when in class mean that the actual daily teaching time In some schools, absence from class is very high, with in schools for Grade 4 students across all types of 20 percent of MoRA Islamic schools and 25.5 percent schools in this survey is on average 2 hours and 56 of MoEC schools facing a class absence rate of 40 minutes, which is 1 hour and 39 minutes less time, or percent or more. 36 percent less, than officially scheduled. It is important to note that the majority of these teacher absences appear to be excused and officially What teachers know allowed. The purpose of the SDI survey is to measure The level of knowledge of teachers is low. Substantial service delivery, and from the perspective of student gaps are observed in the mastery of languages and learning, whether the teacher absence is excused or mathematics evaluated through an assessment with not does not matter. What matters is that their regular competencies based on the Grade 3 and Grade 4 teacher is not there. However, from an accountability curricula of several countries (Johnson et al. 2012). standpoint, most of the absences were excused, Less than 8 percent of all teachers surveyed obtained which suggests that the system allows a very large a score of at least 80 percent in the assessment in number of teacher absences. language (Indonesian language)— the SDI standard threshold estimated to correspond to the level Absence rates of principals are connected to higher of minimum knowledge required for teachers to teacher absence rates. High absence rates appear to perform their job. Mathematics scores are marginally be associated with a lack of accountability within the better, with about one in three teachers (32 percent) school, where the principal’s absence seems to be a scoring 80 percent or more. Teachers in public and large factor associated with teachers’ absence. urban schools perform significantly better than their colleagues in the private sector or in rural schools. Service Delivery Indicator Survey Indonesia 2020 17 Significant differences in teachers’ performance schools (21.5 students vs 15.5 students per teacher, in the evaluation is observed in terms of levels of respectively). These are much lower numbers of education, years of experience and gender. Younger students per teacher compared to the SDI average and more educated teachers present significantly of 40 students per teacher. Even data from other higher performance in the knowledge evaluations. middle-income countries have an average of about Scores continuously decrease with teacher experience 24 students per teacher.1 in all subjects (except Arabic), with teachers with less than 10 years of experience obtaining the highest Important constraints are observed with regard to scores, while those with 30 years of experience and access to textbooks. more achieving the lowest scores in all subjects, including pedagogy. With regard to teachers’ In MoRA Islamic schools, only one level of education, performance in the evaluations in two students had a textbook for monotonically increases with education level in the subject observed (mathematics, all subjects, with teachers with a master’s degree presenting the highest scores, and those with only a Indonesian language, Arabic). The secondary degree or less the lowest scores. Female proportion of students with textbooks teachers obtain significantly better scores than their is also low in MoEC schools, where male colleagues in all subjects evaluated. only about two in three students had a textbook. Figure ES.1. Service Delivery Indicators in the education results chain (All Surveyed Schools) INPUT learning AVAILABIlity • Principals outcomes • School Absence 15% • Teachers • School Absence 19% • Classroom Absence 24% • Infrastructure 56% • Test Score Student Knowledge • Equipment 71% • Mathematics 67% • Mathematics 42% • Students with Textbooks • Indonesian 47% • Indonesian Language 61% • Student-Teacher • Minimum Language 87% Ratio 17 to 1 Knowledge 16% • Arabic 59% What teachers have (inputs) Language textbooks are the rarest, with less than Overall, the observed student-to-teacher ratio one-third of surveyed students having a textbook. in Grade 4 classes is low at 17, with significant Scarcity is more pronounced in schools in rural areas, differences among the strata. Public schools for both where fewer than one in five students has a language MoEC and MoRA have about one-third more students textbook on average for both MoEC and MoRA per teacher than private schools (22.8 students vs schools. 16.6 students per teacher, respectively), while schools under both ministries in urban areas have about 39 percent more students per teacher than rural 1 Student to-teacher ratio, primary schools. https://data.worldbank.org/indicator/SE.PRM.ENRL.TC.ZS?name_desc=false accessed Nov 2, 2020 at 1:39 PM 18 Service Delivery Indicator Survey Indonesia 2020 Access to other basic learning materials for students Over 40% of schools of all types lack minimum is a constraint in primary schools across the country. infrastructure, defined as the availability of working Overall, close to one-third of Grade 4 classes included toilets for students and sufficient light in the in the survey did not have the minimum instructional classroom. materials. Availability of exercise books is the main driver of this. About one in five students did not have In rural areas, about 54 percent of an exercise book on the day of the observation. The the schools do not have minimal shortage is more pronounced in rural schools, where infrastructure, especially access to about one in four students did not have an exercise book compared with one in ten students in urban functional sanitary facilities, i.e., clean areas. toilets that ensure privacy. The problem of non-accessibility to functional toilets for students affects more than one third of schools overall, while in rural areas it affects close to half of table ES.1. schools under both ministries. Service Delivery Indicators at a glance MoRA MoRA MoEC Non - Public Private Urban Rural Islamic Islamic WHAT PROVI D ERS KNOW Minimum knowledge (% of teachers achieving 80% min in 15.6 13.9 43.3 18.6 15.2 17.8 14.4 language and math) Test Score (out of 100 out of language, math 39.6 41.6 50.7 41.1 39.4 41.7 38.5 and pedagogy) What providers DO School absence rate 18.7 19.3 11.1 12.4 19.2 16 19.8 (% of teachers) Classroom absence rate 23.5 25.7 15.1 20.3 23.7 20.1 24.9 (% of teachers) Time spent teaching per day 2h56m 2h38m 3h47m 3h22m 2h55m 3h11m 2h50m W h a t p r o v i d e r s h a v e t o w o r k w i t h (availability of inputs) Observed student-to-teacher ratio 17 21 9 23 17 22 15 Share of students with textbooks 47 71 92 58 47 60 42 (% of students) Minimum equipment availability (% of schools) (90% with pencils 79 80 70 74 70 81 67 and notebooks) Minimum infrastructure availability 55.9 63.9 90 70.8 54.7 68.9 50.6 (% of schools) Student Learning Language and mathematics test 64.4 68 85.8 66.3 64.3 67.1 62.9 score (out of 100) Language test score 86.6 88.6 96.3 88.5 86.5 89.2 85.1 (out of 100) Mathematics test score 42.2 47.4 75.3 44.1 42.1 45.1 40.6 (out of 100) Service Delivery Indicator Survey Indonesia 2020 19 Student Perceptions slightly above the SDI average of 14.6 percent. The Students overall had very positive impressions of average score obtained by teachers in all assessments their teachers. including pedagogy (39.6 percent) is below the SDI average by about 3 percentage points, and far Over 80 percent of the students said below the scores obtained on directly comparable that their teacher encouraged them assessments in Kenya and Tanzania, for example. This is important because students and teachers to do their best; and when they had in wealthier countries tend to perform better than a need for help in the lesson, or had students in lower-income countries, while Kenya’s questions, felt that their teachers GDP per capita was US$1,817 in 2019 and Tanzania’s were supportive. Most students felt US$1,122, compared with Indonesia’s at US$4,136.3 that everyone in the class knew what This adds evidence for Indonesia under-performing in they were supposed to be doing and education relative to its income level. learning in the lesson. The availability of inputs in Indonesian elementary schools is above the mean of SDI countries. Despite Classroom productivity was also perceived to be in shortcomings with respect to the share of students the mid-to-high range by most students. A strong with textbooks and exercise books, input availability majority of students felt that their teachers mostly in Indonesian elementary schools is above the mean of or always explained things in an orderly way and SDI countries, and relatively high compared with the tried different techniques to explain things that were average, especially regarding minimum equipment unclear to them. availability and minimum infrastructure. Furthermore, the observed student-to-teacher ratio is the lowest compared with other SDI countries (17 students per How Indonesia compares with other teacher vs 40.4 students per teacher, on average), countries indicating inefficient teacher distribution and/or a Indonesia performs well, or close to the average, large proportion of small schools often in rural areas compared with the other countries that participated without the use of multi-grade teaching. in SDI surveys, many of which are in Africa and have lower income levels.2 The teacher absence rate from Indonesian Grade 4 students have high learning school is similar to the average level observed in other outcomes in a combined language and mathematics SDI countries. However, once at school, teachers for score compared with other SDI countries. Ultimately, the most part go to class and the classroom absence the learning outcomes of Indonesian Grade 4 students rate is somewhat lower than the average in other in a combined language and mathematics score is SDI countries. Time spent teaching per day is also high compared with other SDI countries, with a score just below the SDI average, at 2 hours 56 minutes, 15 percentage points above the SDI average, with only compared with 3 hours 2 minutes. Given student Kenya showing a higher combined score. However, learning levels measured by this and other surveys, this above average performance is driven by the students would benefit from more instructional time. high scores obtained by students in the Indonesian language assessment. In mathematics, Indonesian Teachers of Indonesian language and math are students score 5 percentage points below the slightly above the SDI average, while in their overall SDI average (42.3 vs 47.3 percent, respectively). assessment they are slight below the SDI average. This underperformance in math relative to reading With only 15.6 percent of teachers in Indonesian is a possible indicator of low levels of school-based language and mathematics having obtained a learning, since households may be more likely to combined score of 80 percent or more (considered as support reading skills development, resulting in the minimum knowledge threshold), Indonesia is still higher scores. 2 System health at the frontline: Using SDI Data to analyze education quality in Sub-Saharan Africa. World Bank. 2016. http://pubdocs. worldbank.org/en/812821457978473769/Session-8-Deon-Filmer.pdf 3 https://data.worldbank.org/indicator/NY.GDP.PCAP.CD 20 Service Delivery Indicator Survey Indonesia 2020 table ES.2. Comparison between Indonesia and other SDI countries INDONESIA Mauritania Morocco Madagascar Kenya Mozambique Nigeria** Tanzania Togo Uganda Afganistan Punjab Mean SDI 2019 2017 2016 2016 2012 *2014 2013 2014 2013 2013 2017 2019 Teacher Knowledge and Ability Minimum knowledge (% of teachers achieving 15.6 14.6 0.3 3.1 0.1 40.4 0.3 3.7 21.5 1.6 19.5 ... 56+ 80% min in language and math) Test Score (out of 100 out of language, 39.6 43 30.9 44.4 33.2 57.1 26.9 32.9 48.3 35.6 45.3 ... ... math and pedagogy) Teacher Effort School absence rate 18.7 18.6 13.6 4.4 30.6 14.1 44.8 13.7 14.4 20.5 26 10 14 (% of teachers) Classroom absence rate 23.5 39.8 27.8 5.5 37.8 42.1 56.2 19.1 46.7 35.8 52.8 15 18 (% of teachers) Scheduled teaching time 4h35m 5h34m 4h36m 4h37m 5h12m 5h37m 4h17m 4h53m 5h54m 5h29m 7h18m 3h25m ... per day Time spent teaching per day 2h56m 3h02m 3h04m 4h09m 3h09m 2h49m 1h41m 3h26m 2h46m 3h29m 3h18m 2h18m ... availability of inputs Observed student-to- 17 40.4 37 22.7 17.6 35.2 21.4 21.6 43.5 29.7 47.9 43 ... teacher ratio Share of students with text- 47 37.1 17.5 87.7 10.3 48 68.1 38.2 25.3 68.5 5 86 ... books (% of students) Minimum equipment avail- ability (% of schools) (90% 71 60.5 36.3 67.5 65 78.8 76.8 54.8 61.4 26.4 80.6 36 ... with pencils and notebooks) Minimum infrastructure 55.9 38.1 4.2 48.6 20.2 59.5 29.1 18.5 40.4 22.3 53.7 35 82++ availability (% of schools) Student Learning Language and mathematics 64.5 49.6 25.9 52.5 50.6 72 20.8 32.2 40.1+* 45.7 48.6 ... ... test score (out of 100) Language test score 42.3 47.3 34.4 57 56.8 59 25.1 31.9 58.2 44.6 43.4 ... ... (out of 100) + Minimum knowledge of teachers in Punjab includes assessments on English, math, or Urdu. The assessment was different with the one used in SDI. Service Delivery Indicator Survey Indonesia 2020 ++ Minimum infrastructure in Punjab means that the classroom has a blackboard, enough visibility, chairs and desks, pencils, and all students have textbooks and exercise books. * In Mozambique, only public schools were surveyed. 21 ** Values for Nigeria are the weighted average of the four states surveyed, namely Anambra, Bauchi, Ekiti, and Niger. Note on cross-country comparisons: Students and teachers in SDI countries were sometimes tested in the language they spoke at home, and sometimes in an official or national language that did not match the language spoken at home. This means that the language tests were not equally difficult for all test-takers, and so country scores in language are less comparable than scores in mathematics. introduction “ This work complements related research, particularly the Subnational Education Public Expenditure Review, which examines how financing is allocated at the subnational level, while “ this report examines the quality of services purchased. 1. This report is a national survey of education service This SDI survey complements the Subnational quality in Indonesia focusing on MoRA schools. Education Public Expenditure Review (Revealing At the request of the Ministry of Religious Affairs How Indonesia’s Subnational Governments Spend (MoRA) and the Ministry of Education and Culture their Money on Education). This work complements (MoEC), the World Bank undertook a national survey related research, particularly the Subnational of education service quality in Indonesia with support Education Public Expenditure Review, which examines from Australia’s Department of Foreign Affairs and how financing is allocated at the subnational level, Trade (DFAT). The goal was to measure and describe while this report examines the quality of services the quality of education service delivery, with a focus purchased. on MoRA schools. The Education SDI survey was conducted in For Indonesia, the standard Education SDI survey is Indonesia with the objective of supporting system expanded to include additional modules. This survey improvements at the GoI’s request for both MoRA expands on the World Bank’s SDI survey by including and MoEC schools, with a focus on MoRA’s school additional modules such as the Development World system. Management Survey and a Parent Motivation Survey. 22 Service Delivery Indicator Survey Indonesia 2020 Around 15 percent (8 million) of 1. What are student learning outcomes and what are the correlates of student performance? Indonesian primary and secondary 2. What are the levels of teacher effort and school students are enrolled in schools competences? What are the correlates of teacher under the jurisdiction of MoRA. While effort and competence? the curriculum is required to be the 3. How does Indonesia compare with other countries same for schools under MoRA as that have implemented the SDI? under MoEC, additional hours are 4. What autonomy on decision-making is given assigned for religious instruction in to which levels (school, regional or central) and what accountability exists for key variables such MoRA schools.4 as teacher attendance, textbook availability and student learning? The SDI survey tries to answer six research 5. What factors motivate parents to choose a MoRA questions. The aim of the Indonesia Education SDI school or a MoEC school? survey, implemented from February to March 2019, is 6. What are students’ perceptions of their teachers, to answer the following research questions regarding and what are teachers’ perceptions of themselves? in particular MoRA’s education system: 4 Religious school under the jurisdiction of MoRA are for the most part private (95 percent according to EMIS 2019/2020, http://emispen- dis.kemenag.go.id/dashboard/?content=data-statistik) and include: (i) madrasahs, which are Islamic schools; and (ii) schools of other religions, such as Catholic, Hindu, etc. Many of the non-public schools under MoRA are accurately described as community-based schools and are non-profit, but we use the more common term ‘private’ in this report. Non-religious schools are overseen by MoEC. Service Delivery Indicator Survey Indonesia 2020 23 chapter 1. Introduction The Service Delivery Indicators provide a set of The indicators are presented together, as well as by measures to compare the performance of schools ministry (MoRA and MoEC) and religious affiliation, and aim to identify the sources of constraints for public and private sector, and urban and rural areas. the provision of quality education services. The The sampling weights are taken into account in order analysis focuses on indicators that reflect the level to derive the estimates and standard deviations.5 and quality of primary education services, particularly Learning levels for students were found to be low, the inputs, effort and competence of teachers. The about 1.5 years behind where they are expected to survey includes a module for evaluating Grade 4 be on the national curriculum on average. Teacher students in language, mathematics and non-verbal knowledge was also low in both content and reasoning, as well as a module for evaluating teachers pedagogy, which are essential for student learning. in language, mathematics and pedagogy. These modules have been adapted to the specificities of the In terms of management practices, sampled school Indonesian education system. In particular, in MoRA administrators were seen to display medium levels Islamic school (commonly called madrasah), students of leadership and operations management skills were also assessed for their knowledge of the Arabic compared with other middle-income countries. language, in addition to the Indonesian language Furthermore, school infrastructure and equipment tests. availability were low, with significant room for improvement. Access to textbooks presents an important constraint, especially in MoRA Islamic schools, where only one in two students had a textbook for the subjects observed (mathematics, Indonesian language and Arabic). The proportion of students without textbooks was lower but still very 5 Analysis is also provided for non-madrasah (other religious) MoRA schools, but because of their small representation in the sample, the results are not representative of these categories of schools. 24 Service Delivery Indicator Survey Indonesia 2020 concerning in MoEC schools, where about one-third of This report is organized as follows: Section II the students did not have a textbook for the subjects presents the context of the survey as well as the observed. Taken together, these findings point to the organization of the Indonesian school system. Section urgent need for student-learning-focused reform of III discusses the methodology of the SDI survey and the Indonesian education system. the perspective and analytical framework underlying the Service Delivery Indicators in education. In terms of school choice, we found that MoRA parents Section IV presents the main results of the surveys, claimed the availability of more religious content was distinguishing the different indicators together and their top reason for sending their children to MoRA according to sub-sectors and settings. It also presents schools, whereas MoEC parents valued proximity the results of the learning assessments of Grade 4 to home the most. Based on this survey, religious students in mathematics, Indonesian language (and content seems to be driving MoRA enrollment, on Arabic among MoRA students). Section V examines average, instead of cost or convenience. the correlates of student performance through an analysis of the individual correlations associated with student and teacher characteristics. Section VI analyzes certain determinants of school learning. Section VII compares the results in Indonesia with those obtained in other countries that participated in the SDI survey. Section VIII presents the conclusions and recommendations. Service Delivery Indicator Survey Indonesia 2020 25 context “ According to the Human Capital Index, Indonesian workers of the next generation will only be 54 percent as productive as they could have been under the benchmark of 14 years of “ quality education and full health. 2. According to the Human Capital Index (HCI), Background Indonesian workers of the next generation will only Indonesia is a large diverse and highly decentralized be 54 percent as productive as they could have been country. With 267 million people (Susenas 2019, under the benchmark of 14 years of quality education Badan Pusat Statistik [BPS]), Indonesia is the fourth and full health.6 The descriptive and analytical work most populous country in the world and the third- presented here, together with other data, can serve largest democracy. Indonesia’s system of government as a benchmark for the development of human is highly decentralized, but it is nonetheless a capital and human resources, which was announced decentralized unitary state and not a federal system. as a priority by President Joko Widodo in his 2019 Indonesia is a diverse country, with over 300 distinct inauguration speech. Progress on student learning ethnic groups, and more than 700 languages and outcomes has been disappointing, as measured by dialects. This diversity extends to religion: 87 percent low scores in the national exam (Ujian Nasional, UN) of the total population is Muslim, while other major and international assessments (PISA), which suggest religions include Protestant (6.9 percent), Catholic that many Indonesian students failing to achieve the (2.9 percent), Hindu (1.7 percent), and Buddhist (0.7 minimum levels of knowledge and skills needed to percent). Indonesia is classified as an upper middle- participate fully in society. This study aims to uncover income country, with a gross national income per the teaching, learning and management processes of capita of US$4,050 (2019).7 the system at the Grade 4 level, with a focus on MoRA schools. 6 Human Capital Index: https://www.worldbank.org/en/publication/human-capital 7 Kewarganegaraan, Suku Bangsa, Agama, dan Bahasa Sehari-hari Penduduk Indonesia: Hasil Sensus Penduduk 2010. BPS. 2011. https://bit. ly/2h40yWa and World Development Indicators database. World Bank. 2018. https://data.worldbank.org/indicator/ny.gnp.pcap.cd 26 Service Delivery Indicator Survey Indonesia 2020 Indonesia’s Human Capital8 International Reading Literacy Study (PIRLS) data The World Bank’s HCI assesses countries based from 2011, given that prior to the SDI survey there was on their education and health outcomes, and the no nationally representative test of student learning impact on productivity. Indonesia’s score on the outcomes at the primary level (World Bank, 2019).9 2020 HCI was 0.54 (Figure 1).i This means that, on This, combined with the fact that there is currently average, Indonesian workers of the next generation no standardized national test for Indonesia until will be only 54 percent as productive as they could Grade 9, makes the below work especially relevant be under the benchmark of 14 years of education and to understanding learning quality at the primary full health. level. Helpfully, there are plans to establish a national assessment for Indonesian primary students. Learning Poverty Indonesia’s learning poverty rate is estimated at 35 percent and aligns with other measures such as the Early Grade Reading Assessment (EGRA) (USAID, 2014). Learning poverty is defined as the share of children in a country that is either not enrolled in school, or not able to read and understand a short, age-appropriate text at age 10. The data used for the learning poverty calculation make use of Progress in 8 Human Capital Index: https://www.worldbank.org/en/publication/human-capital 9 World Bank. 2019. Ending Learning Poverty: What Will It Take? World Bank, Washington, DC. Service Delivery Indicator Survey Indonesia 2020 27 chapter 2. context Figure 2.1. HCI for middle-income countries in East Asia Pacific, 2020 Vietnam 0.69 China 0.65 Thailand 0.61 Malaysia 0.61 Mongolia 0.61 Indonesia 0.54 Tonga 0.53 Phillipines 0.52 Kiribati 0.49 Cambodia 0.49 Myanmar 0.48 Lao PDR 0.46 Timor - Leste 0.45 Vanuatu 0.45 Tuvalu 0.45 Papua New Guinea 0.43 Solomon Island 0.42 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Governance attended formal pre-tertiary education, up from 43.3 Indonesia is governed through a tiered system of million in 2001. The net enrollment rate (NER)12 is 97.6 central ministries, provinces and districts. About 63 percent for primary, while lower secondary is 79.4 percent of total government spending on education— percent and upper secondary is 60.8 percent (Central excluding subsidies and interest payments—is Bureau of Statistics [BPS], 2019). There are 270,810 undertaken by subnational governments.10 There schools of all types in Indonesia, of which 175,028 are 34 provinces and 514 districts, as well as several are primary, 58,735 junior secondary, and 37,047 specially designated subnational regions. Law No. senior secondary.13 Law No. 20/2003 on the National 22/1999 on Local Government and Law No. 25/1999 Education System and the 2002 Amendment IV on Fiscal Balance between Central and Local to the Constitution emphasize that all Indonesian Government began the process of decentralization citizens have the right to education, and that the GoI for most public services, while religious affairs, has an obligation to finance basic education without foreign relations, and other functions such as defense, charging fees, and reaffirm the constitutional target remained centralized. of allocating at least 20 percent of the GoI’s budget to education. Education Sector Indonesia’s education sector is the fourth-largest in the world, and great progress has been made in enrollment levels over the past one and a half decades. In 2019, a total of 53.8 million students11 10 Indonesia Economic Quarterly, World Bank. 2017. http://pubdocs.worldbank.org/en/657051513163708686/IEQ-Dec-2017-ENG.pdf 11 Statistik Sekolah 2019/20 and EMIS 2019. 12 Total number of students in the expected age group for a given level of education enrolled in that level, expressed as a percentage of the total population in that age group (UNESCO UIS, 2018). 13 Statistik Sekolah 2019/20 and EMIS 2019. 28 Service Delivery Indicator Survey Indonesia 2020 Indonesia’s education sector is overseen by two private schools where the average UN score is lower ministries. The roughly 54 million Indonesian children by 12 percentage points than MoRA public schools.17 enrolled in formal education attend either public or Schools under MoRA’s authority are legally required private schools overseen by the Ministry of Education to deliver the same national curriculum as schools of Culture (MoEC) and the Ministry of Religious Affairs under MoEC, although the number of teaching hours (MoRA), with 15 percent (8 million) of all primary and for religious instruction is higher in MoRA schools. secondary students enrolled in schools administered by MoRA. The COVID-19 pandemic The COVID-19 pandemic has led to school closures Ministry of Religious Affairs (MoRA) in Indonesia, forcing more than 68 million young Religious education institutions in Indonesia provide Indonesians out of their classrooms,18 negatively education services to Muslim, Hindu, Protestant, impacting learning. Distance education mechanisms, Catholic, Buddhist and Confucian students, as well such as online learning and educational television, as studies in Al Quran, Pondok Pesantren, Diniyah, are in place to mitigate the effects of school closures, early childhood education and tertiary education.14 but with significant variations in access and uptake The MoRA system is comprehensive and extends across students and families.19 from pre-primary through tertiary, meaning that a child can enter the system at the pre-school level We estimate that Indonesian children and graduate from university, and then continue as a will have lost 21 points on the PISA teacher or administrator until retirement as a part of reading scale and US$484 in future a self-sustaining system. Nationally, MoRA oversees 52,576 education institutions. They provide services annual individual earnings due to to 8,004,640 students from primary through to senior the eight-month closure period from secondary, the majority of which are madrasahs.15 March 24 to the end of November 2020. These estimated losses are About 63 percent—the majority of primary and expected to increase in the coming secondary schools in Indonesia—are public, serving months as schools gradually re-open 37 million children, while private schools serve 17 million children16 (Figure 3). However, private (and possibly re-close), exacerbating schools make up 95 percent of all schools under Indonesia’s learning crisis.20 MoRA, accounting for 11.8 percent of the Indonesian pre-tertiary student population. Private schools under MoRA tend to be less expensive and have a higher level of religious instruction than public schools (M. Woodward, Rohmaniyah, Amin, & Coleman, 2010). Current performance measures are low for MoRA 14 Since a single student may for example attend madrasah in the morning and Al-Qur’an studies in the afternoon, these are not unique stu- dents and there is some double-counting; 28 million includes enrollment in formal, non-formal and informal education. 15 EMIS, 2019. 16 MoRA Education Management Information System [EMIS], 2019; MoEC Statistik Sekolah, 2019/2020 17 Calculated using UN score data 2018. 18 MoEC Circular Letter No. 4/2020 on Implementation of Education activities in the COVID-19 Emergency Period, which closed all schools and suspended exams on March 24, 2020 19 Indonesia High-frequency Monitoring of COVID-19 Impacts Education (World Bank forthcoming); INOVASI April 2020 study; https:// blogs.worldbank.org/eastasiapacific/indonesias-education-technology-during-covid-19-and-beyond 20 Yarrow, Masood and Afkar World Bank 2020 Estimates of COVID-19 Impacts on Learning and Earning in Indonesia: How to Turn the Tide Service Delivery Indicator Survey Indonesia 2020 29 chapter 2. context Figure 2.2. Proportion of students in MoEC and MoRA public and private schools PRIMARY SCHOOL STUDENTS SECONDARY SCHOOL STUDENTS MoEC Private MoEC Private 14% 35% 86% 65% MoEC Public MoEC Public MoRA Public MoRA Public 14% 27% 73% 86% MoRA Private MoRA Private Source: EMIS 2019; Statistik Sekolah 2019/20. Note: Secondary school students consist of junior secondary students and senior secondary students (includes vocational schools). Quality only about one-third of students attained a basic-to- Indonesia has increased enrollment levels, intermediate level of proficiency (30 percent reading, while student learning remains low. Indonesia 28 percent math, and 40 percent science). Less has participated in the OECD’s Programme for than 0.5 percent of students attained a high level of International Student Assessment (PISA) since proficiency in any subject. 22 2000. Student enrollment has grown significantly in the intervening years, with upper secondary gross Student learning levels vary by socioeconomic enrollment rates increasing from 39.8 percent in 2000 background and gender. In terms of equity and to 80.2 percent in 2018.21 Indonesia has made a strong gender, socioeconomically advantaged students effort to represent the country’s 15-year-olds in the outperformed disadvantaged ones in reading by 52 PISA sample; it achieved 85 percent representation points on the 2018 PISA assessment, which is less than in 2018, which is a huge improvement on 46 percent the OECD average (89 points). Girls scored slightly in 2003, and much higher than the Southeast Asian higher than boys in reading (by 25 points), math (by developing countries’ average of 70.5 percent 10 points), and science (by 7 points).23 representation. Actual performance on PISA 2018 indicated serious challenges in learning, with the Most students do not meet the national learning scores in reading, math, and science decreasing by 26 targets that Indonesia sets for itself. Measures points, 7 points, and 7 points, respectively, compared of learning show challenges in the first years, and with Indonesia’s performance in 2015. This means that learning remains low as students move across 21 https://databank.worldbank.org/reports.aspx?source=Education%20Statistics# 22 PISA Results from PISA 2018 – Indonesia. OECD. 2019. https://www.oecd.org/pisa/publications/PISA2018_CN_IDN.pdf 23 PISA Results from PISA 2018 – Indonesia. OECD. 2019. https://www.oecd.org/pisa/publications/PISA2018_CN_IDN.pdf 30 Service Delivery Indicator Survey Indonesia 2020 grades. Learning is low both in absolute terms, below provinces and the three lowest performing provinces national targets and in relative terms compared on the Grade 12 exam for senior secondary schools with neighboring countries, with half of 15-year-olds (SMA) is 21 points on a 100-point scale. Only two unable to read and understand a paragraph of text of the 34 provinces had an average Grade 12 score (World Bank, 2018a). above the minimum passing score of 55. The results are even lower for the Grade 9 exam for junior Learning poverty and learning inequality are both secondary schools (SMP), and for the Grade 12 exam high. Some provinces in Indonesia, especially those for technical and vocational schools (SMK). Districts in the central region, perform poorly on the national with higher incomes, large urban centers, and exam, while others, often in the east and far west, greater implementation capacity tend to do better perform very poorly (Figure 4.; we note the national than lower-income, more rural districts with lower exam has been discontinued).24 The difference implementation capacity. between the average of the three top performing Figure 2.3. Most provincial national exam scores miss the national passing grade PASSING GRADE aceh 40.0 maluku utara 40.6 Sulawesi Barat 40.7 Papua 41.0 Maluku 41.2 Sulawesi utara 42.1 Sulawesi tengah 42.2 Nusa Tenggara Barat 42.2 Sulawesi Selatan 42.5 Nusa Tenggara Timur 42.6 Gorontalo 42.9 Sumatera Selatan 42.9 Jambi 43.4 Sulawesi Tenggara 43.8 SUmatera Utara 43.9 Kalimantan Tengah 44.2 Lampung 44.8 Bengkulu 44.8 Kalimantan Utara 44.9 Kalimantan Barat 45.3 Riau 45.9 Papua Barat 46.2 National 46.7 Banten 47.0 Jawa timur 48.5 Jawa Barat 48.7 Kalimantan Timur 48.9 Sumatera Barat 49.2 Kepulauan Bangka Belitung 49.3 Kalimantan Selatan 49.4 Kepulauan riau 50.2 bali 52.3 jawa tengah 53.3 d. i. yogyakarta 58.4 dki jakarta 60.8 0.0 10.0 20.0 30.0 40.0 50.0 60.0 Source: World Bank, based on Ministry of Education and Culture data, 2019. https://hasilun.puspendik.kemdikbud.go.id. 24 Ministry of Education and Culture statement, December 2019, https://www.kemdikbud.go.id/main/blog/2019/12/mendikbud-tetapkanem- pat-pokok-kebijakan-pendidikan-merdeka-belajar. Service Delivery Indicator Survey Indonesia 2020 31 methodology & implementation “ The perspective adopted by the SDI survey in education is that of citizens using education services and facing potential constraints in the services available to them. “ 3. This section presents the methodology of the SDI The SDI survey aims to establish a portrait of the survey, as well as the survey instruments and sampling quality of the provision of primary education approach. services. By examining the entire education service delivery system from inputs such as funding, school Perspective and Objectives infrastructure and textbook availability, to outcomes The Service Delivery Indicators (SDI) survey is a such as teachers’ pedagogical approaches in the global tool developed by the World Bank to measure classroom and student learning, the SDI survey aims the quality of service delivery in the key areas of to establish a portrait of the quality of the provision education and health. The perspective adopted by of primary education services, and to identify the the SDI survey in education is that of citizens using source of the constraints for the provision of quality education services and facing potential constraints in education services. The input indicators map to three the services available to them. The SDI indicators are basic categories, and are quantitative and ordinal in therefore presented as a service delivery report card. nature to allow for cross-country and country-specific They aim to measure the performance and quality of comparisons: primary education services as experienced by citizens • Teacher competence and knowledge; and service users. However, rather than relying on • Proxies for effort; and individual perceptions to assess performance, they • Availability of key infrastructure and inputs. rely on objective and quantitative information based on a survey of schools. 32 Service Delivery Indicator Survey Indonesia 2020 The SDI survey aims to inform public decision- The survey approach is based on a few guiding makers and citizens. Ultimately, the SDI survey aims principles: direct observation, unannounced visits to to inform public decision-makers and citizens about measure absence, and assessment of the knowledge the performance of education service providers in of teachers and students. To the extent possible, all order to put in place measures aimed at improving information is based on quantitative data and direct education services. A definition of the indicators can observation by investigators. be found in Appendix B. The survey includes two visits to schools. The first Survey Instruments and Modules visit is an announced visit to the school administration The SDI education survey is a facility-based survey and collects most of the information required for the where the facilities are schools. There are few investigation. The second visit is an unannounced visit, standardized facility surveys that are repeated with which takes place shortly after the first visit and aims predictable frequency, and the SDI survey is designed to gather information on the teachers’ attendance at to fill this gap in the data landscape. The information the school. is collected on the supply side, and the intent is to inform the demand side with accountability and The SDI survey in Indonesia uses the standard SDI results by consumers, i.e., parents, policy analysts survey questionnaires with several adjustments to from public and private sector, and policy-makers, the Indonesian context. The SDI survey in Indonesia including parliamentarians and government officials.25 uses the standard SDI survey questionnaires consisting of six modules administered to different stakeholders 25 https://www.sdindicators.org/methodology#samp_desn Service Delivery Indicator Survey Indonesia 2020 33 chapter 3. METHODOLOGY AND IMPLEMENTATION within the school, with several adjustments to the In addition to the standard SDI modules, the Indonesian context, such as the inclusion of the type SDI survey in education in Indonesia included of teacher employment, teacher certification and socioemotional modules for teachers, and student allowance status, and the type of financial resources perceptions of teachers. The survey included a provided to schools. The SDI survey also includes household module among school parents aimed at a module for evaluating student achievement in assessing parental school choice and socioemotional language (Indonesian language), mathematics and conditions. Furthermore, the survey included the non-verbal reasoning (NVR). Student achievement assessment of the school management quality using in Arabic was also assessed in MoRA Islamic schools. the Development World Management Survey (DWMS) Teacher knowledge was assessed across these same questionnaire.26 The various survey instruments are subjects. presented in Table 3.1.27 TABLE 3.1. Education SDI survey instruments Module Description Administered to the head of the school to collect information Module 1: about school type, facilities, school governance, student School Information numbers, and school hours. Includes direct observations of school infrastructure by enumerators. Administered to head teacher and individual teachers to Module 2a: obtain a list of all school teachers, to measure teacher Teacher Absence and Information absence, and to collect information about teacher characteristics. Module 2b: Unannounced visit to the school to assess absence rate. Teacher Absence and Information Administered to the school principal to collect information Module 3: about school management using the DWMS and on school DWMS and School Finances finances. Module 4: An observation module to assess teaching activities and Classroom Observation classroom conditions in two classrooms per school. Module 5: A test of students to have a measure of student learning Student Assessment outcomes in mathematics and language in Grade 4. Module 6: A test of teachers covering mathematics and language Teacher Assessment subject knowledge and teaching skills. Administered at the household level among three parents of Module 7: school students and three other households with children in Parent interviews other schools. 26 Lemos and Scur, 2017. https://developingmanagement.org/ 27 More information on the SDI survey instruments and data, and more generally on the SDI initiative can be found at: www.SDIndicators.org and www.worldbank.org/SDI, or by contacting SDI@worldbank.org. 34 Service Delivery Indicator Survey Indonesia 2020 box 3.1. Analytical underpinnings of the survey Source: Service Delivery Indicators, World Bank, 2017, https://www.sdindicators.org/indicators Service delivery outcomes are determined by the relationships of accountability between policy- makers, service providers, and citizens (Figure 2, World Bank 2004). Human development outcomes are the result of the interaction between various actors in the multi-step service delivery system, and depend on the characteristics and behavior of individuals and households. While delivery of quality education is contingent first and foremost on what happens in classrooms, a combination of several basic elements has to be present for quality services to be accessible and produced by teachers at the frontline, which depend on the overall service delivery system and supply chain. Adequate financing, infrastructure, human resources, material, and equipment need to be made available, while the institutions and governance structure provide incentives for the service providers to perform. box figure 3.1.1. Relationships of accountability between citizens, service providers, and policy-makers Service Delivery Production Function Consider a service delivery production function, f, which maps physical inputs, x, the effort put in by the service provider, e, as well as his/her type (or knowledge), , to deliver quality services into individual level outcomes, y. The effort variable, e, could be thought of as multidimensional and, thus, includes effort (broadly defined) of other actors in the service delivery system. We can think of this type as the characteristic (knowledge) of the individuals who are selected for a specific task. Of course, as noted above, outcomes of this production process are not just affected by the service delivery unit, but also by the actions and behaviors of households, which we denote by . We can therefore write: y = f(x,e, ) + . To assess the quality of services provided, one should ideally measure f(x,e, ). Of course, it is notoriously difficult to measure all the arguments that enter the production, and would involve a huge data collection effort. A more feasible approach is, therefore, to focus instead on proxies of the arguments which, to a first-order approximation, have the largest effects. Service Delivery Indicator Survey Indonesia 2020 35 chapter 3. METHODOLOGY AND IMPLEMENTATION box 3.1. Continued Indicator Categories and the Selection Criteria The proposed choice of indicators takes its starting point from the recent literature on the economics of education and service delivery, more generally. Overall, this literature stresses the importance of provider behavior and competence in the delivery of education services (as opposed to water and sanitation services and housing that rely on very different service delivery models). Conditional on service providers exerting effort, there is also some evidence that the provision of physical resources and infrastructure has important effects on the quality of service delivery. The somewhat weak relationship between resources and outcomes documented in the literature has been associated with deficiencies in the incentive structure of schools and education systems. Indeed, most service delivery systems in developing countries present frontline providers with a set of incentives that negate the impact of pure resource-based policies. Therefore, while resources alone appear to have a limited impact on the quality of education in developing countries, it is possible inputs are complementary to changes in incentives, so coupling improvements in both may have large and significant impacts (Hanushek, 2006). In addition to measuring student learning outcomes, the SDI survey proposes three sets of indicators: (i) teacher effort; (ii) teacher competence and knowledge; and (iii) availability of key infrastructure and inputs at the frontline service provider level. Providing countries with detailed and comparable data on these important dimensions of service delivery is one of the main innovations of the Service Delivery Indicators. Additional considerations in the selection of indicators are: (i) quantitative (to avoid problems of perception biases that limit both cross-country and longitudinal comparisons); (ii) ordinal in nature (to allow within and cross-country comparisons); (iii) robust (in the sense that the methodology used to construct the indicators can be verified and replicated); (iv) actionable; and (v) cost effective to collect. Education indicators Teacher Effort School absence rate Classroom absence rate Time spent teaching per day Teacher Knowledge and Ability Minimum knowledge in language, Minimum knowledge in mathematics, Minimum knowledge in pedagogy Availability of Inputs Minimum infrastructure availability Minimum equipment availability Share of students with textbooks Observed student-to-teacher ratio 36 Service Delivery Indicator Survey Indonesia 2020 Sample The survey sample includes both public and private The SDI survey sample in Indonesia is described primary schools (38 and 62 percent, respectively), in Table 3.3 (see also Table A2 in the Appendix for among which 46 percent are urban. The survey details). In total, 350 primary schools were visited, focuses more specifically on MoRA Islamic schools the mathematics, language, and pedagogical skills (madrasahs), which comprise 72 percent of the of 1,838 mainly Grade 4 teachers were assessed, sample and are nationally representative of that and 3,169 teacher absences from all grades were school system. The sample also includes MoEC examined. In addition, 2,107 households were managed schools, which comprise 25 percent of surveyed to assess parental school choice decisions. the sample. A small number of non-madrasah (other In order to examine the relationship between student religious) private MoRA schools (3 percent) are also learning outcomes and school quality, 3,368 Grade included in the survey sample.28 As noted, the size 4 students were evaluated in Indonesian language, of this last strata is too small to be representative of mathematics and non-verbal reasoning. In addition, these other religious schools, but they are presented 2,462 Grade 4 students from MoRA Islamic schools separately in the analysis below to indicate the likely were evaluated in Arabic language. performance of this sub-sector. 28 As mentioned, MoRA schools include: (i) madrasahs, which are Islamic schools; and (ii) schools of other religions such as Catholic, Hindu etc. Service Delivery Indicator Survey Indonesia 2020 37 chapter 3. METHODOLOGY AND IMPLEMENTATION The SDI education survey in Indonesia is SDI survey in Indonesia, public schools account representative of MoRA Islamic primary schools for 92 percent of the MoEC sample, vs. 88 percent at the national level. The SDI survey also offers nationally. The average MoEC school in our sample the possibility of generating statistics on public has 213 students and 13 teachers, while the average and private schools, as well as on urban schools MoEC school in Indonesia has 169 students and 11 versus rural schools. Unlike MoRA, MoEC schools teachers. The observed student-to-teacher ratio are predominantly public, rather than private (or in MoEC sample is 21, which is lower than the ratio community-based). Nationally, public schools of 28 for MoEC schools nationally. Female students account for 88 percent of all MoEC schools. account for 48 percent out of the total students in both our sample and MoEC schools in Indonesia. The The sampled MoEC schools are similar to MoEC sampling strategy is explained in detail in Appendix A schools nationally and can be fairly an important of this report, along with the weighting coefficients. comparison group for the study, even though the sample is not nationally representative. In the table 3.2. Survey Sample MoRA MoRA MoEC Non - Public Private Urban Rural All Islamic Islamic Schools 253 87 10 133 217 160 190 350 Teachers Absence 2327 768 74 1233 1936 1494 1675 3169 Teachers (Evaluations) 1470 338 30 788 1050 989 849 1838 Students 2465 837 66 1297 2071 1557 1811 3368 Households 1523 526 58 802 1305 964 1143 2107 38 Service Delivery Indicator Survey Indonesia 2020 box 3.2. MoRA Non - Islamic Religious Schools There are 10 non-Islamic MoRA schools included in the SDI survey sample in Indonesia. They were intentionally selected, and the small sample size of surveyed schools in this category precludes statistically representative results, but they still contain some noteworthy information to indicate the likely performance of this school category. Students in MoRA non-Islamic schools show better results on each component of student assessment both in mathematics and Indonesian language compared with MoRA Islamic and MoEC schools. On the overall score, they scored more than 33 percentage points higher in mathematics than students in MoRA Islamic schools, and 10 percentage points higher in Indonesian language. These students in MoRA non-Islamic schools scored 28 percentage points higher in mathematics and about 7 percentage points higher in language than MoEC students. Teachers in MoRA non-Islamic schools show greater effort compared with their colleagues from MoRA Islamic and MoEC schools. MoRA non-Islamic schools present significantly lower school absence rate compared with MoRA non-Islamic and MoEC schools, with a gap of about 8 percentage points. A significantly lower average absence rate of teachers from the classroom is also observed in MoRA non-Islamic schools, with 10 percentage points less than MoEC school and 8 percentage points below MoRA Islamic schools. Non-Islamic MoRA schools has the highest time spent teaching per day, which was 3 hours 47 minutes, while both in MoRA Islamic and MoEC schools, while it was less than 3 hours in both MoRA Islamic and MoEC schools. Teachers in the 10 non-madrasah schools in MoRA tend to score higher than MoRA Islamic and MoEC schools. Teachers in MoRA non-Islamic schools showed better results than their colleagues in the MoRA Islamic and MoEC schools (43.3 vs. 15.6 and 13.9 percent, respectively). Over half of the teachers (53 percent) in MoRA non-Islamic schools had the minimum knowledge criteria for math. Furthermore, despite the level of pedagogical knowledge being low across all types of schools at 23 percent, the MoRA non-Islamic schools still presented significantly higher scores. Results of Teacher Assessment, by Subject Minimum knowledge – 80% threshold (% teachers) MoRA non-Islamic Language and math: 80% 43.3 Language: 80% 26.7 Math: 80% 53.3 Average score (%) Language and math 76.1 Language 73.6 Math 78.7 Pedagogy 33.8 Language, math and pedagogy 50.7 Service Delivery Indicator Survey Indonesia 2020 39 results “ Fourth graders in this study on average performed 1.5 years behind where they are expected to on the Indonesian national curriculum. On average across all schools in the sample, students in the second half of their “ 4th grade year performed at the 2.6 grade level in literacy, and at the 1.5 grade level in math, demonstrating a crisis in learning. 4. This section presents the results of the SDI survey of their 4th grade year performed at the 2.6 grade in education. The indicators are presented across level in literacy, and at the 1.5 grade level in math, all types of schools, as well as by ministry, public demonstrating a crisis in learning. These average and private sector, and urban and rural areas. The figures conceal significant levels of heterogeneity sampling weights are taken into account in order to between groups of students. derive the estimates and standard deviations. Student learning outcomes are presented first, followed by Curriculum-adjusted years of schooling: Overall, indicators of provider effort and knowledge, followed Grade 4 students in all schools do not meet by school inputs. curriculum requirements. A. ASSESSMENT OF STUDENT LEARNING Across all schools included in the survey, Grade 4 students, on average, do not reach the curriculum Overview requirements for their grade level. In fact, Grade 4 Students are learning far less than they are supposed students on average have not mastered the Grade to across all types of schools. 3 curriculum requirements in Indonesian language literacy and Arabic literacy, while for mathematics, Fourth graders in this study on average performed students have not mastered the Grade 2 curriculum 1.5 years behind where they are expected to on the requirements. Indonesian national curriculum. On average across all schools in the sample, students in the second half 40 Service Delivery Indicator Survey Indonesia 2020 The following provides a brief overview of student Arabic literacy (MoRA madrasahs only): While iii.) performance across the three subject areas. We note 91 percent of students could identify letters and that the results are nationally representative for MoRA 89 percent of students could identify words on Islamic schools, and indicative but not nationally the student assessment, students performed representative for MoEC and other types of schools. poorly on all other areas: identifying pictures (10 i.) Indonesian language: While nearly all Grade 4 percent), reading a sentence (76 percent), reading students could identify words and pictures, and a passage (70 percent), and comprehension (45 read sentences and passages, only 72 percent percent). However, the majority of students (65 could read with comprehension. Overall, 53 percent) were able to perform at a Grade 3 level, percent of students performed at either Grade meaning they were able to identify 100 percent of 3 or Grade 4 level, while the remaining students Arabic letters and words, and could answer at least performed at Grade 2 or below. a quarters of Grade 4-level questions correctly. ii.) Mathematics: Grade 4 students struggled the most in mathematics, relative to Indonesian literacy and Arabic literacy. Only 3 percent of students could perform at a Grade 4 level and only 3 percent at Grade 3. The majority of students performed at a Grade 2 level (45 percent) and Grade 1 (40 percent). Service Delivery Indicator Survey Indonesia 2020 41 chapter 4. results FIGURE 4.1. Curriculum - adjusted Years of Education 2.55 2.50 years years 1.54 years LITERACY MATHEMATICS ARABIC Sample size: 350 schools. Student weights are used. Literacy 1) Grade 1: 100 percent of correct answers in identifying words and identifying pictures; Grade 2: Student is able to read aloud all words from a sentence correctly and 100 percent of correct answers in Grade 1 level questions; Grade 3: Student is able to read aloud 80 percent words from a passage correctly in 45 seconds and 100 percent of correct answers in Grade 1 and Grade 2 level questions; Grade 4: Student is able to correctly answer at least 2/3 of reading comprehension questions and 100 percent of correct answers in Grade 1, Grade 2 and Grade 3 level questions. Math 2) Grade 1: 100 percent of correct answers in identifying numbers and correct answer of single-digit addition or single-digit subtraction, Grade 2: Correct answer in ordering from smallest to largest and correct answer of double-digit addition or double-digit subtraction and 100 percent of correct answers in Grade 1 level questions, Grade 3: At least 5/6 of correct answers in triple-digit addition, single-digit, double-digit and triple-digit multiplication, single-digit and double-digit division and 100 percent of correct answers in Grade 1 and Grade 2 level questions, Grade 4: At least 2/3 of correct answers in ability to divide double-digits and identify the smallest result, solve a simple math problem, and complete a sequence, 100 percent of correct answers in Grade 1 and Grade 2 level questions and at least 5/6 of correct answers in Grade 3 level questions. Arabic 3) Grade 1: At least 3/6 of correct answers in identifying letters and identifying words. Grade 2: At least 5/6 of correct answers in Grade 1 level questions. Grade 3: At least 25 percent of correct answers in identifying pictures, words correctly read from a sentence, words correctly read from a passage and reading comprehension questions, and 100 percent of correct answers in Grade 1 level questions. Grade 4: At least 75 percent of correct answers in Grade 3 level questions and 100 percent of correct answers in Grade 1 level questions. 42 Service Delivery Indicator Survey Indonesia 2020 Indonesian Language percent. Lastly, a passage containing 58 words was The Indonesian language assessment measured the given to be read aloud by the students and timed language skills of students from Grade 1 to Grade 4. by the enumerator. The students were then given The Indonesian language assessment measured the three reading comprehension questions relating to language skills of students ranging from the Grade the passage. The first two questions were inferential 1 level, such as their ability to recognize letters and and required students to read the lines to find the words, and identify pictures, to the Grade 4 level, answer. The third question was more evaluative, i.e., it where they are expected to read a passage and answer required students to read between the lines. reading comprehension questions. In this subsection, we look at the overall performance of students on Students scored an impressive 94 each type of question in the Indonesian language percent on reading out the passage, assessment, and then observe the distribution of the with 41 percent of the students students’ performance in Indonesian language as per the requirements of the national curriculum.29 managing to read the entire passage without any mistakes. Across the surveyed schools, students were one year behind on average in Indonesian language skills. In the reading comprehension questions, 74 percent The majority of the students assessed (98 percent) of the students answered the first question correctly, were able to correctly identify the three words asked while 93 percent were able to answer the second by the enumerators. Almost all (99 percent) the question correctly. However, we saw the score students were able to correctly identify names of drop to 55 percent in the third evaluative question. the four images shown to them. Students were then Overall, about 43 percent of the assessed students asked to read a short sentence aloud, comprising of were able to answer all three questions correctly, and 10 words in total. We found that students performed the average score for students in all three reading well in this exercise, with an average score of 96 comprehension questions was 72 percent. 29 Despite it being a Grade 1 level exercise, only 70 percent of the students in the MoRA school sample were able to identify all three letters, though significantly more (98 percent) were able to correctly identify the three words asked by the enumerators. This could point to difficulty in hearing the correct letter, or confusion on the part of the students at the beginning of the exercise. To compensate from this conflicting signal where students were unable to identify letters but were able to read words, we have removed the letter score from the analysis. Service Delivery Indicator Survey Indonesia 2020 43 chapter 4. results FIGURE 4.1.A. Distribution of students by Curriculum-Adjusted Years of Education in Literacy All schools Grade 0: Student is unable to reach Grade 1 level criteria Grade 1: 100% of correct answers in identifying words and identifying pictures. Grade 2: Student is able to read aloud all words from a sentence correctly and 100% of correct answers in Grade 1 level questions. Grade 3: Student is able to read aloud 80% words from a passage correctly in 45 seconds and 100% of correct answers in Grade 1 and Grade 2 level questions. Grade 4: Student is able to correctly answer at least 2/3 of reading comprehension questions and 100% of correct answers in Grade 1, Grade 2 and Grade 3 level questions. 28% 27% 25% 13% 7% GRADE 0 GRADE 1 GRADE 2 GRADE 3 GRADE 4 Sample size: 350 schools. Math About 20 percent of the students Across all types of schools surveyed, students understood questions about fractions: performed on average 1.5 years behind where they were expected to be on the national curriculum. The “Which of the following gives the mathematics exercise had questions ranging from smallest answer? a. 81 5, b. 81 Grade 1 level of difficulty (identifying numbers) to 4, c. 81 3”, or were able to solve Grade 4 level of difficulty (completing sequences, the following simple math problem solving simple mathematical word problems, etc.). requiring multiplication: “A box Almost all students were able to identify numbers. contains 26 oranges. How many Single-digit addition and subtraction were answered oranges are contained in 10 boxes?”, correctly by over 80 percent of students. Students seemed to struggle with double-digit subtractions and about one-third of the students managed to (using carry-over), with only 28 percent of correct complete the following sequence with the next responses. The worst performance was observed number: “48 24 12 6 ____” . in double- and triple-digit multiplication questions, which were answered correctly by less than 10 percent of students. 44 Service Delivery Indicator Survey Indonesia 2020 FIGURE 4.1.B. Distribution of students by Curriculum-Adjusted Years of Education in Math All schools Grade 0: Student is unable to reach Grade 1 level criteria. Grade 1: 100% of correct answers in identifying numbers and correct answer of single digit addition or single digit subtraction Grade 2: Correct answer in ordering from smallest to largest and correct answer of double digit addition or double digit subtraction and 100% of correct answers in Grade 1 level questions. Grade 3: At least 5/6 of correct answers in triple digit addition, single-digit, double-digit and triple digit multiplication, single -digit and double digit division and 100% of correct answers in Grade 1 and Grade 2 level questions. Grade 4: At least 2/3 of correct answers in ability to divide double digits and identify the smallest result, solve a simple math problem, and complete a sequence, 100% of correct answers in Grade 1 and Grade 2 level questions and at least 5/6 of correct answers in Grade 3 level questions. 45% 40% 8% 3% 3% GRADE 0 GRADE 1 GRADE 2 GRADE 3 GRADE 4 Sample size: 350 schools. Student Testing Methodology content of the language tests in Indonesian The SDI education survey tested Grade 4 students’ language and Arabic were identical in the two learning levels. versions of the assessment. In each of the schools, 10 students The Grade 4 student assessment is designed randomly drawn from a Grade 4 as an individualized exam with interviewers presenting instructions to the student. Hence, class were assessed in basic reading, each of the 10 randomly selected students in writing, arithmetic skills, and non- the Grade 4 class was assessed individually verbal reasoning (NVR). with an interviewer. Other than for MoRA Islamic schools, the student assessment In addition to the Indonesian language tests, Grade 4 lasted a total of 25 minutes: 10 minutes for the students in MoRA Islamic schools were also assessed Indonesian language section, 10 minutes for for their knowledge of the Arabic language. The the math section, and 5 minutes for the NVR Service Delivery Indicator Survey Indonesia 2020 45 chapter 4. results section. In MoRA Islamic schools, the assessment they had read. The math exam asked students to lasted a total of 35 minutes, the Arabic language test identify and ascend numbers, add numbers from one being presented as the last subject. If students had to three digits, do subtraction with numbers from not completed the Indonesian language section after one to two digits, and do single-digit multiplications. 10 minutes, they should proceed to the mathematics Finally, the NVR part contained four questions on section. Likewise, the math section lasted 10 minutes, the basis of Raven’s matrices, a standard IQ measure after which students had to move on to the NVR that is designed to be valid across different cultures. section. Depending on the section, students could For more information, please see the SDI website.30 give their answers orally or in writing in their student Table 4.1. and Figure 4.2. show the results of notebooks. student assessments in the different subjects of the assessment. This individualized approach aims to be able to develop a differentiated profile of students’ skills, Learning Inequality since such an individual approach makes it possible These average figures conceal significant levels to determine whether students can solve a math of heterogeneity between groups of students problem, for example, even if their reading level is depending on the sector of the school (MoRA, MoEC), so low that they cannot independently attempt to status (public, private) and location (urban, rural). resolve the problem. The Indonesian language and Grade 4 students in the surveyed MoEC schools did Arabic assessments, on the other hand, consisted significantly better in all subjects than those attending of tasks such as recognizing letters of the alphabet MoRA Islamic schools, especially in mathematics, with and words, reading aloud and understanding what a 5-percentage-point difference. FIGURe 4.2. Student results by sector, status and location 100% 96 87 87 89 92 89 90% 87 89 85 80% 74 75 75 72 72 75 72 70 70% 60% 47 50% 44 45 42 42 42 40 40% 30% 20% 10% 0 All MoRA MoEC MoRA Public Private Urban Rural Islamic Non-Islamic Mathematics Indonesian language NVR 30 www.SDIndicators.org and www.worldbank.org/SDI 46 Service Delivery Indicator Survey Indonesia 2020 The differences in results between students attending There are also significant differences between urban public and private schools are also significant, with and rural students, with rural area students scoring an advantage in favor of the public schools in all lower in all assessments, especially mathematics, with subjects except Arabic. It may surprising to observe a difference of 5 percentage points in favor of urban that students attending private schools perform more schools. poorly than students in public schools. However, this is associated with the nature of the schools classified Reasons for variations in assessment results between here as private, which encompass many non-profit students attending various categories of schools and Islamic schools that are managed as community gender in the various subjects in the assessment are schools and, in many cases, lack the resources that examined in greater depth in Section V. public schools have.31 Detailed results of the evaluations Table 4.1 presents the detailed results of the student assessments in mathematics, Indonesian language and NVR according to the assessment subject sub- table 4.1. themes. Student Assessment: Detailed Results (percentage) MoRA Mora MoEC Non - Public Private Urban Rural Islamic Islamic Overall test score 66.9 70.0 87.7 69.2 66.7 69.7 65.2 (math, language, NVR) Math Score 42.2 47.4 75.3 44.1 42.1 45.1 40.6 Single-digit addition 88.1 89.4 97.8 87.6 88.1 91.5 86.1 Single-digit subtraction 80.1 83.3 95.2 83.4 80.1 86.4 77.8 Double-digit addition 63.6 66.4 95.1 67.6 63.2 69.0 60.3 Double-digit subtraction 28.2 35.3 70.5 32.9 27.7 30.6 26.9 Single-digit multiplication 37.2 47.3 80.8 38.3 37.2 41.9 34.4 Double-digit multiplication 9.7 16.8 58.2 11.3 9.6 11.1 8.9 Single-digit division 53.2 64.7 93.9 63.2 52.2 55.7 51.8 Literacy Score 86.6 88.6 96.3 88.5 86.5 89.2 85.1 Identify a letter 89.5 90.5 97.6 86.0 89.9 88.4 88.4 Identify basic words 97.8 98.6 99.6 98.8 97.7 98.7 97.2 Read sentence 96.2 97.4 98.8 97.6 96.0 97.4 95.4 Read paragraph 94.5 95.5 97.2 95.9 94.3 93.3 96.4 Comprehension score 72.2 75.8 92.5 77.0 71.7 69.5 76.7 NVR score 71.6 74.0 91.5 75.0 71.5 74.9 69.9 Note: Weighted means using sampling weights. The small sample size of MoEC and MoRA non-Islamic schools surveyed precludes nationally representative results for these categories. 31 Realizing Education’s Promise Project Appraisal Document 2019 (World Bank). Service Delivery Indicator Survey Indonesia 2020 47 chapter 4. results Scores in mathematics were much lower than those Single-digit multiplication were successfully obtained in Indonesian language. Shortcomings performed by one student in about 3.5 students (28 in the mathematics section were observed for percent), fewer than one student in 10 (9.7 percent) basic operations. While single-digit addition and was capable of two-digit multiplication, which is a subtraction were successfully performed by between core part of the Grade 3 curriculum. The assessments 80 and 90 percent of students, were given to students during the second semester of Grade 4. Based on the expectations of the Indonesian one-third of the students overall could education system, the students should have mastered not complete a double-digit addition these specific skills in the previous year. and more than two-thirds of students could not complete a double-digit subtraction, both of which are part of the math curriculum for Grade 2. 48 Service Delivery Indicator Survey Indonesia 2020 B. TEACHER EFFORT types of schools. The average Grade 4 student in this study receives an average of 2 hours and 56 minutes Overview of actual teaching per day, which corresponds to 1 There were on 12.28 teachers per school and the hour and 39 minutes less teaching per day than what share of civil servant (‘PNS’) teachers was 13 percent is officially planned. on average. Schools in the sample had on average 12.28 teachers per school. Of the teachers in all Three indicators aim to capture the effort of schools, the share of PNS teachers was 13 percent, the teachers in their work. These indicators are: (i) the rest being non-PNS contract teachers. Public schools school absence rate; (ii) the class absence rate; and employed more than six times as many PNS teachers (iii) the teaching time per day. as private schools. Most of the teachers are female (60 percent) and work full-time (86 percent). The share School absence rate of certified32 teachers was about 41 percent. Overall, Teachers are the largest school-based factor in public schools had a 16-percentage-point higher student learning (WDR 2018). If teachers are not share of certified teachers than private schools, and at school, children are much less likely to learn. In urban schools had a 14-percentage-point higher share order to measure teacher absence, the presence of of certified teachers than rural schools. The teachers 10 teachers (at most) drawn at random during the interviewed were on average 36 years of age and had first visit among all the teachers in the school was 13 years of teaching experience. Public schools and checked during the second unannounced visit to urban area schools had teachers who were older in schools. Hence, a sample of 1,838 teachers from 350 age, with more years of experience on average than schools was used to measure teacher attendance, private and rural area schools. both in school and in the classroom. Teacher effort on average is low. The absence rate of The absence rate of teachers from school is high. teachers from school for both excused and unexcused As shown in Table 4.2, the absence rate of teachers reasons is high for an upper middle-income country, from school is high, with close to one in five teachers with close to one in five teachers (18.7 percent) on (18.7 percent) on average not present during the average not present during the unannounced visit unannounced visit, for reasons other than scheduling across all types of schools observed. The absence rate reasons or days when the teacher was not teaching. from class is even higher at 23.5 percent across all The sample is strictly representative at a national level only for MoRA Islamic schools, but also includes table 4.2. a large indicative sample of MoEC schools.33 Teacher Effort MoRA Mora MoEC Non - Public Private Urban Rural Islamic Islamic School absence rate 18.7 19.3 11.1 12.4 19.2 16.0 19.8 (% teachers) Classroom absence rate 23.5 25.7 15.1 20.3 23.7 20.1 24.9 (% teachers) Time spent teaching 2h 56min 2h 38min 3h 47min 3h 22min 2h55min 3h11min 2h 50min per day Scheduled teaching time 4h 35min 4h29min 5h 7 min 5h 1min 4h 33min 4h 51min 4h 28min per day (Difference btw scheduled 1h39min 1h51min 1h20min 1h39min 1h38min 1h40min 1h38min and actual) Note: Weighted means at the school level using sampling weights. Results for absence rate based on observations from 3,169 sampled teachers from 350 schools. Results for time on task based on observations from 700 sampled class. The small sample size of MoEC and MoRA non-Islamic schools surveyed precludes nationally representative results for these categories. 32 Teacher certification encompasses two areas of requirements: (i) academic qualifications (minimum bachelor’s degree); and (ii) pass the PPG (Pendidikan Profesi Guru/Teacher Professionalism Training) program. 33 Teachers in the sample who had no lessons on the schedule at the time of the unannounced visit or who had no lessons on the schedule that day were removed from the sample to establish the absence rate. Service Delivery Indicator Survey Indonesia 2020 49 chapter 4. results The teacher absence rate from school is about 7 Just under one-quarter of teachers are absent from percentage points higher in private schools than their classroom. The absence rate of the class of the level observed in public schools (19.2 vs 12.4 teachers is on average 23.5 percent, or just under one percent, respectively). A significantly higher absence teacher in four is absent from their classroom (Table rate from school is also observed in rural schools 4.2 and Figure 4.3). The highest level of classroom with a gap of about 4 percentage points with urban absence is observed in MoEC schools, with slightly schools (19.8 vs 16 percent, respectively). more than one in four teachers not in class teaching Methodological Note School absence rate is measured as the share of teachers who are absent from school at the time of an unannounced visit. It is measured in the following way: during the first announced visit, a maximum of 10 teachers are randomly selected from the list of all teachers (excludes volunteer and part-time teachers) who are on the school roster. The whereabouts of these 10 teachers is then verified in the second unannounced visit. Teachers found to be anywhere on the school premises are marked as present. Classroom absence rate during the unannounced visit. The level of class Even though a teacher may be in school, they do not absence rate in MoEC schools is 2 percentage points necessarily provide classroom instruction during higher than that observed in MoRA Islamic schools, this time. The classroom absence rate indicator aims though the sample of MoEC schools is not fully to measure this situation. A teacher is considered representative at the national level. These absence absent from class if they are not at school or, if at rates are very similar to rates found in a 2019 World school, they are not present in their classroom with Bank survey of West Kalimantan and East Nusa their students. Teachers who were not on their shift to Tenggara.34 teach are excluded from the absence rate calculation. FIGURe 4.3. Absence rates by sectors, status and location 30 24 24 26 24 25 25 19 19 19 20 19 20 20 20 16 15 15 12 11 10 5 0 All MoRA MoEC MoRA Public Private Urban Rural Islamic Non-Islamic School Absence Rate Class Absence Rate 34 Primary Education in Remote Indonesia: Survey Results from West Kalimantan and East Nusa Tenggara. (World Bank 2019) reports average teacher absence rates from class to be 23 percent. 50 Service Delivery Indicator Survey Indonesia 2020 Methodological Note Classroom absence rate is measured as the share of teachers not in the classroom at the time of an unannounced visit. The indicator is constructed in the same way as the school absence rate indicator, with the exception that the numerator now is the number of teachers who are either absent from school, or present at school but absent from the classroom. While about one in five teachers were absent from There is also a distribution in teacher absence from their classroom on the day of the unannounced visit class rates. in urban schools, about one in four teachers was absent in rural schools. Absence from class is striking, with 19.8 percent of MoRA Islamic schools Private schools presented significantly higher with an absence from class rate of 40 absence rates, at 23.7 percent, compared with public schools at 20.3 percent. percent or higher. Given these rates in almost one-fifth of schools, it is Absence rate distribution crucial to understand why teachers Some schools have far higher teacher absence rate do not come to school or to class, than others. Table 4.3. provides information on the and to put in place incentives as well distribution of teacher absences; some schools have as support to reduce absence so that much higher absence rates than others. children in these schools can learn. About one-third of schools in Indonesia have a teacher absence rate from school of above 20 percent. table 4.3. Distribution of teacher absence rates Distribution of Absence Rates Distribution of Absence Rates from Class from School Less than 20% More than Less than 20% More than 20% to 40% 40% 20% to 40% 40% MoRA Islamic 68.3 17.8 13.9 59.4 20.8 19.8 MoEC 66.3 17.4 16.3 50 24.4 25.5 MoRA non-Islamic 90 0 10 80 10 10 Public 69.7 24.6 5.7 50.2 36.2 13.6 Private 68.2 17.3 14.5 60.0 19.6 20.4 Urban 70.1 18.9 11 63.4 18.8 17.8 Rural 67.5 17.4 15.1 57.7 21.6 20.7 Note: Weighted means using sampling weights. The small sample size of MoEC and MoRA non-Islamic schools surveyed precludes nationally representative results for these categories. Service Delivery Indicator Survey Indonesia 2020 51 chapter 4. results Where were the teachers during unannounced Characteristics of absent teachers and reasons for visits? absence Figure 4.4 shows the location of teachers during the Teacher absence is associated with various teacher unannounced visit, as well as the proportion giving characteristics. Table 4.4 presents the level of school lessons, by comparing public and private schools, and class absence according to characteristics such and rural and urban schools. These figures show a as gender, position at school, and birth location. more detailed view of where the teachers were at the time of the unannounced visit. In private schools, 68 Civil servant teachers have a 2-percentage-point out of 100 teachers were observed to be teaching in lower rate of absence from school than their non- the classroom and two were teaching outdoors, six civil servant colleagues. However, no difference is were in a classroom but doing something unrelated observed between these employment statuses with to teaching, five were in the school but not in the regard to overall classroom absence rate (about 23 classroom, and the remaining 19 were not in school percent). This may be because one component of at all. In contrast, 12 public school teachers were not civil servant salary (Tunjangan Kinerja) is conditional on school premises and 70 were teaching in their on attendance. Teacher civil-servant presence at classrooms and three were teaching outdoors. The the school may be more closely monitored as a contrast is not as stark between urban and rural result of this conditionality. While this could be an schools, but there is clearly more teaching in urban indication of the positive impact of this conditional schools. FIGURe 4.4. Location and activity of the teachers during unannounced visits PUBLIC PRIVATE 12% 19% 8% 3% 5% 6% 2% 70% 6% 68% RURAL URBAN 20% 16% 4% 5% 4% 2% 8% 6% 69% 68% In Classroom In Classroom In School In School Absent - Teaching - not Teaching - Teaching Outdoor - not in from School the Classroom 52 Service Delivery Indicator Survey Indonesia 2020 payment, the fact that these teachers are no more likely to be in the classroom teaching means it is not The main reason principals gave for teachers’ helpful for improving student learning. With regard absence was illness. During the survey, school to teachers’ place of birth, teachers born in the principals were asked about the reasons for teachers’ district where the school is located are significantly absence. Some of these reasons were considered more frequently absent from school and from the to be justified, while others were not. The reasons classroom. This seems likely be related to family and for teachers’ absence given by the principals are, social responsibilities that teachers place above their primarily, illness (18 percent of the total absence), professional duties. maternity leave (13 percent), and training outside the school (6 percent), as well as various other School principals have lower rates of school absence approved reasons (36 percent). Among the other and class absence than teachers. Principals present approved reasons include accompanying students to somewhat lower rates of school absences and class competition, leave, and praying. Overall, 17 percent absences (when they have to serve as a substitute of the reasons for absences could not be explained teacher) than teachers, but male principals are more by the principal (“Other” and “Don’t know”) (Figure than twice as likely to be absent from school as female 4.5). While the majority of the absences appear to be principals (19.2 vs 8.1 percent, respectively), and justified, an educational system that permits such a present a 6-percentage-point higher absence rate high rate of teacher absence is less likely to deliver on from classrooms (24.2 vs 18.2 percent, respectively). learning for students. table 4.4. Absence rates by gender, functions and origin School absence rate Class absence rate Robust (95% Robust (95% Percent standard confidence Percent standard confidence deviations interval) deviations interval) Teachers 18.2 0.9 16.6 20 22.8 0.9 21 24.7 Female 17.5 0.9 15.7 19.4 21.8 1 19.8 23.8 Male 19.6 1.5 16.6 22.5 25.8 1.7 22.5 29.1 Civil servant 16.5  1.2  14.1  18.9  23.2  1.4  20.4  25.9  Non civil servant 18.5  1.0  16.5  20.6  23.1 1.1 20.9  25.3  Born in the district 18.3 0.9 16.5 20.1 23.3 1 21.3 25.2 Not born in the district 17.7 1.7 14.4 21 22.3 1.8 18.7 25.9 Principal 15 2.2 10.7 19.2 21.9 2.5 17 26.9 Female 8.1 2.7 2.7 13.5 18.2 3.8 10.6 25.9 Male 19.2 3 13.2 25.2 24.2 3.3 17.7 30.7 Note: Weighted means at the individual level using sampling weights. Results for absence rate based on observations from 3,169 sampled teachers from 350 schools. Service Delivery Indicator Survey Indonesia 2020 53 chapter 4. results FIGURe 4.5. Reasons for teacher absence 40 36 35 30 25 18 20 15 13 9 10 8 7 6 3 5 0 0 Sick Maternity In Training Field Funeral Other Meeting Don’t Other /Paternity Trip Approved Outside Know Leave Absence School meeting outside the school, and another 36 percent The survey also reveals the absence of the principal had other reasons approved but not specified. About from school during the unannounced visit. Among 26 percent of the reasons for absences of the principal the 15 percent of principals absent the day of the could not be explained by the person in charge the unannounced visit to the school, 20 percent were day of the visit (“Other” and “Don’t know”) (Figure reported sick, 13 percent were either in training or 4.6.). FIGURe 4.6. Reasons for principal absence 40 36 35 30 25 20 20 20 15 9 10 6 3 4 5 2 0 Sick Maternity In Training Funeral Other Meeting Don’t Other /Paternity Approved Outside Know Leave Absence School 54 Service Delivery Indicator Survey Indonesia 2020 Teacher absence rates according to the presence of Since the principals’ absence was not approved or the principal unexplained almost half of the time, it seems unlikely The absence of the principal is strongly associated that this effect was driven primarily by school-wide with teacher absence. In line with principles of trainings or other official functions that would require accountability and observation, it is interesting to both teachers and the school director to be absent note that the absence of the principal is strongly simultaneously. associated with the absence of teachers in a school. As shown in Figure 4.7, High levels of unapproved teacher absences are associated with school principals’ absences. In the school absence rates of teachers schools where the principal was absent, we found depend on the presence of the that teachers were more than twice as likely to also be absent. This dramatic effect is observed in all principal at the school: when the types of schools. In urban schools, for instance, about principal is absent, the absence of one in 10 teachers (9 percent) was absent when the teachers from the school increases principal was at school compared with one teacher in dramatically from 18.7 to 45 percent. about 2.4 teachers (42 percent) when the principal was absent. Hence, the high rates of absence and high levels of unapproved absences appear to be highly correlated with the principals’ absence, and associated with a lack of accountability within the school. This is explored further through multi-varied analysis below. FIGURe 4.7. Teacher absence rates according to the presence of the principal 70% 67% 60% 46% 46% 50% 45% 45% 46% 42% 18.7% 37% 40% 30% 18.7% 19.3% 19.2% 19.8% 20% 15% 12.4% 16% 15% 13% 13% 11.2% 13% 8% 9% 10% 6% 0 All MoRA MoEC MoRA Public Private Urban Rural Islamic Non-Islamic Principal Present Principal Absent School Absence in School in School Rate Service Delivery Indicator Survey Indonesia 2020 55 chapter 4. results Methodological Note Time spent teaching per day reflects the typical time that teachers spend teaching on an average day. This indicator combines data from the staff roster module (used to measure absence rate), the classroom observation module, and reported teaching hours. The teaching time is adjusted for the time teachers are absent from the classroom, on average, and for the time the teacher teaches while in classroom, based on classroom observations. While inside the classroom, distinction is made between teaching and non-teaching activities. Teaching is defined very broadly, including actively interacting with students, correcting or grading students’ work, asking questions, testing, using the blackboard or having students working on a specific task, drilling or memorization. Non-teaching activities include working on private matters, maintaining discipline in class, or doing nothing and thus not ensuring that students are paying attention. Time spent teaching per day This expected average duration was then weighted The third indicator used to measure teacher effort by the proportion of teachers present at the school. is that of daily teaching time. The effective daily The intuition being that if 10 teachers were each teaching time indicator measures the time a teacher supposed to teach 4 hours and 35 minutes per day, teaches in a school on a typical day (See Table 4.2.). but 18.7 percent of them were not in school, the This indicator takes into account various factors that expected teaching time was reduced to 3 hours and affect teaching time, including teacher absence and 44 minutes (i.e., 275 minutes x 0.813). The second the use of teaching time for purposes other than step was to adjust the teaching time for absences teaching. To this end, the indicator takes into account from the classroom (i.e. teachers could be in school the officially planned daily teaching time, according without being teaching in class). The scheduled to the schedule reported by the school director, teaching time has once again been reduced to 3 weighted by teachers’ absence in class, as well as on hours and 35 minutes. The last step was to remove the observation of a Grade 4 classroom to measure the time teachers spent on non-teaching activities the actual time spent teaching in a day. Figure 4.8. in the classroom. Indeed, even when they are in the breaks down the steps for measuring this indicator classroom, teachers are not necessarily teaching. from the officially planned time to the actual time. The percentage of class spent on non-instructional activities was measured by observing two Grade About 18 percent of the time in a typical lesson was 4 class lessons in each school surveyed. About 18 spent on non-teaching activities. The first step in percent of the time in a typical lesson was spent calculating this indicator of actual daily teaching time on non-teaching activities. To account for this, our was to record the expected teaching time per day measure was multiplied by the proportion of a typical (Module 1) from school records, as reported by the course duration spent teaching. In our case, the school principal. The expected daily teaching time teaching time of 3 hours and 35 minutes has again is 4 hours and 35 minutes nationally in Indonesia. been reduced to 2 hours and 56 minutes (215 minutes x 0.817). 56 Service Delivery Indicator Survey Indonesia 2020 FIGURe 4.8. From Official to Actual Teaching Time 6.0 5.1 5.0 4.9 5.0 4.6 4.6 4.4 4.6 4.5 4.5 4.4 4.2 3.7 3.8 3.8 3.8 4.0 3.8 3.5 3.6 4.0 3.5 3.5 3.6 3.5 3.4 3.4 3.4 2.9 2.9 2.9 2.9 2.8 3.0 2.6 2.0 1.0 0 All MoRA MoEC MoRA Public Private Urban Rural Islamic Non-Islamic Expected Adjustment Adjustment Adjustment Teaching School Class Non Pedagogical Duration Absence Absence Activities level, years of experience, whether Grade 4 students receive 1 hour and 39 minutes they were born in the district), school less teaching per day than they should. Thus, across characteristics (school location, in- all types of schools, a Grade 4 student receives an average of 2 hours and 56 minutes of actual teaching frastructure, equipment, whether per day, which corresponds to 1 hour and 39 minutes the school is on a paved road) and less teaching per day than what is officially planned, management variables (principal according to what is reported by school principals absence, whether the teacher had (Table 4.2). The lowest effective teaching time per received at least one delayed day was observed in rural schools, at only 2 hours payment in the past year), as well as and 50 minutes, which was about 21 minutes less than district fixed effects. urban school on average. The main factor associated with teachers’ absence Factors associated with teacher effort is the performance of their school principal. Figure A multivariate analysis tries to understand the 4.8 shows selected significant estimates from the factors associated with teacher effort. To try to regressions between absence from school and from better understand the factors explaining the effort the classroom. The main factor associated with of teachers, Table D1 in Appendix D presents a teacher absence is school principal performance, multivariate analysis of the factors explaining teacher namely presence. We observe that teachers in absence from school and from the classroom. schools where the principal was absent had about 26 percent higher probability of being absent from The regressions control for teacher the school and classroom even after controlling for characteristics (e.g., gender, certifi- teacher and school characteristics, as well as external cation, type of contract, education management capability. Service Delivery Indicator Survey Indonesia 2020 57 chapter 4. results FIGURe 4.9. Correlates of Teacher Absence from School and Classroom Absence Principal NBR_Students_obs Salary Paved Road Female Teacher Other Religious Schools MoEC Schools -.2 0 .2 .4 -.2 0 .2 .4 Better paid teachers are absent less often. Having a paved road access to school also makes teacher absence more likely. The presence of a paved We observe that higher-paid teach- road access to school is associated with about a 5- ers show lower absence rates, as to 6-percentage-point higher probability of absence from school and classroom. Teachers in MoEC there is a significant negative associ- schools are associated with about a 6-percentage- ation between salary level and school point higher probability of absence from school and and classroom absence. classroom than teachers in MoRA Islamic schools. (see Appendix D1 for further details). However, we found that teachers’ civil servant status and certification were not significantly associated with absence, after controlling for teachers and school characteristics, as well as experience levels. Also, female teachers are significantly less absent from the classroom than their male colleagues, with about 4 percentage points lower probability of being absent from the classroom. 58 Service Delivery Indicator Survey Indonesia 2020 C. TEACHER COMPETENCE the classroom and linked to gains in student learning, including: (i) Structuring lessons by introducing topics Overview and learning outcomes at the start of the lesson, and The presence of teachers in class is not enough for reviewing them at the end; (ii) Assigning, reviewing, student to improve outcomes if they do not have or collecting homework during the class; (iii) Varying good pedagogical skills. Even if the teachers are the cognitive level of questions by mixing lower - present in class and use the teaching time allocated and higher - order questions; (iv) Creating a positive for learning activities as expected, they must master learning environment and providing substantive the subject they teach and have pedagogical skills feedback to students; and (v) Making use of lesson to be able to transmit their knowledge to students. plans. Identified by trained enumerators observing Previous research in Indonesia has shown this intuitive one full 60-minute lesson per teacher, teachers link clearly (de Ree, J. 2016). This section presents the overall performed worse than the SDI global average indicators used to measure teachers’ knowledge and for this measurement (Bold et al. 2017). their ability to teach, called “minimum knowledge”. Teacher Knowledge Fewer than 1 in 5 teachers surveyed meet the Teachers’ subject-matter knowledge is measured minimum subject knowledge requirements. Only by standardized tests. The proportion of teachers 15.6 percent of teachers scored higher than who have at least minimum levels of subject-matter 80 percent on the combined mathematics and knowledge is calculated from a standardized test Indonesian language tests, meaning that fewer than administered to teachers of Indonesian language and 1 in 5 teachers surveyed meet the minimum subject mathematics teaching to the student cohorts of 2019. knowledge requirements. In Indonesian, only 7.6 In addition, an Arabic language assessment was also percent of teachers reached the minimum knowledge given to Arabic language teachers in MoRA Islamic threshold, while in math, 31.9 percent demonstrated schools. minimum knowledge. Teachers are considered to have a minimum Almost none of the teachers assessed in the survey knowledge in the subject if they score over 80 managed to achieve a minimum knowledge score percent. The purpose of the teacher assessment is to of 80 percent on the pedagogy assessment. This examine whether teachers have the basic skills that highlights teachers’ poor performance in pedagogy, primary school students need in reading, writing and where the average score on the pedagogy exercise arithmetic in order to advance in school. Teachers was only 22 percent. Given that the pedagogy are considered to have a minimum knowledge in assessment required teachers to conduct tasks that the subject if they score at least 80 percent in the they are normally required to do as part of their assessment, which is interpreted as the minimum level teaching profession, such as prepare a lesson, assess of knowledge required for teachers to be effective in and evaluate student performance, the low level of this subject.35 proficiency demonstrated is surprising for a middle- income country such as Indonesia and can help In addition, assessments examine a teacher’s explain low levels of student achievement. mastery of academic content beyond the level that he/she teaches, as well as their mastery of teaching Indonesian teachers’ performance overall was skills. As a courtesy to teachers, the assessment is below the SDI global average for “good teaching designed as a correction of copies of a hypothetical practices”. Teachers were directly observed teaching student’s test. The language test was administered to in their classrooms, and almost none followed the teachers who teach Indonesian language, and similarly complete set of SDI “good teaching practices” in their for the other sections of the test (mathematics and observed lessons. These practices are observable in Arabic). The assessment, initially based on the Grade 35 This 80 percent threshold is the standard used in SDI surveys to determine the minimum knowledge expected of the teacher. The sensitivity of the results to the variation of this cut-off is examined later in this section. Service Delivery Indicator Survey Indonesia 2020 59 chapter 4. results box 4.3. Assessment of teachers’ knowledge The SDI survey included an assessment of teachers’ knowledge of language and mathematics. Teacher assessment was administered collectively during the teachers’ lunch break to minimize the impact on teaching time. Minimum knowledge is measured as the percentage of teachers who can master the curriculum they taught. It is based on a mathematics and language tests covering the primary curriculum administered at the school level and is calculated as the percentage of teachers who score more than 80 percent on the language and mathematics portion of the test. The test is given to all mathematics or language teachers who taught Grade 3 last year or Grade 4 in the year the survey was conducted. Teachers were assessed for their mastery of the primary school level mathematics, Indonesian language curriculum and Arabic for those in MoRA Islamic schools on the one hand, and teaching skills on the other. To test for teacher knowledge in mathematics, Indonesian language, and Arabic, teachers were given an indirect test. The test involved asking teachers to mark standardized tasks done by a student and suggest a correct answer whenever they indicated that the student had given an incorrect answer. Thus, they were assessed on their ability to identify and suggest a correct answer. The student tasks that teachers were asked to mark covered various topics, giving a complete picture of the assessed teachers’ mastery of the curriculum. The test for teaching skills asked teachers to perform tasks they are expected to do to enhance student learning, such as preparing a lesson plan, evaluating students, and tracking progress in student performance. For example, teachers were presented with a short story about accidents and asked to prepare a lesson on the reasons why road accidents happen and the consequences. Among other things, they were then asked to: (i) specify the learning objectives of the lesson; (ii) suggest questions they would ask to determine that students understood the lesson and can apply what they have learned; and (iii) list points for two sides of an argument for a group activity. To test their ability to compare and evaluate student performance, teachers were presented with compositions written by two students and asked to identify the strengths and weaknesses of each student. 4 primary curriculum of 12 countries (Johnson et al. more advanced questions in the different subjects, as 2012), was previously validated with the curriculum of well as a pedagogy section. As observed in Table 4.5, primary teachers in Indonesia. minimum content knowledge among teachers was very low. Only 15.6 percent of teachers scored higher Minimum knowledge than 80 percent on the combined mathematics and The minimum knowledge indicator is calculated Indonesian language test. as the percentage of teachers who obtain more than 80 percent on the Indonesian language and mathematics test. The assessment also contained 60 Service Delivery Indicator Survey Indonesia 2020 table 4.5. Teacher assessment Minimum knowledge MoRA MoEC MoRA Public Private Urban Rural (80% threshold) Islamic non- (% teachers) Islamic Language and math : 80% 15.6 13.9 43.3 18.6 15.2 17.8 14.4 Language : 80% 7.6 9.2 26.7 9.6 7.3 8.2 7.2 Maths : 80% 31.9 34.3 53.3 39.7 30.9 37.3 29.0 Average score (%) Language and Math 64.2 64.5 76.1 65.8 64.0 66.8 62.8 Language 61.0 62.4 73.6 62.0 60.9 63.2 59.9 Math 67.6 66.6 78.7 70.0 67.1 70.4 65.7 Pedagogy 23.2 26.3 33.8 24.6 23.0 24.9 22.3 Language, math and pedagogy 39.6 41.6 50.7 41.1 39.4 41.7 38.5 Note: Weighted means using sampling weights. Results for minimum knowledge based on observations from 1,469 sampled teachers from 350 schools. Results for Indonesian language based on observations from 1,837 sampled teachers. Results for Arabic based on observations from 1,469 sampled teachers. Results for mathematics based on observations from 1,837 sampled teachers. Results for pedagogy based on observations from 1,819 sampled teachers. The small sample size of MoEC and MoRA non-Islamic schools surveyed precludes nationally representative results for these categories. Test scores of questions. If we look more specifically at the scores Test scores measure the proportion of questions that obtained by teachers in the different subjects of the teachers were able to answer. Table 4.5 and Figure assessment, the results in mathematics are slightly 4.10 also show the scores for these tests. Unlike the higher than those observed in Indonesian language minimum knowledge indicator, which measures the (67 vs 61 percent), as shown in Table 4.6. Considering proportion of teachers scoring above the applicable math and Indonesian language together, teachers threshold, this score simply measures the proportion correctly answered on average 45 out of 71 questions of questions that teachers as a whole were able to (64 percent) on the entire test. Teachers consistently answer, expressed as a proportion of the total number obtained much lower scores in pedagogy (23.2 percent). FIGURe 4.10. Average teacher scores in math, Indonesian language, Arabic and pedagogy 80 75 72 73 67 67 70 67 71 70 66 70 70 61 62 63 62 61 60 60 50 Percent 40 30 25 25 23 26 23 22 20 10 0 MoRA MoEC Public Private Urban Rural Islamic Math Indonesian language Arabic Pedagogy Service Delivery Indicator Survey Indonesia 2020 61 chapter 4. results Detailed test scores Arabic Mathematics The test in Arabic was similar to the one in Indonesian Private and rural school teachers score far lower on language. In addition to the Indonesian language the math assessment. Table C1 in the Appendix shows evaluation, teachers working in MoRA Islamic schools the performance of teachers on the mathematics test, were also assessed in Arabic. The test in Arabic was together with a detailed analysis of specific questions. similar to the one in Indonesian language. Table As mentioned, private and rural school teachers C3 in the Appendix presents the average scores in present much lower scores in the mathematics Arabic, as well as a detailed analysis according to the assessment. different sections of the test. The results for the first and last sections of the test in Arabic grammar and Overall, more than 20 percent of composition39 were relatively high, while the average teachers were unable to subtract reached 75.4 and 78.8 percent, respectively, of correct answers. However, a somewhat lower performance two-digit numbers, while almost 40 was recorded in the Cloze task (68.9 percent). percent of teachers could not add numbers with decimals. Pedagogy The overall level of pedagogical knowledge was Almost 45 percent of teachers could not read a watch very low across all types of schools. Although the and one-quarter could not interpret a Venn diagram. pedagogy component of the teacher evaluation is not Only six out of 10 teachers could solve algebraic taken into account in the calculation of the minimum equations with one variable. knowledge indicator, the results in pedagogy provide information on the capacity of teachers to prepare Indonesian language and deliver their lessons, as well as their ability to Teachers performed worst in composition, with an evaluate their students (see Table C4 in the Appendix average score of only 40 percent. Table C2 in the for the performance of teachers on the pedagogy Appendix presents the average scores in Indonesian test and sub-components). The overall level of language, as well as a detailed analysis according to pedagogical knowledge was very low across all types the different sections of the Indonesian language test. of schools, with an overall score of 23 percent. There Results for the first section of the test in grammar36 are were relatively small differences between sectors, significantly higher than for the rest of the assessment status and location of schools.40 in Indonesia language, while the average reaches 93.6 percent across the entire sample. Scores in the Cloze Teachers are ill-prepared both in pedagogy and task (’blank test’),37 which assesses vocabulary and subject content knowledge. The weak pedagogical reading comprehension, are somewhat lower with an results mixed with the low scores obtained in the average across all teachers tested of 79.2 percent. language and mathematics sections indicate that The third section of the Indonesian language test, teachers have limited content knowledge and that, which consists of a composition exercise, presents even for what they do know, they are not well the lowest performance with an average of only 40 prepared to teach it. percent.38 36 The grammar section asked teachers to complete the sentences by choosing the correct term(s) from those presented. 37 The Cloze task consisted of a short story where some words were missing and the teacher had to correct the answers given by the student, indicating if they were correct and if not, to write the correct answers. 38 In this exercise, the teacher had to correct a letter composed by a student and identify errors in grammar, punctuation, spelling, syntax and salutation and indicate the corrections. The text to be corrected contained 46 errors and on average the teachers identified and corrected less than 18 errors. 39 As for the Bahasa evaluation, the composition section consisted of the teacher correcting a letter written by a student. The teacher was called upon to identify errors in grammar, punctuation, spelling, syntax and salutation and indicate the appropriate corrections. 40 The pedagogical test consisted of three sections which were designed to take into account all the skills that teachers should systematically use in teaching. The first section asked teachers to prepare a lesson plan on road accidents based on a simple informational text. The average mark for this exercise was 22.6 percent. The second exercise asked teachers to assess and compare student writing based on two form letters. The average mark for this exercise was 25.4 percent. The final exercise asked teachers to examine the test scores of 10 students, compile them, and come up with some conclusions about learning trends. The score for this task was 21.6 percent (see Table C4 in the Appendix for details). 62 Service Delivery Indicator Survey Indonesia 2020 box 4.4. Good Teaching Practices Across all observed MoRA teachers in the SDI survey, only 1 percent were seen to follow the complete set of “good teaching practices” in their observed lessons. “Good teaching practices” can be measured across several areas that are observable in the classroom and linked to gains in student learning, including: (i) Structuring lessons by introducing topics and learning outcomes at the start of the lesson, and reviewing them at the end; (ii) Assigning, reviewing, or collecting homework during the class; (iii) Varying the cognitive level of questions by mixing lower - and higher - order questions; (iv) Creating a positive learning environment and providing substantive feedback to students; and (v) Making use of lesson plans (Mujis et al. 2014). Identified by trained enumerators observing one full 60-minute lesson per teacher, the following describes a breakdown of teacher practices across the aforementioned areas observed in the survey: i. Lesson structure: Less than one-third (30 percent) of all teachers both introduced and summarized the lesson to their classrooms. Evidence reveals that learning is maximized when teachers structure lessons at the beginning with overviews of objectives, outlining content to be covered, calling attention to the main ideas, and reviewing these ideas at the end. These elements help students to understand information gained in the classroom as an integrated whole, while also understanding the relationships between various parts of the lesson (Mujis et al., 2014). ii. Homework: Only 21 percent of teachers assigned homework to the class during the observation period, and roughly 17 percent of the teachers reviewed or collected homework from the class. Homework, as a part of continuous assessment, allows teachers to collect data on student knowledge and skills, and to analyze and identify student needs. iii. Questions: Only 39 percent of observed teachers asked a mixture of higher- and lower-order questions to check for student understanding. A combination of such questions is important, as it requires students to recall information that was covered in the lesson, demonstrate understanding of what they learned in the lesson, and apply the information to new topics, requiring them to use their creativity and imagination. iv. Learning environment: Only 11 percent of classrooms were rated as having a positive learning environment. A positive learning environment is observed if the teacher refers to their students by name (86 percent), gives feedback with praise at least once during the lesson (72 percent), corrects mistakes at least once (82 percent), does not use any physically abuse (98 percent) to control students and does not scold students when they make mistakes (29 percent). Positive classroom learning environments are indicative of strong teaching skills and high trust in the classroom, and thus make overall learning more conducive for students. v. Lesson Plan: Only 37 percent of teachers were observed to have a lesson plan. Lesson plans are a key component to strong teaching practices, as having a lesson plan provides teachers with structure and goal-setting to achieve targeted learning outcomes. Service Delivery Indicator Survey Indonesia 2020 63 chapter 4. results box 4.4. Continued While only 1 percent of MoRA teachers were seen to follow the complete set of “good teaching practices,” key differences were identified across teachers of various employment and certification status, education level, and gender. For example, Pegawai Negeri Sipil (PNS), or civil servant teachers, were 18 percentage points more likely to have a lesson plan available in the classroom than non-PNS teachers; PNS teachers were also roughly 18 percentage points more likely to assign, review or collect homework from students. Regarding education level, university- graduated teachers (bachelor’s degree or above) were significantly more likely to exhibit good teaching practices, compared with non-university graduated teachers—university degree teachers were 25 percentage points more likely to ask a mixture of higher- and lower-order questions. This group of teachers also outperformed their non-degree holding counterparts in terms of introducing and summarizing lessons, assigning, reviewing or collecting homework, and creating a positive learning environment. Across gender, male teachers were more likely to introduce and summarize lessons, ask a mixture of higher - and lower - order questions, and create a positive learning environment in the classroom, while female teachers were more likely than male teachers to make use of lesson plans in the class. Overall, most teachers in MoRA schools are not utilizing essential pedagogic practices to transfer knowledge to students. Beyond understanding their subject matter, effective teachers should also understand and engage in teaching practices that allow them to ensure student learning. While some teachers do exhibit “good teaching practices” across the five measured areas of lesson structuring, assigning and collecting homework, asking higher- and lower-order questions to check for understanding, creating positive learning environments, and lesson planning, only 1 percent of MoRA teachers were observed implementing this complete set of good practices, while 17 percent were observed practicing at least three of them. However, as further explained in Box 4.5 on student perception of teachers, the survey found that most students actually had positive perceptions of teachers’ efficacy and pedagogic skills, especially in the case of emotional support; this may suggest that, while most teachers are not utilizing essential pedagogic practices, students still trust and feel comfortable in their learning environments. Sensitivity of minimal knowledge passing threshold of around 45 percent. Below this Only 10 percent of teachers score above the 45 percent threshold, the proportion of teachers minimum passing level in pedagogy. To analyze performing above the threshold increases slightly the sensitivity of the minimum knowledge indicator more in Indonesian language and mathematics. At according to the threshold used to establish the this threshold, about 85 percent of teachers obtained minimum knowledge indicator, Figure 4.10 shows for a score of 45 percent or more in the evaluation in each subject the proportion of teachers who scored all subjects other than pedagogy. In pedagogy, the above the pass mark for passing grades between 0 proportion of teachers obtaining results above a 45 and 100 percent. We observe that teachers performed percent threshold was only about 10 percent. better in Arabic relative to the other subjects up to a 64 Service Delivery Indicator Survey Indonesia 2020 FIGURe 4.11. Sensitivity of the minimal knowledge indicator to passing grade Proportion of Teachers Above Passing Grade 1.00 0.80 0.60 0.40 0.20 0.00 100 90 80 70 60 50 40 30 20 10 Passing Grade Math Indonesian Language Arabic Pedagogy Determinants of teacher knowledge Arabic, with older teachers with more than 30 years of Civil servant teachers have slightly higher results in experience showing the lowest scores in all subjects, mathematics and pedagogy compared with non- including pedagogy. This may be partly a result of civil servant teachers. Table 4.6 presents the results improvements in teacher selection processes over of teacher assessments by level of education and the last forty years. In Arabic, the highest score was gender, as well as the level of statistical significance of obtained for teachers with between 20 and 30 years a test of the differences between the means of these of experience, while teachers with less than 10 years subgroups. With regards to years of education, scores of experience obtained the second-highest scores monotonically increased with teacher education among their peers. levels in all subjects which is expected, with teachers with a master’s degree presenting the highest scores, Teachers consistently achieve low scores in while those with only a secondary education or less pedagogy. As noted above, teachers correctly presented the lowest scores in the evaluations. Small answered on average 45 out of 71 questions (64 differences are observed in teacher’s performance percent) on the test when math and Indonesian on the evaluation between civil servant and non- language are considered together. Teachers civil servant teachers, with slightly higher results for consistently obtained much lower scores in pedagogy civil servant teachers in mathematics and pedagogy, (23.2 percent). while the latter present slightly higher results in mathematics and Arabic. Female teachers obtained significantly better results than their male colleagues in all evaluation Regarding experience, teachers with less than 10 subjects. The differences were about 4 percentage years of experience presented higher scores in all points in favor of women teachers relative to their subjects except Arabic. Teacher scores monotonically male colleagues in mathematics, Indonesian language decreased with experience in all subjects except and pedagogy, and 3 percentage points in Arabic. Service Delivery Indicator Survey Indonesia 2020 65 chapter 4. results table 4.6. Teacher assessment results by gender, employment status and level of education (percentage) Indonesian Mathematics Arabic Pedagogy Language Female 68.9 62.8 72.9 24.8 Male 65.1 58.4 69.7 20.8 Civil servant 83.1 52.0 85.9 27.0 Non civil servant 65.8 61.0 71.1 22.7 Experience (years) Less Than 10 68.8 63.8 71.7 24.9 10 to 20 67.4 60.9 70.9 23.5 20 to 30 66.1 58.1 74.1 21.1 30 and More 62.0 54.9 69.7 16.8 Senior Secondary Diploma and Less 59.6 59.6 65.7 16.9 Diploma D1, D2, D3 64.1 63.4 69.9 22.9 D4/S1 Bachelor Degree 68.4 61.6 72.3 23.9 S2 Master Degree 73.9 61.4 77.4 29.4 Note: Weighted means using sampling weights based on observations from 1,832 sampled teachers from 350 schools, including 635 male and 1,197 female teachers. Results for experience include 517 sampled teachers with experience less than 10 years; 812 with experience 10 to 20 years, 334 with experience 20 to 30 years, and 174 with more than 30 years. Education based on a sample of 116 teachers with level of senior secondary and less, 46 with Diploma D1, D2, D3; 1600 with education D4/S1 bachelor, and 74 with education level master’s level. Econometric analysis Teachers with higher levels of education were To better understand the factors associated with associated higher teacher test scores in mathematics. poor teacher performance in their subject knowledge Teachers with D4/S1 level equivalent to a Bachelor’s evaluations, we estimate a multivariate model whose degree were associated with about 6 percentage dependent variable is the teachers’ performance points higher scores compared with teachers with in the different subjects of the evaluation. The only a secondary diploma or less, and those with a regressions control for teacher characteristics (e.g., master’s degree, 11 percentage points higher. We gender, certification, civil servant status, education observe that female teachers are associated with level, years of experience, whether they were born significantly higher teacher test scores than their male in the district), teacher pedagogical practices (e.g., if colleagues. Also, teachers with longer experience the teacher introduces lessons, summarizes lessons, presented significantly lower evaluation results in review homework in class, and if the teacher uses all subjects. With regard to teaching practices, we local languages); and school characteristics (types find unexpectedly that teachers who tend to give of schools, school location, as well as district fixed homework to their students are associated with effects). Regression results are presented in Table D2 significantly lower teacher test results in all subjects in the Appendix. Figure 4.12 below shows selected (except mathematics). Regardless of the specific significant estimates from the regressions of teachers’ association, almost all teachers did very poorly on the Indonesian language evaluation and mathematics assessment. evaluations. 66 Service Delivery Indicator Survey Indonesia 2020 FIGURe 4.12. Correlates of Teacher Evaluations in Language and Mathematics D4/S1 Bachelor S2 Master Experience More _30 Years Experience_20_30 Years Experience_10_20 Years Female Teacher_Homework Other Religious School MoEC School Public Rural -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 Teachers in public and urban schools were associated favor of public schools. Finally, compared with MoRA with significantly higher knowledge results than Islamic schools, MoEC teachers showed significantly those in private schools. We observe that teachers lower scores in all subjects (2 percentage points in public and urban schools were associated with in Indonesian language and 10 percentage points significantly higher knowledge results than those in in mathematics). Teacher certification and civil private schools of about 3 percentage points after servant status did not have a significant effect on controlling for teacher and school characteristics. In teacher scores in the evaluations (see Table D2 in the mathematics, the estimated gap between public and Appendix for details). private school teachers was 7 percentage points in Service Delivery Indicator Survey Indonesia 2020 67 chapter 4. results box 4.5. Student perceptions of teachers Overall, Grade 4 students have positive perceptions of their teachers’ efficacy and pedagogic skills, especially in the case of emotional support. Measured by questioning students on their perceptions regarding their teachers’ ability to understand and support the students, manage the classroom, engage the students, provide opportunities for improved learning, and summarize the lesson, these insights provide valuable information into students’ beliefs about the classroom environment that teachers create.1 This is linked to the level of trust and comfort that students have with their teachers, which can impact learning outcomes. However, positive student perceptions do not guarantee strong learning outcomes. As seen in Box 4.4 on teaching practices, teachers must also engage in “good teaching practices,” and only 1 percent of teachers in the study were found to do so. Below is a brief summary of students’ perceptions of their teachers in various areas: i. Nurturing ability: Over 80 percent of the students said that their teacher encouraged them to do their best; and when they had a need for help in the lesson, or had questions, their teachers were supportive. However, in terms of teachers knowing when something was bothering the students, there is an equal split among the student responses, which may signal non - school - related care (i.e., when the problems that students face are not academic, their teachers might be less likely to either perceive their concerns, or take active measures to provide care and help in these matters). ii. Class management ability: Most students felt that everyone in the class knew what they were supposed to be doing and learning in the lesson, and that students mostly behaved the way teachers wanted them to behave. Classroom productivity was also perceived to be in the mid - to - high range by most students. iii. Aligning instruction with learning ability: Around 80 percent of students felt that their teachers mostly or always explained things in an orderly way and tried different techniques to explain things that were unclear to them, highlighting students’ perception that MoRA teachers had good instructional capacity and practices. box figure 4.5.1. Student Perceptions of Aligning Instruction with Learning Ability of MoRA Teachers If you don’t understand something, my... This class is neat - everything has a place and... My teacher knows when the class... I understand what I am supposed to be... My teacher has several good ways to explain... My teacher explain difficult things clearly... In this class, we learn to correct our mistakes My teacher explains things in very orderly ways 0% 20% 40% 60% 80% 100% No, never Mostly not Maybe, sometimes Mostly yes Yes, always All MoRA Schools Avg. score: 4. 04 out of 5 68 Service Delivery Indicator Survey Indonesia 2020 box 4.5. Continued iv. Teachers’ expectations for students: Overall, students mostly believe that teachers set high expectations for their students. When asked if their teacher pushed them to work hard, 83 percent of students responded ‘mostly yes’ or ‘always. When asked if their teacher pushed them to think hard about things they read in the lesson, 76 percent of students said that this statement was mostly or always true. v. Ability to engage with students: Overall, students responded positively. Only 20 percent of the students said that they mostly or never enjoyed schoolwork, while roughly 50 percent of the students said that schoolwork was always interesting. vi. Ability to check in with students: Students reported that teachers often check for student understanding by asking them if they understood what was being taught and if they were following the progression of the lesson, though teachers were less likely to ask students to share their thoughts and explain to the class what they thought. For example, only 61 percent of students said their teacher either mostly or always “wants me to explain my answers – why I think what I think.” vii. Ability to summarize and review lessons: On average, teachers scored the highest on this measure of socioemotional skill. Students reported that teachers tended to check students’ work and provide feedback, in addition to summarizing lessons daily mostly or always (84 and 85 percent, respectively). box figure 4.5.2. Student Perceptions of The Ability of MoRA Teachers to Check in with Students My teacher wants me to explain my answers - why I think what I think Students speak up and share their ideas about class work My teacher wants us to share our thoughts My teacher tell us what we are learning and why My teacher checks to make sure we understand what s/he is teaching My teacher asks questions to be sure we are following along when s/he is teaching When s/he is teaching us, my teacher ask us whether we understand 0% 20% 40% 60% 80% 100% No, never Mostly not Maybe, sometimes Mostly yes Yes, always All MoRA Schools Avg. score: 4. 04 out of 5 Service Delivery Indicator Survey Indonesia 2020 69 chapter 4. results D. SCHOOL INPUTS Overall, the observed schools did very poorly on measures of inputs, despite a budgeting system that The availability of inputs at school is measured is largely input-based.41 using four main indicators. These are: the minimum availability of teaching and learning materials; the Over 40% of schools in the sample minimum availability of infrastructure; the proportion of students with textbooks; and the observed student- did not have minimal infrastructure, to-teacher ratio. These indicators are constructed defined as the availability of functional using data collected through a visual inspection toilets for the students and sufficient of a typical Grade 4 classroom and the equipment light in the classroom, while fewer available in each of the schools. In each school than half of the students observed on visited, two Grade 4 classes were observed. Table the day of the data collection visit had 4.7 presents the results for the various indicators and sub-indicators according to the strata. textbooks. Language textbooks were the rarest, with fewer than one-third of students having one. In addition, about one in five Grade 4 students did not have an exercise book the day of the observation. Methodological Note Minimum equipment availability is a binary indicator capturing the availability of: (i) functioning blackboard and chalk; and (ii) pens, pencils and exercise books in Grade 4 classrooms. In one randomly selected Grade 4 classroom in each school, the enumerator assessed if there was a functioning blackboard by looking at whether text written on the blackboard could be read at the front and back of the classroom, and whether there was chalk available to write on the blackboard. We consider that the classroom met the minimum requirement of pens, pencils and exercise books if both the share of students with pen or pencils and the share of students with exercise books were above 90 percent. 41 See ‘How Do Subnational Governments Spend their Education Money in Indonesia,’ Afkar, Nomura and Luque, World Bank 2020. 70 Service Delivery Indicator Survey Indonesia 2020 table 4.7. Minimum teaching equipment availability MoRA MoRA All MoEC non- Public Private Urban Rural Islamic Islamic Minimum teaching equipment availability Equipment index (% of classrooms) 70.6 78.9 79.9 70.0 74.2 70.3 80.5 66.5 Students with pencils (% Students) 99.0 99.0 99.0 100 99.6 98.9 99.7 98.7 Students with exercise book (% Students) 80.9 80.9 89.3 85.0 79.1 81.1 90.2 77.1 Classroom with board (% classrooms) 99.3 99.3 100 85.0 100 99.3 100 99.1 Classroom with chalk (% classrooms) 95.7 95.7 98.9 85.0 99.4 95.4 96.5 95.4 Contrast to read the board (% classrooms) 93.8 93.8 96.0 100 98.2 93.4 95.2 93.2 Note: Weighted means using sampling weights. Results for equipment index based on observations from 350 sampled schools. Results for students with pencils based on observations from 700 sampled classrooms. The small sample size of MoEC and MoRA non-Islamic schools surveyed precludes nationally representative results for these categories. One-third of observed classrooms did not have chalk, and students with pens. pencils and notebooks. minimum teaching materials available. As shown in Note that the SDI survey measures whether all the Table 4.7, close to one-third of all observed Grade teaching materials are available simultaneously in 4 classrooms did not have the minimum teaching the classroom, rather than measuring the individual materials for students. The main constraint was presence of each of the elements or the average the availability of exercise books. About one in five presence of these elements. Grade 4 students did not have an exercise book the day of the observation. The SDI survey measures Minimum infrastructure availability whether teaching materials are available in the Grade Table 4.8 presents the average minimum availability 4 classes observed. Minimum instructional material is of infrastructure in schools, as well as the averages defined as a classroom with a blackboard and usable for each of the sub-indicators. Service Delivery Indicator Survey Indonesia 2020 71 chapter 4. results table 4.8. Minimum Infrastructure Availability MoRA MoRA MoEC non- Public Private Urban Rural Islamic Islamic Functioning school infrastructure (% schools) 55.9 63.9 90 70.8 54.7 68.9 50.6 Visibility (% classrooms) 89.3 91.4 100 97.3 88.7 89.3 89.3 Toilet accessible (% schools) 73.6 81.4 90 82.4 72.9 82.2 70.1 Toilet clean (% schools) 79.5 76.7 100 87.2 78.9 90.3 75.0 Toilet private (% schools) 95.5 97.7 100 97.9 95.4 97.8 94.6 Water & soap (% schools) 49.7 55.8 80 63.8 48.6 65 43.5 Observed Student - teacher ratio 17.0 21.1 8.5 22.8 16.6 21.5 15.2 Textbook availability (% students) 47.3 71.1 91.7 57.8 46.6 60.0 42.3 Textbook availability (mathematics) 44.5 56.5 100 50.0 44.2 64.0 38.0 (% students) Textbook availability (language) (% students) 32.9 79.4 75.0 36.7 32.3 60.7 19.8 Note: Weighted means using sampling weights. Results for functioning school infrastructure based on observations from 350 sampled schools. Results for textbook based on observations from 700 sampled classroom from 350 schools. The small sample size of MoEC and MoRA non-Islamic schools surveyed precludes nationally representative results for these categories. Over 40% of schools of all types lack minimum One-fourth of MoRA Islamic schools had no infrastructure, defined as the availability of functioning toilets available for students, in both functional toilets for the students and sufficient urban and rural areas, especially in rural areas (17.8 light in the classroom. This proportion is higher percent in urban area vs 29.1 percent in rural area). in rural areas, where half of the schools of all types On average, the other establishments had about 3.5 do not have minimal infrastructure. Public schools toilets accessible to both students and teachers. In tend to have better infrastructure than non-public rural area schools, there was about one toilet for schools (70.8 vs 54.7 percent have minimum school 69 students, while in urban area schools this figure infrastructure, respectively) went up to 92 students per bathroom. This ratio is regulated in Permendikbud No. 24/2007, which The main constraint for infrastructure is access to states that one toilet should be available for every functional sanitation facilities, i.e., accessible toilets, 60 male students and one toilet for every 50 female clean and ensuring privacy (enclosed space and students.42 Only half of the schools contained proper separated by gender). Sufficiently clean facilities handwashing facilities, i.e., the presence of water and are only available in about 80 percent of schools, soap for the bathrooms. This is an important finding and in rural area in about 75 percent of schools. linked to health and safety, particularly during the COVID-19 pandemic. 42 MoEC Regulation No. 24/2007 on school infrastructure and facilities regulates that toilet requirements in schools: one toilet for every 60 boys, one toilet for every 50 girls. Size of the toilet: 1x2 m2, presence of water, and the toilet should be with walls, roof, and a lock. 72 Service Delivery Indicator Survey Indonesia 2020 Proportion of students with textbooks Less than half of the students had textbooks for the relevant subject. The SDI survey also measures the availability of mathematics, Indonesian language and Arabic textbooks in a typical Grade 4 class. Table 4.8 shows the average proportion of students with a textbook on average for each of the subjects observed according to the setting and the sector. Fewer than half of the students observed on the day of the data collection visit had textbooks for the subject in which they were observed. The proportion of students without textbooks was slightly lower in MoEC schools, where about one-third of the students did not have a textbook for the subject observed. Language textbooks were the rarest, with fewer than one-third of students having one. This scarcity was the most pronounced in rural schools, where fewer than one in five students had a language textbook on average. This appears to highlight a basic and fundamental breakdown in education service delivery. Student to teacher ratio The student-to-teacher ratio in the observed Grade 4 classes was 17 nationally (Table 4.8). There were significant differences between the strata, as the number of students per class in public schools was one-third more than in private schools (22.8 students vs. 16.6 students per teacher, respectively). Furthermore, Grade 4 classes in urban schools had 39 percent more students per teacher than schools in rural areas (21.5 students vs. 15.5 students per teacher, respectively). Service Delivery Indicator Survey Indonesia 2020 73 correlates of students learning “ outcomes The quality of learning depends not only on what takes place inside the classroom but also on the students’ external environment. “ 5. Uncovering the main factors associated with student observed reported not eating breakfast the day of performance. What are the main factors associated the observation. Girls scored higher than boys on all with the performance of students on standardized subjects except one. learning tests and, in particular, how can we explain the important differences in learning between the The quality of learning depends not only on what public and private sectors, as well as between rural takes place inside the classroom but also on and urban school students? In this section, we examine the students’ external environment. Academic the correlates of student performance through an performance can be linked to characteristics of the analysis of the individual correlations associated with students themselves, the home environment, and student and teacher characteristics. peer groups, as well as the school context and the interrelationships between them. We first analyze the Attendance of pre-primary is associated with learning level of students according to the following significantly higher scores in all subjects of the determinants: (i) early childhood education; (ii) student assessment. The differences between a student characteristics and (iii) teacher and school child who attended early childhood education for characteristics. any length of time and one who did not are large: in Indonesian language was 8 percentage points, in math was 11 percentage points, in Arabic was 16 percentage points, and in non-verbal reasoning was 8 percentage points. Fourteen percent of all students 74 Service Delivery Indicator Survey Indonesia 2020 Early-childhood Education A majority of students attended some early Data collected during the SDI survey allow us to childhood education. As observed in Table 5.1, 83.7 examine the association of previous attendance percent of the Grade 4 students previously attended in early childhood education of any duration with any type of pre-primary education for any length learning outcomes of Grade 4 students. Table 5.1 of time. About 85 percent of students in public shows the previous attendance of early childhood schools attended early childhood education, about 3 education in our sample among the various types of percentage points higher than in private schools. A schools and locations. The table also presents the difference of over 11 percentage points was observed percentage of female students, the percentage of for urban and rural schools. students who had breakfast on the day of the survey and those who received a PIP (need-based) student scholarship. Table 5.2 presents students’ average scores for each of the assessment subjects according to preschool attendance, as well as according to gender, having had breakfast and receiving a PIP scholarship. Service Delivery Indicator Survey Indonesia 2020 75 chapter 5. CORRELATES OF STUDENT LEARNING OUTCOMES table 5.1. Student gender, pre-primary, breakfast and scholarship, by school type (percentage) All MoRA MoEC MoRA Public Private Urban Rural Islamic non- Islamic Female student % 48.8 48.6 49.2 50.0 49.7 48.2 47.6 49.8 Ate breakfast today % 86.0 85.6 86.3 97.0 87.5 85.0 87.9 84.4 Attended preschool % 83.7 82.8 83.1 100 85.2 82.2 89.4 78.3 PIP scholarship % 21.9 18.5 31.4 21.4 28.1 17.9 16.3 26.5 Note: Weighted means using sampling weights. Results for female student evaluation based on observations from 1,641 sampled students from 3,365 students. Results for breakfast based on observations from 2,894 sampled students from 3,365 students. Results for preschool based on observations from 1,034 students. Results for scholarship based on observations from 1,034 students. The small sample size of MoEC and MoRA non-Islamic schools surveyed precludes nationally representative results for these categories. Attendance of pre-primary is associated with respectively), in math was 11 percentage points significantly higher scores in all subjects of the (45.3 vs 34.4 percent, respectively), in Arabic was 16 student assessment (Table 5.2 and Figure 5.1). In percentage points (61.9 vs 46 percent, respectively), fact, the differences between a child who attended and in non-verbal reasoning was 8 percentage points early childhood education for any length of time (73.8 vs 66 percent, respectively). and one who did not are, in Indonesian language was 8 percentage points (88.5 vs. 80.8 percent, table 5.2. Results of student assessment, by gender, pre-primary, breakfast and scholarship (percentage) Indonesian Mathematics Arabic NVR Language All 42.3 86.6 59.0 71.8 Female student 45.0 88.9 61.6 71.4 Male student 39.8 84.8 56.2 72.4 Attended preschool 45.3 88.5 61.9 73.8 Did not attend preschool 34.4 80.8 46.0 66.0 Ate breakfast today 40.9 85.7 58.4 71.3 Not eaten breakfast today 39.3 84.8 55.3 68.7 With PIP scholarship 40.8 86.3 56.1 69.7 Without PIP scholarship 43.8 87.2 59.6 73.0 76 Service Delivery Indicator Survey Indonesia 2020 FIGURe 5.1. Student results by attending any early childhood education 100 89 90 81 74 80 66 70 62 Percentage 60 45 46 50 34 40 30 20 10 0 Mathematics Indonesian Language Arabic NVR Attended Early Childhood Education Did not Attend Early Childhood Education Service Delivery Indicator Survey Indonesia 2020 77 chapter 5. CORRELATES OF STUDENT LEARNING OUTCOMES box 5.1.1. Parental choices: Which schools do parents send their children to and why? In explaining why they chose to send their children to MoRA schools, parents cited: (i) more religious content knowledge; (ii) closer to home; and (iii) higher quality education as the top three reasons. In comparison, parents in the neighborhood who were not sending their children to MoRA schools provided the following three reasons for sending their children to their respective schools: (i) closer to home; (ii) higher quality; and (iii) children in the neighborhood/community/ family study there. BOX FIGURe 5.1.1. Parents’ reasons for sending their children to MoRA schools (madrasah and other religious schools) 85% 69% 71% 47% 39% 30% More Closer Children Close Higher Children Religious to Home They Know to Home Quality They Know Content Study There Study There Parents in Sample Parents in Neighborhood For most parents of children who attend MoRA schools, religious education was the top educational priority. In addition, when given five different priorities: (i) basic reading and math skills; (ii) strong values and religious beliefs; (iii) ability so solve problems and think for yourself; (iv) preparation to find good jobs; and (v) foundation to be a good citizen, parents of MoRA school children overwhelmingly (64 percent) chose religious education as the top educational priority of a good education. Interestingly, 54 percent of parents who did not send their children to MoRA schools also expressed that religious education should be the main priority, highlighting the overall significance of religion. However, parents who sis send their children to MoRA schools had a higher demand for good religious education in their children’s schooling, compared with parents who did not send their children to MoRA schools. Individual student characteristics male and female disadvantage between and within Various student characteristics can also influence provinces (Afkar et. al 2020). academic achievement. We examine the roles of In all subjects except non-verbal gender, eating breakfast before coming to school, reasoning, female students outper- and receiving a PIP scholarship. formed male students. While overall Gender student scores were low compared There are significant differences in learning results with what students are expected to by gender. Table 5.3. and Figure 5.3. present know in Grade 4, this female per- the averages score in the various subjects of the formance advantage was especially assessment by student gender and school type. This pronounced in mathematics, with a national averages mask important variations at the 5-percentage-point difference. subnational level, including variations of significant 78 Service Delivery Indicator Survey Indonesia 2020 table 5.3. Student results according to gender, by type of school and subject (percentage) Indonesian Average Results Mathematics Arabic NVR Language in Percentage Male Female Male Female Male Female Male Female All schools 39.8 44.9 84.8 89 56.2 61.6 72.4 71.4 MoRA 39.8 44.9 84.7 88.9 56.5 61.8 72.3 71.4 Islamic 74.3 76.6 96.6 96 92.2 90.7 non Islamic 74.3 76.6 96.6 96 - - 92.2 90.7 MoEC 44.2 50.7 86.6 90.5 - - 74.7 73.2 Public 43.1 45.1 88.2 88.8 53.2 57.7 75.1 74.8 Private 39.5 44.9 84.4 88.9 56.6 62 72.1 71.1 Urban 42.4 48 87.5 91.1 56.4 63.5 75.8 74.1 Rural 38.3 43.1 83.1 87.5 56.2 60.4 70.3 69.8 Note: Weighted means using sampling weights from 3,368 sampled students from 350 schools. The small sample size of MoEC and MoRA non-Islamic schools surveyed precludes nationally representative results for these categories. FIGURe 5.2. Student results according to gender, by status and subject (percentage) 100 89 88 89 89 90 85 84 80 70 62 62 60 58 56 53 57 50 45 45 45 43 40 40 40 30 20 10 0 All Schools Public Private All Schools Public Private All Schools Public Private MATH Indonesian Language ARABIC Male Female Service Delivery Indicator Survey Indonesia 2020 79 chapter 5. CORRELATES OF STUDENT LEARNING OUTCOMES These differences in favor of female students are (Table 5.1.). The proportion was slightly higher in rural seen across school type and location. These very schools, with one in 6.4 students not eating breakfast. significant differences of learning results in favor Students who had had breakfast, whether at home, of female students in all subjects of the evaluation outside or at school on the day of the evaluation except NVR were observed across the various types obtained better results in all subjects (Table 5.2). of schools and locations. In mathematics, for instance, female students outperformed male students by Overall, students who ate breakfast 6.5 percentage points in MoEC schools, while they scored about 2 percentage points outperformed male students by 5.6 percentage higher in the evaluation in mathemat- points in urban schools. ics. Breakfast In MoRA Islamic schools, the math score difference Childhood nutrition and stunting are major was 5 percentage points between those who had had challenges for Indonesia. breakfast and those who did not, while the difference was 6 percentage points among all public school One in seven students (14 percent) students (Figure 5.4). Not having a meal before reported not eating breakfast the coming to school in the morning is associated with day of the evaluation lower student learning levels. FIGURe 5.3. Student results in mathematics according to eating breakfast, by type of school (percentage) 60 48 45 50 43 43 45 40 45 42 43 41 41 41 41 40 40 30 20 10 0 All Schools MoRA MoEC Public Private Urban Rural Islamic MATHEMATICS Ate Breakfast Did not ate Breakfast 80 Service Delivery Indicator Survey Indonesia 2020 PIP scholarships this support, students from low-income families Despite support from scholarships, students from performed significantly worse in all the subjects of the low-income families performed significantly worse in evaluation compared with the other students. Overall, all the subjects of the evaluation than other students. and in MoRA Islamic schools, students who received Overall, about one-fifth of the Grade 4 students a PIP scholarship scored about 3 percentage points surveyed receive a financial needs-based (PIP) lower in the math evaluation (Table 5.2.). In MoEC scholarship (Table 5.1). Among public MoEC schools, schools, as well as urban schools, the gap was about the proportion is close to one-third of the students 8 percentage points lower in mathematics between receiving the PIP scholarship (31.4 percent). Despite those with the scholarship and those without (Figure 5.5.). FIGURe 5.4. Student results in mathematics according to receiving a PIP scholarship, by type of school (percentage) 60 50 46 50 44 44 44 46 43 41 41 41 41 42 38 39 40 30 20 10 0 All Schools MoRA MoEC Public Private Urban Rural Islamic MATHEMATICS With PIP Without PIP Service Delivery Indicator Survey Indonesia 2020 81 chapter 5. CORRELATES OF STUDENT LEARNING OUTCOMES Characteristics of schools with the highest average The high performing schools in terms of student scores in student assessment. Before examining learning assessments also had higher scores across multivariate analysis to assess the factors associated all of the main Service Delivery Indicators (Figure with student test scores in the various subjects, 5.6). including student, teacher and school characteristics, we examine the characteristics of schools with the The best performing schools showed highest average scores in student assessment and teachers with significantly higher the association with the Service Delivery Indicators. subject competence test scores than High performing schools other schools in math (80 vs 65 per- The top 5 percent of schools in terms of student cent, respectively), language (68 vs learning significantly outperform the rest of the 60 percent, respectively), and peda- sample. The characteristics of top 5 percent schools gogy (29 vs 22 percent, respectively). in terms of student learning as measured by combined average results in language and mathematics are Also, these schools presented significantly lower shown in Figure 5.6, compared with the other teacher absence rates from school and from schools in the sample. The average combined score classroom than other schools (12 vs 18 percent, and 17 in mathematics and language scores of students vs 23 percent, respectively), as well as absence of the attending the highest performing schools was 80.1 school principal (8 vs 13 percent, respectively). The percent, compared with 61.7 percent for the rest of best performing schools were also distinguished by the sample. longer actual teaching per day (3 hrs and 53 mins vs 2 hrs and 56 mins)—a difference of about 1 hour per day—and a higher ratio of students with textbooks (67 vs 47 percent, respectively). FIGURe 5.5. Characteristics of the 5 percent highest performing schools compared with the others 120 100% 100 89% 80% 83% 80 76% 68% 70% 65% 63% 67% 60% 60 47% 40 29% 22% 23% 22 18% 17 20 17% 12% 13% 8% 0 Teacher Teacher Teacher Teacher Teacher Teacher Principal Teaching Textbooks Sufficient Student Knowledge Knowledge Knowledge Knowledge School Class School Time Light Teacher _Math _Indonesian _Arabic _Pedagogy Absence Absence Absence Per Day Ratio Language 5% Highest Performing Schools Remaining Schools 82 Service Delivery Indicator Survey Indonesia 2020 box 5.2. Development World Management Survey The SDI survey included the implementation of the DWMS among school directors to assess school management quality. Created in 2008 as an adaptation of the original World Management Survey, the Development World Management Survey (DWMS) is a tool used to measure management and leadership skills in developing country contexts. The DWMS covers 15 questions across the following five main areas: Leadership, Operations Management, Performance Monitoring, Target Setting, and People Management. Through an interview style questionnaire aimed at school administrators (either principals or their second-in-command), trained and certified enumerators assign scores ranging from 1 to 5 along the five aforementioned areas. A score of 1 indicates practically no structured management practices implemented, while 5 refers to best practice in the sector. The survey was conducted in 262 out of 263 MoRA schools. The mean overall management score for MoRA schools in the sample was 2.53, indicating that they were intermediate between having informal mechanisms in place with a mostly relative approach to management, and having more formal processes being implemented with some weaknesses. The disaggregated mean scores across the five management areas also hover around 2.5. While no statistically significant differences between urban and rural schools were identified, public schools scored slightly higher in all management areas. This dataset will be analyzed in a separate publication (forthcoming). Quality of overall management practices in MoRA schools Mean Mean All Urban Rural diff. Test Public Private diff. Test MoRA School School (Urban - School School (Public - School Rural) Private) Management 2.53 2.48 2.56 -0.07 2.87 2.51 0.36*** Leadership 2.48 2.42 2.50 -0.08 2.74 2.46 0.27* Operations Management 2.57 2.53 2.58 -0.06 2.90 2.54 0.35** Performance Monitoring 2.68 2.61 2.71 -0.10 2.97 2.66 0.31** Target Setting 2.27 2.25 2.27 -0.02 2.58 2.25 0.33** People Management 2.68 2.60 2.71 -0.11 3.16 2.65 0.51*** Service Delivery Indicator Survey Indonesia 2020 83 chapter 5. CORRELATES OF STUDENT LEARNING OUTCOMES Correlations between SDI indicators and outcomes the relationship between student learning and the Examining how the Service Delivery Indicators observed independent variables using a multivariate correlate with learning achievement. We now regression model. Table D4 in the Appendix shows examine the relationship between student the results of this analysis for all students, and then performance and the Service Delivery Indicators. according to gender. The estimates take into account The SDI survey measures key inputs in an education several student characteristics, and subject teacher production function. The outcome of this production characteristics, as well as school characteristics that function is student learning. With outcome data in may influence student performance. Figures 5.7 and education as measured from student evaluations 5.8 show statistically significant coefficient estimates in the different subjects, we can also examine from the regressions of student test results in whether and how the input measures are related to Indonesian language and mathematics, respectively. learning outcomes. Despite not implying causality, it is interesting to examine how the Service Delivery Teachers with higher level education show better Indicators correlate with learning achievement to student outcomes in math. Teachers who themselves help answer the question of how to improve student have higher levels of education, especially D4/S1 level learning outcomes. compared with secondary level and lower (reference category), show positive association with student Strong relationships exist between the indicators achievement in mathematics after controlling for and student learning results. Table D3 in the Appendix student, teacher and school characteristics. The depicts unconditional correlations between student relationship is an increase of 3 percentage points in learning achievement and the education indicators. student score in mathematics on average, and about Panel A pools school-level data from all schools, 4 percentage points more for female students. Civil while Panel B uses data from the student level. We servant teachers are associated with higher test observe that there are strong relationships between scores among math students compared with non- the indicators and student learning results, with all civil servant teachers by about 5 percentage points. the correlations having the expected directionality. Eating breakfast is associated with higher test Regression analyses confirm the descriptive results in mathematics, especially for female statistics. In the next section, we make use of students. Female students significantly outperform multivariate analysis to assess the factors associated male students in both Indonesian language and with the determinants of student performance in mathematics by about 3.5 percentage points in the various subjects, including student, teacher and language and 4.5 percentage points in mathematics, school characteristics. even after controlling for student, teacher and school characteristics. With regard to student age, 10-year- In general, the regression analyses old students are associated with higher test results confirm the descriptive statistics pre- than 9-year-old students in both subjects, while students above 10-years-old present significantly sented above. lower results. The gap is especially important for 12-year-old students and associated with lower Factors associated with student test scores by scores by between 6 and 14 percentage points in subject both subjects, compared with 9-year-old students. Looking at student, subject teacher and school characteristics to understand what influences Teacher subject knowledge in itself is not a student performance. To better understand major predictor of student achievement. Teacher the factors associated with the determinants of competence as measured by subject knowledge student performance in assessments, we estimate test scores is associated with higher student 84 Service Delivery Indicator Survey Indonesia 2020 achievement in Indonesian language, especially for for their students, and those with more than 20 years male students, while in mathematics the association of experience and more with about 10-percentage- is significant only for female students. However, point lower scores. This inverse association the estimate is very small, suggesting that teacher between teacher experience and student learning in subject knowledge in itself, on average, is not a major mathematics is especially observable among male predictor of student achievement, and needs to be students. In MoEC schools, math teachers with 10 combined with pedagogic knowledge and teaching years of experience and more are associated with skill. Other characteristics have stronger associations 15- to 17-percentage-point lower scores among their with student scores. male students compared with teachers with less than 10 years of experience (8 to 12 percentage points Teachers’ years of experience in education are in MoRA Islamic schools). This inverse relationship inversely related with student learning results in between teacher age and student learning is likely due language and especially in mathematics. Compared to improvements in teacher recruitment processes as with teachers with less than 10 years of experience, well as improvements in Indonesia’s human capital mathematics teachers with 10 to 20 year of experience over the last thirty-plus years. are associated with 7-percentage-point lower scores FIGURe 5.6. Correlates of student performance: Indonesian language Female Student 10 Year Old 11 Year Old 12 Year Old Mother Tongue_Javanese Teacher Experience_10_20_Years Teacher Experience_20_30_Years Teacher_Test_Score Teacher_use_Local_Language Public Rural Other Religious School MoEC Schools -.15 -.1 -0.5 0 0.5 Service Delivery Indicator Survey Indonesia 2020 85 chapter 5. CORRELATES OF STUDENT LEARNING OUTCOMES FIGURe 5.7. Correlates of student performance: Mathematics Female Student 10 Year Old 11 Year Old 12 Year Old Ate Breakfast Today Mother Tongue_Bahasa Mother Tongue_Javanese D4/S1 Bachelor Teacher Experience_10 - 20 Years Teacher Experience_20 - 30 Years Teacher Experience_30_More_Years Female_Teacher Certified_Teacher Civil Servant Public Rural Other Religious Schools -.2 -.1 0 .1 .2 .3 Higher test results in math are associated with Students in public and urban schools do better too. female teachers. We also note that in mathematics, higher test results are associated with female Students attending private (non- teachers, especially for male students. With regard to public) schools as well as those at- language, the use of a local language by a teacher tending rural schools are associated is associated with lower student achievement in Indonesian language for both genders. This should not with significantly lower achievement be taken as evidence that local-language instruction levels than students in public and ur- does not work, rather it is likely that students spoke a ban schools.43 language other than Indonesian at home, and without this assistance in Indonesian language their scores Household level data sample may have been even lower. The SDI survey in Indonesia included data collection at the household level among a sub-sample of students participating in the student evaluation in each school. Data included information on students’ pre-primary attendance and reception of a need-based student scholarship. Estimations results performed using this smaller sub-sample are presented in Table D5 in the Appendix. Figures 43 However, as previously noted, the small sample of MoRA non-Islamic schools surveyed preclude statistically representative results for that category. 86 Service Delivery Indicator Survey Indonesia 2020 D1 and D2 in the Appendix also present statistically student, teacher and school charac- significant estimates from the regressions of student teristics. test results in Indonesian language and mathematics, respectively. In particular, boys appear to benefit from pre-primary education, with a 7.7-percentage-point higher test Children who have attended early childhood score in Indonesian language and a 6-percentage- education achieve significantly higher test scores. point higher score in mathematics compared with Regression analyses confirm the descriptive statistics those not having attended pre-primary education. presented above. Among female students, those who attended pre- primary are associated with a 6-percentage-point Children who have attended early higher score in both subjects. With regard to need- childhood education are associated based scholarship (PIP), regression results show that with significantly higher test scores in students receiving the transfer are associated with both Indonesian language and math- a 6-percentage-point lower score in mathematics after controlling for student, teacher and school ematics, even after controlling for characteristics. Service Delivery Indicator Survey Indonesia 2020 87 comparing indonesia with other sdi countries “ Indonesia performs above or close to the average compared with other SDI countries, however it was expected to do even better given its much higher GDP per-capita. “ 6. SDI surveys have been undertaken in about 20 The teacher absence rate from school is similar to countries. Since the launch of the SDI initiative the average level observed in other SDI countries. in 2010, SDI and associated surveys have been However, once at school, teachers for the most completed in about 20 countries, mainly in Africa but part go to class and the classroom absence rate is also in Afghanistan, Guatemala, Lao PDR, Pakistan somewhat lower than average in other SDI countries. as well as Iraq (forthcoming) and others. Table 6.1 Time spent teaching per day is also just below the SDI presents the value of Service Delivery Indicators in average of 3 hours 2 minutes. several countries that participated in the survey, as well as the SDI average. Despite its low results, Indonesia is still slightly above the SDI average. With a proportion of only Indonesia performs somewhat well, or close to the 15.6 percent of teachers in Indonesian language average, compared with the other countries that and mathematics who obtained a combined mark participated in the SDI surveys. of 80 percent or more (considered as the minimum knowledge threshold), despite these low results, Indonesia has a much higher GDP Indonesia is still slightly above the SDI average of 14.6 per-capita than most of the other percent. countries and so would be expected to do even better in comparison. The average score obtained by teach- ers in overall assessments is below the SDI mean by about 3 percent- age points, and far below the scores 88 Service Delivery Indicator Survey Indonesia 2020 obtained in Kenya and Tanzania, for Among SDI countries, only Kenya shows a higher combined student learning score than Indonesia. instance, both of which have much Ultimately, the learning outcomes of Indonesian Grade lower GDP per capita. 4 students in combined language and mathematics is 15 percentage points above the SDI average, with only The observed student-to-teacher ratio is the Kenya showing a higher combined score. However, lowest compared with other SDI countries. Despite this is primarily due to the high scores obtained by shortcomings, especially with respect to the share students in the Indonesian language assessment. of students with textbooks and exercise books, input availability in Indonesian elementary schools is above the mean of SDI countries and relatively high In mathematics, Indonesian students compared with the average, in particular, regarding score 5 percentage points below the the minimum equipment availability and minimum SDI average (42.3 vs 47.3 percent, infrastructure. respectively), with most of the other SDI countries having much lower The observed student-to-teacher ra- levels of income than Indonesia. tio is the lowest compared with other SDI countries (17 students per teach- er vs 40.4 students per teacher, on average). Service Delivery Indicator Survey Indonesia 2020 89 chapter 6. COMPARING INDONESIA WITH OTHER SDI COUNTRIES table 6.1. Comparison between Indonesia and other SDI countries Indonesia Mean SDI Mauritania Morocco Madagascar 2019 2017 2016 2016 Teacher Knowledge and Ability Minimum knowledge (% of teachers achieving 80% min in language 15.6 14.6 0.3 3.1 0.1 and math) Test Score 39.6 43 30.9 44.4 33.2 (out of 100 out of language, math and pedagogy) Teacher Effort School absence rate 18.7 18.6 13.6 4.4 30.6 (% of teachers) Classroom absence rate 23.5 39.8 27.8 5.5 37.8 (% of teachers) Scheduled teaching time per day 4h35m 5h34m 4h36m 4h37m 5h12m Time spent teaching per day 2h56m 3h02m 3h04m 4h09m 3h09m availability of inputs Observed student-to- 17 40.4 37 22.7 17.6 teacher ratio Share of students with textbooks (% of students) 47 37.1 17.5 87.7 10.3 Minimum equipment availability (% of schools) 71 60.5 36.3 67.5 65 (90% with pencils and notebooks) Minimum infrastructure availability (% of schools) 55.9 38.1 4.2 48.6 20.2 Student Learning Language and mathematics test score (out of 100) 64.5 49.6 25.9 52.5 50.6 Language test score 42.3 47.3 34.4 57 56.8 (out of 100) 90 Service Delivery Indicator Survey Indonesia 2020 Kenya Mozambique Nigeria** Tanzania Togo Uganda Afganistan Punjab 2012 *2014 2013 2014 2013 2013 2017 2019 40.4 0.3 3.7 21.5 1.6 19.5 ... 56+ 57.1 26.9 32.9 48.3 35.6 45.3 ... ... 14.1 44.8 13.7 14.4 20.5 26 10 14 42.1 56.2 19.1 46.7 35.8 52.8 15 18 5h37m 4h17m 4h53m 5h54m 5h29m 7h18m 3h25m ... 2h49m 1h41m 3h26m 2h46m 3h29m 3h18m 2h18m ... 35.2 21.4 21.6 43.5 29.7 47.9 43 ... 48 68.1 38.2 25.3 68.5 5 86 ... 78.8 76.8 54.8 61.4 26.4 80.6 36 ... 59.5 29.1 18.5 40.4 22.3 53.7 35 82++ 72 20.8 32.2 40.1+* 45.7 48.6 ... ... 59 25.1 31.9 58.2 44.6 43.4 ... ... + Minimum knowledge of teachers in Punjab includes assessments on English, math, or Urdu. The assessment was different with the one used in SDI. ++ Minimum infrastructure in Punjab means that the classroom has a blackboard, enough visibility, chairs and desks, pencils, and all students have textbooks and exercise books. * In Mozambique, only public schools were surveyed. ** Values for Nigeria are the weighted average of the four states surveyed, namely Anambra, Bauchi, Ekiti, and Niger. Note on cross-country comparisons: Students and teachers in SDI countries were sometimes tested in the language they spoke at home, and sometimes in an official or national language that did not match the language spoken at home. This means that the language tests were not equally difficult for all test-takers, and so country scores in language are less comparable than scores in mathematics. Service Delivery Indicator Survey Indonesia 2020 91 conclusion and recommendations “ Most students had positive perceptions of their teachers. There were other positive findings as well, for example, more than 80 percent of the students said that their teachers encouraged “ them to do their best. 7. The SDI survey results show major shortcomings the MoRA school classrooms had classroom ‘corner in the education system for a recently designated libraries,’ while more than 70 percent of observed upper middle-income country.44 Many of the findings MoEC school classrooms had one. This was mostly discussed in the report are systemic in nature, for driven by public MoEC schools, as only 29 percent of example, less than one-third of surveyed Grade 4 the private MoEC schools had corner libraries in the students in MoEC and MoRA schools read at a Grade classrooms. 4 level, and none of the surveyed teachers met the minimum knowledge criteria in all key subjects. Key infrastructure measures were Teacher effort varies widely and inequality in teacher also lacking and unevenly distribut- pay is high; civil servants earn almost three times the ed; while 91 percent of public MoRA salaries of non-civil servants, though both are asked to lead classrooms and teach children; both have classrooms in the survey had an elec- students who on average have very low levels of tricity connection, only 61 percent of learning. Teacher classroom absence is between 20 private MoRA schools did. and 26 percent, with much higher rates in non-public and rural schools than in urban areas. Only about half of the schools visited had running water and soap available for students. Official WHO The disparities in infrastructure between school types guidelines to mitigate the spread of the coronavirus are concerning, if indeed all children have a right to note that it is important to wash hands regularly with the same quality of education. Only 49 percent of soap. Unclean hands and contaminated surfaces can 44 https://blogs.worldbank.org/opendata/new-world-bank-country-classifications-income-level-2020-2021 92 Service Delivery Indicator Survey Indonesia 2020 lead to transmission of the virus, making cleanliness scored among the highest in the world on measures and hygiene essential in schools—more than ever of happiness, even though their scores on academic now. subjects were low.46 These findings imply large invest- Spending more on education does not necessarily ments and effort are required to im- mean better education outcomes. prove handwashing facilities before schools can safely reopen.45 A complementary study on subna- tional education public expenditure Most students had positive perceptions of their ‘Revealing How Indonesia’s Subna- teachers. There were positive findings as well, for tional Governments Spend their Ed- example, more than 80 percent of the students said ucation Money’(World Bank 2020) that their teachers encouraged them to do their finds that more funding does not best. Most students felt that everyone in the class necessarily lead to better education knew what they should be doing and learning in the lesson, and classroom productivity was perceived outcomes. to be in the mid-to-high range by most students. This matches with evidence from other studies, for example, the 2015 PISA in which Indonesian students 45 https://www.who.int/emergencies/diseases/novel-coronavirus-2019/advice-for-public; https://www.who.int/publications/i/item/ considerations-for-school-related-public-health-measures-in-the-context-of-covid-19 46 https://www.oecd.org/pisa/PISA-2015-Indonesia.pdf Service Delivery Indicator Survey Indonesia 2020 93 chapter 7. COnclusion and recommendations Despite the major increases in funding for education Over 40% of schools do not have minimal in Indonesia in recent years, the low levels of teacher infrastructure, defined as the availability of subject and pedagogical knowledge found here, functional toilets for the students and sufficient light along with ineffective use of funds, suggest that in classrooms. Whether schools are public or non- Indonesia should focus on spending more effectively public, Indonesian students are trying to learn inside and more efficiently. their classrooms. Help all schools achieve minimum levels of infrastructure by making sure infrastructure Schooling is not the same as learning—an insight funds already flowing to subnational governments are repeatedly stressed by the 2018 World Development targeted to the highest-need schools. Report. To build on its education reforms and achieve better results, continuing reform in Indonesia can Fewer than half of the students observed on the day focus on some key areas based on the SDI results. of the data collection visit had textbooks, while about While noting that the findings are associational and one in five Grade 4 students did not have an exercise not causal, the data collected here identify key areas book the day of the observation. The unavailability for action at the national and subnational level to of textbooks and exercise books despite BOS improve student learning outcomes: financing is hard to understand, unless we consider that: (i) teacher salaries and other expenses are being Almost one teacher in four is absent from their prioritized for BOS funds instead of textbooks; and (ii) classroom. The absence rate of the class of teachers Curriculum 2013 introduced new textbooks that were is on average 23.5 percent, very high for an upper unpopular with many educators and not distributed middle-income country. There are promising from a central printer to all schools, which meant that examples of ways to improve teacher attendance many teachers continued to use pre-2013 textbooks in Indonesia.47 Act to hold teachers and principals out of both preference and necessity (The Promise of accountable for using student time effectively. This Education in Indonesia, World Bank 2020). These and starts with being present and teaching during class other factors have led to critical shortages in textbook time, as well as minimizing official non-teaching duty availability for students, something that MoEC and assignments. MoRA can work together with subnational districts to address urgently. Only 15.6 percent of teachers scored higher than 80 percent on the combined mathematics and While associational and not causal, the survey found Indonesian language tests, while the level of that pre-primary education attendance is associated pedagogical knowledge was very low, with an with significantly higher scores in all subjects of overall score of 23 percent. Teachers who themselves the student assessment even after controlling for have higher levels of education, especially D4/S1 level student, teacher and school characteristics. Districts compared with secondary level and lower (reference and cities can allocate larger portions of their existing category), show positive association with student education budget to provide two or three years of achievement in mathematics. Use the current wave access to high-quality ECED services for every child. of teacher retirement to hire only teachers with high levels of subject-matter and pedagogy knowledge. 47 Gaduh, A., Pradhan, M., Priebe, J., and Susanti, D. 2020. “Scores, Camera, Action? Incentivizing Teachers in Remote Areas.” RISE Working Paper Series. 20/035. https://doi.org/10.35489/BSG-RISE-WP_2020/035. 94 Service Delivery Indicator Survey Indonesia 2020 Female students outperformed male students by Indonesia has some high performing schools where important margins in almost all subjects. When students are learning. The best performing schools designing improvements in teacher training and showed teachers with significantly higher subject curriculum reform, MoEC and MoRA can consider competence test scores than other schools in math, ways to make instruction more appealing to male language, and pedagogy. These schools presented students to help them stay engaged and learn more. significantly lower teacher absence rates from school This approach should attempt to improve learning for and from classroom than other schools, as well as all students, both male and female. lower absence of the school principal. The best performing schools were also distinguished by longer There is a gap between what students perceive and actual teaching (about 1 hour per day) and a higher the actual quality of service delivery. Over 80 percent ratio of students with textbooks. of the students said that their teacher encouraged them to do their best; and when they had a need for Indonesia has several key human capital challenges help in the lesson, or had questions, their teachers in the coming years, including i) Supporting, were supportive. Around 80 percent of students felt incentivizing and holding qualified teachers that their teachers mostly or always explained things accountable for teaching, ii) providing every student in an orderly way and tried different techniques to with a textbook and adequate learning facilities, and explain things that were unclear to them. Indonesian iii) offering quality early childhood education. These students have reported high levels of satisfaction goals are both important and achievable for a rising in the face of low levels of learning in other studies middle-income country. (OECD 2015, World Bank 2019). Service delivery and most importantly, student learning outcomes, may Additional recommendations for the Indonesian improve by providing parents, school committees, education sector based on other analyses. Based and communities with relevant and targeted on the SDI analysis and other research undertaken information on student learning achievement, while by the World Bank, as well as other researchers and simultaneously supporting teachers to identify and partners, the World Bank has developed 12 core address learning gaps. recommendations for the Indonesian education sector. These are presented in the Promise of One in seven students reported not eating breakfast Education in Indonesia (Indonesian Education the day of the evaluation. Students who had had Flagship report) (link) and summarized here: breakfast, whether at home, outside or at school on the day of the evaluation obtained better results in all subjects. Efforts at all levels of government to address malnutrition and its effects, including stunting, can be increased. Service Delivery Indicator Survey Indonesia 2020 95 chapter 7. COnclusion and recommendations Twelve recommendations from the Promise of Education in Indonesia • Ensure that students reach at least minimum learning and development standards at each level of the system. • Make quality early childhood education accessible to all. • Act to guarantee equitable access to good quality education and learning by children most excluded from the system. • Act to improve learning outcomes of the lowest performers. • Ensure that all students, including those with disabilities, succeed. • Improve the quality of preservice institutions and the candidates that enter them; recruit the best teacher candidates and distribute them effectively. • Improve professional development and calibrate incentives. • Strengthen accountability mechanisms through better data tracking and verification. • Support existing institutions to improve service delivery. • Expand access to and improve the quality and relevance of TVET. • Improve the quality, relevance, and equity of the tertiary education sector. • As a part of the COVID-19 response and recovery, strengthen the system for future shocks and stresses. These recommendations align with many of those MoEC’s ‘Freedom to Learn’ policy48 outlines major currently pursued by the Government of Indonesia. reforms in student assessment, teacher selection, school financing and the use of technology. MoRA is In a speech following his 2019 ree- actively working on a program of change to improve lection, Indonesian President Joko student assessment, teacher training, school financial management and data systems management through Widodo declared his aim to develop the Realizing Education’s Promise project (link). an adaptive, productive, innovative, and competitive Indonesia that would The challenges identified in this survey and the Edu- make the country one of the strong- cation Flagship combine with the COVID-19-induced est in the world, highlighting that the learning crisis to pose a serious threat to human key to this more prosperous future capital, but one that can also be an opportunity.49 lay in developing human resources (State Address, August 2019). 48 Ministry of Education and Culture statement, December 2019. https://www.kemdikbud.go.id/main/blog/2019/12/mendikbud-tetapkan-empat-pokok-kebijakan-pendidikan-merdeka-belajar 49 Yarrow, Noah; Masood, Eema; Afkar, Rythia. 2020. Estimated Impacts of COVID-19 on Learning and Earning in Indonesia: How to Turn the Tide. World Bank. 96 Service Delivery Indicator Survey Indonesia 2020 As a part of ‘building back better’, ers in the decentralized system for MoEC, and the centralized but largely non-public system of MoRA. investments in distance learning can improve support to rural and remote This SDI survey can be a benchmark to measure teachers and schools, improve data future progress. If investments in education can be collection and sharing, and increase maintained in the coming fiscal years, while at the same accessibility of high-quality learn- time increasing their effectiveness and efficiency, we ing materials for students, while also hope to be able to measure improvement five years opening up new options for parental from now in a subsequent round of the survey. The MoEC sample could be expanded to be nationally engagement and support. representative. These data can also be used in the interim to continue to identify areas that are in They can also make the system resilient to climate, particular need of attention. natural and manmade disasters. Improving face-to- face learning as schools reopen to recover from the effects of the COVID-19 pandemic and build a strong- er system in the long term will require coordination with other ministries and a broad array of stakehold- Service Delivery Indicator Survey Indonesia 2020 97 APPENDIX A: SAMPLING STRATEGY The sample design is a stratified random sample Non-madrasah MoEC schools were chosen as a design utilizing a comprehensive inventory of schools function of their proximity to madrasah schools, while nationally. The sample was implicitly stratified by non-madrasah schools under MoRA were selected urban/rural and type of school, boys, girls, mixed, and intentionally to represent multiple faiths. From the implicitly stratified by province and public/private selected sample of madrasah schools, a sub-sample strata. of 87 madrasahs were randomly selected to be the base for selection and interview of MoRA schools. Due to the missing urban/rural classification in the MoRA non-Islamic schools were selected based sampling frame of schools, an additional “unknown” on their proximity to these madrasahs. Similarly, a stratum was created and a number of schools were random sub-sample of 10 madrasahs was selected selected from this stratum. The sample size of 253 as the selection base for “other religious” schools. madrasahs was allocated roughly proportionally to the These “other religious” schools were again selected student population within the explicit strata. Schools and interviewed based on their proximity to the sub- were randomly selected within stratum randomly and sample of madrasah schools. with probability proportional to size. Replacement schools were selected prior to implementation. table A1. Sampling strata Strata Total Percent PPS Allocation Urban 1,523,369 0.41520 105 Rural 1,866,236 0.50865 145 Unknown 279,380 0.07615 3 Strata School Type Student Percent PPS Allocation Urban Boys 5,208 0.0034187 2 co - ed 1,512,885 0.9931179 101 Girls 5,276 0.0034634 2 Boys 2,449 0.0013123 2 co - ed 1,861,418 0.9974183 141 Rural Girls 236,900 0.0012694 2 Boys 1,391 0.0049789 - co - ed 277,216 0.9922543 3 Unknown Girls 773 0.0027668 - 98 Service Delivery Indicator Survey Indonesia 2020 table A2. Number of schools by type of school Number of schools by school type Urban Rural All Public Private Urban Rural Public Private Public Private All Schools N 350 133 217 160 130 64 96 69 121 % 100% 38% 62% 46% 54% 40% 60% 36% 64% All MoRA Schools N 263 53 210 117 146 26 91 27 119 % 75% 20% 80% 44% 56% 22% 78% 18% 82% Madrasah within MoRA N 253 53 200 109 144 26 83 27 117 % 72% 21% 79% 43% 57% 24% 76% 19% 81% Other Religious Schools N 10 0 10 8 2 0 8 0 2 % 3% 0% 100% 80% 20% 0% 100% 0% 100% MoEC Schools N 87 80 7 43 44 38 5 42 2 % 25% 92% 8% 49% 51% 88% 12% 95% 5% Service Delivery Indicator Survey Indonesia 2020 99 APPENDIX B: DEFINITION OF INDICATORS S ch o o l abs e n c e rat e Share of a maximum of 10 It is measured as the share of teachers who are absent from school at a time of an randomly selected teachers unannounced visit. It is measured in the following way: During the first announced absent from school during an visit, a maximum of 10 teachers are randomly selected from the list of all teachers unannounced visit (excludes volunteer and part-time teachers) who are on the school roster. The whereabouts of these 10 teachers are then verified in the second, unannounced, visit. Teachers found anywhere on the school premises are marked as present. classr o o m abs e n c e rat e Share of teachers who are The indicator is measured as the share of teachers not in the classroom at the time present in the classroom during of an unannounced visit. The indicator is constructed in the same way as school scheduled teaching hours as absence rate indicator, with the exception that the numerator now is the number of observed during an unannounced teachers who are either absent from school, or present at school but absent from the visit classroom. T i m e sp e n t t e ach i n g p e r da y Amount of time a teacher spends This indicator reflects the typical time that teachers spends teaching on an average teaching during a school day day. This indicator combines data from the staff roster module (used to measure absence rate), the classroom observation module, and reported teaching hours. The teaching time is adjusted for the time teachers are absent from the classroom, on average, and for the time the teacher teaches while in classrooms based on classroom observations. While inside the classroom distinction is made between teaching and non-teaching activities. Teaching is defined very broadly, including actively interacting with students, correcting or grading student’s work, asking questions, testing, using the blackboard or having students working on a specific task, drilling or memorization. Non- teaching activities includes working on private matters, maintaining discipline in class or doing nothing and thus leaving students not paying attention. M i n i m u m k n o wl e d g e Share of teachers with minimum It is measured as the percentage of teachers who can master the curriculum they knowledge taught. It is based on mathematics and language tests covering the primary curriculum administered at the school and is calculated as the percentage of teacher who score more than 80 percent on the language and mathematics portion of the test. The test is given to all mathematics or language teachers that taught Grade 3 last year or Grade 4 in the year the survey was conducted. Test score It is measured as the overall score of a mathematics, language and pedagogy tests covering the primary curriculum administered at the school level to all mathematics and language teachers that taught Grade 3 last year or Grade 4 in the year the survey was conducted. 100 Service Delivery Indicator Survey Indonesia 2020 APPENDIX C: DETAILED TEACHER ASSESSMENTS table C1. Teacher Mathematics Assessment Minimum knowledge MoRA MoRA (80% threshold) All non- MoEC Public Private Urban Rural Islamic (% teachers) Islamic Maths: 80% 31.9 31.8 53.3 34.3 39.3 30.9 37.3 28.9 Average Score (%) Score Maths (complete test) 67.4 67.4 78.7 66.6 69.5 67.1 70.4 65..7 Double-digit addition 98.0 98.0 100.0 97.9 97.2 98.1 99.1 97.4 Double-digit subtraction 89.0 89.0 85..0 91.7 92.6 88.6 91.0 88.0 Triple-digit addition 91.1 91.1 100.0 89.1 89.6 91.3 92.0 90.6 Double-digit division 79.6 79.6 90.0 84.5 83.0 79.2 80.6 79.1 Double-digit multiplication 85.8 85.8 98.3 91.4 88.5 85.5 88.3 84.5 Adding with decimals 61.0 61.0 78.3 62.9 66.2 60.4 63.9 59.5 Comparison of fractions 76.4 76.4 85.0 74.1 75.1 76.5 78.1 75.4 Geometry1 92.9 92.9 96.7 90.7 93.7 92.8 94.7 91.9 Geometry2 35.0 35.0 58.3 39.3 38.6 34.5 38.1 33.3 Venn diagram 72.5 72.5 88.3 66.4 70.3 72.8 76.8 70.2 Hours 56.1 56.1 69.2 52.2 60.5 55.6 60.5 53.7 Algebra 39.9 39.9 48.3 35.7 42.9 39.5 44.5 37.4 Area 61.3 61.3 70.0 58.6 65.2 60.8 67.5 57.9 Perimiter 55.4 55.4 63.3 57.4 58.6 55.0 61.9 51.8 table C2. Teacher Indonesian Language Assessment Minimum knowledge MoRA MoRA (80% threshold) All non- MoEC Public Private Urban Rural Islamic (% teachers) Islamic Indonesian Language: 80% 7.6 7.6 26.7 9.2 9.6 7.3 8.2 7.2 Average Score (%) Indonesian Language (complete test) 61.1 61.0 73.6 62.4 62.0 60.9 63.2 59.9 Grammar 93.6 93.6 96.8 93.5 94.0 94.0 94.1 93.3 Close Task 79.2 79.1 89.7 79.7 80.5 79.0 80.3 78.6 Composition 40.1 40.1 57.6 42.3 41.1 40.0 43.3 38.3 Service Delivery Indicator Survey Indonesia 2020 101 table C3. Teacher Arabic Assessment Minimum knowledge MoRA MoRA (80% threshold) All non- MoEC Public Private Urban Rural Islamic (% teachers) Islamic Arabic: 80% ---- 53.5 ---- ---- 58.9 52.6 51.1 0 Average Score (%) Arabic (complete test) ---- 71.7 73.0 71.4 75.1 69.7 Grammar 75.4 74.7 75.2 76.4 74.5 Close Task 68.9 71.3 68.3 73.1 66.3 Composition 78.8 80.3 78.6 83.3 76.4 table C4. Teacher Pedagogical Assessment Minimum knowledge MoRA MoRA (80% threshold) All non- MoEC Public Private Urban Rural Islamic (% teachers) Islamic Pedagogy: 80% 0.1 0.1 0 0 0.1 0 0.2 0 Average Score (%) Pedagogy (complete test) 23.2 23.2 33.8 26.3 24.6 23.0 24.9 22.2 Lesson Preparation 22.6 22.6 26.4 23.2 24.1 22.4 22.9 22.4 Student Comparisons 25.4 25.4 36.1 28.7 25.9 25.3 27.7 24.2 Student Evaluations 21.6 21.6 38.9 26.9 23.9 21.3 24.0 20.3 102 Service Delivery Indicator Survey Indonesia 2020 APPENDIX D: Regression Results table D1. Correlates of teacher efforts: Regression results (1) (2) (3) (4) Variables Absent_School Absent_School Absent_Class Absent_CLass Teacher characteristics Female -0.0175 -0.0189 -0.0467** -0.0436** (0.0167) (0.0179) (0.0188) (0.0206) Certificate -0.0226 -0.0218 -0.00250 -0.00236 (0.0191) (0.0215) (0.0215) (0.0247) Born_District 0.00156 -0.00198 -0.0234 -0.0255 (0.0188) (0.0213) (0.0212) (0.0245) Experience_Years -0.00158 -0.00196 -0.00136 -0.00271 (0.00138) (0.00164) (0.00156) (0.00188) civil -0.142 -0.0721 -0.201 -0.0858 (0.122) (0.134) (0.137) (0.154) Isalary_Month -0.0506*** -0.0390*** -0.0404*** -0.0259* (0.0105) (0.0119) (0.0119) (0.0137) Civil * lsalary_Month 0.0268* 0.0162 0.0322* 0.0152 (0.0157) (0.0174) (0.0178) (0.0200) Full Time -0.000460 0.00518 -0.0114 -0.0202 (0.0344) (0.0382) (0.0389) (0.0439) Educ_5 -0.0406 -0.0333 -0.0473* -0.0341 (0.0248) (0.0269) (0.0280) (0.0309) Educ_6 0.0194 0.0498 -0.00323 0.0517 (0.0562) (0.0616) (0.0635) (0.0706) School characteristics Principal_School_Absence 0.273*** 0.265*** (0.0255) (0.0292 Public -0.0209 0.00425 (0.0319) (0.0366) Rural 0.0198 0.0352 (0.0210) (0.0241) nbr_Students_obs_per_Teach -0.00231** -0.00350*** (0.00112) (0.00128) equip_avail -0.00147 -0.00914 (0.0257) (0.0295) electricity -0.000733 -0.0189 (0.0253) (0.0290) Shr_w_txt_book 0.0386 0.0162 (0.0285) (0.0326) Paved Road 0.0454* 0.0613** (0.0250) (0.0287) Traveling_Time -0.0215 -0.0368 (0.0204) (0.0234) Salary_Delay_nbr -0.00189 -0.00347 (0.00608) (0.00698) MoRA non-islamic -0.0878 -0.127* (0.0634) (0.0727) MoEC 0.0550** 0.0611* (0.0275) (0.0315) Constant 0.587*** 0.801*** 0.616*** 0.828*** (0.0779) (0.163) (0.0880) (0.187) Observations 2,301 1,873 2,301 1,873 R-squared 0.021 0.119 0.013 0.094 Note: Robust standard errors in parentheses; *** p<0.01, ** p<0.05, * p<0.1 Service Delivery Indicator Survey Indonesia 2020 103 table D2. Correlates of teacher evaluations: Regression results (4) (1) (2) (3) (4) Variables avg_Lang_ Language Math avg_pedag avg_all Math Teacher characteristics Female 0.0509*** 0.0535*** 5.568*** 5.217*** 5.428*** (0.00825) (0.0112) (0.925) (0.845) (0.791) Certificate -0.0111 -0.0157 -0.384 -1.338 -0.765 (0.0111) (0.0156) (1.225) (1.164) (1.069) Civil servant -0.0101 0.000749 -0.816 -0.468 -0.676 (0.0122) (0.0158) (1.413) (1.205) (1.170) Experience_10_20 -0.0267** -0.0337** -3.547*** -3.020*** -3.336*** (0.0105) (0.0141) (1.180) (1.085) (1.017) Experience_20_30 -0.0391*** -0.0499*** -8.341*** -4.452*** -6.786*** (0.0136) (0.0187) (1.516) (1.392) (1.302) Experience_30_More -0.0986*** -0.125*** -11.74*** -11.16*** -11.51*** (0.0174) (0.0231) (1.940) (1.736) (1.648) Educ_level5 0.00963 0.0619*** 3.085** 3.578** 3.282** (0.0137) (0.0212) (1.566) (1.582) (1.418) Educ_level6 0.0121 0.112*** 10.61*** 6.207*** 8.849*** (0.0221) (0.0277) (2.631) (2.140) (2.210) Teaching practices Teacher_Intro_Lesson 0.00611 0.0131 4.233*** 0.963 2.925** (0.0136) (0.0177) (1.478) (1.346) (1.252) Teacher_Summ_Lesson 0.0144 0.0105 2.690** 1.249 2.114** (0.0106) (0.0141) (1.220) (1.070) (1.036) Teacher_Homework -0.0337** -0.0223 -3.302* -2.798* -3.100** (0.0161) (0.0218) (1.792) (1.636) (1.541) Teacher_Review 0.00692 -0.00764 -1.501 -0.0361 -0.915 (0.0142) (0.0185) (1.619) (1.388) (1.370) Teacher_use_Local_Language -0.0127 -0.0201 -2.137* -1.644* -1.940** (0.00954) (0.0132) (1.186) (0.966) (0.962) School characteristics MoRA non-Islamic 0.0691** 0.0864* 7.237* 7.774** 7.452** (0.0277) (0.0455) (4.348) (3.381) (3.562) MoEC -0.0224* -0.0987*** -2.235* -6.056*** -3.763*** (0.0116) (0.0156) (1.339) (1.126) (1.107) public 0.0284** 0.0747*** 6.771*** 5.158*** 6.126*** (0.0113) (0.0139) (1.312) (1.075) (1.065) Rural -0.0321*** -0.0316*** -3.835*** -3.181*** -3.573*** (0.00829) (0.0110) (0.952) (0.845) (0.803) Constant 0.586*** 0.572*** 17.72*** 57.93*** 33.80*** (0.0232) (0.0349) (2.793) (2.535) (2.419) Observations 1,332 1,332 1,332 1,332 1,332 R-squared 0.214 0.270 0.249 0.283 0.299 Note: Robust standard errors in parentheses; *** p<0.01, ** p<0.05, * p<0.1 104 Service Delivery Indicator Survey Indonesia 2020 Table D3.: Correlations between the Service Delivery Indicators and student test scores Teacher Share of Teacher Absence Time spent test score Minimum Minimum Observed School teachers with test score Students per from class (at teaching per (language, equipment infrastructure student-to- absence rate minimum (language and textbook school) day mathematics availability availability teacher ratio knowledge mathematics) and pedagogy) PANEL A School Level Results -11.11*** -8.475** 2.705*** 3.468* 0.447*** 0.442*** 5.682*** 6.772*** 0.336*** 7.583*** All Schools (2.955) (2.472) (0.422) (2.087) (0.0443) (0.0463) (1.427) (1.096) (0.0660) (1.588) Observations 349 349 348 349 349 349 350 328 350 350 PANEL B School Level Results -9.606*** -7.355*** 2.428*** 6.480*** 0.216*** 0.241*** 5.545*** 6.183*** 0.362*** 6.835*** All Schools (1.406) (1.210) (0.217) (0.927) (0.0177) (0.0183) (0.719) (0.572) (0.0315) (0.886) Observations 3,360 3,360 3,360 3,360 3,360 3,360 3,368 3,149 3,368 3,368 Service Delivery Indicator Survey Indonesia 2020 105 table D4. Correlates of student performance: Regression results (1) (2) (3) (4) (5) (6) Variable Lang_all Lang_Girl Lang_Boy Num_all Num_Girl Num_Boy S t u d e n t C haract e r i st i cs Girl 0.0355*** 0.0456*** (0.00491) (0.00713) Age_10_year 0.0107** 0.00588 0.0139* 0.0125 0.0105 0.0132 (0.00477) (0.00601) (0.00776) (0.00769) (0.0104) (0.0112) Age_11_year -0.0254** -0.0473** -0.0101 -0.0296** -0.0426* -0.0227 (0.0108) (0.0193) (0.0136) (0.0133) (0.0217) (0.0179) Age_12_year -0.0820** -0.144** -0.0601* -0.103*** -0.146*** -0.0962*** (0.0335) (0.0603) (0.0337) (0.0248) (0.0531) (0.0288) Breakfast 0.000628 0.00615 -0.00248 0.0155* 0.0224* 0.00486 (0.00735) (0.00842) (0.0115) (0.00838) (0.0118) (0.0127) Students_nonverb100 0.00144*** 0.00114*** 0.00164*** 0.00266*** 0.00259*** 0.00274*** (0.000127) (0.000167) (0.000180) (0.000135) (0.000211) (0.000185) Stud_lang_bahasa 0.00216 0.00656 -0.00105 0.0298*** 0.0327** 0.0270* (0.00873) (0.0109) (0.0120) (0.0114) (0.0155) (0.0145) Stud_lang_javanese 0.0213** 0.0138 0.0244* 0.0711*** 0.0793*** 0.0611*** (0.00923) (0.0120) (0.0131) (0.0156) (0.0196) (0.0189) TEA C HER C HA RACTERISTICS Teacher Knowledge_score_lang 0.000556** -0.000139 0.00118*** (0.000231) (0.000272) (0.000334) Teacher Knowledge_score_math 0.000254 0.000712* -0.000196 (0.000320) (0.000401) (0.000363) Female 0.00437 0.00597 0.00365 0.0212* 0.0139 0.0282* (0.00693) (0.00764) (0.00964) (0.0124) (0.0160) (0.0145) Civil Servant 0.0101 -0.00356 0.0231 0.0418** 0.0338 0.0535*** (0.0114) (0.0119) (0.0152) (0.0163) (0.0213) (0.0186) Experience_10_20 Years -0.0149* -0.00513 -0.0252** -0.0675*** -0.0455** -0.0881*** (0.00894) (0.0103) (0.0120) (0.0149) (0.0192) (0.0182) Experience_20_30 Years -0.0253** -0.0134 -0.0361** -0.105*** -0.0799*** -0.127*** (0.0116) (0.0131) (0.0162) (0.0187) (0.0259) (0.0227) Experience_30 Years And More -0.00861 -0.0106 -0.00500 -0.0949*** -0.0699** -0.119*** (0.0136) (0.0170) (0.0185) (0.0239) (0.0331) (0.0258) Certificate 0.00724 0.00406 0.0102 0.0380** 0.0319 0.0408** (0.00989) (0.0102) (0.0133) (0.0163) (0.0209) (0.0189) Educ_level5_d4/S1 Bachelor 0.0204 0.0265 0.0124 0.0354** 0.0419* 0.0359* (0.0138) (0.0184) (0.0171) (0.0155) (0.0215) (0.0200) Educ_level6_master 0.0126 0.0125 0.0157 -0.0209 -0.00536 -0.0256 (0.0219) (0.0220) (0.0312) (0.0373) (0.0398) (0.0412) Training Last 2 Years 0.00388 0.0152* -0.00508 -0.0145 -0.00694 -0.0221 (0.00688) (0.00807) (0.00893) (0.0113) (0.0143) (0.0138) Teacher_use_local_language -0.0206*** -0.0172* -0.0230** -0.00924 -0.0120 -0.00732 (0.00775) (0.0102) (0.0101) (0.0133) (0.0171) (0.0158) S ch o o l C haract e r i st i cs Principal Female -0.000786 -0.00973 0.00924 0.00701 0.0180 -0.00346 (0.00606) (0.00764) (0.00849) (0.0108) (0.0134) (0.0134) Public 0.0249** 0.0147 0.0382** 0.0450*** 0.0291 0.0552** (0.0115) (0.0140) (0.0152) (0.0168) (0.0210) (0.0241) Rural -0.0321*** -0.0256*** -0.0389*** -0.0257** -0.0246 -0.0260* (0.00740) (0.00784) (0.0108) (0.0118) (0.0153) (0.0142) Mora Non Islamic 0.0423*** 0.0459*** 0.0253* 0.203*** 0.182*** 0.231*** (0.0112) (0.0129) (0.0142) (0.0428) (0.0449) (0.0459) Moec -0.0150 -0.00382 -0.0297* -0.00795 0.0194 -0.0322 (0.0108) (0.0128) (0.0157) (0.0159) (0.0190) (0.0236) Constant 0.683*** 0.716*** 0.782*** -0.0275 -0.0250 -0.0867* (0.0359) (0.0388) (0.0508) (0.0427) (0.0607) (0.0496) Observations 3,063 1,481 1,582 2,997 1,468 1,529 R-Squared 0.204 0.193 0.229 0.293 0.292 0.303 Note: School clusters and regional controls; Robust standard errors in parentheses; *** p<0.01, ** p<0.05, * p<0.1 106 Service Delivery Indicator Survey Indonesia 2020 table D5. Correlates of student performance. Household Sample (1) (2) (3) (4) (5) (6) Variable Lang_all Lang_Girl Lang_Boy Num_all Num_Girl Num_Boy S t u d e n t C haract e r i st i cs Girl 0.0365*** 0.0540*** (0.00803) (0.0118) Age_10_year 0.0208** 0.0211* 0.0203 0.0286** 0.0379* 0.0203 (0.00870) (0.0116) (0.0138) (0.0140) (0.0204) (0.0200) Age_11_year -0.0226 0.00576 -0.0393* -0.0129 0.00651 -0.0247 (0.0181) (0.0340) (0.0225) (0.0261) (0.0457) (0.0352) Age_12_year -0.0273 -0.0330 0.00267 -0.0650* -0.0407 -0.0432 (0.0429) (0.0614) (0.0520) (0.0342) (0.0542) (0.0497) Pre-Primary 0.0375** 0.0149 0.0689*** 0.0492*** 0.0519* 0.0525** (0.0162) (0.0191) (0.0225) (0.0174) (0.0302) (0.0221) Scholarship -0.0170 -0.0186 -0.0251 -0.0353** -0.0179 -0.0605*** (0.0112) (0.0131) (0.0160) (0.0137) (0.0185) (0.0206) Breakfast -0.000818 0.00558 0.00332 -0.00196 0.0175 -0.0213 (0.0125) (0.0153) (0.0189) (0.0169) (0.0224) (0.0249) Students_nonverbal 0.00101*** 0.000866*** 0.00102*** 0.00244*** 0.00273*** 0.00214*** (0.000200) (0.000289) (0.000298) (0.000251) (0.000374) (0.000338) Stud_lang_bahasa -0.0174 -0.00158 -0.0275 0.0326* 0.0147 0.0491** (0.0124) (0.0190) (0.0182) (0.0173) (0.0262) (0.0239) Stud_lang_javanese 0.00229 0.0171 -0.00182 0.0530** 0.0512* 0.0530* T e ach e r C haract e r i st i cs (0.0135) (0.0189) (0.0195) (0.0219) (0.0296) (0.0312) Teacher Knowledge_avg_lang 0.000867*** 0.000221 0.00107** (0.000281) (0.000432) (0.000450) Teacher Knowledge_avg_math 0.000223 0.000670 -0.000468 (0.000405) (0.000535) (0.000574) Experience_10_20 Years -0.0162 -0.0191 -0.00515 -0.0821*** -0.0817*** -0.0836*** (0.0109) (0.0149) (0.0165) (0.0205) (0.0279) (0.0296) Experience_20_30 Years -0.0273* -0.0140 -0.0290 -0.125*** -0.109*** 0.131*** (0.0154) (0.0184) (0.0236) (0.0268) (0.0397) (0.0369) Experience_30 Years And More 0.00102 -0.00841 0.0191 -0.0789** -0.0532 -0.0943** (0.0197) (0.0282) (0.0309) (0.0311) (0.0461) (0.0408) Certified -0.0101 -0.00625 -0.0172 0.0425* 0.0242 0.0456 (0.0121) (0.0155) (0.0174) (0.0224) (0.0296) (0.0293) Female -0.00111 -0.00551 0.00451 0.0252 -0.00564 0.0488** (0.00965) (0.0116) (0.0146) (0.0157) (0.0216) (0.0198) Civil Servant 0.0201 -0.00237 0.0371* 0.0364* 0.0546* 0.0351 (0.0142) (0.0184) (0.0213) (0.0216) (0.0295) (0.0284) Educ_level5_d4/S1 Bachelor 0.0335* 0.0585** 0.00290 0.0493** 0.0799** 0.0475* (0.0177) (0.0264) (0.0278) (0.0208) (0.0326) (0.0286) Educ_level6_master 0.0175 0.0521 -0.0131 0.00143 0.0575 -0.0254 (0.0318) (0.0320) (0.0555) (0.0518) (0.0577) (0.0637) Training Last 2 Years -0.0146 -0.000739 -0.0281* -0.00503 0.0204 -0.0327* (0.00951) (0.0118) (0.0145) (0.0146) (0.0210) (0.0196) Teacher_use_local_language -0.0296*** -0.0118 -0.0494*** -0.00825 0.0133 -0.0296 S ch o o l C haract e r i st i cs (0.0105) (0.0129) (0.0162) (0.0178) (0.0250) (0.0239) Principal Female -0.00200 0.00496 -0.00618 0.0220 0.0742*** -0.0237 (0.00821) (0.00982) (0.0131) (0.0150) (0.0207) (0.0208) Public 0.00426 -0.0103 0.0134 0.0466** 0.0267 0.0572* (0.0138) (0.0183) (0.0195) (0.0231) (0.0349) (0.0331) Rural -0.0293*** -0.0235* -0.0276** 0.00272 -0.00108 0.0145 (0.00948) (0.0120) (0.0139) (0.0157) (0.0207) (0.0209) Mora Non-Islamic 0.0184 0.0178 0.00904 0.112** 0.0943 0.116* (0.0165) (0.0277) (0.0346) (0.0476) (0.0598) (0.0635) Moec -0.00824 0.00303 -0.0134 -0.0191 -0.0257 -0.0239 (0.0127) (0.0166) (0.0190) (0.0215) (0.0310) (0.0337) Constant 0.669*** 0.648*** 0.676*** -0.0200 -0.140 0.0906 (0.0410) (0.0570) (0.0696) (0.0574) (0.0936) (0.0728) Observations 939 443 496 919 439 480 R-Squared 0.241 0.281 0.297 0.311 0.348 0.329 Note: School clusters and regional controls; Robust standard errors in parentheses; *** p<0.01, ** p<0.05, * p<0.1 Service Delivery Indicator Survey Indonesia 2020 107 FIGURe D1. Correlates of student performance: Indonesian Language Household sample Female Student 10 Year Old Pre-Primary Scholarship D4/S1 Bachelor Teacher Experience_10 - 20 Years Teacher Experience_20 - 30 Years Teacher Experience_30_More_Years Teacher_Test_Language Teacher_use_Local_Language Rural -.05 0 .05 .1 FIGURe D2. Correlates of student performance: Mathematics Household sample Female Student 10 Year Old Pre-Primary Scholarship Breakfast Bahasa Mother Tongue Javanese Mother Tongue D4/S1 Bachelor Experience_10 - 20 Mathly Experience_20 - 30 Mathly Experience_30_More_Mathly Certificated Teacher Civil Mathly Other Religious Schools -.2 -.1 0 .1 .2 108 Service Delivery Indicator Survey Indonesia 2020 References Afkar, R; Luque, J; Marshall, J; Nomura, S. 2020. Revealing How Indonesia’s Subnational Governments Spend their Money on Education, World Bank. Jakarta. © World Bank Afkar, R., N. Yarrow, S. Surbakti, and R. Cooper. 2020. Inclusion in Indonesia’s Education Sector: A Subnational Review of Gender Gaps and Children with Disabilities. Policy Research Working Papers. Washington, DC: World Bank. https://doi.org/10.1596/1813-9450-9282. Bold, Tessa, Deon Filmer, Gayle Martin, Ezequiel Molina, Brian Stacy, Christophe Rockmore, Jakob Svensson, and Waly Wane. 2017. “Enrollment without Learning: Teacher Effort, Knowledge, and Skill in Primary Schools in Africa.” Journal of Economic Perspectives, 31 (4): 185-204.DOI: 10.1257/jep.31.4.185 Data for the Sustainable Development Goals, UNESCO Institute of Statistics. 2018. data.uis.unesco.org/ Index.aspx?DataSetCode=EDULIT_DS&popupcustomise=true&lang=en. Daniel Muijs, Leonidas Kyriakides, Greetje van der Werf, Bert Creemers, Helen Timperley & Lorna Earl. 2014. State of the art – teacher effectiveness and professional learning, School Effectiveness and School Improvement, 25:2, 231-256, DOI: 10.1080/09243453.2014.885451 De Ree, J. 2016. How much teachers know and how much it matters in class. World Bank Policy Research Working Paper 7556 De Ree, Joppe Jaitze; Muralidharan, Karthik; Pradhan, Menno Prasad; Rogers, F. Halsey. 2017. Double for nothing? Experimental evidence on an unconditional teacher salary increase in Indonesia (English). Policy Research working paper; no. WPS 8264. Washington, DC: World Bank Group. http://documents.worldbank. org/curated/en/616961512396126770/Double-for-nothing-experimental-evidence-on-an-unconditional- teacher-salary-increase-in-Indonesia Dobbie, W., & Fryer, Jr., R. G. 2013. Getting beneath the veil of effective schools: Evidence from New York City. American Economic Journal: Applied Economics, 5(4), 28–60. Duckworth, Angela Lee, and Patrick D. Quinn. 2009. “Development and validation of the Short Grit Scale (GRIT–S).” Journal of personality assessment 91.2: 166-174. Duckworth, Angela Lee, and Stephanie M. Carlson. 2013. “Self-regulation and school success.” Self-regulation and autonomy: Social and developmental dimensions of human conduct 40: 208. Dweck, Carol S. 1999. “Essays in social psychology.” “DWMS Scoring Guide.” Development World Management Survey, September 13, 2017, developingmanagement.org/wp-content/uploads/2016/12/DWMS-scoring-grid-2017-external.pdf. “Education Statistics - All Indicators.” DataBank - Education Statistics, World Bank, databank.worldbank.org/ reports.aspx?source=Education Statistics#. EMIS 2019. http://emispendis.kemenag.go.id/dashboard/?content=data# Service Delivery Indicator Survey Indonesia 2020 109 Filmer, D., Molina, E., & Stacy, B. 2015. Teacher knowledge, behavior and student achievement. Evidence from 5 countries in Africa. Filmer, D. and Rogers, H., 2018. Learning to realize education’s promise. World Development Report. The World Bank. https://www.worldbank.org/en/publication/wdr2018 Gaduh, A., Pradhan, M., Priebe, J., and Susanti, D. 2020. “Scores, Camera, Action? Incentivizing Teachers in Remote Areas.” RISE Working Paper Series. 20/035. https://doi.org/10.35489/BSG-RISE-WP_2020/035. Gauthier, Bernard,  Mohamed Salem Tfeil,  Waly Wane and Yahya Abou Ly (2019) “Education Service Delivery in Mauritania: Results of 2017 Service Delivery Indicators (SDI) Survey”, World Bank, Africa Region, mimeo, March. “GNI per Capita, Atlas Method (Current US$).” Data, World Bank Group, data.worldbank.org/indicator/ ny.gnp.pcap.cd. Hanushek, E. A., & Rivkin, S. G. 2006. Teacher quality. Handbook of the Economics of Education, 2, 1051–1078. Hanushek, E. A., & Woessermann, L. 2007. The role of education quality for economic growth. World Bank Policy Research Working Paper, (4122). “Human Capital.” World Bank, www.worldbank.org/en/publication/human-capital#Data. Johnson, David, Andrew Cunningham, and. Rachel Dowling. 2012. “Teaching Standards and. Curriculum Review.” John, Oliver P., and Sanjay Srivastava. 1999. “The Big Five trait taxonomy: History, measurement, and theoretical perspectives.” Handbook of personality: Theory and research 2.1999: 102-138. Laporan Hasil Ujian Nasional: Kementerian Pendidikan Dan Kebudayaan. 2019. hasilun.puspendik.kemdikbud. go.id/#2019!smp!capaian_nasional!99&99&999!T&T&T&T&1&!1!&. Lavy, V. 2010. Do Differences in Schools Instruction Time Explain International Achievement Gaps? Evidence from Developed and Developing Countries. National Bureau of Economic Research, Inc. Lemos, Renata, and Daniela Scur. 2015. “Developing Management: An Expanded Evaluation Tool for Developing Countries.” Jan. 2016, doi:10.35489/bsg-rise-wp_2016/007. Molina, Ezequiel, and Gayle Martin. 2015. Education Service Delivery in Mozambique. October. World Bank. Molina, Ezequiel et al. 2018. “SABER Service Delivery: The Learning Crisis in Afghanistan.” SABER Service Delivery: The Learning Crisis in Afghanistan ( )| , 5 Sept. 2018, documents. vsemirnyjbank.org/curated/ru/588881536147087211/SABER-Service-Delivery-The-Learning-Crisis-in- Afghanistan. 110 Service Delivery Indicator Survey Indonesia 2020 “Mozambique - Service Delivery Indicators Education Survey 2014 - Harmonized Public Use Data.” Mozambique - Service Delivery Indicators Education Survey 2014 - Harmonized Public Use Data (SDI-E 2014) | Data Catalog, World Bank, 6 Jan. 2017, datacatalog.worldbank.org/dataset/mozambique-service-delivery- indicators-education-survey-2014-harmonized-public-use-data. Muijs, Daniel et al. 2014. “State of the art–teacher effectiveness and professional learning.” School effectiveness and school improvement 25.2: 231-256. “Nationality, Ethnicity, Religion, and Daily Language of Indonesian Population.” Badan Pusat Statistik, 2010, www.bps.go.id/publication/2012/05/23/55eca38b7fe0830834605b35/kewarganegaraan-suku-bangsa- agama-dan-bahasa-sehari-hari-penduduk-indonesia.htm. OECD. 2016. Country Note – Results from PISA 2015 – Indonesia. https://www.oecd.org/pisa/PISA-2015- Indonesia.pdf OECD. 2019. Country Note – Results from PISA 2018 – Indonesia. https://www.oecd.org/pisa/publications/ PISA2018_CN_IDN.pdf Service Delivery Indicators, World Bank, 2017, www.sdindicators.org/methodology#samp_desn. State Address, August 2019. Statistik Pendidikan Indonesia (BPS) accessed August 7, 2020 https://bit.ly/30B78JI Statistik Sekolah, Ministry of Education and Culture, accessed August 7 2020, http://publikasi.data. kemdikbud.go.id/index.php?thn=all&sek_id=&bpid=&pageNum=3 Trako, Iva et al. Making Great Strides Yet a Learning Crisis Remains in Tanzania. wbgfiles.worldbank.org/ documents/hdn/ed/saber/supporting_doc/AFR/Tanzania/SDI/Tanzania-SDI_SABER SD Report_Oct7.pdf. USAID. 2014. “Indonesia 2014: The National Early Grade Reading Assessment (EGRA) and Snapshot of School Management Effectiveness (SSME) Survey Report of Findings.” Rep. Findings. World Bank. 2015. “Assessing the Role of the School Operational Grant Program (BOS) in Improving Education Outcomes in Indonesia”, World Bank. World Bank. 2017. “Decentralization That Delivers.” Indonesia Economic Quarterly, World Bank. December 2017, pubdocs.worldbank.org/en/657051513163708686/IEQ-Dec-2017-ENG.pdf. World Bank. 2018. “The Human Capital Project.” Open Knowledge Repository, World Bank, Washington, DC, October 11, 2018. openknowledge.worldbank.org/handle/10986/30498. World Bank. 2019. “Ending Learning Poverty.” Open Knowledge Repository, World Bank, Washington, DC. 16 Oct. 2019. openknowledge.worldbank.org/handle/10986/32553. Service Delivery Indicator Survey Indonesia 2020 111 World Bank. 2019. Primary Education in Remote Indonesia: Survey Results from West Kalimantan and East Nusa Tenggara World Bank. 2019. Realizing Education’s Promise Project Appraisal Document “World Development Indicators.” World Development Indicators (WDI) | Data Catalog, World Bank, June 11, 2010, datacatalog.worldbank.org/dataset/world-development-indicators. Woodward, Mark & Rohmaniyah, Inayah & Amin, Ali & Coleman, Diana. (2010). Muslim Education, Celebrating Islam and Having Fun As Counter-Radicalization Strategies in Indonesia. Perspectives on Terrorism. 4. 28-50. Yarrow, Noah; Masood, Eema; Afkar, Rythia. 2020. Estimated Impacts of COVID-19 on Learning and Earning in Indonesia: How to Turn the Tide. World Bank, Jakarta. © World Bank. Yeager, David S. et al. 2019. “A National Experiment Reveals Where a Growth Mindset Improves Achievement.” Nature News, Nature Publishing Group, August 7, 2019. www.nature.com/articles/s41586-019- 1466-y. 112 Service Delivery Indicator Survey Indonesia 2020