91321
How can
Bulgaria improve
its education
system?
An analysis of PISA 2012
and past results
Education Global Practice
Europe and Central Asia Region
Acknowledgments
This report was prepared by Lucas Gortazar, Katia Herrera-Sosa, and Daniel Kutner, with analytical support
from Martin Moreno and editing by Amy Gautam. The report is part of the World Bank’s education sector
knowledge and advisory services provided to the Government of Bulgaria between 2013 and 2015, led by
Plamen Danchev. This report is the first volume of the PISA Country Series conducted by the Education
Unit in the World Bank’s Europe and Central Asia Region. The team is deeply grateful to Alberto Rodriguez,
Andrea Guedes, Naveed Hassan Naqvi, Cristian Aedo, Omar Arias, and Shagun Mehrotra for their helpful
guidance, comments, and assistance. Juan Manuel Moreno and Igor Kheyfets kindly peer reviewed the
report; Christian Bodewig, Ismail Radwan, Valerie Morrica, and Allison Berg kindly provided comments.
Finally, the team would like to acknowledge the contribution of art director Nicholas Dehaney.
Contents
Abbreviations and Acronyms
Executive Summary 01
1 Why is PISA Important? An Overview of Bulgaria’s Performance on the PISA 05
2 What Determines the Quality of Education in Bulgaria and How Can It Be Improved? 13
The importance of pupils’ origin: the lifelong impact of unequal opportunities 14
Gender disparities 16
Urban-rural disparities 16
Linguistic minorities 17
What can policy makers do to improve education in Bulgaria? 18
A stratified school system: the importance of peers and tracking mechanisms 18
Has the 2007 school autonomy reform worked? 22
Early childhood policies, teacher practices, and the school environment 25
3 Drilling Down Further Into Math and Reading Skills 27
Math skills in PISA 2012 28
Reading skills in PISA 2009 29
Findings and Recommendations 33
4.
References 37
Annex 39
Boxes
Box 1 Bulgaria’s Education System 08
Box 2 PISA’s Index of Economic, Social, and Cultural Status 11
Box 3 Bulgaria’s 2007 Autonomy Reform 23
Figures
Figure 1 PISA scores and public expenditures per pupil 07
Figure 2 PISA 2012 scores for Bulgaria and comparator countries versus ECA and OECD averages 07
Figure 3 Distribution of students by proficiency level in math: (a) Bulgaria’s progress in 2006-2012;
(b) Bulgaria and comparators in 2012 09
Figure 4 Student performance in math on PISA 2006-2012 by socioeconomic group 11
Figure 5 Index of equality of opportunities: Bulgaria and other ECA countries, 2012 15
Figure 6 Math and reading performance of Bulgarian students by gender 16
Figure 7 Index of School Social Stratification in PISA 2012-participating countries 18
Figure 8 Evolution of PISA math scores by school type 19
Figure 9 Decomposition of general profiled-vocational school PISA 2012 math score gaps
into different factors by student achievement group 21
Figure 10 Decomposition of changes in PISA math scores gaps between 2006 and 2012
into factors and by student achievement group 22
Figure 11 Math contents and process categories in PISA 2012 29
Figure 12 PISA 2012 performance on different math subscales compared to
the average math performance 30
Figure 13 PISA 2009 performance on different reading subscales compared to the
combined reading performance 31
Tables
Table 1 PISA performance byscale for Bulgaria in 2000-2012 10
Table 2 Determinants of achievement in PISA by categories 15
Table 3 Characteristics of students in Bulgaria by language group in 2012 17
Table 4 Rate of affirmative responses on the responsibility over resources and pedagogy
of each body, as reported by school principals 23
Abbreviations
& Acronyms
ESCS
ECA
Economic, Social, and Cultural Status
Europe and Central Asia
EU European Union
GDP Gross domestic product
OECD Organization for Economic Co-operation and Development
OLS Ordinary least squares
PIRLS Progress in International Reading Literacy Study
PISA Programme for International Student Assessment
RIF Re-centered influence functions
TIMSS Trends in International Mathematics and Science Study
UN United Nations
UNESCO United Nations Educational, Scientific and Cultural Organization
VET Vocational education and training
analysis of PISA 2012 and past results 1
Executive Summary
Bulgaria’s performance on all three disciplines of the PISA1 2012 was slightly better than its PISA
2000 performance, after having dropped between 2000 and 2006. The improvements post-2006
were greater in reading and math than in science. In the latest PISA (2012), Bulgarian students scored an
average of 34 points more on reading and 26 points more on math than in 2006. This represents gains
equivalent to almost one year of schooling in reading and a little more than half a year of schooling in math.
Despite the recent improvement in achievement, Bulgaria has not made significant progress since 2000
and its performance gap with the OECD accounts for more than one year of schooling. Moreover, around
39 percent of 15-year-old students in Bulgaria are considered functionally illiterate, as they are not able to
understand and analyze what they read. Similarly, about 44 percent of Bulgarian students are considered
functionally innumerate.
The improvements in performance between 2006 and 2012 promoted shared prosperity, but
equality of opportunities is still a major challenge. The gains in Bulgaria’s education system between
2006 and 2012 were such that students in the bottom 40 percent of the socioeconomic status made
improvements comparable to those of average students (that is, the average score of all students who took
the PISA). However, a persistent challenge is the PISA score differentials between students in the highest
and lowest socioeconomic quintiles. For instance, in math the difference is approximately 115 points, much
higher than OECD standards. Moreover, in Bulgaria, students’ predetermined individual characteristics
play a disproportionately high role in explaining PISA scores. Gender, age, and socioeconomic status2
account for almost one-third of students’ differences in reading performance. This reflects the low equality
of students’ educational opportunities, as an important share of performance is predefined by students’
backgrounds, potentially limiting social mobility.
In fact, disaggregating students’ PISA scores across a number of variables — e.g., location and
ethnicity — shows that large inequalities exist in Bulgaria’s education system. Students living in
urban areas score as high as 90 points more (or more than two years of schooling) than students in rural
areas. There are discrepancies for linguistic minorities as well: Bulgarian-speaking students perform the
equivalent of three years of schooling higher in reading and two years of schooling higher in math and science
than students who speak a foreign language at home.
Peer characteristics and school segregation are the key drivers of the Bulgarian education
system’s performance. In Bulgaria, peer characteristics explain more of the differences in PISA test scores
than do individual characteristics. That is, the performance of a child on the PISA test depends more on
the type of his or her classmates than on his or her own individual factors. This is because the system sorts
students into schools populated by other students with similar socioeconomic status, rendering Bulgaria
with one of the most stratified educational systems among PISA participating countries.
2 How Can Bulgaria Improve Its Education System?
Disparities in performance by school type are large and are exacerbated by the early streaming
of students. In Bulgaria, students are streamed into either general profiled or vocational tracks after
they take a high-stakes national exam at age 13. The consequence is that most students in general profiled
schools, which have a very low share of disadvantaged students, tend to fare quite well. General profiled
school students tend to come from families with higher socioeconomic status and interact with similarly
better off peers. But over half of Bulgaria’s 15-year-old student population struggles in the worse performing
vocational or general non-profiled schools. The analysis of the learning gap between general profiled and
vocational education students in Bulgaria shows that socioeconomic status and peer effects explain most of
the differences in student outcomes for low-, medium-, and high-achieving pupils.
The effects of Bulgaria’s 2007 school autonomy reform on student achievement are mixed and
worse than expected. The 2007 governance reform in Bulgaria was a major effort that delegated several
responsibilities to school principals—particularly in setting teacher salaries, handling student assessment
and admission, undertaking more financial responsibilities, and determining textbook use and course
contents. Results of the reform vary by type of autonomy. On one hand, the results show that principals’
greater autonomy in the allocation of resources (such as policies regarding teachers or budget decisions) had
a moderately positive impact on all students’ performance (6 PISA points on average), and especially that
of low-achieving students. This impact was stronger in urban than in rural areas. On the other hand, the
impacts of principals’ greater curriculum and assessment responsibilities on students’ PISA performance
were slightly negative, especially in rural areas. Finally, the analysis showed the importance of the quality of
educational resources as a key driver of the student performance increase since 2006.
An in-depth analysis into math and reading skills shows imbalances in performance in Bulgaria.
PISA rotates the in-depth assessment of skills by subject area each time it is administered. PISA 2009
focused on reading, while PISA 2012 focused on math; PISA 2015 will focus on science. Compared with the
combined math performance, results in Bulgaria show slightly higher variation across subscale assessments
than is found in OECD countries. Students performed better in problems related to space and shape and
algebra, and not as well in problems related to data and statistics. In reading, students performed better
with more traditional text than with text contained in sample lists, graphs, or diagrams. Moreover, the PISA
subscale assessments reveal that Bulgarian students are not good at relating information presented in a text
to their own experiences.
analysis of PISA 2012 and past results 3
The main areas in which Bulgaria can further improve its educational system involve:
Delaying the tracking3 of students to reduce segregation in schools. Bulgaria streams its students
1
into general profiled, general non-profiled, and vocational education schools when they are 13 years old
through a high-stakes exam. Existing admission policies on a number of primary schools suggest that
this mechanism leads to sorting as early as grade 1. Most countries do this at a later stage, usually when
students are 16 years old. A recent World Bank report (2013a) found that the prospective of high-stakes
exams creates incentives for parents to invest in private tutoring to help their children increase their scores,
leading to sorting among families, which raises important equity concerns. Indeed, PISA score differences
between the three streams are fully explained by socioeconomic background and peer effects. Moreover,
early tracking hampers the skill development and future long-term employability of students in vocational
schools, as they will lack the basic reading and math skills needed for success in a dynamic and rapidly
changing labor market. Finally, alternatives to the high-stakes exam that implicitly select students into
schools more randomly could further reduce segregation based on students’ abilities.
2
Continue improving the quality of educational resources to ensure that all students learn in
an environment with books, lab equipment, and technological hardware and software.
The analysis of the improvement in performance in math and reading between 2006 and 2012 shows
that the two key drivers were the evolution of students’ socioeconomic status and the improved quality of
educational resources. The impact of educational resources was especially important for low-achieving
students, indicating a low-hanging fruit for improving the quality and equity of the education system.
Continuation of this would include better provision of lab equipment, computer and software materials,
library materials, and instructional materials and/or the renovation of buildings and grounds.
Encouraging longer pre-primary education for all children. Pre-primary education increases
3
school readiness and has a positive and significant effect on the student achievement of Bulgarian 15-year-
olds. This study found that attending at least two years of preschool education raises low achievers’
scores by up to 10 points and the scores of those who speak a different language at home by up to 19
points. In Bulgaria, the pre-primary gross enrollment rate for children aged three to six is 84 percent, but
disadvantaged students and minorities still face challenges in accessing this education stage. Promoting
early childhood education for all is critical, as cognitive and character skills gaps start opening during early
life and inequalities in access to pre-kindergarten perpetuate learning gaps across income groups.
4 How Can Bulgaria Improve Its Education System?
4
Learning from successful schools to improve accountability mechanisms for schools country-
wide, particularly in rural areas. There is a need to further understand: (i) why the autonomy reform did
not function as expected; (ii) why the reform was more successful in urban areas; and (iii) why PISA scores
were positively affected by greater autonomy in the management of school resources, but not by greater
autonomy in curriculum development and assessment. Learning from successful schools could help the
Government of Bulgaria augment the impact of the reform in rural areas over the coming years.
5
Reevaluating the curriculum and assessment framework to better align student learning to the
envisaged country goals. PISA results shed light on Bulgaria’s large discrepancies with other countries
within different reading and math skills. PISA results present a good opportunity to engage in an in-depth
debate about a curriculum and assessment framework reform, as well as how to better align the education
system with national social and economic development goals.
Promoting effective classroom management and strengthening teaching practices. The
6
analysis shows that a class that is orderly and has fewer disruptions to students is more conducive to
learning and therefore improves PISA scores. The government could use classroom observation methods
and international best practices on classroom management to help teachers identify opportunities to
improve their performance in the classroom. Teacher development programs could be implemented to
improve management techniques in the classroom for the current and future teaching workforce, yielding
rapid improvements in the quality of learning.
analysis of PISA 2012 and past results 5
Why
1. is PISA
important An Overview
?
of Bulgaria’s
Performance on the PISA
6 How Can Bulgaria Improve Its Education System?
Education and skills are critical for the development of both countries and individuals.
International evidence suggests that quality of education is one of the most important determinants
of long-term economic growth.4 Hanushek and Woessman (2007 and 2012) looked at a wide range of
student assessment surveys from 1960 onward, including the Trends in International Mathematics
and Science Study (TIMSS), the Programme for International Student Assessment (PISA), and the
Progress in International Reading Literacy Study (PIRLS). They estimated that an improvement of 50
points in PISA scores would imply an increase of 1 percentage point in the annual growth rate of GDP
per capita.5 Top-quality education systems are also associated with democratic governments. Beyond
economic growth, education improves the living standards of individuals, as the more educated are
able to acquire more and higher-order skills, making them more productive and employable and
extending their labor market participation over their lifetime, which in turn leads to higher earnings
and better quality of life. Formal schooling also contributes to development of socio-emotional skills like
attention, motivation, self-confidence, and physical and emotional health, all important determinants
of socioeconomic mobility. Individuals equipped with more education and skills are better prepared to
become civically engaged, improve the democratic capital of their country, and create and make use of
opportunities. Education is a key ingredient for reducing inequality and increasing shared prosperity.
The analysis of detailed data is critical for understanding the determinants of education quality and can
play an important role in shaping effective evidence-based education policy. The PISA database is a great
resource in the pursuit of this analysis.
PISA is a tool for measuring education quality across countries. Introduced in 2000 by the
Organisation for Economic Co-operation and Development (OECD), PISA is a worldwide study of 15-year-
old school students’ performance on three different disciplines: math, science, and reading. PISA focuses
on the competence of students and their ability to tackle real-life problems in those three disciplines
and emphasizes skills that are critical for individuals’ personal and professional development. PISA
only assesses students who are in the education system, making it the most internationally comparable
snapshot available of a country’s education system. However, if dropout rates are high, the results may
not be representative of a country’s cohort of 15-year-olds. PISA’s scoring system is standardized so that
the mean score for each discipline among OECD countries in year 2000 is 500 points, with a standard
deviation of 100 points. According to OECD, 40 points in PISA is equivalent to what students learn in one
year of schooling.6 Bulgaria’s education system (Box 1) was assessed in the PISA rounds of 2000, 2006,
2009, and 2012. Bulgaria’s participation in PISA allows us to benchmark it with other countries, measure
the extent to which the country has succeeded in promoting education quality, and gauge whether system
inequities have been reduced over time.
analysis of PISA 2012 and past results 7
Figure 1 PISA scores and public expenditures per pupil
580
East Asia
Korea Hong Kong
540 Japan
Europe & Switzerland
Central Asia Estonia Finland Netherlands
Poland
PISA score in mathematics, 2012
Belgium
Czech Republic New Zealand Australia Austria
500 Latvia Slovenia France Ireland
United Kingdom
Denmark
Portugal Iceland Norway
Lithuania Italy Sweden
Croatia Slovak Republic Spain United States
Hungary
460 Isreal
Serbia Cyprus Western Europe
Romania & US/Canada
Kazakhstan Bulgaria
420 Thailand Chile Malaysia
Mexico
Argentina Latin American
380 Indonesia
Colombia and Caribbean
Peru
340
0 2000 4000 6000 8000 10000 12000 14000 16000
Public expenditures per pupil (in PPP dollars), UNESCO 2012 or latest
Source: PISA 2012 and UNESCO 2012. Note: The curve represents a logarithmic approximation of the scatter plots.
Figure 2 PISA 2012 scores for Bulgaria and comparator countries versus ECA and OECD averages
530
510
PISA 2012 Scores
490 One year of
schooling
470
450
430
410
390
Bulgaria
Romania
Turkey
Serbia
ECA
EU12
OECD
Romania
Turkey
ECA
Serbia
ECA
EU12
OECD
Poland
Serbia
Bulgaria
Turkey
EU12
OECD
Poland
Bulgaria
Romania
Poland
Source: PISA 2012. Math Reading Science
8 How Can Bulgaria Improve Its Education System?
Box 1 Bulgaria’s Education System
Bulgaria has a population of 7,36 million people (2011), with three large ethnic groups. Those of
Bulgarian ethnicity comprise 85 percent of the population; those of Turkish ethnicity, 9 percent;
and those of Roma ethnicity, 5 percent. The education system serves over 1.2 million students from
pre-primary school through tertiary education. According to UN estimates, Bulgaria’s school-age
population is projected to shrink by 10 percent between 2015 and 2030, reflecting the impact of low
fertility and migration.
Bulgaria’s education system consists of four levels. Pre-primary education is offered to children
between three and six (or seven) years old and since 2010, two years of pre-schooling are
compulsory, starting from age five. Basic education comprises grades 1 to 8, usually starts at age
seven, and is offered by state, municipal, and private school providers. Although lower secondary
does not end until the end of grade 8, most students change schools after grade 7, once they take
a high-stakes exam that streams students into general profiled schools, vocational education and
training (VET) schools, or general non-profiled schools. Upper secondary education is provided by
non-profiled, profile-oriented, and technical (vocational) schools. General profiled schools (often
referred as “elite schools”) offer general education with additional focus on a selected subject
(e.g., a foreign language, mathematics, information and communication technologies (ICT),
etc.). General non-profiled schools provide education without extra focus on a given subject, while
vocational schools incorporate vocational subjects into the curriculum, often at the expense of time
allocated to general curriculum subjects. Education is compulsory for students up to the age of 16.
Source: National Statistical Institute and Ministry of Education, Youth and Science, and World Bank (2014).
Bulgaria’s performance is slightly below what PISA 2012 recovered to levels slightly above those
should be expected given its current level of of 2000, after having dropped between 2000 and
public expenditure per student (Figure 1). In 2006 (Table 1). On PISA 2000, Bulgarian students’
addition, Bulgaria’s performance is worse than performance in science was substantially better than
expected given its income level. Comparator in reading and math. The drop in 2006 was more
countries like Serbia, Romania, and Turkey acute for math and reading, and the recovery in these
performed better than Bulgaria on PISA 2012. disciplines was stronger between 2006 and 2012.
While a certain level of financial resources is
important to ensure access to a minimum standard Bulgaria’s performance is worse than that of
of quality, higher levels of expenditures and regional comparator countries (Figure 2).
development do not necessarily imply better learning Despite its improved performance since 2006,
outcomes. In the case of upper-middle-income Bulgaria’s scores are still lower than those in many
countries like Bulgaria, more investment can still Europe and Central Asia (ECA) region countries, and
help improve quality, but additional policy efforts its math and reading scores lag 30 points behind the
are needed to take education quality to the next level ECA average. While PISA score changes in Bulgaria
and make the improvement sustainable. between 2000 and 2012 were not statistically
significant, countries such as Turkey and Poland
Bulgaria has not made significant progress in carried out sustained and systemic reforms and
achievement since 2000. Bulgaria’s performance on saw their scores go up by 30 (Turkey) to 40 (Poland)
analysis of PISA 2012 and past results 9
Figure 3 Distribution of students by proficiency level in math: (a) Bulgaria’s progress in 2006-2012;
(b) Bulgaria and comparators in 2012
60%
50%
40% n 2006
n 2009
n 2012
30%
20%
10%
0%
Below level 2 Level 2 Level 3 Level 4 Level 5+
100%
90%
n Level 5+
80%
n Level 4
70%
n Level 3
n Level 2
60% n Below
Level 2
50%
40%
30%
20%
10%
Bulgaria Turkey Romania Serbia ECA OECD EU12 Poland
0%
Source: PISA 2006, 2009, and 2012.
10 How Can Bulgaria Improve Its Education System?
2+2+6=10
4+4+5=13
Table 1 PISA performance by scale for Bulgaria in 2000-2012
2000 2006 2009 2012
Reading 430 402 429 436
Math 430 413 428 439
Science 448 434 439 446
Source: PISA 2000, 2006, 2009, and 20127
points. Finally, Bulgaria’s scores are about 40 points 15-year-old students in Bulgaria score below level 2
below those of EU12 new-member states, and need in math (Figure 3a), meaning that they are not able to
to increase by about 60 points to reach the OECD understand and solve simple math problems, severely
average in all disciplines (equivalent to one and limiting their development and skill acquisition
a half years of schooling). process. The picture is similar for reading: about
39 percent of Bulgarian students are considered
Bulgaria has reduced the share of students functionally illiterate. That said, an important part of
below basic proficiency levels since 2006, the progress made by Bulgaria since 2006 was due to
although it remains high. PISA categorizes scores the improvements of students performing below level
in six levels of proficiency; students who score below 2. Countries like Poland have a much lower share of
level 2 in the reading and math tests are considered students below level 2 (Figure 3b) and their progress
functionally illiterate and innumerate, respectively. in the last decade was also mainly driven by the
According to the 2012 data, around 44 percent of improvements of low achievers.
analysis of PISA 2012 and past results 11
Figure 4 Student performance in math on PISA 2006-2012 by socioeconomic group
450
440
n Average
430 student
n 439
420 428
Math PISA Performance
410 n
400 413
Bottom ESCS
390 40% student
397
380 388
370
360 369
350
2006 2009 2012
The World Bank’s mission has recently been articulated into two main goals: boosting the end of extreme poverty and promoting
shared prosperity. The definition of the latter focuses on the income of the bottom 40 percent. This number has been arbitrarily
chosen given that: (i) in many low-income countries, the bottom income quintile coincides with the percentage of people in
extreme poverty so that this group needed to be expanded; and (ii) this indicator expands this notion to also capture the people
considered moderately poor in middle-income countries.
Source: Data from PISA 2006, 2009, and 2012.
Box 2 PISA’s Index of Economic, Social, and Cultural Status
Created by OECD, PISA’s Index of Economic, Social, and Cultural Status (ESCS)
is a multidimensional measurement that takes into account information
reported by students on their family’s wealth and occupational, educational, and
cultural background. It is derived from a combination of three other indexes: (i) an
index of the highest occupational status of parents, indicating not only labor market
status, but also the type of job held by parents; (ii) an index based on the highest
level of parental education in years of schooling; and (iii) an index of family home
possessions, which itself consists of a combination of the family’s possessions (such
as cars, bathrooms, or technological devices) and educational resources (such
as desks, computers, textbooks, the number of other books), as well as the type of
cultural possessions (such as the type and genre of books or works of art). The ESCS
Index is the most important determinant of student achievement and is therefore
crucial for analysis of the quality of education.
Source: PISA 2012 results (OECD 2014).
12 How Can Bulgaria Improve Its Education System?
Without sustained
improvements for all,
disadvantaged students are
unlikely to experience an
increase in their future
living standards
Improvements since 2006 promoted shared development (see Box 2). Results (in Figure 4)
prosperity for the bottom 40 percent, but show that since 2006, the bottom 40 percent of
the gap between students of privileged students in terms of socioeconomic status have
socioeconomic background and the made advancements in math comparable to those
disadvantaged remains high.8 Without of average students (and similar trends are seen in
sustained improvements for all, disadvantaged reading and science). However, the differences in
students are unlikely to increase their future living math and reading scores between students in the
standards. While average score growth is important, highest and lowest quintiles of socioeconomic status
it is also crucial to foster improvements among the are 115 and 150 points, respectively (representing
bottom 40 percent of a country’s student population. between three and four years of education), while
From the PISA data, the OECD’s Index of Economic, the OECD average differences between these income
Social, and Cultural Status (ESCS) is used herein as quintiles are 100 points in math and 90 points in
a measure of student wealth and level of household reading.
analysis of PISA 2012 and past results 13
2.What determines
the quality of education
in Bulgaria
improved?
and can it be
14 How Can Bulgaria Improve Its Education System?
In this section, we analyze the determinants and drivers of education quality in Bulgaria. We use
PISA student achievement as a measure of education quality and relate it to the variables in the PISA student
and school questionnaires that can determine quality in an education system. We use different analytical
techniques for this purpose, and broadly divide variables into individual and school characteristics, with
subgroups of variables within school characteristics: peer characteristics, school resources, and system
variables like school autonomy (Table 2).9
The importance of pupils’ origin:
the lifelong impact of unequal opportunities
PISA results suggest that the opportunities for obtaining a good education are highly unequal
in Bulgaria, and mostly depend on students’ background characteristics. As seen in the previous
section, the difference in math scores between students in the highest and lowest quintiles of socioeconomic
status is very large. Analysis indicates that the importance of certain individual characteristics (gender,
age, and socioeconomic status) to students’ performance in Bulgaria is among the highest in the region
(Figure 5), explaining 33 percent of the difference in reading achievement,10 and reflecting the low equality
of educational opportunities. Disaggregating test scores reveals important differences in the effects of a
number of variables, such as gender, school location (rural or urban), and language spoken at home.
I love *
science
analysis of PISA 2012 and past results 15
Figure 5 Index of equality of opportunities: Bulgaria and other ECA countries, 2012
0.35
0.30
More equality
of opportunities
PISA 2012 Scores
0.25
0.20
0.15
0.10
Kazakhstan
Estonia
Serbia
Russia
Poland
Lithuania
Turkey
Slovenia
Montenegro
Croatia
Czech Republic
Romania
Latvia
Hungary
Slovak Republic
Bulgaria
Source: Authors’ calculations based on PISA 2012.
Note: The index is the percent of the variance in reading scores explained by the main predeter mined charactristics
(age, gender, and socioeconomic status) in a linear regression (Ferreira and Gignoux 2011).
Table 2 Determinants of achievement in PISA, by categories
Individual Characteristics Age
Gender
Socio-Economic Status (ESCS Index)
Ethnicity
Grade
Participation in Pre-Primary
Education
School Characteristics Peer Characteristics School average socioeconomic
status Index (ESCS Index)
School dropout rate
Share of minorities
School Resources Quality of Educational Resources (Index)
Student-Teacher Ratio
Location (Urban or Rural)
Parental Engagement
Type of school (Public or Private)
School Autonomy Responsibility over Curriculum and
Assessment (Index)
Responsibility over Human and
Financial Resources (Index)
Source: Greenwald, Hedges and Laine 1996; Hanushek 2009
16 How Can Bulgaria Improve Its Education System?
Figure 6 Math and reading performance of Bulgarian students by gender
445 470 n
n
440 n n
n 450
PISA Reading Perfromance
PISA Reading Perfromance
435
n 430 n
430 n
n
425 n
410
n n
420 n
390
415 n
410 n 370 n
2000 2003 2006 2009 2012 2000 2003 2006 2009 2012
Source: PISA 2012. n Female
Note: Results in 2003 were estimated by linear interpolation.
n Male
Gender disparities Urban-rural disparities
Bulgarian girls outperform boys by almost 70 The disparity between the PISA scores of urban
PISA points in reading, while performance in and rural students is high for all three disciplines.
math does not vary significantly by gender. In Bulgaria, around 25 percent of PISA-takers live
Differences in performance between girls and boys in in rural areas, in municipalities with a population
both math and reading have not changed significantly smaller than 15,000. The difference between rural
since 2000 (Figure 6). In OECD countries, girls and and urban students’ scores is 89 points in reading
boys also perform at similar levels in math. And in and 65 points in math. The difference in math
other neighboring countries, girls tend to score higher scores between urban and rural locations is very
than boys on the reading scale, as in Bulgaria. For high compared to the ECA average of 27 points.
example, girls score 45 points more on reading in As this only provides an absolute number without
Serbia, 46 points more in Turkey, and 40 points more taking into account several other differences in the
in Romania. Relative to these countries, the difference characteristics of these two subpopulations, the
in Bulgaria is very high. In particular, Bulgarian girls’ Annex further explores the key factors behind the
enrollment in general profiled schools is higher than urban-rural disparity. Results show that individual
boys’: 56 percent of girls study in these programs and peer characteristics as well as school resources
versus 40 percent of boys, a streaming process that are the main drivers explaining the differences
may be exacerbating the gender gap. between urban and rural students.
analysis of PISA 2012 and past results 17
Table 3 Characteristics of students in Bulgaria by language group in 2012
Bulgarian Linguistic
speaking minority
students students
Enrolled in general profiled schools (percent) 51.9 16.0
Live in rural areas (percent) 19.8 44.9
Mother working (percent) 82.0 57.2
Father working (percent) 87.8 76.3
Mother’s education (years) 11.8 9.1
Father’s education (years) 11.5 9.2
Source: PISA 2012.
Linguistic
minority
Linguistic minorities
students are
In Bulgaria, linguistic minority students lag
significantly behind Bulgarian-speaking
much less likely
students. In 2012, almost 11 percent of students
reported speaking a language other than Bulgarian
at home. PISA data did not identify which language
was spoken by these language minority students, but
given the population structure, it is likely that they to be enrolled in
general profiled
were mostly Turkish and Roma ethnic minorities.
Students from linguistic minorities lag behind
schools, tend to
Bulgarian-speaking students the equivalent of three
years of schooling in reading (121 points) and two
be concentrated
years of schooling in math (75 points) and science
(82 points). A more detailed picture shows that the
language groups in Bulgaria do not share the same
socioeconomic and geographical characteristics (see
Table 3). In particular, linguistic minority students more in rural
areas, and have
are much less likely to be enrolled in general profiled
schools, tend to be concentrated more in rural areas,
parents who are
and have parents who are less educated and less likely
to participate in the labor market. Overall, the large
gap in educational opportunities between language
groups can be summarized by large differences in
their socioeconomic backgrounds. less educated
18 How Can Bulgaria Improve Its Education System?
Figure 7 Index of School Social Stratification in PISA 2012-participating countries
0.65
0.60
0.55
0.50
0.45
0.40
0.35
0.30
Great Britain
Finland
Norway
Sweden
Montenegro
Canada
Estonia
Serbia
Russia
USA
Greece
Poland
Turkey
Slovakia
Denmark
Latvia
Croatia
Lithuania
Slovenia
Czech Republic
Germany
Romania
Hungary
Bulgaria
Source: Authors’ calculations based on PISA 2012.
Note: The index goes from 0 to 1. A higher index indicates a higher correlation between students’ and
schools’ socioeconomic status. The figure includes a selected number of PISA countries.
What can policy makers Moreover, there is a strong relationship between each
do to improve education student’s individual characterstics and those of other
students in the same school.
in Bulgaria?
Social stratification in Bulgarian schools is
A stratified school system: the the highest among EU countries (Figure 7).
importance of peers and tracking We define the Index of School Social Stratification
as the correlation between the PISA student’s
mechanisms socioeconomic status and the average school’s
Peer effects are a fundamental driver of student socioeconomic status.12 In a world without social
achievement in Bulgaria. The previous section stratification (thus an index equal to zero), families
studied the importance of individual predetermined from different socioeconomic backgrounds would
characteristics, which explained 33 percent of randomly settle across the country and students from
students’ differences in reading. However, individual different backgrounds would study together, making
characteristics averaged at the school level (i.e., peer
schools more diverse. However, households tend to
characteristics) explain more of the differences in
scores (48 percent) than do individual characteristics. co-locate in neighborhoods with other households
This critical finding suggests that a student’s similar to them, and students tend to attend school
performance depends more on where he or she attends with peers who have a similar socioeconomic status
school than on his or her individual characteristics.11 as a result of spatial inequalities.
analysis of PISA 2012 and past results 19
Figure 8 Evolution of PISA math scores by school type
480
470
460
450 n General profiled
Math PISA Points
n Vocational
440
430
420
410
461 416 475 416
400
2009 2012
Source: PISA 2009 and 2012.
Disparities in performance by school type are Delaying student tracking reduces school
large and have recently increased. Figure 8 stratification and allows for better opportunities
shows the average math scores by type of school for low achievers. Several factors may lead to
for the two largest streams of 15-year-old students: segregated schools. Some have to do with the
the difference in math scores between general geographic assignment of students to schools
profiled school students (comprising 48 percent of (e.g., when wealthier people are concentrated in a
the sample in 2012) and vocational school students particular neighborhood). Another factor is the use of
(41 percent) increased from 45 to 60 points in math exams to select and stream students at early stages.
(equivalent to one and a half years of schooling). The Moreover, parents in high and low socioeconomic
situation is similar for reading, with an increase in the groups may have different access to information
difference from 68 to 86 points in reading (two years or different priorities when they make schooling
of schooling). This indicates that slightly less than decisions. Bulgaria streams its students at age 13
half of Bulgarian children have good opportunities into general profiled, general non-profiled, and
in general profiled schools, while most of the other vocational education schools based on a high-stakes
half struggle in typically lower-quality vocational exam. Examination of the existing admission
or general non-profiled schools. General profiled policies of a number of primary schools suggests
schools have a very low share of disadvantaged low- that this mechanism leads to sorting as early as
achieving students13 relative to vocational schools. grade 1. A recent World Bank report (2013a) found
This means that not only are general profiled students that the prospective of high-stakes exams creates
better off in terms of their family background, but incentives for parents to invest in private tutoring,
they also have the privilege of interacting with leading to sorting of students and raising important
similarly better off peers. equity risks. Most countries with better education
20 How Can Bulgaria Improve Its Education System?
Most countries
with better education
systems stream
students at later
stages of schooling,
usually at age 16.
analysis of PISA 2012 and past results 21
Figure 9 Decomposition of general profiled-vocational school PISA 2012 math score gaps into
different factors by student achievement group
140
120
100
n Individual
80 characteristics
n Peer characteristics
60 n School resources
n School autonomy
40 n Unexplained
n Actual difference
20
0
-20 High
Average Low Middle
achiever achiever achiever
achiever
-40
Source: Authors’ calculations based on PISA 2012.
Note: Results decomposition was done using an Oaxaca-Blinder method on RIF-regressions for each quantile of the
distribution of performance (Firpo, Fortin and Lemieux 2009). Low achievers are students in the 20th percentile.15
systems stream students at later stages of schooling, individual effects due to their high relation (already
usually at age 16. Hanushek and Woessman (2006) explained in this section), it is clear that the ability
used previous PISA data to show how early tracking based selection of students through national tests
systems lead to a systematic increase in inequality after grade 7, which in practice is implicitly sorting
of student performance without affecting average students according to their socioeconomic status,
performance levels. This suggests that there are no determines differences between general profiled
efficiency gains from introducing early streaming and vocational schools.14 This finding has important
of students. policy implications. While little can be done about
individual characteristics, policy levers can be
Individual and peer socioeconomic used to reduce school segregation and promote
characteristics are the major determinants more interaction between children of different
of the difference in student achievement backgrounds, which may lead to major improvements
between general profiled and vocational in student achievement.
schools in Bulgaria. Econometric analysis shows
that socioeconomic status and peer characteristics
explain most, if not all, of the differences in student
outcomes, no matter how students performed in
each school (Figure 9). In fact, peer effects appear to
be more important than individual characteristics.
Although it is difficult to disentangle peer from
22 How Can Bulgaria Improve Its Education System?
Figure 10 Decomposition of changes in PISA math scores gaps between 2006 and 2012 into factors
and by student achievement group
40
35
30
25
n Individual
characteristics
20
n Peer characteristics
15 n School resources
n School autonomy
10 n Unexplained
n Actual difference
5 Average Low Middle High
achiever achiever achiever achiever
0
-5
-10
-15
Source: Authors’ calculations based on PISA 2006 and 2012.
Note: Results decomposition was done using an Oaxaca-Blinder method on RIF-regressions for each quantile of the distribution
of performance (Firpo, Fortin and Lemieux 2009). Low, middle, and high achievers are students in the 20th, 50th, and 80th percentile,
respectively.14
Overall, Has the 2007 school autonomy
reform worked?
the results By linking student outcomes to school
derived from
information, PISA data offer a great opportunity
to assess Bulgaria’s 2007 school autonomy
the governance
reform for the first time. In 2007, the Government
of Bulgaria engaged in an ambitious reform to
reform were
decentralize education management from the central
to the school level (Box 3). Evidence suggests that
it takes time for such autonomy reforms to yield
not the game- tangible results, such as an increase in student test
scores. Borman et al. (2003) showed that school-
changer that
based management reforms need about five years to
bring fundamental changes at the school level and
policy makers
about eight years to show up in indicators such as test
scores. As the PISA 2012 test was taken five years
after the beginning of the 2007 reform, it provides a
expected. great opportunity to make an initial assessment of
the reform’s impact on Bulgarian student outcomes
analysis of PISA 2012 and past results 23
Box 3 Bulgaria’s 2007 Autonomy Reform
In 2007, the Government of Bulgaria introduced a decentralization reform to promote greater
autonomy in schools with respect to financial and personnel management. The education system
became highly decentralized in resource allocation matters after the reform. Schools now have
the autonomy to manage their own budgets, a role transferred from the central government to
municipalities and from municipalities to schools based on per-capita financing principles. Schools
may have their own revenues in addition to those received from the government, although the share of
schools’ own revenues in their budgets is modest. School principals have the authority to hire and fire
teachers and to decide individuals’ workloads, remuneration, and bonuses within broadly defined
central regulations. School principals are hired by the Ministry of Education and its regional structures.
However, there is still room for improving the reform’s implementation. Relationships of accountability
between principals and parents need further development. School Boards are composed of parents
and representatives of the local community, but do not have the legal authority to participate in
school decisions, budget preparation, or supervision. Further, student assessments and school-
specific assessment data are known to education authorities (central and regional) and schools, but
are not disclosed to the public. Assessment results are used to track performance and inform decisions
for administrative and pedagogical adjustments, but are not part of a long-term national plan for
school improvement, as they are outside the accountability framework.
Source: World Bank 2011b
Table 4 Rate of affirmative responses on the responsibility of each body over
resources and pedagogy, as reported by school principals.
Principal School Regional Central
(%) Governing Authority Authority
Board (%) (%) (%)
2006 2012 2006 2012 2006 2012 2006 2012
Responsibility for teacher hiring 100 99 2 4 6 4 2 2
Responsibility for teacher firing 99 95 3 3 7 2 1 2
Responsibility for teachers’
starting salaries 15 79 0 6 8 2 89 41
Responsibility for teachers’
salary increases 19 90 4 10 10 1 87 25
Responsibility for formulating budget 56 65 4 9 48 26 33 53
Responsibility for budget allocations 83 91 18 31 26 5 20 6
Responsibility for student discipline 37 48 93 93 9 5 55 21
Responsibility for student assessment 24 59 27 39 13 17 91 63
Responsibility for student admission 52 77 47 19 33 20 34 9
Responsibility for textbook use 66 83 61 41 6 2 23 21
Responsibility for course content 20 36 15 7 7 4 90 88
Responsibility for courses offered 18 19 47 55 11 8 88 82
Source: PISA 2006 and 2012 School Questionnaire. The percentage indicates the percentage of principals that reported
some responsibility of each administrative body over different resources.
24 How Can Bulgaria Improve Its Education System?
and of the reform’s strengths and weaknesses. The Exploratory analysis of changes between
model employed to do this decomposes the change in 2006 and 2012 shows little changes in results
scores between PISA 2006 (baseline) and PISA 2012 associated with the school autonomy reform,
to make a preliminary assessment of the impact of the but highlights the importance of school
shift in responsibility from the government to school resources as a driver of improvements for
principals. low achievers. Using an approach similar to that
followed to identify the factors associated with the
Moderate to significant changes in school gap between general profiled and vocational schools,
autonomy between 2006 and 2012 allow for the increase in math performance between 2006 and
assessment of the reform’s impact. As part of the 2012 is mainly explained by improved socioeconomic
PISA, school principals are given a questionnaire conditions and the quality of educational resources
in which they respond to questions related to the (Figure 10).16 The improvement in socioeconomic
organization of the school, the school’s student conditions (through individual households and peer
and teacher bodies, the school instruction and effects) explained most of the performance increase
curriculum, the school climate, school policies and for high-achieving students. Similarly, improvements
practices, and the school financing. The section in school resources – through increased availability
on school policies and practices includes the of quality library materials, lab equipment, and
following question: “Regarding your school, who has computer materials – played a crucial role for low
considerable responsibility for the following tasks?” achievers (explaining almost half the increase). One
For each task, the principal can indicate which of hypothesis for this is that improvements in the school
four educational institutions have responsibility learning environment are particularly important for
(with more than one response possible): Principals, children who lack materials at home. This finding
School Governing Board, Regional Authority, or draws important policy lessons for future decisions.
Central Authority. Table 4 displays the percentage The overall effects of the school autonomy reform are
of responses given by principals for each of the not statistically significant (see Annex) and suggest
educational institutions by specific autonomy that had the reform not been implemented, PISA
responsibility in 2006 and 2012. Although this performance would have been essentially the same in
does not reflect the exact responsibility of each 2012. Overall, the results derived from the governance
educational stakeholder, it displays a major shift reform were not the game-changer that policy makers
in responsibility towards principals, mainly in expected.
the decision of teacher salaries and in student
assessment and admission, and also moderate shifts Although the overall results are limited, the
in principals’ responsibilities for budgets, textbook effects of different types of autonomy vary by
use, and development of course content. The increase urban and rural settings. The overall impact of the
in principals’ decision making allows us to identify reform can be disaggregated by type of autonomy.18
if the reform helped explained the changes in PISA This decomposition includes the interaction of
results between 2006 and 2012. autonomy indexes with a rural variable indicator to
analysis of PISA 2012 and past results 25
Global evidence shows that
providing quality preschool
education is important for
promoting children’s social,
emotional, physical, and
cognitive development
allow the impact of the reform on rural and urban Early childhood policies, teacher
schools to be disentangled (see Annex). On one practices, and the school environment
hand, the results show that the shift in autonomy
for allocation of resources (such as teacher salaries There is room for policy interventions that have
and budget allocation) had a positive and very the power to improve the quality of education.
significant impact on all students’ scores (6 points The previous section emphasized how individual and
on average), and especially on those of low achievers peer characteristics are an important determinant
(11 points). This impact was stronger in urban than of student achievement. In this part of the study, a
in rural areas; a possible reason may be better and multilevel analysis of determinants first includes
more accountable school administration in urban individual characteristics, peer characteristics,
areas. On the other hand, the impact of principals’ and school resources variables (such as quality of
greater curriculum and assessment responsibilities educational resources and shortage of teachers). In
on students’ performance was slightly negative the next step, the two autonomy measures discussed
(although not very significant), outweighing the gains in the previous section are also included in the model
made from greater autonomy in resource allocation. of determinants of learning (see the Annex for a
The fact that the impact of the reform was higher for summary of results).
low-achieving students (especially in urban areas)
suggests that greater autonomy allowed principals The analysis finds that early childhood
and teachers to focus on those students who lagged education (ECE) has a positive and significant
behind or who needed more support. effect on student achievement (see Annex).
About 77 percent of 15-year-old students taking
the PISA in Bulgaria have more than a year of pre-
primary education. This is due to the increased
26 How Can Bulgaria Improve Its Education System?
efforts of the Government of Bulgaria to expand
the coverage of preschool education during the last
decade. Results show that having attended at least
a 2-year pre-primary education program increases
PISA math scores by an average of 7 points relative
to having attended one year or none at all. The effect
of ECE is greatest for low achievers (10 points on
average) and students who speak a different language
at home (19 points on average), while its effect on
high achievers is not significant.19 Global evidence
shows that providing quality preschool education is
important for promoting children’s social, emotional,
physical, and cognitive development; it also increases
school readiness, which helps learning (Heckman
and LaFontaine 2010; Heckman 2008; Engle et al.
2011). Cognitive skills gaps start opening during An orderly school (i.e., one where teachers can teach
early life and inequalities in access to early childhood effectively, and students listen to their teachers and
perpetuate learning gaps. Given that attendance work well) offers fewer disruptions to students and is
in early childhood programs is correlated with more conducive to learning.
higher educational attainment, policies improving
access to and quality of ECE in Bulgaria for the mostTeaching practices are another important
disadvantaged students (who still face challenges in determinant of learning. For instance, effective
starting education early) have the highest potential teacher management of a classroom (such as
to increase student achievement. This would help keeping the class orderly, getting students to listen,
improve the cognitive and social skills of the entirestarting lessons on time, or ensuring that there
population, translating into higher human capital are no disruptions) has a positive and significant
and productivity and likely contributing to an overall
effect on the PISA math score (about 5 points).21
reduction in learning inequality. Nonetheless, changing teaching practices to improve
service delivery in education is not straightforward.
The school and classroom environment affects Therefore, developing relevant policies to tackle this
student achievement. Disciplinary climate issue – such as effectively reforming teacher pre-
measures the frequency and severity of disruptions and in-service training or attracting more qualified
by students in a school and is an important variable teachers to the teaching force – is a challenge for
20
in explaining students’ academic performance (about the medium and long run. Finally, other school-
6 points on average). Disciplinary climate depends related variables, like class size, were not found to be
not only on the student body but also on the social significant in determining student achievement as
and managerial abilities of teachers and principals. measured by students’ PISA math scores.
analysis of PISA 2012 and past results 27
Drilling
3.
down further into
math & reading
skills
28 How Can Bulgaria Improve Its Education System?
PISA offers the opportunity to fully explore one subject area every three years, even though all
three subjects are assessed every time PISA is administered. PISA seeks to assess not merely whether
students can reproduce knowledge, but also to examine how well they can extrapolate from what they have
learned and apply it in unfamiliar settings, both in and outside of school. The detailed test of “subscale” skills
of a given subject area is an in-depth assessment with a larger set of questions. The detailed assessment was
on reading in 2000 and 2009, on math in 2003 and 2012, and on science in 2006. The 2015 round will focus
again on science.
Math skills in PISA 2012
The PISA math 2012 subscale assessment measured individuals’ abilities to formulate, employ,
and interpret mathematics in a variety of contexts and content areas. In PISA, the concept of
mathematical literacy includes: (i) mathematical reasoning; (ii) usage of mathematical concepts, procedures,
and facts; (iii) tools to describe, explain, and predict phenomena; and (iv) the role that mathematics plays in
the world and the need to make well-founded judgments and decisions needed by constructive, engaged, and
reflective citizens. Furthermore, mathematic literacy as defined by PISA is not an attribute that an individual
has or does not have; rather, it can be acquired to a greater or lesser extent, and it is required in varying
degrees in society. PISA seeks to measure not just the extent to which students can reproduce mathematical
content knowledge, but also how well they can extrapolate from what they know and apply their knowledge of
mathematics in new situations.
PISA’s math framework is a sophisticated tool for connecting students’ mastery of mathematical
processes and contents. The math subscale assessment evaluates capacity in four content categories
(Figure 11): quantity (incorporates the quantification of attributes of objects, relationships, situations, and
entities); uncertainty and data (understanding messages embedded in data, and appreciating variability that
is inherent in many real processes); change and relationships (temporary and permanent relations among
objects and circumstances); and space and shape (phenomena encountered in patterns, object properties,
positions, representations, visual information, navigation, and dynamic interactions). Figure 11 also shows
a schematic of the stages faced by a student when solving a real life problem through the mathematical
modelling cycle. The action begins with identifying the problem in context and finishes when the results of
the problem are found in a context and again are reflected in the problem context. This process involves four
skills that PISA defines as “processes,” and were assessed in 2012 as: formulate a mathematical situation
according to the concepts and relationships identified; employ mathematical facts, procedures, and
reasoning to obtain a result (usually involving calculation, manipulation, and computation); interpret the
results in terms of the original problem to obtain the “results in context”; and finally, evaluate the outcomes
and their reasonableness in the context of the problem.22
analysis of PISA 2012 and past results 29
Figure 11. Math contents and process categories in PISA 2012
ç
problem in context
ç formulate mathematical problem
ç employ
quantity
uncertainty and data
change and relationships
evaluate
space and shape
results in context
Source: OECD 2014.
ç interpret mathematical results
Reading skills in PISA 2009
Students in Bulgaria performed better in The PISA 2009 subscale assessment of readings
problems related to space and shape and skills measured students’ ability to actively,
quantity, but not as well in problems related purposefully, and functionally apply reading in
to data and statistics (Figure 12). Compared a range of situations. PISA defines reading literacy
with the average score of all math subscales, as understanding, using, reflecting on, and engaging
Bulgaria’s results show slightly higher variation with written texts to achieve one’s goals, to develop
across subscale assessments than is found in OECD one’s knowledge and potential, and to participate in
countries. Students successfully solved problems society. Understanding refers to the reader’s ability
related to space and shape and quantity, usually to construct meaning from text; using refers to the
related to geometry, algebra, and physics. However, kind of reading that is directed toward applying
students underperformed when they needed to use information in a text to an immediate task; reflecting
their ability to solve data problems or to appreciate means that readers relate what they are reading with
variability and uncertainty in real life problems. their thoughts and experiences. Although texts are
differentiated in different characteristics (medium,
environment, type and format), performance on text
format is the only one reported in PISA through two
30 How Can Bulgaria Improve Its Education System?
Figure 12. PISA 2012 performance on different math subscales compared to the average math performance
6
Change & Space & Quantity Uncertainty Formulating Employing
Performance difference between each content/process
Interpreting/
relationships shape and Data Evaluating
4
subscale and the average mathematics scale
2
0
-2
-4
-6
Contents Processes
-8
n OECD total
Source: PISA 2012. n Bulgaria
There is a
types: continuous texts (sentences organized into
paragraphs, which may fit into even larger structures)
need to
and non-continuous texts (smaller sentences, usually
in sample lists, graphs, diagrams, or catalogues),
improve
although there are also mixed and multiple texts.
Aspects are measured as PISA reading subscales
with three categories: access and retrieve (skills
the reflection associated with finding, selecting, and collecting
information); integrate and interpret (which involves
and evaluation
understanding the relations between different parts
of a text, or making meaning from something that
skills in
is not stated in the text); and reflect and evaluate
(which involves drawing on knowledge, ideas, or
values external to the text). Finally, situations intend
reading to maximize the diversity of content included in
the PISA reading survey; for example, personal,
public, educational, and occupational situations are
represented.
analysis of PISA 2012 and past results 31
Figure 13. PISA 2009 performance on different reading subscales compared to the
combined reading perfor mance
10 Continuous Non- Access Integrate Reflect
Texts Continuous & retrieve & Interpret & Evaluate
Performance Difference between each content/process
Texts
5
subscale and the combined mathematics scale
0
-5
-10
-15
Texts Aspects
n OECD total
Source: PISA 2009. n Bulgaria
32 How Can Bulgaria Improve Its Education System?
8-4+1=5
120-10=110
The 2009 subscale assessment for reading
revealed that Bulgarian students have a better
understanding of continuous text compared
with non-continuous text, while there is a need
to improve their reflection and evaluation skills
in reading. Comparing the reading subscale results
with the average score across all reading subscales,
Bulgaria shows much more variation across
subscales compared with OECD countries, which
means there is large room for improvement in some
subscales. In particular, students perform better with
more traditional texts rather than texts contained
in sample lists, graphs, or diagrams. Moreover,
students’ ability to relate their own experiences to
the text is weak, reflecting a disconnect between
what students learn and their ability to apply this
knowledge in real life situations.
analysis of PISA 2012 and past results 33
4. Findings
recommendations
&
34 How Can Bulgaria Improve Its Education System?
After a drop between 2000 and 2006, Bulgaria’s PISA scores improved in all three disciplines.
To sustain the recent success, new policies are required. A large share of the improvement since 2006 is
explained by the improvement in students’ socioeconomic status, which translated into better test scores,
as well as the better quality of educational resources. It is now necessary to devise a new set of effective
policies to continue narrowing the gap in scores with OECD and other countries in the region. Investment in
educational resources is important to ensure minimum standards, but is not sufficient to sustain continuous
improvement.
Although it is difficult to affect students’ predetermined characteristics in the short term, there is still
an important role for policy. In Bulgaria, the difference in performance between students in the bottom
and top socioeconomic quintiles is much larger than in OECD countries. The significance of predetermined
factors in affecting students’ educational performance can be discouraging, as these factors generally take
time, often generations, to improve.
Inequality of educational opportunities in Bulgaria is the highest in the region and the EU.
Disadvantaged groups, such as rural populations and linguistic minorities, perform much worse on the PISA
than urban populations and Bulgarian-speaking students. Moreover, the performance gap between girls and
boys on the PISA reading score is the highest in the region.
An assessment of Bulgaria’s 2007 school autonomy reform shows little impact. This report is the first
analysis of the impact of Bulgaria’s school-based management reform, which shifted more responsibility to
school principals. The results show that overall the results have been more limited than expected, especially
given the amount of effort expended on the 2007 reform. A detailed analysis shows that principals’ greater
autonomy over curriculum and assessment policies had a slight negative impact on Bulgaria’s 2012 PISA
math scores, while their greater autonomy over management of resources (teachers and budget allocation)
had a positive impact. The impact of the reform was higher in urban schools, suggesting better and more
accountable school administration in urban areas. Overall, the results indicate the need to further improve
the management capacity of principals in rural areas while also strengthening accountability mechanisms.
Students performed better on problems related to space and shape and quantity, and not as
well on problems related to data and statistics. Compared with the average math performance of all
subscales, Bulgaria’s results show slightly higher variation across subscale assessments than is found in
OECD countries. Students successfully solved problems related to space and shape and quantity, usually
related to geometry, algebra, and physics. However, students underperformed when they needed to use their
ability to solve data problems or to appreciate variability and uncertainty in real life problems.
analysis of PISA 2012 and past results 35
Bulgarian students have a better understanding secondary schools, like streaming students at the
of continuous text than of non-continuous text, end of compulsory education (age 16), could raise
and there is a need to improve their reflection the overall education quality of the less favored
and evaluation skills in reading. Comparing the without lowering average performance.
reading subscale results with the average reading
performance of all subscales, Bulgaria shows much 2
Continue to improve the quality of
more variation across subscales compared with educational resources to ensure that all
OECD countries, which means there is large room students learn in an appropriate environment
for improvement in some subscales. In particular, of books, libraries, lab equipment, and
students perform better with more traditional texts technological resources. The analysis of
than with texts contained in sample lists, graphs, the improvement in performance in math and
or diagrams. Moreover, PISA reveals students’ reading between 2006 and 2012 shows that the
weaknesses in relating their own experiences to the two key drivers were the evolution of students’
text, reflecting a disconnect between what they learn socioeconomic status and the improved quality of
and their ability to apply that knowledge in real life educational resources. The impact of educational
situations. resources was especially important for low-
achieving students, indicating a low-hanging
If adequate policies are pursued, Bulgaria is fruit for improving the quality and equity of the
likely to succeed in increasing the equality education system. Continuation of this would
of opportunities to achieve its “Learning For include better provision of lab equipment,
All” goals. With this in mind, six main policy computer and software materials, library
recommendations arise as a result of this study: materials, and instructional materials and/or the
renovation of buildings and grounds.
1 Delay the tracking of students into different
types of schools as it leads to school 3
Expand preschool education for the most
stratification with no benefits. School disadvantaged students, as analysis shows
stratification – the concentration of students with it is especially beneficial for the less favored.
similar socioeconomic status in the same schools The study found that the expansion of preschool
– is a result of the inequalities in the Bulgarian education to at least two years raises low achievers’
education system combined with use of a high- and minorities’ scores by up to 10 and 19 points,
stakes exam that channels students into different respectively (even after taking into account other
schools according to their socioeconomic status. As relevant individual and school factors). Universal
a consequence, disadvantaged students suffer not preschool education would provide a great
only from their own situation but are also penalized opportunity to effectively narrow the skills gap
by having to interact with similarly disadvantaged from the early stages of children’s lives.
peers. Thus, it is plausible that the implementation
of adequate selection mechanisms for students in
36 How Can Bulgaria Improve Its Education System?
4
Learn from successful schools to improve 6
Promote effective classroom management
accountability mechanisms for schools and strengthen teaching practices. The
country-wide, particularly in rural areas. analysis shows that a class that is orderly, with
There is a need to further understand: (i) why the fewer disruptions to students, is more conducive
autonomy reform did not function as expected; to learning and therefore improves PISA scores.
(ii) why the reform was more successful in urban The government could use classroom observation
areas; and (iii) why PISA scores were positively methods and international best practices on
affected by greater autonomy in the management classroom management to help teachers identify
of school resources, but not by greater autonomy opportunities to improve their performance in
in curriculum development and assessment. the classroom. Teacher development programs
Learning from successful schools could help the could be implemented to improve management
Government of Bulgaria augment the impact of the techniques in the classroom for the current
reform in rural areas over the coming years. and future teaching workforce, yielding rapid
improvements in the quality of learning.
5
Reevaluate the curriculum and assessment
framework to better align student learning
to the envisaged country goals. The PISA full
assessment analysis derives important lessons
for policy makers in Bulgaria. Results shed light
on the large discrepancies (as compared to other
countries) within reading and math skills. PISA
results present a good opportunity to engage
in an in-depth debate about a curriculum and
assessment framework reform, as well as how to
better align the education system with national
social and economic development goals.
analysis of PISA 2012 and past results 37
References
Amermueller, A. 2004. “PISA: What Makes the Difference? Explaining the Gap in Pisa Test Scores between
Finland and Germany.” ZEW Center for European Economic Research Discussion Paper No. 04-004.
Barrera-Osorio, F., V. Garcia-Moreno, H.A. Patrinos, and E. Porta. 2011. “Using the Oaxaca-Blinder
Decomposition Technique to Analyze Learning Outcomes Changes Over Time: An Application to Indonesia’s
Results in PISA Mathematics.” World Bank Working Paper 5584. World Bank, Washington, DC.
Borman, G.D., G.M. Hewes, L.T. Overman, and S. Brown. 2003. “Comprehensive School Reform and
Achievement: A Meta-Analysis.” Review of Educational Research 73 (2): 125-230.
Engle, P., L. Fernald, H. Alderman, J. Behrman, C. O’Gara, A. Yousafzai, M. Cabral de Mello, M. Hidrobo,
N. Ulkuer, I. Ertem, S. Iltus, and Global Child Development Steering Group. 2011. “Strategies for reducing
inequalities and improving developmental outcomes for young children in low-income and middle-income
countries.” The Lancet 8 October Vol. 378, Issue 9799: 1339-1353 ). DOI: 10.1016/S0140-6736(11)60889-1.
Ferreira, H.G., and J. Gignoux. 2011. “The Measurement of Educational Inequality: Achievement and
Opportunity.” IZA Discussion Paper No. 6161.
Firpo, S., N. Fortin, and T. Lemieux. 2009. “Unconditional Quantile Regressions.” Econometrica Vol. 7,
No 3: 953-973.
Greenwald, R., L. V. Hedges, and R. Laine. 1996. “The Effect of School Resources on Student Achievement.”
Review of Educational Research Vol. 66, No. 3 (Autumn): 361-396.
Hanushek, E. 2009. “School policy: Implications of recent research for human capital investments in South
Asia and other developing countries.” Education Economics 17(3): 291-313.
Hanushek, E. 2010. “The High Cost of Low Educational Performance. The long-run economic impact of
improving PISA outcomes.” OECD Publications.
Hanushek, E., and L. Woessmann. 2006. “Does Educational Tracking Affect Performance and Inequality?
Differences-in-differences evidence across countries.” The Economic Journal Vol. 116, Issue 510: C63-C76.
Hanushek, E., and L. Woessmann. 2007. “The Role of Education Quality in Economic Growth.” World Bank
Policy Research Working Paper 4122. World Bank, Washington, DC.
38 How Can Bulgaria Improve Its Education System?
Hanushek, E., and L. Woessmann L. 2012. “Do Better Schools lead to more growth? Cognitive skills,
economic outcomes, and causation.” Journal of Economic Growth Vol. 17: 267-321.
Heckman, J. 2008. “Schools, skills, and synapses.” Economic Inquiry 46(3): 289-324.
Heckman, J., and P. LaFontaine. 2010. “The American High School Graduation Rate: Trends and Levels.”
Review of Economics and Statistics 92(2): 244–262.
OECD. 2012. PISA 2009 Technical Report. Paris: OECD. Retrieved April 10, 2014 from http://www.oecd.
org/pisa/pisaproducts/50036771.pdf
OECD. 2014. PISA 2012 Results: What Students Know and Can Do. Paris: OECD. Retrieved April 10, 2014
from http://www.oecd.org/pisa/keyfindings/pisa-2012-results-volume-I.pdf
Sala-i-Martin, X., G. Doppelhofer, and R.I. Miller. 2004. “Determinants of long-term growth: A Bayesian
averaging of classical estimates (BACE) approach.” American Economic Review 94 (4), 813-835.
World Bank. 2011a. “Assessing the Quality of Education in Bulgaria using PISA 2009 data.” World Bank,
Washington, DC.
World Bank. 2011b. “Benchmarking School Autonomy and Accountability in Selected European Countries.”
World Bank, Washington, DC.
World Bank. 2013a. “Improving the transition between lower and upper secondary schools in Turkey:
Recommendations for the design of a more equitable system.” World Bank. Processed Papers (Unpublished).
World Bank, Washington, DC.
World Bank. 2013b. “Promoting Excellence in Turkey’s Schools.” World Bank, Washington, DC.
World Bank. 2014. “Bulgaria: SABER Workforce Development.” World Bank, Washington, DC.
analysis of PISA 2012 and past results 39
Annex
The analytical approach used in Section 2 of this report is based on the Firpo, Fortin, and Lemieux
(2009) methodology. Typically, the literature on decomposition of student scores in PISA through groups
(Amermueller 2004) and years (Barrera et al. 2011) has focused on the mean differences, with little attention
to what happens at the tails of the distribution. The Firpo, Fortin, and Lemieux (FFL) method allows one to
decompose gaps in student performance not only for the mean but also for other statistics of the distribution.
Traditionally, the problem with quantile regressions has been that the law of iterated expectations does not
apply, thus making it impossible to interpret the unconditional marginal effect of each independent variable
on a student’s performance. However, recent econometric techniques, such as the one proposed by FFL,
have solved this methodological difficulty. The FFL technique is based on the construction of re-centered
influence functions (RIF) of a quantile of interest, , as a dependent variable in a regression:
⌧ − D(I q⌧ ))
RIF (I ; q⌧ ) = q⌧ +
f I ( q⌧ )
where is an indicator function and is the density of the marginal distribution of scores. A crucial
characteristic of this technique is that it provides a simple way of interpreting the marginal impact of an
additional unit of a certain factor on students’ PISA scores. Once the unconditional quantile regression
has been computed for different quantiles of the distribution, the results can be decomposed following the
Oaxaca-Blinder approach.
40 How Can Bulgaria Improve Its Education System?
Table A. 1. Decomposition of urban-rural PISA math score gaps by
student achievement groups.
Variables Average Percentile 20 Percentile 50 Percentile 80
Rural 400.2*** 307.0*** 401.2*** 449.2***
(6.980) (17.49) (6.230) (9.039)
Urban 459.1*** 397.8*** 451.3*** 542.5***
(5.518) (7.648) (4.715) (12.50)
Difference -58.87*** -90.72*** -50.10*** -93.33***
(8.897) (19.09) (7.813) (15.43)
Unexplained -38.70*** -52.55** -30.34*** -53.32***
(9.693) (21.38) (9.874) (19.15)
Explained -20.17* -38.17 -19.76** -40.01***
(10.39) (25.32) (10.07) (15.08)
Individual Characteristics -16.05*** -40.66*** -12.14*** -14.03***
(3.084) (9.369) (2.924) (5.130)
Peer Characteristics 1.814 19.35 -3.437 -22.04
(9.272) (24.80) (9.199) (13.65)
School Resources -4.670 -14.78 -3.335 -2.714
(5.128) (14.11) (5.236) (6.678)
Autonomy -1.263 -2.083 -0.851 -1.225
(1.875) (4.290) (1.768) (2.383)
Constant 17.62 77.70 -136.9 156.9
(161.0) (489.4) (174.7) (268.2)
Observations 4,501 4,501 4,501 4,501
Note: Robust standard error in parentheses and clustered at the school level. *** p<0.01, **p<0.05,*p<0.1. Variable effects are
grouped and include individual characteristics (age, gender, grade, language at home, participation in ECE, and socioeconomic
status), peer characteristics (socioeconomic status, school dropouts, and minorities at school), school resources (school owner
ship, location, quality of educational resources, teacher shortage, and parental pressure), and school autonomy (autonomy in
resources, and autonomy in curriculum and assessment).
analysis of PISA 2012 and past results 41
Table A.2. Decomposition of general-vocational PISA math score
gaps by student achievement group
Variables Average Percentile 20 Percentile 50 Percentile 80
Year 2012 444.9*** 373.7*** 439.3*** 522.1***
(4.532) (6.902) (3.789) (9.214)
Year 2006 420.0*** 337.2*** 417.4*** 501.3***
(6.251) (3.469) (6.773) (9.796)
Difference 24.88*** 36.50*** 21.86*** 20.81
(7.721) (7.724) (7.760) (13.45)
Explained 17.20*** 29.84*** 13.15** 26.91**
(6.601) (10.42) (5.725) (13.39)
Unexplained 7.680 6.658 8.710 -6.102
(5.654) (9.553) (6.281) (10.79)
Individual Characteristics -0.464 1.295 0.117 -2.025
(1.125) (2.597) (1.022) (1.995)
Peer Characteristics 9.348** 8.612* 7.611** 21.61**
(4.647) (4.542) (3.790) (10.71)
School Resources 6.980*** 16.68*** 4.277 6.838
(2.709) (6.176) (2.614) (6.170)
Autonomy 1.333 3.252 1.141 0.487
(2.467) (5.117) (2.384) (5.442)
Constant -89.58 -443.1*** 55.61 -418.0*
(114.6) (170.3) (139.0) (228.5)
Observations 8,749 8,749 8,749 8,749
Note: Robust standard error in parentheses and clustered at the school level. *** p<0.01, **p<0.05,*p<0.1. Variables effects are
grouped and include individual characteristics (age, gender, grade, language at home, participation in ECE, and socioeconomic
status), peer characteristics (socioeconomic status, school dropouts, and minorities at school), school resources (school owner
ship, location, quality of educational resources, teacher shortage, and parental pressure), and school autonomy (autonomy in
resources, and autonomy in curriculum and assessment).
42 How Can Bulgaria Improve Its Education System?
Table A.3. Decomposition of 2006-2012 PISA math score gaps by
student achievement group
Variables Average Percentile 20 Percentile 50 Percentile 80
Year 2012 444.9*** 373.7*** 439.3*** 522.1***
(4.532) (6.902) (3.789) (9.214)
Year 2006 420.0*** 337.2*** 417.4*** 501.3***
(6.251) (3.469) (6.773) (9.796)
Difference 24.88*** 36.50*** 21.86*** 20.81
(7.721) (7.724) (7.760) (13.45)
Explained 17.20*** 29.84*** 13.15** 26.91**
(6.601) (10.42) (5.725) (13.39)
Unexplained 7.680 6.658 8.710 -6.102
(5.654) (9.553) (6.281) (10.79)
Individual Characteristics -0.464 1.295 0.117 -2.025
(1.125) (2.597) (1.022) (1.995)
Peer Characteristics 9.348** 8.612* 7.611** 21.61**
(4.647) (4.542) (3.790) (10.71)
School Resources 6.980*** 16.68*** 4.277 6.838
(2.709) (6.176) (2.614) (6.170)
Autonomy 1.333 3.252 1.141 0.487
(2.467) (5.117) (2.384) (5.442)
Constant -89.58 -443.1*** 55.61 -418.0*
(114.6) (170.3) (139.0) (228.5)
Observations 8,749 8,749 8,749 8,749
Note: Robust standard error in parentheses and clustered at the school level. *** p<0.01, **p<0.05,*p<0.1. Variables effects are
grouped and include individual characteristics (age, gender, grade, language at home, participation in ECE, and socioeconomic
status), peer characteristics (socioeconomic status, school dropouts, and minorities at school), school resources (school owner
ship, rural, quality of educational resources, and parental pressure), and school autonomy.
analysis of PISA 2012 and past results 43
Table A.4. Decomposition of 2006-2012 PISA math score gaps by student achievement
group, detailed autonomy variables
Variables Average Percentile 20 Percentile 50 Percentile 80
Year 2012 444.9*** 373.7*** 439.3*** 522.1***
(4.532) (6.902) (3.789) (9.214)
Year 2006 420.0*** 337.2*** 417.4*** 501.3***
(6.251) (3.469) (6.773) (9.796)
Difference 24.88*** 36.50*** 21.86*** 20.81
(7.721) (7.724) (7.760) (13.45)
Unexplained 7.680 6.658 8.710 -6.102
(5.654) (9.553) (6.281) (10.79)
Explained 17.20*** 29.84*** 13.15** 26.91**
(6.601) (10.42) (5.725) (13.39)
Individual Characteristics -0.464 1.295 0.117 -2.025
(1.125) (2.597) (1.022) (1.995)
Peer Characteristics 9.348** 8.612* 7.611** 21.61**
(4.647) (4.542) (3.790) (10.71)
School Resources 6.980*** 16.68*** 4.277 6.838
(2.709) (6.176) (2.614) (6.170)
Autonomy Curriculum -1.891 -2.598 -2.242 -2.278
(1.495) (3.105) (1.428) (3.574)
Autonomy Curriculum
(Interaction with Rural) -0.996 -2.118 -0.555 -2.055
(0.698) (1.743) (0.629) (1.470)
Autonomy Resources 7.008*** 13.95*** 7.304*** 10.02*
(2.360) (4.572) (2.172) (5.851)
44 How Can Bulgaria Improve Its Education System?
Variables Average Percentile 20 Percentile 50 Percentile 80
Autonomy Resources
(Interaction with Rural) -2.788** -5.981* -3.367*** -5.203**
(1.271) (3.053) (1.244) (2.342)
Constant -89.58 -443.1*** 55.61 -418.0*
(114.6) (170.3) (139.0) (228.5)
Observations 8,749 8,749 8,749 8,749
Note: Robust standard error in parentheses and clustered at the school level. *** p<0.01, **p<0.05,*p<0.1. Variables effects are
grouped and include individual characteristics (age, gender, grade, language at home, participation in ECE, socioeconomic
status), peer characteristics (socioeconomic status, school dropouts, minorities at school), school resources (school ownership,
rural, quality of educational resources, parental pressure), and school autonomy.
Table A.5. Share of variation in mathematics scores: multilevel models
Model 1 Model 2
Individual characteristics (gender, ESCS, Grade YES YES
School characteristics (Disciplinary climate,
peer characteristics, and teacher shortage) YES YES
System characteristics (autonomy variables—
autonomy in resources and in curriculum
and assessment) YES
Explained variation (%) 0.52 0.53
Source: PISA 2012.
analysis of PISA 2012 and past results 45
Table A 6. Determinants of math performance: a multilevel approach
Model 1 Model 2 Model 3
ESCS 7.57*** 7.58***
(1.23) (1.23)
Kindergarten 6.89*** 6.91***
(2.41) (2.41)
Female -15.42*** -15.45***
(2.07) (2.07)
Foreign language at home -17.22*** -17.53***
(3.95) (3.95)
Age 6.34* 6.38*
(3.47) (3.47)
Mathematics anxiety -20.04*** -20.03***
(1.04) (1.04)
Sense of belonging 2.06* 1.99*
(1.15) (1.15)
ESCS-school 44.16*** 43.20***
(5.12) (5.06)
Teacher shortage 6.18 4.71
(5.71) (5.62)
Student-teacher ratio -0.15 -0.12
(0.13) (0.13)
Student-teacher relations -4.98*** -4.96***
(1.09) (1.08)
46 How Can Bulgaria Improve Its Education System?
Table A.6. Determinants of math performance: a multilevel approach
Model 1 Model 2 Model 3
Teacher support -1.87 -1.84
(1.21) (1.21)
Disciplinary climate 6.05*** 6.04***
(1.25) (1.25)
Grade 19.71*** 19.71***
(4.14) (4.14)
Classroom management 5.38*** 5.41***
(0.97) (0.97)
Rural -6.72 -6.05
(7.49) (7.33)
Educational Resources 3.84*** 3.59***
(1.33) (1.31)
-
Program 2 -9.12 9.23
(11.31) (11.28)
Program 3 7.68 7.52
(10.37) (10.34)
Program 4 -2.24 -2.42
(10.74) (10.68)
Autonomy curriculum -7.09**
(3.10)
Autonomy resources 5.80**
(2.85)
analysis of PISA 2012 and past results 47
Table A.6. Determinants of math performance: a multilevel approach
Model 1 Model 2 Model 3
_cons 429.75*** 177.29*** 174.84***
(5.19) (59.50) (59.48)
ICC
(Intraclass correlation,
% of variance attributable to schools) 0.58 0.30 0.28
Source: PISA 2012 Bulgaria.
Note: Multilevel models are able to analyze data in nested structure (students within classrooms, within schools) and allow
correlation of observations within clusters. For this exercise, we use a random coefficient model at the school level (disciplinary
climate). ECE is measured as two years of pre-primary education, and the baseline is one year or less of pre-primary education.
Standard errors in parenthesis, *** p<0.01, **p<0.05,*p<0.1
48 How Can Bulgaria Improve Its Education System?
Endnotes
1 Socioeconomic status is 7 Ferreira and Gignoux 11 Although results show the
measured in PISA with the (2011) propose a measure weight of peer effects to be
OECD’s Economic, Social, and of educational opportunity more important than that of
Cultural Status Index (ESCS). using the share of variance in individual socioeconomic
test scores that is explained characteristics, this should be
2 Tracking of students refers by individual predetermined interpreted with caution, as the
to separating students into circumstances. If a significant high correlation between them
different academic paths. share of the results is explained indicates that both matter.
by these characteristics, then
3 See Sala-i–Martin, the equality of opportunities is 12 Decomposition included
Doppelhofer, and Miller (2004). low. individual characteristics,
peer characteristics, school
4 See Hanushek and Woessman 8 In fact it depends on with whom resources, and autonomy.
(2007) and Hanushek (2010). he or she attends school. Student and peer characteristics
Using these tests as measures were the most important
of cognitive skills of the 9 See World Bank (2013b). characteristics in the regression
population, they show that (full results can be found in
countries that had better quality 10 According to PISA data, Table A.2 in the Annex). By
of education in the 1960s a student is classified as a decomposing differences,
experienced faster economic disadvantaged low achiever if he one often finds that one of the
growth during the years 1960- or she is in the bottom quarter explanatory factors is negative
2000, controlling for other of the PISA ESCS Index in a or higher than the actual
factors. country and performs in the difference, meaning that other
bottom quarter of students from factors outweighed their impact.
5 PISA 2009 Technical Report all countries/countries, after
(OECD 2012). accounting for socioeconomic 13 In this analysis, parental and
status. Only 2.8 percent of teacher engagement in the
6 Note: Countries that students in general profiled school community were used
participated only once in schools are disadvantaged as proxies to control for school
PISA between 2000 and 2012 low achievers, while the figure accountability.
were not considered for the increases to 12.5 percent in
ECA average trend. Linear vocational schools. 14 By decomposing differences,
interpolations were made for one often finds that one of the
Albania, Bulgaria, and Romania explanatory factors is negative
in missing years. or higher than the actual
difference, meaning that other
factors outweighed their impact.
analysis of PISA 2012 and past results 49
15 OECD aggregates all the
autonomy measurements
shown in Table 4 into two
indexes: an index that relates to
autonomy in resource allocation
(RESPRES), such as teachers
and budget preparation,
and an index that relates to
curriculum and assessment
policies (RESPCURR), such as
course content, textbooks, or
assessment policies.
16 Low achievers were classified
as those students at the
bottom 20% of the learning
distribution.
17 The Disciplinary Climate Index
is derived from students’ reports
on how often the followings
happened in their lessons: (i)
students don’t listen to what the
teacher says; (ii) there is noise
and disorder; (iii) the teacher
has to wait a long time for the
students to quiet down; (iv)
students cannot work well; and
(v) students don’t start working
for a long time after the lesson
begins.
Education Global Practice
Europe and Central Asia Region