Early learning in Ethiopia: equitable access and learning Early Learning Partnership Ethiopia System Diagnostic Report June 2018 { All dates Gregorian Calendar } System Diagnostic Report: a summary for early learning in Ethiopia .................................................... 1 1. Research purpose and structure of this report ................................................................................ 4 2. Context ............................................................................................................................................... 6 2.1. Substantial recent policy reform for early learning in Ethiopia ............................................... 6 2.2. Policy reform and rapid change creates current implementation challenges...................... 12 3. The system diagnostic approach..................................................................................................... 14 3.1. A model of accountability relationships to guide analysis .................................................... 14 3.2. Applying a model of ‘accountability relationships’ to Ethiopia’s early learning system ...... 16 4. Three potential levers for reform in Ethiopia’s early learning system.......................................... 18 4.1. Coherence for equitable access and learning within the early learning system .................. 18 5. Scope of Programmes...................................................................................................................... 19 5.1. Which early learning services are offered? ............................................................................ 20 5.2. What do we see in terms of programmes and coverage? ..................................................... 21 5.3. A closer look at enrolment reveals wide variation in O-Class coverage across woredas ..... 23 5.4. ‘Downwards’ delegation from federal and regional levels to woredas and schools suffers coordination and communication weaknesses ................................................................................. 29 5.5. Local preferences, negotiation and decision-making lead to ‘upwards’ delegation to schools that is incoherent with national policy for O-Class .............................................................. 30 5.6. Scope of Programmes: a lever for increasing access to quality early learning, equitably.... 33 6. Finance ............................................................................................................................................. 35 6.1. What sources of finance are available for O-Class? ............................................................... 36 6.2. The distribution of School Grant finance across regions ....................................................... 40 6.3. Finance: a lever for increasing access to quality early learning, equitably ........................... 43 7. Monitoring and Assuring Quality. ................................................................................................... 44 7.1. School readiness and child development goals of early learning in Ethiopia ....................... 45 7.2. A monitoring process that prioritises enrolment and checklists of resources ..................... 45 7.3. Implementation experience and preliminary findings from a MELQO field test.................. 49 7.4. Monitoring and assuring quality: a lever to reframe the understanding of early learning progress in terms of child development ............................................................................................ 61 8. References........................................................................................................................................ 63 8.1. Documents ............................................................................................................................... 63 8.2. Fieldwork sources .................................................................................................................... 65 9. Annexes ............................................................................................................................................ 66 List of abbreviations An attempt has been made to minimise the number of abbreviations used in this report. Any abbreviation that is used is first introduced in full. ECCE Early Childhood Care and Education EDRI Ethiopian Development Research Institute ELP Early Learning Partnership EMIS Education Management and Information System ESDP Education Sector Development Programme GER Gross Enrolment Rate GEQIP General Education Quality Improvement Program GEQIP-E General Education Quality Improvement Program for Equity InEHD Institute for Education Health and Development MELE Measure of Early Learning Environments MELQO Measuring Early Learning Quality and Outcomes MODEL Measure of Development of Early Learning MOE Ministry of Education NEAEA Ethiopian National Educational Assessment and Examinations Agency NGO Non-Governmental Organisations REAL Research for Equitable Access and Learning RISE Research on Improving Systems of Education SABER Systems Approach for Better Education Results SNNP Southern Nations, Nationalities and Peoples’ Region UNICEF United Nations Children’s Fund USD United States Dollar List of figures Figure 1: members of the ECCE taskforce in 2006-07 which contributed to ECCE policy development............ 9 Figure 2: four pillars in the National Policy Framework for ECCE and the ministry with lead responsibility .... 10 Figure 3: rapid GER enrolment increases from 2005-06 to 2015-16 (all early learning programmes) ............ 11 Figure 4: strategies and targets for early learning, to be achieved by the 2019-20 academic year ................. 12 Figure 5: basic actor map for Ethiopia’s early learning system ........................................................................ 14 Figure 6: a stylised accountability framework for Ethiopia’s education system (Iyer & Rossiter, 2018) .......... 17 Figure 7: summary of potential levers of reform and their relation to incoherence identified in the system . 18 Figure 8: coherence across accountability relationships: inconsistent delegation .......................................... 19 Figure 9: current access targets for early learning, to be achieved by 2020 .................................................... 20 Figure 10: four modalities for early learning in Ethiopia .................................................................................. 20 Figure 11: increase in early learning enrolment, by modality, from 2005-06 to 2015-16 ............................... 21 Figure 12: changes in GER during period of rapid expansion, by region (all early learning programmes) ....... 22 Figure 13: GER by region and gender, 2015-16 (all early learning programmes) ............................................ 23 Figure 14: share of primary schools with an O-Class, by region, 2014-15 & 2016-17...................................... 24 Figure 15: the relationship between Grade 2 to Grade 1 enrolment ratio and O-Class coverage, 2016-17 .... 26 Figure 16: GER in O-Class and share of schools with O-Class, by region .......................................................... 26 Figure 17: 6-year-olds as a share of all children enrolled in O-Class, by region, 2016-17 ................................ 27 Figure 18: 6-year-olds enrolled in O-Class as a share of 7-year-olds enrolled in Grade 1, by region and woreda, 2016-17.............................................................................................................................................. 28 Figure 19: O-Class children according to age group, by region, 2016-17......................................................... 29 Figure 20: O-Class enrolment by region, showing ‘on-age’ enrolment rates across woredas, 2016-17 .......... 31 Figure 21: example indicators might be used to classify woredas for attention .............................................. 34 Figure 22: coherence within accountability relationships: delegation flows, without sufficient associated finance ............................................................................................................................................................. 35 Figure 23: federal and regional distribution of expenditure as a percent of Gross Domestic Product (left) alongside the finance ‘loop’ from the House of the Federation down to schools (right) ................................ 36 Figure 24: sectoral breakdown of total woreda expenditures, financial years 2013 to 2016 .......................... 37 Figure 25: sub-sector financing for education across three sector plans since 2005-06 ................................. 38 Figure 26: estimated School Grant equivalent amounts per child in O-Class and per child in the population 41 Figure 27: shares of population, O-Class enrolment and financing, by region................................................. 42 Figure 28: coherence within the accountability relationships: delegation flows, without appropriate associated information .................................................................................................................................... 44 Figure 29: data collection content, frequency and process across seven regions ........................................... 46 Figure 30: enrolment in early learning and changes in dropout + repetition since introduction of 2010 National Policy Framework for ECCE ............................................................................................................... 47 Figure 31: final MELQO sample for 2017-18 field test ..................................................................................... 50 Figure 32: example rubric item for ‘Maths Learning Activity’ .......................................................................... 52 Figure 33: summary scores across all learning activity items in observation rubric ........................................ 53 Figure 34: example rubric item for ‘Grouping Types’ compared with ‘Individualised Instruction’ .................. 53 Figure 35: example ratings for ‘Grouping Types’ and ‘Individualised Instruction’ items ................................. 54 Figure 36: engagement with materials across categories ................................................................................ 54 Figure 37: Simple Addition and Subtraction against Mental Transformation exercises, which contribute to the early numeracy domain ................................................................................................................................... 56 Figure 38: large differences in Letter Identification across languages, suggesting areas for revision ............. 56 Figure 39: Forward and Backward Digit Span exercises, which contribute to the executive function domain 57 Figure 40: fieldwork sources referenced in this report .................................................................................... 65 Figure 41: population projections for 0-14-year-olds in Ethiopia, 2017 to 2100 ............................................. 66 Figure 42: three models of O-Class expansion ................................................................................................. 67 Figure 43: histogram of urban-share for all woredas (after excluding Addis Ababa, Harari and Dire Dawa) ... 68 Figure 44: the path diagram of the final hierarchical multi-domain direct assessment model (Koziol, 2018) . 69 Figure 45: MODEL exercises and items ............................................................................................................ 70 Figure 46: MELE domains................................................................................................................................. 70 List of boxes Box 1: research approach and methods ............................................................................................................ 5 Box 2: the Ethiopian government’s ambitious vision for Early Childhood Care and Education (ECCE), as stated in ESDP V ............................................................................................................................................................ 6 Box 3: population, location and poverty in Ethiopia .......................................................................................... 8 Box 4: example of an ‘accountability relationship’ and terms used in this report ........................................... 15 Acknowledgements This report has been written by the Ethiopia ELP team led by Tassew Woldehanna. It has been prepared by Jack Rossiter with other members of the team including Belay Hagos and Tirussew Teferra and Pauline Rose, together with Andy Dawes and John Hoddinott. Rebecca Gordon provided support to the analysis of interview transcripts for the report and Yohannes Wogasso provided support to the completion of SABER tables. The NEAEA and MELQO Global teams provided helpful guidance and support to implementation and analysis of the MELQO tools. We are grateful to members of the Research on Improving Systems of Education (RISE) Ethiopia team and others for their advice and feedback throughout the process, including Shelby Carvalho, Padmini Iyer, Janice Kim and Louise Yorke. We would also like to thank Young Lives for providing access to interview transcripts that have been drawn upon in this report. We would also like to thank Shawn Powers and Amanda Devercelli at the World Bank for their guidance and support throughout the process. System Diagnostic Report: a summary for early learning in Ethiopia Ethiopia is in a phase of extensive and ongoing policy reform in the early learning sector, which started with the introduction of a National Policy Framework for Early Childhood Care and Education (ECCE) in 2010. An early learning system that for so many years provided for a few hundred thousand children in wealthier circumstances has grown to provide for almost 4 million young children, spread across all regions and from all backgrounds. It is inevitable that substantial challenges arise during such periods of rapid expansion following any large-scale policy reform. This System Diagnostic Report is concerned with identifying these challenges, particularly as they pertain to O-Class, and it does so through a systems lens. This approach focuses on the incentives and barriers faced by individuals during the design and implementation of programmes, and how this influences equitable service delivery, at scale. The report combines findings from Early Learning Partnership (ELP) Phase 1 research in Ethiopia, which come together in the identification of three ‘levers for reform’. We suggest that these levers may serve as entry points under which specific policy options can improve readiness to learn in school, equitably and cost- effectively at scale. They also serve as starting points for future research efforts in Ethiopia’s early learning system. Levers include: Scope of programmes: i.e. how actors in the early learning system have managed a process of rapid 1 programme expansion and what that means for equity. Finance: i.e. how available finance has influenced a process of rapid expansion and quality improvement in early learning and what that means for equity. Monitoring and assuring quality: i.e. how the system is working to measure and improve quality in early learning service provision, how this relates to policy objectives for child development and school readiness, and what that means for equity. Education systems research is a lightly studied field and early learning system research lighter still. In order to make sufficient and substantial links between qualitative and quantitative research activities conducted during Phase 1, this report is structured around a model of accountability which illustrates the actors and relationships that exist in Ethiopia’s early learning system. Within this model – and based on the premise that “governmental (sub) systems of education ‘work’ when there is an adequate flow of accountability in the system” (Pritchett, 2015) – the above levers stem from three areas of the system in which objectives, finance, motivation and/or information are misaligned to deliver equitable access to quality early learning. These can be summarised as follows: Scope of programs: goals for O-Class that are delegated from the Ministry to Regions are different 1 from the expectations of citizens. These citizen expectations are ‘delegated’ to schools and influence who gains access and the services offered. The identified incoherence leads to inequitable access to early learning across regions and woredas, despite rapid enrolment increases. Finance: goals for early learning expansion, which are being realised in large part through the expansion of O-Class, have been and continue to be achieved with no dedicated budget for early learning services. Instead regions rely on local contributions and the more recent introduction of 1 enrolment-based capitation grants. As a result, the amount and structure of finance available to regions does not support the national goals and leads to inequitable access to early learning. Monitoring and assuring quality: early learning goals are concerned with improving child cognitive and socio-emotional development before children enter Grade 1, but the information collected, which is used to monitor performance against goals, is geared in favour of counting enrolment and resource availability. As a result, the information available does not work to improve understanding of early learning service quality, child development or school readiness with which to make evidence-based decisions for improvement. In response to these areas, steps may be taken to improve system alignment towards goals for equitable access to quality early learning. In that respect, this diagnostic report suggests policy options under each lever which might account for the following: Scope of Programs 1 The enrolment of 4-5-year-olds in O-Classes is both a challenge and an opportunity. Programs do not have to exclude children younger than 6-years-old so long as regions and woredas are also able to prioritise reaching 6-year-olds and achieving broad location coverage. A revised framework for programmes can, however, support regions to develop context-specific strategies to improve equitable access to services, which reinforce commitments in national policy to prioritise 6-year-olds so that they receive at least one year of early learning before starting school. An effective framework will be developed collaboratively, capturing expectations and demand from community stakeholders and considering regional perspectives from innovations over the past 5-10 years and on which programmes are most efficient to deliver. Within any framework for programmes, caution is needed to avoid the possibility that a differentiated approach reinforces inequalities of birth (e.g. by providing less effective services in harder-to-reach areas). A revised framework will also need to consider cost implications of assorted options for structuring programs, including the opportunities and implications of operating O-Classes as ‘school satellites’ in community centres. Finance plans in ESDP V assume that O-Class serves 6-year-olds only, with less expensive non-formal programs feeding into this. This is not the current pattern of expansion in O- Class, with parents enrolling children of different ages and so stretching provision. Finance Enrolment rates have increased rapidly but with substantial variation across regions, resulting in wide variation in finance available for O-Class, favouring those O-Class centres that are already established. This approach is both inequitable and unsustainable. The most disadvantaged schools and areas will require more resources to initiate O-Class and will require substantially different amounts of finance to reach full enrolment of 6-year-olds, from their current positions. GEQIP-E aims, to an extent, to address this with some prioritisation for emerging regions. More importantly, finance to early learning is insufficient to reach government enrolment goals set out in ESDP V, with scarce resource from ‘core’ sources such as block grants. Regions are, in 2018, in hugely different positions with respect to achieving targets for service delivery. A costed roadmap of financing for early learning – linked to a revised framework for programmes – can support regions to plan for equitable service expansion. As a first step, this could include assessments of basic resources required (including salaries) to deliver services to expected minimum standards. In addition, linking to indicators of coverage, it would enable prioritisation of funding to regions and areas that are currently under-served. Regions and woredas could use this assessment to inform costed targets and strategies for equitable early learning expansion. 2 Monitoring and Assuring Quality If child progress on preparedness for formal schooling or social-emotional competence is seen as an important part of early learning service delivery, then there must be some information about these. As a means to assess an approach to this, MELQO – a tool aimed at promoting feasible, accurate and useful measurement of children’s development and learning at the start of primary school, and of the quality of their pre-primary learning environments – was tested with 1,144 children in 36 schools across six of Ethiopia’s regions. This was a first attempt at establishing whether a tool of this type could fill this information gap. It has provided strong indications that modules may, with some further revision and adjustment, help to establish a culture of monitoring and supervision that is based on quality of the early learning environment rather than on enrolment and checklists of resources and facility standards. Aspects of the tool that appeared to offer most potential include the direct assessment of child development across literacy, numeracy, executive function and fine motor skills domains. Preliminary results indicated that the direct assessments were, overall, reliable and valid, suggesting that this tool could be further adapted for use in large-scale national assessments. Next steps will include additional work on establishing standards and analysis of variance according to age and language of implementation (a huge challenge in Ethiopia). Following this, a review of the tool against expectations for O-Class, which were updated in 2018 (as reflected in the 32-week teachers’ guide), will be required to gauge assessment validity for the context. Experience from that field test also suggests that the classroom observation tool aimed at identifying the quality of early learning environments currently faces challenges in relation to reliability in how items are understood by different observers. A revision of the tool that addresses these problems could (a) be used by cluster supervisors and school leaders in local efforts to observe and supervise O-Class teachers’ pedagogical practices, thereby linking local measurement to accountability and expectation; and (b) inform minimum standards for O-Class which contribute to an inspection framework for early learning. The System Diagnostic Report is structured as follows: in Section 1 we introduce the research purpose and structure of the report and in Section 2 we provide context on the early learning system in Ethiopia. We then proceed, in Section 3, to summarise our research approach and methods, which have informed all activities. Section 4 introduces levers proposed, then Sections 5, 6 and 7 proceed to look at these in turn, drawing on relevant research findings. 3 1. Research purpose and structure of this report The Early Learning Partnership (ELP) is a multi-donor trust fund managed by the World Bank and supported by the UK Department for International Development. It works with countries to improve early learning opportunities and outcomes for young children, through both research and operational support (World Bank, 2017a). In 2017, ELP funding was made available for a programme of Systems Research with two overall goals: (1) to provide policymakers in a set of focus countries with actionable information to help guide the delivery of quality, equitable early learning at scale; and (2) to build the international evidence base in the emerging field of systems research in early childhood education. This report focuses on the first phase of research to address to understand the levers for strengthening early learning service provision in Ethiopia. The work has been led by the Ethiopian Development Research Institute (EDRI) in collaboration with the Institute for Education, Health and Development (InEHD) in Addis Ababa, and the Research for Equitable Access and Learning (REAL) Centre at the University of Cambridge. The four other countries in which the research is taking place include Liberia; Pakistan (the province of Punjab); Tanzania and Jamaica. In a second phase, we anticipate that ELP research will evaluate the effects of early learning programmes and reforms and take guidance from the levers identified. Early learning programmes in Ethiopia are concerned with child development (i.e. socio-emotional development), child learning (i.e. their development of numeracy and literacy skills) and school readiness (i.e. a mix of academic foundations, socio-emotional skills and knowledge of health, safety and their environment). Throughout this system diagnostic report, we use the term ‘learning’ to reflect this (MOE, 2010a). The report addresses ELP’s four cross-cutting research themes: quality at scale, equity, cost- effectiveness and non-state actors; with a focus on priority areas identified by stakeholders in Ethiopia, including: quality of state service delivery, provision for children under age six and cross-cutting themes related to equity, notably geography. This System Diagnostic Report identifies three ‘Levers for Reform’ (hereafter ‘levers’) around which the report is structured. We suggest these levers may serve as entry points under which specific policy options can improve readiness to learn in school, equitably and cost-effectively at scale. In elaborating each lever, we do not seek to provide a solution (as would be assumed from a lever that might be pulled, leading to an outcome), rather we provide a description of the challenge in the early learning system and relate this to strategies for change. Levers include: • Scope of programmes: i.e. how actors in the early learning system have managed a process of rapid expansion and what that means for equity. • Finance: i.e. how available finance has influenced a process of rapid expansion and quality improvement in early learning and what that means for equity. • Monitoring and assuring quality: i.e. how the system is working to measure and improve quality in early learning service provision, how this relates to policy objectives for child development and school readiness, and what that means for equity. In Section 2 of this report we provide context on the early learning system in Ethiopia. We then proceed, in Section 3, to summarise our research approach and methods, which have informed all activities. Section 4 introduces levers proposed, then Sections 5, 6 and 7 proceed to look at these in turn, drawing on relevant research findings. 4 Box 1: research approach and methods All findings in this report are based on research activities conducted in Phase 1, which include: Actor map: stakeholder mapping of actors, programs and policies in the early learning system Situation analysis of the early learning system: based on the World Bank’s Systems Approach for Better Education Results (SABER)1 initiative, relevant modules were selected and completed from SABER Early Childhood Development, SABER Private Sector, SABER School Accountability and Autonomy, SABER School Support and SABER Teachers. Service quality and outcome measurement in O-Class: in collaboration with the Ethiopian National Educational Assessment and Examinations Agency (NEAEA), the Measuring Early Learning Quality and Outcomes (MELQO) suite of tools was adapted and field tested, including the assessment of child development and learning prior to entering Grade 1 and the quality of O-Class learning environments. Secondary data analysis: to contextualise change and political economy of reform, using Education Management and Information System (EMIS), Welfare Monitoring Survey and Young Lives2 quantitative and qualitative data. Political economy analysis: desk review and adaptation of accountability framework for Ethiopia, followed by semi-structured interviews conducted with officials at the Ministry of Education (MOE), donor partners at federal level and with early learning experts and focal persons from seven Regional Education Bureaus. Cost and finance review: preliminary review of funding to O-Class, including teacher remuneration and funding for early learning available at school level. 1 For more information on SABER, see: http://saber.worldbank.org/index.cfm 2 See: https://www.younglives.org.uk/ 5 2. Context 2.1. Substantial recent policy reform for early learning in Ethiopia Large-scale government involvement in early learning was formalised in Ethiopia’s fifth Education Sector Development Programme (ESDP V, MOE, 2015a). This represented a substantial policy departure from a long-standing commitment to provide guidance and oversight to early learning services delivered by non- state actors. An ambitious vision for early learning, with the aim of addressing equity in the education system, established in ESDP V (Box 2), arrived following a decade of policy negotiation and implementation support to early learning from federal and regional governments. Box 2: the Ethiopian government’s ambitious vision for Early Childhood Care and Education (ECCE), as stated in ESDP V “Quality, targeted, ECCE provision will be used as a tool to increase equity i n the education system. Without continued government expansion of opportunities, especially for the most disadvantaged children, ECCE will favour those children from relatively wealthy backgrounds, in predominantly urban areas. By focusing ECCE expansion first in the areas with lower educational attainment (and on the children most at risk of exclusion, drop-out and under-achievement within those areas), the government will seek to improve the performance of children who can benefit the most from the support in order to transition more successfully into Grade 1.” So urc e: M OE , 201 5a: 77 2.1.1. The policy development process Ethiopia’s 1994 Education and Training Policy remains the lead policy document for education and includes provision for early learning, emphasising “all round development of the child in preparation for formal schooling” (Government of Ethiopia, 1994). Until ESDP V, early learning has not, however, been a government priority in sector plans and programmes since 1994. Education Sector Development Programmes (ESDP I to ESDP IV, 1996-97 to 2014-15) have taken a common stance on the supply of early learning, emphasising no direct government involvement. Instead, the government has encouraged the private sector, and facilitated long-standing faith-based suppliers and Non-Governmental Organisations (NGOs) by providing a common curriculum and syllabus materials and operating within a framework of oversight and licensing for non-state providers (Source 1).3 The Government of Ethiopia has long-held education as a priority sector, as evidenced by the steady share of education budget in the total government budget, at 20 percent over the past decade (World Bank, 2016). The resource demands from primary and tertiary levels, however, left little scope for government engagement in the provision of early learning (Hoot et al., 2004). Since 2000, Education for All and Millennium Development Goals (MDGs) influenced education expansion, with the government seeking to universalise primary enrolment (Source 1). At the same time, rapid university expansion and upgrading was ongoing, as part of efforts to strengthen the education decentralization process (MOE, 2005a). This saw undergraduate enrolment more than quadruple from 42,132 to 192,165 between 1996-97 and 2004-05 (MOE, 2005a), and, as a sector, command almost one quarter of the education budget between 2002-03 and 2010-11 (MOE, 2002a; MOE, 2005a). In 2005, during this period of university expansion, pre-primary education remained in the hands of the non- state sector: “Government policy for this [pre-primary] sub-sector is not to establish and run pre-schools in the next five years. However, [government] has a critical role to play in policy development, curriculum design, standard setting, supervision, etc.” (MOE, 2005a). Private providers operated in towns and cities and 3 See Figure 40 for an explanation the different sources referred to as references. 6 did not extend to rural areas: as an indicator, kindergarten enrolments in the two City Administrations of Addis Ababa and Dire Dawa had already reached 29 percent for the 4-6-year-old population, in contrast to a rate of 1.6 percent elsewhere (MOE, 2005b). However, questions of fairness in the supply of early learning services – if left to non-state providers – were being raised by regional politicians and within the education administration (Source 1). This concern with unequal early learning service provision related also to Education for All goals, which the government reported against in international monitoring events in 2007 (Sources 1, 2). The goals encouraged service expansion by equitable means: “[1] Expanding and improving comprehensive early childhood care and education, especially for the most vulnerable and disadvantaged children”; and “[2] Ensuring that by 2015 all children, particularly girls, children in difficult circumstances and those belonging to ethnic minorities, have access to a complete free and compulsory primary education of good quality” (UNESCO, 2000). In 2007, the United Nations Children’s Fund (UNICEF) – which had been advocating for greater government investment in ECCE for many years – began to support the government in the development of an ECCE strategy (Source 2). The existing sectoral guidelines relating to services for infants and children were not supported by an overarching early childhood care and education policy framework and services available for this age cohort were “inadequate, but also fragmentary and lacking in coordination” (MOE, 2010a: 18). A new multi-sectoral strategy would bring together Ministries of Education, Health and Women’s and Children’s Affairs4 and could serve as a common framework for implementation across Ethiopia’s nine regional states and two city administrations (hereafter ‘regions’) which differed substantially in their capacity and availability of services (Source 4 – see also Box 3 which provides a snapshot of regional differences in population and poverty in Ethiopia). 4 Formerly, and at that time, Ministry of Women, Youth and Children’s Affairs. 7 Box 3: population, location and poverty in Ethiopia In 2018, Ethiopia is home to 7.8 million children aged 4-6-years old, approximately 80 percent of whom live in rural areas (Ethiopian Central Statistical Agency). This age group is eligible for early learning programmes prior to joining school at age seven, although there is no requirement for compulsory attendance at this or any other level of education. Ethiopia’s population is far from evenly distributed across its regions, such that 94 percent of this 4-6-year-old cohort5 lives in five regions: Oromia, Amhara, Southern Nations, Nationalities and Peoples’ Region (SNNP), Somali and Tigray (see Annex 1 for projected national youth population growth, which emphasises the scale that the early learning system might have to reach). Population of children eligible for early learning programmes (aged 4-6-years-old) by region, 2018 3.5 3.09 Millions 3.0 2.5 2.0 1.73 1.60 1.5 1.0 0.52 0.39 0.5 0.13 0.09 0.20 0.03 0.02 0.03 0.0 Tigray Afar Amhara Oromia Somali Ben-G SNNP Gambella Harari Addis Dire Ababa Dawa So urc e: Mi ni st ry o f E duca tio n o f E th iop ia , Ed ucat io n Sta tist ics Ann ual A bs trac t , 201 5 -1 6. N ot e: ‘B en - G ’ s tan ds fo r B en isha n gul - Gu mu z. Location – often region – will be used regularly in this report to examine issues of equity. Frequently there is an overlap between location and poverty, but it is important to highlight that although Ethiopia identifies four ‘emerging regions’ as those with lower levels of development (namely Afar, Benishangul- Gumuz, Gambella and Somali), poverty is widespread. According to the multidimensional poverty index, six predominantly rural regions have a similar level of poverty as the national average. This is why, where possible, we will look within regions at equitable access to early learning. Multidimensional Poverty Index (higher value indicates more poverty) by Ethiopian region, 2017 0.8 0.6 0.4 0.2 0 Tigray Afar Amhara Oromia Somali Ben-G SNNP Gambella Harari Addis Dire Dawa Ababa MPI value National average So urc e: O xf o rd P o ve r ty and Hu man D e ve lo pm ent Ini tia ti ve , C oun t ry B ri ef in g 2 01 7: E thi opia . N ot e: ‘B en - G ’ s tand s fo r B en isha n gul - Gu mu z. 5All population statistics are estimates, projected from a national census conducted in 2007. The distribution of the 4-6-year-old population represents the distribution of total population (any age) across regions. 8 Although progress in policy development started slowly, the state minister for General Education was instrumental in coordinating sectors towards an integrated approach to early childhood care (Sources 1, 2) – e.g. the opportunity to link early stimulation with health, nutrition, and child protection. An ECCE taskforce (Figure 1) was established and commissioned a baseline report on the status of ECCE in Ethiopia (Source 2). That 2007 baseline report identified major challenges in the delivery of ECCE, which could be tackled with a national strategy, including: “high payment that is requested to attend the preschool, lack of proper training of preschool teachers; lack of standard curriculum and guidelines; lack of culturally relevant story books; quasi non-existence of alternative care and education services for the majority of the children population in the rural areas; lack of access to early childhood education for almost all children and especially children from low socio-economic backgrounds; lack of awareness about the value and type of care and education of young children; low salary for teachers, causing high staff turnover; lack of early childhood education professionals; misconception about teaching children ‘because they are children’ and the use of foreign languages (mainly English) as a medium of instruction” (MOE, 2010a). Figure 1: members of the ECCE taskforce in 2006-07 which contributed to ECCE policy development This is an excerpt from a full actor map, shown and explained in Figure 5. In this diagram, members in Green are federal stakeholders, including the three signatory ministries, members in Blue are development partners and members in Yellow are interest groups. This figure imposes no structure among members. N ot e: m em be rs of the ECCE Task fo rc e in vol ved i n th e d e ve lo pm en t of th e Nat io nal P olicy Fra me wo rk f or E CCE fr om 2 00 6- 0 7 . Th is has no t a lways b e en th e s e t o f E CCE Task fo rce M emb er s . Th e M ini st ry of Wo me n ’s and Chi ld re n ’s Af fai rs , a t th e ti me wa s th e M ini st ry o f Wom en , Yo uth an d Chi ldr e n ’s Af fai r s . So urc e: ba sed o n docu me nta ry analy si s a nd s tak eho ld er int e r vi ews . During 2008 and 2009 policy development and negotiation continued, led by the ECCE Taskforce (Source 2) and informed by the National Education and Training Policy, The National Health Policy, National Nutrition Strategy and the National Policy and Legal Framework on Child Rights. Alongside policy development, new early learning initiatives emerged. These included a pilot of the Getting Ready for School Child-to-Child approach6 and the first signs of a reception year, referred to as “an interesting initiative… launched with success: it consists of organizing a pre-primary class within an existing primary school [which] has helped 6See: American Institutes for Research, 2013, http://www.childtochild.org.uk/wp-content/uploads/2014/07/AIR-Report-of-Child-to- Child-Pilot-Findings_June_-2013_A4.pdf 9 spreading ECCE into rural areas” (MOE, 2010b). Early interventions such as the Child-to-Child programme, helped to demonstrate the potential of pre-primary for school readiness and revealed strong demand among communities for early learning services in rural areas (Sources 1, 2). 2.1.2. The National Policy Framework for ECCE and its implementation By 2010 The National Policy Framework for ECCE in Ethiopia was ready and would serve to support the government in overcoming three challenges in the early learning system and in the education system more generally: • increase equity: by expanding services to under-served areas and thereby also move closer to international targets for universal access to services (Source 4); • enhance quality: by improving child readiness to learn in school and keep-pace with the curriculum (Source 1); and • improve efficiency: by reducing early grade absenteeism, repetition and dropout (Source 4). Its launch was timely, coinciding with Ethiopia’s first Early Grade Reading Assessment which found that “a significant percentage of children in Grade 2 read zero words correctly”, with considerably lower proficiency in rural areas – e.g. 69 percent non-readers in Sidama Zone – as opposed to urban centres – e.g. only 10 percent non-readers in Addis Ababa (RTI, 2010). These weak results brought further attention to the new policy and the potential for early learning to improve student readiness for school (Source 1). The National Policy Framework for ECCE in Ethiopia presents a “comprehensive, integrated, quality, developmentally appropriate and culturally responsive service for the holistic development of all children” until they reach age seven (MOE, 2010a:21). It provides a frame of reference for key sectors involved in the provision of services for infants and young children. It was expected to form a springboard from which other sector policies will be strengthened, developed or reviewed, particularly in the areas of health and nutrition, education, water and sanitation and social services. The policy framework sets out to: “establish a coherent governance structure for ECCE and ensure mainstreaming of ECCE in all relevant national policies and programmes; promote and support development of accessible, equitable and quality ECCE services, for all children, particularly for vulnerable children with special needs and marginalised children…” (MOE, 2010a:22). The National Policy Framework for ECCE identifies four basic pillars for service delivery, which span sectors, ages and stages of development (Figure 2). The framework identifies the Ministry of Health as lead for Pillars 1 and 2, to be delivered with support of the Health Extension Program and Community Health Promoters but including collaboration with the Ministry of Education to integrate parental education within the Adult Literacy Program (MOE, 2010a). The Ministry of Education then takes the lead for Pillars 3 and 4, under which “attention for the cognitive and psycho-social development becomes more formal” (MOE, 2010a:23). Figure 2: four pillars in the National Policy Framework for ECCE and the ministry with lead responsibility Ministry of Health lead Ministry of Education lead Pillar 1 Pillar 2 Pillar 3 Pillar 4 Health and Early Preschools: community- Community-based non- Parental education Stimulation Program based kindergartens (4 to formal school readiness (Prenatal to 3+ years) 6+ years) So urc e: MOE , 201 0a Guided by the new policy framework, from 2010-15, the government of Ethiopia began to provide early learning programmes, supplementing private and other non-state provision. Ambitious targets for expansion were communicated from the centre, such as “at least one Pre-primary class will be opened in all rural and urban primary school premises starting from 2010-11”, based on dual arguments of increased efficiency and skills development: “participating in ECCE helps the young child prepare for the formal school and succeed in 10 the early grades. Through an expansion of ECCE the drop-out rate in early grades, which is very high in many Ethiopian schools, can significantly decrease. At the same time, children who have attended ECCE, more easily acquire the necessary basic reading and writing skills.” (MOE, 2010b) With increased government involvement, new fee-free programme types were formalised, including ‘O- Class’ introduced in 2010, a reception year prior to entering Grade 1 (current programme types are summarised in Figure 10). These new programmes provided opportunities for rapid enrolment expansion, which has been widely reported (e.g. Rossiter, 2016). In the first year following the launch of the policy framework (i.e. 2011-12), O-Class enrolled almost three times as many children as had access to early learning through kindergarten centres the year before (Woodhead et al, 2017). Over the six year period from 2010-11 to 2015-16, Ministry of Education figures report that the Gross Enrolment Rate (GER) for all 4-6- year-olds rose from 5 percent to 50 percent, with rapid increases for both boys and girls (Figure 3).7 Notably, this expansion has taken place at a time in which sector plans for education allocated no more than 3 percent of the education budget to early learning (see Figure 25 for a detailed breakdown).8 In addition, according to OECD DAC data, in 2016, donor spending on early childhood education reached an all-time high of around USD $2.2 million, more than doubling over a period of four years. However, it remains an extremely small share of donor spending: equivalent to less than 1 percent of education aid spending in the country. Only six donors report spending on early childhood education, with UNICEF responsible for over half. Finland, Korea, Ireland, Canada and Germany also all contribute, but relatively small amounts. Figure 3: rapid GER enrolment increases from 2005-06 to 2015-16 (all early learning programmes) 50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% 2005/6 2006/7 2007/8 2008/9 2009/10 2010/11 2011/12 2012/13 2013/14 2014/15 2015/16 Boys Girls Total So urc e: Min is t ry of E duca tio n o f E thi opia , E duca ti on Sta tis tics A nnua l Ab st ract s 2 005 -06 to 20 15- 16 Successes in expansion since 2010-11 encouraged the Ministry of Education to set out an ambitious vision for equitable access to quality early learning in Ethiopia (Box 2). Large-scale government involvement was formalised in the fifth Education Sector Development Programme (ESDP V) with targets, by 2020, to (i) ensure that all children receive at least 1-year of pre-primary education and (ii) reach an 80 percent Gross Enrolment Rate of 4-6-year-olds (MOE, 2015a). ESDP V states strategies to reach targets, reflecting finance projections through to 2020, which prioritise classroom-based O-Class and kindergarten programmes for (i) and delivery via non-formal programmes for (ii) among younger children (Figure 4). 7 Note that throughout this diagnostic report, we do not include enrolment in Accelerated School Readiness programmes. ASR was first piloted in 2015/16 with a cohort of 9,267 children (UNICEF, 2016). That pilot has grown in 2017/18 as part of a randomised controlled trial continuing until 2019. Enrolment data are small in relation to other programmes and have not been captured by the government’s EMIS. 8 Data are not available on early learning expenditure. ESDP IV (2010-11 to 2014-15) allocated 0 percent of its total budget to early learning; the government continued to provide an oversight role and the resource requirement for that was captured in an ‘Advisory & Support Services’ budget line. ESDP V (2015-16 to 2019-20) allocated 3 percent of its total budget to early learning, in addition to any resource that supports oversight via the ‘Admin’ budget line. 11 Figure 4: strategies and targets for early learning, to be achieved by the 2019-20 academic year Strategies Expand O-class and kindergarten provision so that all children have access to at least one year of Access classroom-based pre-primary education Access Expand access to Child-to-Child and Accelerated Child Preparedness Programmes Improved teaching and leadership skills in all institutions, matched with greater motivation and job Quality satisfaction Providing services and resources to schools to improve the physical facilities and foster a safe and healthy Quality environment. Targets Access Percentage of students that receive at least one year of pre-primary education will reach 100% Access GER for pre-primary (age 4–6 years) will rise from 34% to 80% Access National strategy for non-formal Accelerated Child Readiness and Child-to-Child programmes exists Quality Percentage of pre-primary teachers who are qualified with ECCE multiyear diploma Quality Percentage of pre-primary schools with qualified leader (diploma) will reach 100% Quality Percentage of pre-primary schools met and well above the [inspection] standards will reach 60% So urc e: co mpi le d f rom E SD P V ( MOE , 2 01 5a) 2.2. Policy reform and rapid change creates current implementation challenges An early learning system that for so many years provided for a few hundred thousand relatively advantaged children in wealthier circumstances has grown to provide for almost 4 million young children, spread across all regions and from all backgrounds. It is inevitable that substantial challenges arise during such periods of rapid expansion following any large-scale policy reform. Beginning from the government’s vision for early learning (Box 2), this system diagnostic report is concerned with core challenges that affect how the early learning system functions to provide access to quality services, equitably. The report acknowledges the particular challenges of reaching the most disadvantaged groups, linked to (i) poverty, (ii) gender or (iii) location (see Box 3), and including pastoralists, linguistic minorities and children with disabilities. In identifying core challenges, the diagnostic exercise has focused on early learning service provision through O-Class. ESDP V acknowledges that, “[a] mix of modalities will be used to reach this [enrolment] target... [which] has not been defined upfront…. [but] in the first years of ESDP V different approaches will be piloted and lessons learned will be used to inform expansion choices.” Within the ‘mix of modalities’, public O-Class – an optional ‘reception’ year prior to entering Grade 1 – has so far been the most pragmatic way to rapidly scale–up early learning, and the preferred option for government and communities (Woodhead et al., 2017). With a focus on O-Class established, this diagnostic report identifies three core challenges which inform three levers. These levers, we suggest, may serve as entry points under which specific policy options can improve readiness to learn in school, equitably and cost-effectively at scale. Levers relate to: • Scope of programmes: i.e. how actors in the early learning system have managed a process of rapid expansion and what that means for equity. • Finance: i.e. how available finance has influenced a process of rapid expansion and quality improvement in early learning and what that means for equity. • Monitoring and assuring quality: i.e. how the system is working to measure and improve quality in early learning service provision, how this relates to policy objectives for child development and school readiness, and what that means for equity. 12 While this report focuses on current challenges, it is important to note that the early learning system is constantly evolving and far from static: new issues regularly arise and implementation difficulties are tackled. In conducting this system diagnostic analysis, we have aimed to keep track of ongoing changes and adapted to this. Importantly, the General Education Quality Improvement Program for Equity (GEQIP-E) – a revision to the government’s flagship quality improvement package – will formally commence in mid-2018 and includes a Quality Enhancement and Assurance Package for O-Class. This package will, it is hoped, lead to direct improvements in services delivered in O-Classes (World Bank, 2017a). A large programme such as GEQIP-E also affects the functioning of the early learning system, by influencing the roles of actors involved, their targets and objectives and the incentives that exist and motivate change. In addition, the government of Ethiopia is developing an Education Development Roadmap to 2030, which covers all levels from early learning to tertiary and is expected to arrive as the largest policy shift in the sector since 1994. Next steps for research or policy change in Ethiopia’s early learning system may take some direction from the levers identified but will also have to be informed by and work with the frequent changes triggered by government and its development partners. 13 3. The system diagnostic approach The Ethiopian early learning system is vast. In the diagnostic phase, it has been necessary to identify how the early learning system functions to deliver services. This requires an understanding of the individuals and organisations involved, their roles and relationships and which of these are expected to exert strong influence on system function. As a first step, an actor mapping exercise (Figure 5) and the completion of selected components from the World Bank’s SABER suite of tools,9 were used to create an overview of the system. The actor mapping and situation analysis provided a critical understanding of the relationships that exist between people and programmes in the early learning system and set the foundation for our analysis. Figure 5: basic actor map for Ethiopia’s early learning system This figure captures a screenshot of an actor map for early learning in Ethiopia but doesn’t work well in a document. Instead, the interactive map is available online at https://kumu.io/elpethiopia/elp-actors#elp- actors-overview. Each actor is coloured according to their organisation type and a description of their role is included. If you click on any actor, their relationships to others are highlighted. If you click on ‘loops’ such as ‘Finance’, or ‘Politics’, the relevant actors and relationships are highlighted. This actor map is a live document which will be updated and we hope will support others interested in Ethiopia’s early learning system. So urc e: b as ed o n d ocum en tary a nalys is a nd stak eh old e r in te r vie ws . A n in t erac ti ve ver sion of th is map ca n b e acce ss ed a t: ht tp s: / /kum u .io / el pe thi op ia /e lp -ac to rs #el p -act or s-o ve r vie w . 3.1. A model of accountability relationships to guide analysis 10 In seeking to understand how the early learning system ‘works’ to deliver services, ELP research uses, as a starting point, a framework proposed in the 2004 World Development Report “Making Services Work for Poor People” (World Bank, 2004). The framework relies on modelling a series of ‘accountability relationships’ that exist between policymakers, service providers and ‘clients’ and how these relationships may influence service delivery (in the case of early learning ‘clients’ are young children, their families and communities). An update to this framework was proposed in 2015 as part of the launch of the Research on Improving Systems of Education (RISE) programme which “seeks to understand how school systems in the developing world can overcome the learning crisis and deliver better learning for all” (Pritchett, 2015). In that paper, the 9 For more information on SABER, see: http://saber.worldbank.org/index.cfm 10 This section summarises a RISE Insight Note: “Coherent for Equitable Learning? Understanding the Ethiopian Education System” (Iyer & Rossiter, 2018). The note explains in greater detail the adaptation process for a model of accountability relationships in Ethiopia’s education system. 14 core argument is that “governmental (sub) systems of education work when there is an adequate flow of accountability in the system” and the approach is concerned with identifying areas of ‘incoherence’ within relationships which are preventing the transition from systems of schooling to systems of learning (Pritchett, 2015). This framework is the starting point for our attempt to model and understand the Ethiopian early learning system. Box 4 provides an example of an ‘accountability relationship’ which introduces terminology used throughout the report. Research activities have focused on analysing accountability relationships so that cases of incoherence can be identified. Where a case of incoherence exists, there may be an opportunity to make an improvement in that relationship. These circumstances present potential levers for reform. Box 4: example of an ‘accountability relationship’ and terms used in this report In this hypothetical example, the relationship is between the woreda education office and a local school. The woreda education office is a principal which guides what is to be achieved and the school is the agent, which has responsibility to achieve the goal. The woreda informs the school that a ‘Know Your School’ programme is being launched, in response to a study that finds children who enter Grade 1 are often overwhelmed with the experience and new environment, leading to high dropout. The programme brings 6-year-olds to the school monthly, to introduce them gently to the school environment, through games and walking tours. The woreda asks the school to initiate the programme and register 6-year-olds in the community. It provides some guidance documentation and a budget to cover the costs of teachers’ outreach work each month. The woreda emphasises that the regional head is leading the programme and will collect programme data, including the number of children enrolled and a report of activities. Stakeholders have agreed that a team will be sent out to intervene in schools that have not managed to register at least half of 6-year-olds in their community by the end of two years and will request the head teacher to provide a detailed plan for how they will achieve this target. In this ‘accountability relationship’ there are four ‘design elements’ which determine action: Delegation what the woreda wants the school to do, i.e. ‘initiate the ‘Know Your School’ programme and register 6-year-olds in the community’ Finance what the woreda will provide in support of the assignment, i.e. ‘guidance documentation and a budget to cover the costs of teachers’ outreach work each month’ Information what information will be used to determine performance, i.e. ‘programme data, including the number of children enrolled and a report of activities’ Motivation what will happen if outcomes are good, i.e. that ‘children will be better prepared to enter primary school; and what will happen if outcomes are not achieved, i.e. that the regional head will ‘request the head teacher to provide a detailed plan for how they will achieve this target’ In this example, the accountability relationship is ‘coherent’ around the goal of children being prepared to enter primary school. This means that all four design elements are compatible. For example, what the woreda has asked the school to do and what it has provided to achieve this are reasonable; the woreda sets out appropriate information that will be used to monitor performance against the goal and the school can be motivated by benefits to school readiness or by avoiding reputational loss. An ‘incoherent’ relationship would exist if no finance had been provided as then the school would struggle to deliver on the instructions of the woreda education office. Another example of an ‘incoherent’ relationship would exist if instead, the delegation and finance were aligned but there was no interest at the woreda office to collect information about whether the ‘Know Your School’ programme had been set up or not. Then the school might have no motivation in establishing the programme. Ultimately, incoherence leads to a breakdown in the relationship and that prevents the achievement of the goal. 15 3.2. Applying a model of ‘accountability relationships’ to Ethiopia’s early learning system The Ethiopian education system is largely decentralised, with the transfer of expenditure and decision- making authority from upper (federal and regional) to lower (woreda) tiers of government (MOE, 2002b). Schools (service providers) are strictly accountable to woreda governments for producing results and in turn, the woreda authorities are held accountable by the regional and federal governments for delivering basic services and achieving targets (Khan et al., 2014). In the Ethiopian early learning system – as in any system – actors are connected through relationships of ‘accountability’ that provide feedback loops. To capture the relevant accountability relationship within this decentralised system, RISE and ELP teams for Ethiopia have proposed an adaptation of Pritchett’s (2015) framework (for a full description of the adaptation process, see Iyer & Rossiter, 2018). The approach provides a useful ‘vocabulary’ for discussing specific dimensions of the existing system elements, which may provide alternative explanations for performance of agents in the system (Box 4). The stylised accountability framework for Ethiopia’s education system (Figure 6) acknowledges the decentralised nature of the early learning system and is based on a model of ‘subnational shared responsibility’ (Di Gropello, 2004) which places two different political actors (the Ministry of Education and Regional Education Bureaus) at the centre of the decentralisation process and distributes the main responsibility for service delivery among them. This model reflects the way in which the Ethiopian education system was structured following the country’s 1994 Education and Training Policy. Recent early learning reforms have been implemented according to this decentralised model. The three levels of government (federal, regional, and woreda) are organised hierarchically, and are seen as having mostly complementary roles in service delivery (Di Gropello, 2004). Kebele councils are also involved in planning for service delivery at the local level and are a conduit through which the voice of communities is passed upwards to woredas. Similarly, coordination mechanisms such as Parent, Student, and Teacher Associations influence the flow of information from parents and communities to teachers and schools. This sub-national model recognises the importance of the relationships between the federal and regional level and between regional and woreda levels in Ethiopia’s decentralised system (Iyer & Rossiter, 2018). The complexities in systems which involve “more than one political sub-national actor in service delivery with complementary functions makes the creation of an effective accountability system particularly difficult” (di Gropello, 2004:18). Our aim has therefore been to develop a model of accountability relationships within the Ethiopian system which will allow us to capture some of these complexities. Our adaptation of the World Bank and RISE models leads to a stylised accountability framework for Ethiopia’s early learning system (Figure 6). The improved understanding of actors, policies and programmes, and of how services are delivered in Ethiopia have informed research activities in ELP Phase 1. These activities have focused on the relationships from the Ministry of Education to Regional Education Bureaus and Woreda Education Offices and influence the levers identified in this diagnostic report.11 11In some systems, teacher unions play an important role in accountability relationships. In addition, civil society actors can also play an advocacy role in the system. We have not included these here as their engagement and influence is not as apparent in this context, with civil society actors more likely to be involved in service delivery, for example. 16 Figure 6: a stylised accountability framework for Ethiopia’s education system (Iyer & Rossiter, 2018) Key to relationships 17 4. Three potential levers for reform in Ethiopia’s early learning system 4.1. Coherence for equitable access and learning within the early learning system Starting from the model of accountability relationships adapted for the Ethiopian early learning system, and using data gathered across research activities, we aim to identify relationships which are ‘incoherent’ insofar as they do not work to fulfil Ethiopia’s goals for early learning. In the sections that follow we present the data and research findings that underpin our suggested cases of incoherence, specifically as they apply to O-Class. For this phase of ELP research we are concerned with equity with respect to access goals: from a point of view of a process of rapid expansion, and quality goals: from a point of view of child development and school readiness at any scale. We suggest that there are three important sources of incoherence in the early learning system. In these situations, system actors face incentives and barriers that influence the delivery of programmes such that they are unable to provide equitable access to quality early learning. Drawing on this, we propose three potential levers for reform which can modify the way these relationships of accountability work and may improve the delivery of early learning services. Levers are summarised in Figure 7 which shows the source of incoherence, the accountability relationships that are affected and the suggested impact on equitable access or learning. Figure 7: summary of potential levers of reform and their relation to incoherence identified in the system Monitoring and assuring Lever Scope of programmes Finance quality Incoherence between: Incoherence between: Coherence for Incoherence within: • delegation (what is to be • delegation (what is to be equitable access and • delegation, across done) and done) and learning relationships • finance (the resource with • information (what is used to which to do it) monitor performance) • Between Ministry and • Between Ministry and Regions • Between Ministry and Regions Accountability • Between Woredas and Regions • Between Regions and relationship affected Schools • Between Regions and Woredas • Between Communities and Woredas • Between Woredas and Schools Schools Substantial Early learning system is not Early learning system is not Early learning system is not effect coherent for equitable access coherent for equitable access coherent for equitable learning 18 5. Scope of Programmes Incoherence between delegated national objectives and local preferences and capacities, such that O-Class access is provided unequally, for a sub-set of the target population. A system can be incoherent when there are unconnected or contradictory conditions across relationships of accountability within a single design element. In this case, delegation – the roles and responsibilities that principals expect agents to fulfil in delivering goals – is different across the various relationships (Figure 8). The goals for O-Class that are delegated from the Ministry to Regions are different from the expectations of citizens. These citizen expectations are ‘delegated’ to schools and influence who gains access and the services offered. The identified incoherence leads to inequitable access to early learning across regions and woredas, despite rapid enrolment increases. But this source incoherence can be overcome. This incoherence is shown through: (i) a review of the national goals for early learning and the programme types offered by regions; (ii) a description of programme coverage focusing on O-Class, including shares of schools and target population covered; (iii) a discussion of how local and community preferences for early learning conflict with national goals; and (iv) the proposal of a potential lever to improve equitable access to quality early learning. Figure 8: coherence across accountability relationships: inconsistent delegation The left-hand system map represents the route of ‘downwards’ delegation for national goals to schools and frontline providers. The right-hand system map represents the route of ‘upwards’ delegation from citizens to frontline providers and schools, and to woreda officials. Key to relationships 19 5.1. Which early learning services are offered? The Government of Ethiopia’s rationale for involvement in early learning is to offer a cost-effective method of improving children’s readiness for school, focusing on areas with lower educational attainment and the children who can benefit the most from the support to transition more successfully into Grade 1 (see Box 2). This objective is reflected in two headline targets (Figure 9) which form the core of the Ministry of Education’s delegation to regions. Figure 9: current access targets for early learning, to be achieved by 2020 Access Percentage of students that receive at least one year of pre-primary education will reach 100% Access GER for pre-primary (age 4–6 years) will rise from 34% to 80% So urc e: M OE , 201 5a Ethiopia seeks to reach targets by delivering early learning through four programme types (‘modalities’). There are no policy restrictions governing which of the modalities regions employ to reach enrolment and quality targets (Figure 10). ESDP V permits a mix of modalities and investment in all these programmes is intended to advance progress toward both access targets. Current policy is for an O-Class to be attached to every primary school, to achieve 100 percent enrolment of 6-year-olds, as a contribution to 80 percent overall enrolment of 4-6-year-olds (Sources 2, 4). When reflecting on early learning implementation to date, regional stakeholders perceive O-Class as the most ‘effective’ modality to reach access targets (Source 4). Effectiveness here is not necessarily measured in terms of child development outcomes but is argued on the basis that it’s the most feasible to implement across the region and has been ‘accepted’ by communities and local governments (Oromia); because it can be easily integrated into primary schools and is cost-effective (Afar); or because there are qualified teachers and teaching materials and the programme relies on standard classrooms at school sites (Benishangul- Gumuz) (Source 4). Figure 10: four modalities for early learning in Ethiopia Accelerated School Kindergarten Child-to-Child O-Class Readiness Formal or informal Formal Informal Formal Formal Up to three years Duration Up to three years One year Two months (part-time)12 Main source of UNICEF & UNICEF & Private fees Government funding government government UNICEF & UNICEF & Main implementer Private sector Government government government O-Class or Teacher Private teacher Older children O-Class teachers Grade 1 teachers Target age group 4-6 years 4-6 years 6 years 6-7 years So urc e: ada pt ed f ro m Jo ur n ey s t o Sc al e (UN ICE F , 20 16 ) . 12 One session per week for 35 weeks. Each session lasts 2 to 3 hours 20 5.2. What do we see in terms of programmes and coverage? 5.2.1. Rapid enrolment increase has been driven by the new O-Class modality Apart from in the predominantly urban centres of Addis Ababa, Dire Dawa and Harari, early learning enrolments have risen rapidly, with the vast majority of enrolment growth having come via the newer modalities of O-Class in particular and, to a lesser extent, Child-to-Child (Figure 11). Kindergarten enrolments remain low (611,174 children in 2015-16) while doubling roughly every five years, O-Class enrolments jumped to over 1.0 million in their first year and almost tripled within four years, serving 2.6 million children by 2015-16. Figure 11: increase in early learning enrolment, by modality, from 2005-06 to 2015-16 4.0 MILLIONS 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 2005/06 2006/07 2007/08 2008/09 2009/10 2010/11 2011/12 2012/13 2013/14 2014/15 2015/16 Kindergarten O-Class Child-to-Child So urc e: Min is t ry of E duca tio n o f E thi opia , E duca ti on Sta tis tics A nnua l Ab st ract s 2 005 -06 to 20 15- 16 5.2.2. However, large differences have emerged in enrolment rates, favouring more developed regions The first dimension of unequal access depends on location. Within the national average enrolment trend (Figure 3), which shows rapid and consistent growth, there is huge variation in enrolment growth across regions (Figure 12). On introduction of the National Policy Framework for ECCE, SNNP and Tigray reacted quickly and established enrolment rates of 32 percent and 47 percent, respectively. In Tigray, “w hen the ECCE program was implemented a challenge faced the bureau and the bureau decided to establish the ECCE team and experts” who had been involved in the national policy development process, had received ECCE training from Addis Ababa University and were tasked to support early learning expansion in the region (Source 4). Other regions were unequally prepared to respond to the new options and expectations in the policy framework. According to Ministry of Education data, stark differences in enrolment rates emerged within the first year, with Afar, Amhara, Oromia, Somali, Benishangul-Gumuz and Gambella able to enrol, at best, 1 in every 6 target children by 2011-12 (Figure 12). Representatives of Oromia region reported that readiness activities coordinated by federal and regional governments (including raising awareness and training) were sufficient to prepare stakeholders for implementation of early learning at scale, however, this was not a commonly held opinion. Other regional representatives were more critical of the expansion process and cited reasons including a lack of trained teachers, textbooks, classrooms and teacher training colleges (Somali and Afar), to a lack of experts in the region to prepare for implementation (Somali, Tigray and Gambella), as well as capacity and budgetary issues (Afar and Benishangul-Gumuz) (Source 4). The predominantly urban ‘regions’ of Addis Ababa, Harari and Dire Dawa already had high, largely non -state, early learning enrolments (e.g., private kindergartens) which have continued to rise steadily. 21 Figure 12: changes in GER during period of rapid expansion, by region (all early learning programmes) 110% 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Tigray Afar Amhara Oromia Somali Ben.- SNNP Gambella Harari Addis Dire Dawa Gumuz Ababa GER 2011-12 GER 2015-16 GER National 2011-12 GER National 2015-16 So urc e: Min is t ry of E duca tio n o f E thi opia , E duca ti on Sta tis tics A nnua l Ab st ract s 2 011 -12 & 2 01 5-1 6 SNNP and Tigray have continued to expand access in the period up to 2015-16 and, along with them, Amhara and Gambella have made substantial enrolment gains in recent years, all reaching over 50 percent GER by 2015-16. Ethiopia’s other ‘emerging regions’ have experienced slower progress13 with Afar, Somali and Benishangul-Gumuz reaching only 8 percent, 5 percent and 37 percent GER, respectively, by 2015-16. Data from the 2015-16 Welfare Monitoring Survey suggests similar wide inequalities in access across regions, while also potentially pointing at lower rates of enrolment compared with EMIS data. It should be recalled that EMIS enrolment rates are dependent on comparing current enrolment data with extrapolation of age- specific population data from the 2007 census, which could be a source of inaccuracy.14 When asked of their 5-, 6- and 7-year-olds, which had completed an early learning programme in the previous year, 22 percent of this group was reported as having done so (at a time when the GER was 39 percent according to EMIS). Another dimension of unequal access relates to gender which, although appearing less severe than inequalities stemming from location, results in a persistent enrolment gap in favour of boys. This is represented in a Gender Parity Index that has widened slightly as enrolment has expanded, from 0.98 to 0.95 over the period 2010-11 to 2015-16. Compounding location-based variation in access, differences by gender are most noticeable in the ‘emerging regions’. The slightly higher enrolment of boys is evident in all regions except Afar where enrolment for both boys and girls is extremely low (Gender Parity Index = 1.06),15 with the largest differences in the other emerging regions of Somali (0.84), Gambella (0.89) and Benishangul-Gumuz (0.90) (Figure 13). It is possible that other dimensions of disadvantage, notably associated with poverty as well as disability for example, interconnect with gender to reinforce unequal access within regions. However, available data do not currently allow us to provide this level of analysis. 13 Four regions (Afar, Somali, Gambella and Benishangul Gumuz) are known as developing regional states and are referred to by government as ‘emerging regions’; because they have low level of development index as compared to the others (UNICEF, 2015, Organizational Capacity Development Plan in the Four Emerging Regions of Ethiopia). 14 As the following section explains, this likely inaccuracy, combined with lack of birth registration resulting in inaccuracies in reported ages of children, is illustrated by the net enrolment rate being estimated at over 100 percent in some regions which is impossible. 15 The Adjusted Gender Parity Index is used for Afar which has a higher enrolment rate for females. Adjusted parity indices address certain disadvantages of the simple parity indices: the latter are imperfect measures because they are not symmetrical around 1 and have no upper limit, with a theoretical range of 0 to infinity. For example, if the female enrolment rate is 40 percent and the male rate is 50 percent, the Gender Parity Index has a value of 0.8. If the female and male values are reversed, the Gender Parity Index has a value of 1.25, which gives the mistaken impression of greater gender disparity because 1.25 is at a greater distance from 1 than 0.8. (see: http://uis.unesco.org/node/539580) 22 Figure 13: GER by region and gender, 2015-16 (all early learning programmes) 120% 100% 80% 60% 40% 20% 0% Tigray Afar Amhara Oromia Somali Ben.- SNNP Gambella Harari Addis Dire Dawa Gumuz Ababa GER Male GER Female National GER Male National GER Female So urc e: Min is t ry of E duca tio n o f E thi opia , E duca ti on Sta tis tics A nnua l Ab st ract , 20 15 - 16 5.2.3. Non-state continues to play a leading role in urban centres, but new programmes reduce its significance in national enrolment Pre-2010, non-state actors dominated early learning service provision, with 100 percent of recorded early learning enrolment being generated through kindergarten programmes (MOE, 2010c). At that time, around 95 percent of kindergarten enrolment was provided by non-state actors in Ethiopia (Orkin et al., 2012) and most recent estimates keep this as high as 90 percent (MOE, 2013). National policy has repeatedly encouraged the involvement of non-state actors (private investors, faith- based organisations and NGOs) in the delivery of ECCE: “ encouragement for increased private sector provision ECCE services to communities in which low-fee options are realistic, will continue” (MOE, 2015a: 77). Non-state continues to play a leading role in urban centres but with the introduction of new modalities its importance has reduced in relative terms. This is not to say that its dominant role has changed in the urban centres of Addis Ababa, Dire Dawa and Harari, only that as other services grow rapidly, it captures a smaller share of all enrolment. Nationally, kindergarten now accounts for 16 percent of early learning, down from 100 percent in 2009-10, and continues to decrease as O-Class, Child-to-Child and other programmes expand into rural areas (MOE, 2010c; MOE, 2017). The Ministry of Education will continue to encourage and strengthen quality and accountability within private provision of programmes and will do this through (a) incentives to invest, including the provision of land or tax exemptions for learning materials; and (b) regulation and monitoring, including enforcing standards across all early learning centres and requiring a license to operate which depends on the attainment of standards (MOE, 2015a). 5.3. A closer look at enrolment reveals wide variation in O-Class coverage across woredas Latest EMIS data have 3.8 million children enrolled in early learning programmes with roughly 70 percent in O-Class and 15 percent each in Child-to-Child and Kindergarten programmes (MOE, 2017). The enrolment statistics so far in this report have shown total enrolment and its change across regions but do not provide information on (i) how access is distributed within regions; and (ii) whether children enrolled are of the ‘correct-age’ for the programme. It would, for example, be possible to reach 50 percent GER for 4-6-year-olds by enrolling all 4-year-olds, half of 5-year-olds and no 6-year-olds (See Annex 2 for a graphical representation). On the other hand, a national 23 50 percent GER can be obtained by enrolling all 4, 5 and 6-year-olds in exactly half of all schools.16 Concerned with equitable access to O-Class, the following two indicators are reviewed for a sub-set of 625 ‘rural’ woredas in which O-Class is expected to be the main form of early childhood provision and so can be compared (see Annex 3 for a description of which ‘urban’ woredas were excluded from the analysis). • Share of schools with O-Class: to provide access for all 6-year-olds to complete a year of early learning, the government targets an O-Class in every school. The GER provides no information on which schools or what share of schools contributes to aggregate rates, but an indicator of the share of schools with O-Class could help provide an estimate. • Enrolment according to age in O-Class: O-Class is a programme designed for 6-year-olds, but enrolment rates are unconcerned with age. With no attention to age of children enrolled, a high GER can be achieved in a manner that masks substantial inequities. As such, the GER provides no information about progress towards providing for all 6-year-olds. An indicator of age-specific enrolment in O-Class could provide a better estimate to inform progress towards a goal of reaching all 6-year-olds. 5.3.1. What is the share of schools with O-Class in each region? The number of government primary schools has increased from 11,490 (MOE, 2000) in 1999-2000 to reach 32,556 in 2015-16 (MOE, 2017). The government aims to have an O-Class in each government school, which will provide a path to achieving at least one year of early learning for every child before they start Grade 1. Figure 14 shows, for each region, the share of primary schools with an O-Class. Bars are shown for 2014-15 and 2016-17 to provide a sense of change in access and the line at 1.0 indicates full coverage – i.e. that there is an O-Class in every school. Coverage is low in Afar, Gambella and Somali – across these regions about 1 in 3 schools has an O-Class and it is unlikely that children from areas served by schools without an O-Class will travel to sites where services are provided. This impacts the share of 6-year-olds in these areas that can access programmes. In all other regions, by 2016-17 coverage had reached 80 percent – so that at least 4 in 5 schools offered O-Class. In this context it would be possible for most, if not all, 6-year-olds to attend a programme prior to Grade 1 in a school close to their community. Figure 14: share of primary schools with an O-Class, by region, 2014-15 & 2016-17 So urc e: Au th or s ’ ca lcula ti on ba se d on Ed ucat ion Mana gem en t Inf or ma tio n Sy st em da ta , 2 01 4 -1 5 & 201 6-1 7 . No te : So mal i do es n ot ha ve dat a fo r 2 01 4 -1 5 . Regions are innovating in service delivery for disadvantaged groups, including with respect to language. For example, Regional Education Bureaus have adapted O-Class materials for Saho and Kunama languages in Tigray, for Argoba in Afar, for Berta, Gumuz and Shinasha in Benishangul-Gumuz and adaptation is ongoing 16Assuming, for the sake of argument, that schools are all equally sized, or at least that the schools that have full enrolment are on average the same size as the schools with no enrolment. 24 for Agewugna in Amhara (Source 4). Other responses for disadvantaged groups are mixed in their coverage but include the adaptation of non-formal accelerated readiness and child-to-child programmes for pastoralist groups, a mobile education system in Afar, which extends coverage and adapts to needs of pastoralist communities, and in Somali and Benishangul-Gumuz, the purchase of materials for cluster resource centres, specific to the needs of children with disabilities (Source 4). In SNNP, one particularly interesting innovation is the establishment of O-Classes away from school sites. This has resulted in the number of O-Classes exceeding the number of schools in 2016-17 (Figure 14). It is a regional response to extending access to early learning for all communities, with multiple O-Classes then feeding into a single school.17 This is not an approach that is expected in policy documents or financial models, which assume that an O-Class will be constructed at a school site, but was also suggested by representatives of Amhara education bureau, during consultations with Young Lives in 2015, on the basis that there are contexts in which travel distances to the nearest school are prohibitive for the youngest children and their caregivers (Young Lives, 2015). This issue was raised once again during consultations with community stakeholders in Amhara and SNNP regions: “We cannot take our children to school because the school starts early in the morning (8:00am). Early in the morning, mothers have many house chores. We have to take care for smaller children, and we give priority to our household work. Previously, the O-Class teacher came to our home, took our children to O-Class, and returned them back home when they finished class. However, recently, he stopped this activity and we stopped sending our children to school because we are afraid of the car accidents. Our main problem is the main road, which crosses the community. The school is beside the road. Many car accidents have happened so far.” (Young Lives, 2017) In interpreting O-Class coverage rates it is useful to recall the goal set out by the government, which aims for service expansion that favours areas with lower educational attainment and the children who can benefit the most from the support (MOE, 2015a). However, despite government policies of affirmative action towards historically disadvantaged regions (Khan et al., 2014), including long-standing programmes of ‘special support’ in the education sector, three of four ‘emerging regions’ continue to have the lowest rates of O- Class coverage (between 24 percent and 44 percent of schools with an O-Class, see Figure 14). Given that poverty and educational disadvantage are often correlated it would also be useful to compare O- Class coverage to a woreda-level indicator of poverty or of other potential markers of disadvantage, such as linguistic minority status. This is not possible as those indicators currently cannot be linked to access/enrolment data, but it is possible to provide a speculative association between ‘educatio nal development’ and coverage. GEQIP-E has chosen to use the ‘Grade 2 to Grade 1 enrolment ratio’ as a Key Performance Indicator which is a “holistic indicator that can capture dropout, repetition, and readmission by estimating those who are lost in transition between Grade 1 and Grade 2” (World Bank, 2017a). This indicator reflects aspects of educational attainment, advantage and efficiency for each woreda. A lower ‘Grade 2 to Grade 1 enrolment ratio’ suggests schools and woredas with lower levels of educational attainment and efficiency which would classify these as priority areas for services. However, the relationship between ‘Grade 2 to Grade 1 enrolment ratio’ and O-Class coverage among woredas is positive, in favour of higher O-Class coverage where G2/G1 rates are higher (Figure 15). While caution is needed in trying to interpret the significance of this relationship it is consistent with our understanding of O-Class expansion: that it arrives first in the areas that are better coordinated, have some surplus resources (i.e. a spare room that can be repurposed as an O-Class) and more motivation (from the 17It does raise a question, however, of what these ‘O-Class-Schools’ do about managing school grant receipts, how they are managed and staffed and whether they gain from or lose out on the functions typically provided within a school. 25 school or from the community).18 However, the schools and woredas which tend to need the service the most – and which the government has delegated goals to provide services for first – have lower rates of coverage. Figure 15: the relationship between Grade 2 to Grade 1 enrolment ratio and O-Class coverage, 2016-17 Woredas with G2/G1 enrolment ratio: < 0.55 0.60 to 0.70 > 0.90 Woreda O-Class coverage: 62 percent 76 percent 91 percent So urc e: Au th or s ’ c alcu lat ion ba sed on Ed ucat ion Mana gem en t Inf o rma tio n Sy st em d ata , 20 16 - 17 . A question that follows is how O-Class coverage converts into O-Class enrolment, whether this is consistent between woredas and regions and how that might relate to equitable access. As schools provide O-Class, so access increases but the relationship between coverage and enrolment rates varies across regions, suggesting differences in the way that expansion is managed and variation in local demand for programmes.19 For example, in Gambella around one-third of schools offers an O-Class and in Amhara almost universal O-Class coverage has been reached, yet enrolment rates are almost identical across these two regions. Figure 16 compares O-Class coverage and enrolment for all regions, which sort into three groups. 1. Tigray, Amhara, Oromia, Benishangul-Gumuz and SNNP: have high shares of schools with O-Class (> 80 percent) but enrolments between one-third and one-half as high as coverage rates. Access to O- Class at each school is largely unconstrained even if not quite universal. However, regions are only partially enrolling target populations at each school site, leading to low ‘conversion’ rates between access and uptake. This is perhaps because a single O-Class, which is most common at each school (Source 4), cannot accommodate the full target population; or because of limited awareness and/or demand locally (Young Lives, 2017). 2. Afar and Somali: have low shares of schools with O-Class and low rates of ‘conversion’ between O- Class access and uptake, leaving very-low rates of enrolment (5 percent). This suggests that these regions suffer both issues: substantial access constraints in most locations, perhaps stemming from lack of resources or awareness for expansion (Young Lives, 2017); and only partial enrolment in areas that access is provided (for similar reasons as above). 3. Gambella: has a low share of schools with O-Class but an enrolment rate that is higher than this share. Although access is constrained in most schools, where available, uptake for O-Class is high. Figure 16: GER in O-Class and share of schools with O-Class, by region 140% 120% 100% 80% 60% 40% 20% 0% Tigray Afar Amhara Oromia Somali Ben-G SNNP Gambella Gross Enrolment Rate in O-Class (2015-16) Share schools with O-Class (2016-17) So urc e: Au th or s ’ c alcu lat ion ba sed on Ed ucat ion Mana gem en t Inf o rma tio n Sy st em d ata , 20 15 - 16 & 20 16 - 17 . 18 It is also (hopefully) the case that higher progression rates from Grade 1 to Grade 2 is a result of O-Class in the following year, although this temporal relationship cannot be explored reliably with available data. 19 Note that this is not a perfect comparison because gross enrolment data are available for 2015-16 but the share of schools is shown for 2016-17. 26 This sub-section has provided some description of O-Class coverage – which influences the locations in which children can access services. The differences in coverage coupled with the differences in how regions are ‘converting’ O-Class access into enrolment has important implications for equity. The next section investigates age of enrolment in O-Class which, in addition to coverage, will influence both total enrolment rates and equitable enrolment for all 6-year-olds. 5.3.2. What does O-Class enrolment look like among 6-year-olds Where there is no O-Class in the nearby school it is impossible for children of any age to enrol. Where there is an O-Class in the nearby school, the enrolment of 6-year-olds is not necessarily automatic; it will depend on local policies for enrolment, which may not prioritise access for 6-year-olds, and features of demand among families, which will depend on awareness and on barriers such as travel requirements (e.g. in the quotation in section 5.5) and priorities for time-use among children (Tafere & Pankhurst, 2015). By identifying the share of 6-year-olds among those enrolled in O-Class (children of the target age) it is possible to estimate coverage for the 6-year-old population: a group that is prioritised for at least one year of early learning before entering Grade 1. Recalling that any child irrespective of age contributes to a regional GER, multiple paths can lead to the same GER. It would be possible, therefore, for two regions to adopt quite different enrolment policies (i.e. one prioritising 6-year-olds, the other with no prioritisation) yet reach identical rates of enrolment among the 4- 6-year-old population. Even if rates of enrolment were equalised, approaches would have quite different implications for equitable access. Figure 17 shows, for each region, 6-year-olds as a share of all children in O-Class. On average this stands at 42 percent with only Tigray substantially different from other regions. This figure raises two questions: (1) if the proportion of 6-year-olds in O-Class is so low, then what share of 6-year-olds is receiving any early learning service, and how does this vary by location; and (2) who else is attending O-Class? Figure 17: 6-year-olds as a share of all children enrolled in O-Class, by region, 2016-17 So urc e: Au th or s ’ c alcu lat ion ba sed on Ed ucat ion Mana gem en t Inf o rma tio n Sy st em d ata , 20 16 - 17 Woreda-level population data is required to investigate O-Class attendance among the population of 6-year- olds in each woreda, but unfortunately these data are not available.20 Instead, as a proxy, it is possible to take the enrolment of 7-year-olds in Grade 1 as an estimate of the population of 6-year-olds who could be 20 Even if they were available, there would be limitations on their usefulness. Population data in Ethiopia are currently estimated using projections from the 2007 census. At national and regional levels these estimates may be acceptable. However, problems arise from the projections at woreda levels as factors such as internal migration are likely to mean that population projections will be misleading and errors will be quite different across woredas. The enrolment of 7-year-olds in Grade 1 can serve as an improved denominator against which to set 6-year-old enrolment because it (i) does not suffer any of the projection issues, rather it captures something real about the distribution of children in schools and woredas, today; and (ii) suffers the same errors in estimating child age, which we have no reason to suspect are other than randomly distributed across schools. 27 attending O-Class.21 This approach is likely to return a conservative estimate because it’s possible that not all 7-year-olds, particularly those in emerging regions, are enrolled in Grade 1, but is useful for relative comparisons, particularly given the unreliability of other age-based estimates of enrolment.22 Figure 18 shows the box plot for the rate of 6-year-olds enrolled in O-Class as a share of 7-year-olds enrolled in Grade 1 for each region. This provides an indication of age-specific enrolment rates in O-Class and reveals huge variation across regions and between woredas within regions. Each region has a box, the left-hand edge of which represents the woreda at the 25th percentile (i.e. a woreda with a low rate within the region), the middle line of which represents the median (i.e. the middle woreda with average rate within the region) and the right-hand edge of which represents the woreda at the 75th percentile (i.e. a woreda with a high rate within the region). Figure 18: 6-year-olds enrolled in O-Class as a share of 7-year-olds enrolled in Grade 1, by region and woreda, 2016-17 So urc e: Au th or s ’ c alcu lat ion ba sed on Ed ucat ion Mana gem e n t Inf o rma tio n Sy st em d ata , 20 16 - 17 Regions sort into three groups. In Afar, Gambella, Oromia and Somali, the median enrolment rate of 6-year- olds is only around 20 percent and often as low as 10 percent. The second group includes Amhara, Benishangul-Gumuz and SNNP, with an estimated median enrolment rate of 6-year-olds of closer to 45 percent. The third group includes only Tigray and the estimated enrolment of 6-year-olds is around 75 percent but still with variation ranging from 65 to 85 percent. This information is important from the point of view of equity and school readiness because it seems that GER targets are being achieved without a priority for ensuring that all children receive at least one year of early learning before entering Grade 1. While SNNP might have reached a GER of 70 percent, well on the way to national targets, only an estimated 50 percent of 6-year-olds are enrolled in O-Class. If that latter target were a priority at the school level, as it is in national targets, then we would expect to see higher rates of enrolment among 6-year-olds than in total for the 4-6-year-old group. Similarly, in Gambella, which has achieved a GER of 55 percent, only around 20 percent of 6-year-olds are enrolled in O-Class. It is possible for a 4-year-old or a 5-year-old to receive one year of early learning and contribute to the national policy target for ensuring that all children receive at least one year of early learning before entering Grade 1 (i.e. it does not have to take place at age 6). However, O-Class is designed to be developmentally appropriate for 6-year-olds and it is unlikely that a child attending at age 4 and then dropping out until starting school benefits in the same way as a child attending O-Class as a dedicated reception year before 21 Each estimate is dependent on the correct determination of each child’s age, and as birth registration is rare, this is not always correct. We have no reason to suspect that there is any systematic error in the estimation of age which will bias results. 22 Net Enrolment Rates in Grade 1 (MOE, 2017) return figures well above 100 percent in many regions and a national average of 116 percent, which is statistically impossible and demonstrates the combined challenges in (i) estimating the age of children in Ethiopia; and (ii) reliability of population projections for each age-cohort, based on the 2007 census. 28 starting school. Similarly, it is unclear what benefits accrue to younger children who attend a one-year programme for multiple years. Moreover, if 6-year-olds are not prioritised for enrolment as schools introduce O-Class, they will miss out altogether, before reaching Grade 1. In the national case, even if a gross enrolment rate of 50 percent gives a sense of considerable progress having been made in a short time period, in most woredas most 6-year-olds are not yet benefiting from O-Class. In the case of Gambella, low coverage of 6-year-olds links directly to its low coverage of schools with O-Class: if the school does not have O-Class then 6-year-olds in that area cannot attend. This leads onto the second question: who else is enrolled in O-Class and boosts enrolment figures if it is not the 6-year-olds, and why? Figure 19 shows, for each region, O-Class enrolment by age group. Each region has a small share (between 1 percent and 9 percent) of overage children but the majority are underage, accounting for 3 in 5 children in most regions. Figure 19: O-Class children according to age group, by region, 2016-17 So urc e: Au th or s ’ c alcu lat ion ba sed on Ed ucat ion Mana gem en t Inf o rma tio n Sy st em d ata , 20 16 - 17 Early learning enrolment rates have risen rapidly, particularly in O-Class. As has been shown, however, this has been achieved (a) with limited coverage of O-Class among schools and (b) with less than half of O-Class enrolees representing the target population of 6-year-olds. The basic argument for why this might have happened is an incoherence in delegation across relationships of accountability. What we mean by this is that although national goals seek equitable access for all 6-year-old children, this is not the delegation that comes at the local level, either ‘downwards’ from woredas to schools or ‘upwards’ from communities to schools and woredas. This may be the result of design, development and implementation processes within government as well as differences of opinion among community stakeholders in what they want for children before starting school and/or of pressure on schools and woredas to enrol children to reach enrolment targets unconcerned with age. The process for policy formulation provides insights on how and which stakeholder preferences are represented in national policy and targets. 5.4. ‘Downwards’ delegation from federal and regional levels to woredas and schools suffers coordination and communication weaknesses The ECCE framework and sector plans, as with most national policy documents in the education sector, were developed through a process of negotiation involving the Ministry of Education and Regional Education Bureaus, but with little involvement from lower levels (Source 4). Regional Education Bureaus report that they were involved through consultative meetings and validation of the ECCE policy documents, as well as during the baseline assessment conducted in 2007 (Sources 2, 4). This process is consistent with Regional Education Bureaus’ understanding of strategy development in the education system as a joint decision- making process which creates a framework that can then be adapted to suit regional contexts (Source 4). 29 However, perspectives about the level of regional and woreda involvement were different among stakeholder groups and the Ministry of Education described greater involvement than was discussed by regional officials (Sources 2, 4). At regional level, focal persons and experts were invited from Oromia, Amhara and Tigray to contribute to the design of the National Policy Framework, but other regions report that they participated only in the launch of the National Policy Framework, not substantively during design (Afar, Gambella, Benishangul- Gumuz and Somali) (Source 4). Almost all Regional Education Bureaus considered their region to be insufficiently involved in the design process. This occurred because there were delays in calling for representatives and experts from the office (e.g. Somali), or because awareness raising about the framework was ineffective (e.g. Amhara). Tigray was the only region that considered its involvement adequate because experts from the kindergarten programme and bureau heads were involved throughout the ECCE framework development process (Source 4). Regional officials’ perspectives on who sets priorities and targets for early learning were similar but not always consistent. Most regional officials report that targets are set at the regional level, reflecting and adapting a national goal (Oromia, Afar, Gambella, Amhara, BG) while others point to greater involvement of woreda education offices in setting targets (Afar and Somali), perhaps considering the variation among woreda capacity and context in these regions. Linked to decision making for targets, policy documents and implementation plans were cascaded through the system from the Ministry of Education to regions, at which stage adaptations to suit context were made before they were conveyed to woredas. Along with policy materials, a small sub-group of regional/woreda officials received training and were required to disseminate this information in their offices (Source 4). The various levels through which information must flow coupled with high turnover/change of officials at woreda, regional and federal levels mean that there is potential for substantial information to be lost along the way, particularly when a new person enters a new role. Review of Ministry of Education field supervision reports points to overall weak coordination and communication across levels in the implementation of early learning (Source 3). For example, although a Strategic Operational Plan for ECCE envisages technical committees at each level, which will coordinate horizontally and vertically for the achievement of targets, these barely function (Sources 3, 4). Consecutive field supervision reports confirm that the one of the major challenges for the provision of equitable and high quality early learning, is inadequate coordination between regions, zones and woredas, resulting in inconsistent messaging at the most decentralised levels (Source 3). 5.5. Local preferences, negotiation and decision-making lead to ‘upwards’ delegation to schools that is incoherent with national policy for O-Class At the local level, the role of the woreda education offices in all regions is to coordinate and fund training for early learning teachers based on the needs of the woreda, to prepare classrooms and undertake supervision of teaching. Woredas have a role in convincing schools to implement early learning programmes, with changes typically made following consultative meetings with schools (Source 4). Also, at this local level, Kebele Councils play a strong role in encouraging participation and enrolment in early learning and primary education, in part through annual community conferences and mobilisation campaigns (Young Lives, 2015). The school director often has a seat on a Kebele Council along with other influential people in a community and the linkage between school and community is expected to contribute to early learning provision, including community involvement in classroom construction and with schools to improve O-Class conditions (Sources 2, 3, 4). Community ‘investments’ and collaboration in early learning service provision assure strong voice in negotiations of which services are going to be delivered. Some regions (Somali and Amhara) highlighted the role of the community in constructing additional classrooms and fundraising to pay for teachers’ salaries and materials. However, as one region (Amhara) cautioned, this flexibility is intentional but can lead to a lack of 30 coordination and quite different processes for implementation depending on community/kebele/woreda interests, which influence what is delivered and how (Source 4). Some of this influence is captured in Figure 20, which illustrates the substantial variation in ‘on-age’ enrolment across woredas, with far more consistency in Tigray and Amhara than elsewhere and most variability in the four emerging regions. Figure 20: O-Class enrolment by region, showing ‘on-age’ enrolment rates across woredas, 2016-17 So urc e: Au th or s ’ c alcu lat ion ba sed on Ed ucat ion Mana gem en t Inf o rma tio n Sy st em d ata , 20 16 - 17 Although our Phase 1 research was concentrated at the federal and regional levels, Young Lives has conducted exploratory fieldwork with regional education bureaus and community stakeholders. This sought to understand perspectives on what children need prior to going to school and how this relates to services provided. These data offer an insight into the delegation from communities to schools, kebeles and woredas at the local level. A powerful sense of community ownership of O-Class prevails – linked to the substantial contributions that community members are expected to make to start-up and operational costs, financially or in-kind (Young Lives, 2017). School directors share a collective understanding of the O-Class programme’s objectives, here captured by a director from SNNP, and this understanding is broadly consistent with national policy: “We opened O-Class attached to our primary schools… to provide preschool education for children who have little access to kindergarten education [and thereby help] smaller children to read and write before they start primary education [and support] them to improve their social relationship with peers and other school communities. Socialization is an important part of O- Class programme.” (Young Lives, 2017) Sampled parents agree that O-Class is a programme for school readiness, citing the pre-academic benefits of the programme in preparing their children for school: “Our children, who learn at secondary school now, did not get the opportunity to learn in O- Class… They were compelled to join Grade 1 directly at the age of seven without having any early learning experience… Now in O-Class they are trying to count numbers and learn alphabets that are more helpful to them to become ready for Grade 1.” (Young Lives, 2017) But while programme content explained in these exploratory studies is consistent with policy and suits the desires of communities and directors, programme reach falls short with respect to governing which children attend. For example, parents consulted “regret that still large number of O-Class age children stay at home due to distance and [other, nearby] parents’ lack of awareness about the importance of O-Class for the future educational life of their children” (Young Lives, 2017). Other than in Tigray sites, parents proposed O- Class starting ages from age 4 and schools accept all children aged 4, 5 or 6 into O-Class. In sample sites, communities held a shared sense that the O-Class programme is ‘kindergarten for rural areas’, which is perhaps best captured through reference to the programme as “olmaa da’imannii” (kindergarten) among 31 parents in Oromia (Young Lives, 2017). Allowing children of ages other than 6-years-old to participate in O- Class is not a problem per-se but when it crowds out priorities to afford access to all 6-year-olds, it can have equity implications. Not only from an access point of view, but also for quality, O-Class delivery in ‘multi-grade’ classrooms increases the burden on often unqualified and/or underprepared teachers and raises questions about the age-appropriateness of teaching and learning strategies (Young Lives, 2015). Regional Education Bureaus reported that the issue of underprepared teachers begins in Colleges of Teacher Education which are trying to improve programmes for early learning training but themselves suffer capacity constraints and relevant skills among trainers (Source 4). In some schools, adaptations are being made to limit multi-grade teaching, but they seem so far unsatisfactory to ensure equitable access to quality early learning through O-Class: “Last year, students learned in two sections, children aged 4 and 5 years learned in one class and children aged 6 years learned in another class but after she [one teacher] left the school, the two sections have been merged which led to large class size. Now children aged 4-6 years old are merged in one class. There are also some overage children (aged 7 and 8).” (Young Lives, 2017) Overall, the local desire to provide ‘kindergarten’ in rural areas, although consistent with the Ministry of Education’s longer-term objectives for early learning, is often incoherent with current plans to extend access to services equitably, through O-Class for 6-year-olds such that all children receive at least one year of early learning before entering Grade 1. 32 5.6. Scope of Programmes: a lever for increasing access to quality early learning, equitably This section covers research findings on the theme of ‘Scope of Programs’. The incoherence identified in delegation across accountability relationships can be summarised as follows: • O-Class is intended as a classroom-based reception year that 6-year-olds attend before joining Grade 1. Its roll-out and implementation will influence substantially, progress towards national and regional policy goals which have a strong equity focus and target one year of early learning for all children prior to joining school. • Communities provide substantial support to service delivery at school-level – i.e. through Parent, Student, and Teacher Associations. Their preferences vary and influence local expansion and access to O-Class, particularly given weak coordination and communication of objectives ‘downwards’. This leads to different rates of enrolment growth across schools and woredas, linked to variation in community awareness, interest and support to the establishment of early learning programmes. • This variation is captured beneath national aggregate statistics, which reveals rapid progress in enrolling children to programmes along with substantial variation in (i) rates of enrolment across regions and between woredas; (ii) O-Class coverage among schools; and (iii) age among O-Class attendees. • This means that although the national policy aims to admit 6-year-olds to O-Class, they account for only 40 percent of children enrolled in the programme. This is likely to be around half of the population of 6-year-olds.23 Many 6-year-olds are missing out on the programme of school readiness before entering Grade 1, even though total enrolments are very high and rising. • From an equity point of view, there are indications that coverage is greater in schools with better rates of educational attainment overall and in better-resourced regions. While this is, in general, an issue of inequitable access, the multi-age structure of classes can also weaken the linkage between curriculum and age-appropriate pedagogy. This has implications for service quality. 5.6.1. Possible levers for reform In response to the incoherence identified, steps can be taken to improve access to early learning services, equitably. Policy options under this lever for reform might account for the following: • The enrolment of 4-5-year-olds in O-Classes is both a challenge and an opportunity. Programs do not have to exclude children younger than 6-years-old so long as regions and woredas are also able to prioritise reaching 6-year-olds and achieving broad location coverage. • A solution to the enrolment challenge in O-Class will require a revised framework for programmes which can support regions to develop strategies to improve equitable access to services and which can reflect commitments in national policy to prioritise 6-year-olds so that they receive at least one year of early learning before starting school. • An effective framework will need to be developed collaboratively, capturing expectations and demand from community stakeholders and considering regional perspectives on which programmes are most efficient to deliver. Having had an opportunity to innovate in early learning for 5-10 years, regional experiences can inform framework options (e.g. SNNP introducing community O-Class as satellites of schools and Tigray prioritising O-Class enrolment of 6-year-olds). • Within any framework for programmes, caution is needed to avoid the possibility that a differentiated approach reinforces inequalities of birth (e.g. services that are perceived to be less effective in harder-to-reach areas, which is a common strategy among regions consulted). A 23More precise estimates are not possible, given reliability of population projection data and difficulties in estimating child age, explained in section 5.3.2. 33 framework should be consistent with government policies for early learning expansion which prioritise early learning expansion “in the areas with lower educational attainment (and on the children most at risk of exclusion, drop-out and under-achievement within those areas)” (MOE, 2015a) • A framework does not require new programme types but will benefit from guidance that supports regions and woredas to plan and implement with equity in mind. This will also need to consider cost implications of assorted options for structuring programs. Finance plans in ESDP V, which afford a modest 3 percent of the sector plan budget to early learning, do so on the assumption that O-Class serves 6-year-olds only, with less expensive non-formal programs feeding into this (MOE, 2015a). The mode of early learning expansion to date has seen a far higher share of 4-5-year-olds enrolled in O-Class than financial projections expected, adding pressure to an already extremely tight budget. • With current data, coverage indicators may be added to monitor progress against equity targets. Indicators for coverage of O-Class among schools and coverage in schools based on age can be achieved with existing EMIS data and provide a far richer picture of where attention is needed. Figure 21, for example, provides an example of coverage indicators, which can support regional and woreda officials to prioritise action towards early learning goals. Further use of household survey data, such as the Welfare Monitoring Survey, can also potentially provide rich information on where intersecting inequalities within regions might be apparent, such as related to poverty, rural residence and gender. Tracking progress for these most disadvantaged groups towards national targets will be vital to identify if the targets are being met. Figure 21: example indicators might be used to classify woredas for attention A ‘low school coverage’ flag might capture woredas in which fewer than half of schools has O-Class and a ‘low enrolment coverage’ flag might capture woredas in which enrolment among 6 -year-odls is estimated at lower than 40 percent. As improvements are made towards national targets, flags can adjust. So urc e: Au t h or s ’ c alcu lat ion ba sed on Ed ucat ion Mana gem en t Inf o rma tio n Sy st em d ata , 20 16 - 17 34 6. Finance Incoherence between national goals and the finance that is allocated to achieve these, which results in an inequitable process of early learning expansion. A system can be incoherent when the design elements of a single relationship of accountability are internally incoherent. In this case, delegation – what the principals have as their objectives – is disproportionate relative to the amount and structure of finance available – what the agents use to fulfil objectives (Figure 22). The goals for early learning expansion, which it has been shown are being realised in large part through the expansion of O-Class, have been and continue to be achieved with no dedicated budget for early learning services. Instead regions rely on local contributions and the more recent introduction of enrolment-based capitation grants. As a result, the amount and structure of finance available to regions does not support national goals and leads to inequitable access to early learning. But this source incoherence can be overcome. This incoherence is shown through: (i) a review of the finance sources available for early learning; (ii) a description of how the main source(s) of finance are distributed across regions and relate to target population and enrolments in programmes; and (iii) a description of how enrolment-linked finance, during a process of rapid expansion, may increase inequity. Figure 22: coherence within accountability relationships: delegation flows, without sufficient associated finance The system map represents the organisations, from Ministry of Education to schools, within which delegation and finance flow, but are incoherent around national goals . Key to relationships 35 6.1. What sources of finance are available for O-Class? We are aware of four sources of finance available to O-Class: block grants, school grants, community contributions and direct grants/support from NGOs and donor agencies. Sources vary in scale and in nature, entering and travelling through the early learning system based on different rules and priorities. Block grants represent core government budgets which move downwards through national and regional finance infrastructure (see Figure 23), with disbursement decisions at each tier of government, while school grants – although initiated at the federal level – are assigned directly to schools (albeit they pass through bank accounts at higher levels). Other sources, in the form of community contributions and support from NGOs are typically transferred directly to schools, although NGOs and donor agencies also sometimes provide implementation support to woredas and regions (for example in the case of UNCIEF’s programmes supporting national and regional governments and selected woredas). 6.1.1. Block Grant Ethiopia’s system of blocks grants is intended to overcome a large vertical fiscal imbalance. For example, in 2008-09, about 57 percent of the public expenditure of the country was incurred by the Federal government while 80 percent of the revenue collection was made at the same level of government. On the other hand, about 43 percent of the public expenditure and only 20 percent of the revenue collection was made by the regional governments, with different propensities to raise revenue across regions (World Bank, 2010). The share of budget expenditure at regional level continues to increase, to the point that current expenditure shares are approximately equal between regional and federal levels (Figure 23). This reflects a longer-term trend linked to progressive decentralisation and transfer of responsibility and authority to lower levels of government for the delivery of general education (MOE, 2002b). Figure 23: federal and regional distribution of expenditure as a percent of Gross Domestic Product (left) alongside the finance ‘loop’ from the House of the Federation down to schools (right) So urc e: Wo rl d Ban k E thi opi a Pu blic Ex pe ndi tu r e R e vi ew (2 01 6 ) a nd a uth o rs ’ imp r es sio n bas ed on act or map pin g, SABER ana lys is an d key sta keh ol de r in te r view s . No t e th at i n t he le ft -han d cha rt , to tal ex pe ndi tu re as a s har e o f G r os s Dom e sti c P r oduc t has fall en o ve r ti me . Th is f oll ow s a p lann ed f isca l co nt rac tio n , pr ima ri ly th ro ugh r e duct io ns in fe de ral s pe ndi ng s inc e 20 02- 03 (W or ld Bank , 201 6 ). The distribution of block grants across regions is determined by a formula that is set by the House of the Federation and which assigns grants to regions after the federal budget has been deducted. The current formula for allocations to regions is based on an assessment of revenue potential, expenditure needs, and a “disability” factor (covering security, federal government related assignments undertaken by regions, weather conditions, cost of service and physical location) (DFID, 2012). Population features prominently (i.e. through expenditure needs) in block grant formulation such that regions receive approximately the share of the block grant represented by their share in the population (refer to Box 3 for an indication of population breakdown across Ethiopia).24 ‘Emerging regions’ with disproportionately greater needs, and/or regions with 24An analysis of the allocation in 2012 suggested that regions with smaller populations receive around two to three times their population share (e.g. 2.1 percent rather than 1.1 percent in the case of Benishangul-Gumuz), which reflects that the smaller regions 36 significant “disability” factors, are eligible for a greater share of the block grant than suggested by their share of the population. These adjustments are not large in percentage point terms but can be large proportionally, for smaller regions. For example, estimates for 2012 suggest that Amhara received 23.3 percent of total block grant finance (compared with a population share of 22.6 percent), Somali received 8.2 percent of block grant finance (compared with a population share of 6.1 percent) and Afar received 3.2 percent of block grant finance (compared with a population share of 1.9 percent) (DFID, 2012). Consistent with the increase in block grant transfers to regions, woreda level expenditures have grown significantly over the past few years (World Bank, 2017b). Woreda Cabinets decide how money will be allocated across sectors (see Figure 24 for a summary across all woredas) and, within sectors, budgets are used for recurrent (e.g. salaries) and capital (e.g. construction) spending. Greater decision-making responsibility at the woreda level allows budgets to be more responsive to local needs but it also requires woredas to have substantial and accurate information to make optimal spending decisions. Figure 24: sectoral breakdown of total woreda expenditures, financial years 2013 to 2016 Agriculture, 11% Other, 43% Education, 35% Health, 12% So urc e: W or ld Bank 20 17b . R ou ndi n g mea ns t hat sha re s su m t o 10 1 p erc en t . Even though block grants represent the core budget that regions and woredas use to deliver services, in the case of early learning, regions reflect on a lack of funds available to deliver the programme. Every region reported that they have no dedicated budget line for early learning delivery (Young Lives, 2015) and face a scarcity of funding for early learning through the block grant channel (Source 4). The lack of a consistent budget prevents effective planning for early learning expansion (Source 4). From time to time regional representatives can negotiate small budget shares via savings in the allocations to primary education, for example to implement a monitoring or supervision activity, but these situations are rare (Young Lives, 2015; Source 4). Concern over the inadequacy of funds is not surprising as the sector plan for the period 2010-11 to 2014-15 allocated 0 percent of its total budget to early learning (MOE, 2010b). Even with the ambitious expansion targets established in ESDP V, the share of spending on early learning, within general education, reached only 7 percent (MOE, 2015a). This represents 3 percent as a share of total education spending, in addition to any resource that supports early learning oversight via the Advisory and Support budget components. Figure 25 shows the sub-sector distribution which illustrates sector priorities for education expansion and improvement over a 15-year period. are typically facing greater development challenges and that there are fixed costs of conducting some activities such that budgets may need to be larger than population where populations are very small (UK Department for International Development, Promotion of Basic Services Business Case, 2012). 37 Figure 25: sub-sector financing for education across three sector plans since 2005-06 Pre-primary Primary Secondary TVET Tertiary Admin & Sector plan (%) (%) (%) (%) (%) other (%) ESDP III (2005-06 to 2009-10) 0 51 12 9 24 4 ESDP IV (2010-11 to 2014-15) 0 55 9 6 25 5 ESDP V (2015-16 to 2019-20) 3 27 12 17 34 7 So urc e: E SDP III, E SD P IV and E SD P V fi nancia l tab le s (MOE , 2 00 5 a; MOE , 2 010 b; M OE , 20 15 a) . E SDP V d oe s no t p r o vide t he o ve ral l sha re fo r pr e - p rima ry , p ri mary an d sec onda ry, so fi gu r es h er e a r e a ut ho rs ’ cal cula ti ons ba se d on t he ‘ge ne ra l e duca tio n ’ s har e a nd s har e s t o each l e vel r epo r ted in E SDP V . N ot e tha t ‘P r ima ry ’ b ud ge t fo r E SD P V wa s du e to b e inc re as ed by th e Go ve rnm en t of E th iop ia wit h a s ubs tan tial i ncr eas e t o teac he r sal ar ie s wh ich we r e n ot p lann ed n or b ud ge t ed f or i n th e s ect o r plan . Moreover, the vast majority of block grant funding that is allocated to education is used to pay salaries and is unavailable for expansion of services or the purchase of materials. As such, there is limited discretion that woredas have over the allocation of spending. An education public expenditure review from 2010 estimates that of regional government education budgets only 10 percent is used for capital items and that within the recurrent budget, an estimated 83 percent is used to pay salaries, as is common in other countries given teachers make up the bulk of recurrent expenditure (DFID, 2010). 6.1.2. School Grant During consultations, regional representatives reported that School Grants are the source of financing for early learning via O-Class, given the lack of resources allocated through regular channels. For many years School Grants have been allocated directly to schools based on enrolment in primary and secondary grades, as a resource for “implementing the school improvement plan aimed at improving learning conditions [in schools]” (MOE, 2015b). School Grants for O-Class were introduced in the 2016-17 academic year, as a means of increasing finance to early learning from the donor funded and government implemented General Education Quality Improvement Program (GEQIP). Because School Grants for the next academic year are calculated using data with a one- or two-year time lag, a top-up to School Grants was added in 2015-16. This allows for increased enrolment and the opening of new schools and is equivalent to 10 percent of enrolment in the non-emerging regions and 15 percent of enrolment in the emerging regions (MOE, 2015b). School Grants are spent based on decisions made by a school committee and can be used on any activity that will advance their priorities, with a few exclusions.25 An exit study of GEQIP-I and subsequent Joint Review Missions conducted in GEQIP-II show that School Grant resources have helped to improve conditions in classrooms and at school sites (generally, with no specific investigation for O-Class). More common changes included improvements to libraries, improvements to infrastructure and school compounds for play and the construction of separate toilets for boys and girls (Pfaffe et al., 2013; MOE, 2015b). The process of School Grant allocation to O-Class under GEQIP-II is identical across regions: annually, 60 Ethiopian Birr (approximately $2 USD) is transferred to the school for each 6-year-old child enrolled in O- Class. It is important to note that this means that any child that is not 6-years-old is not counted in School Grant transfers, even though a significant proportion of enrolment in some regions is among 4- and 5-year- olds (Sources 2, 4).26 The O-Class allocation serves as a top-up to the ‘core’ school grant that each school is due to receive based on enrolment in primary grades. There are also some special adjustments to ‘core’ school grant payments in the case of small schools where a minimum payment is made if enrolment is less than 200 children and for improving resources and facilities for children with special educational needs 25 Exclusions include the construction of buildings, the purchase of fuel or weapons, televisions or DVD players or as payments to individuals. This will exclude the option to use School Grant funds to build classrooms, but they can be used for the purchase of teaching and learning materials and facilities such as indoor/outdoor play areas and age-appropriate toilets. 26 Note also that calculations using current enrolments are not possible as it takes the Ministry of Education well over a year to gather and process all enrolment data. As a result, enrolment data used for calculation in 2020 will be from the 2018 or 2019 academic year, reducing slightly average funding per child in a context of rising enrolments. 38 (where a top up of 1 percent of the total school grant assigned to each region, for officials at that level to disburse based on regional priorities) (MOE, 2015b). Indicative data from 36 schools included in our field test of an early learning quality measure conducted in early 2018 (and discussed fully in Section 7), show that all 36 schools identify a school grant receipt, with 28 of these schools reporting funds specifically for O-Class. During consultation with regional representatives, participants expressed mixed opinions about the impact and fairness of School Grants to O-Class. In some cases, regions are content and feel that the system is fair (Amhara), in other cases representatives point out (i) that funds are not allocated equally, because of the fact they depend on the enrolment of students (Somali); and (ii) that regions have no flexibility in how they will spend the money and so cannot allocate to groups who require extra support, such as children with extra educational needs, or to balance the variations in block grant expenditures at woreda levels (Gambella, also cited in MOE, 2015b). Other regions argue (iii) that an enrolment- or population- based funding formula is inadequate “because the regions are not equally developed… other regions (emerging regions) need [extra] support” (Benishangul- Gumuz during consultations); or (iv) that because donor contributions to targeted woredas are large in comparison to core budgets, even though donors do a good job in targeting their support “there is a question of equity [in] woredas who are not supported by [donors]” (Tigray). Finally, (v) regions criticise a lack of advice and guidance on the utilisation of School Grants in O-Class because manuals were prepared before O-Class existed, have not yet been updated and contain no advice on how to utilise School Grants for O-Class (Sources 2, 4). School Grants will continue in GEQIP-E, at least until 2021-22, with continued donor support but with a requirement for greater national government contributions to the resource envelope (World Bank, 2017a). That programme strives to improve equity in access and outcomes within the education sector and grants will contribute to the attainment of key performance indicators, some of which will require priority support to struggling schools and regions facing greater challenges to provide early learning services (World Bank, 2017a). 6.1.3. Community contributions and NGO or donor support Community contributions, NGO and direct27 donor support can be financial (e.g. small financial contributions to the salary of an O-Class teacher, as identified via the MELQO study reviewed in Section 7 and as reported elsewhere, e.g. Teferra & Hagos, 2016), or in-kind (e.g. the provision of age-appropriate reading books by an NGO to a group of O-Classes in a selected kebele). What they tend to have in common is that for early learning they often transfer resources directly to schools, not through the government system and can offer targeted and context-specific support. Some donors and NGOs implement their own programmes, or at least provide substantial support to tens or hundreds of early learning centres (e.g. Save the Children, Plan International and World Vision). These early learning centres are not the focus of this diagnostic report as they reach a comparatively small share of the 30,000+ schools in which the government expects to implement O-Class (MOE, 2017). While block grant school grant allocations follow standardised formulae for disbursements to regions and woredas, the actions of external actors vary substantially across regions. NGOs In Oromia mainly conduct research and provide funds for the implementation of early learning while in Somali, UNICEF has provided budget for training ECCE stakeholders and purchasing early learning materials. Other NGOs and faith-based organisations have supported the financing and construction of schools (Source 4). UNICEF also works in 27Direct donor support excludes contributions to programmes like GEQIP or other large-scale donor-funded programmes. In the case of GEQIP, those resources are addressed in School Grants and it is beyond the scope of this diagnostic to map every donor contribution to programmes which work in the early learning system. 39 Tigray, supporting curriculum development, training of woreda and school principals and the purchase of indoor materials (Source 4). Donor inputs are seen as positive, as children are more motivated to play with resources, but support is only provided in selected woredas, based on a needs assessment (Source 4). In Amhara, UNICEF, Save the Children and other NGOs provide financial and technical support. In Gambella, UNICEF is the only donor engaged on a day-to-day basis, supporting O-Class training, provision of materials and finance, which was perceived positively by the region’s representatives. In Afar, UNICEF has four target woredas, which it has provided resources to, while in BG, UNICEF provides short term training, technical support, budget allocation and the purchasing of teaching aids (Source 4). There is a long tradition of community contributions to the education system, mostly through time and resources for construction and the supply of materials, rather than as cash or direct financial contributions. For example, over the period 2003-08, during a phase of rapid school expansion, capital spending on primary and secondary schools rose by about 20 percent and the number of additional classrooms created per year rose even more impressively, because of community contributions to complement capital budget resources (DFID, 2010). Such contributions were estimated to account for 40 percent of construction costs in primary schools of Oromia in 2007-08 and close to 30 percent nationally (DFID, 2010). Regions cite the role of communities in financing, managing (through a committee) and taking ownership for schools – and more recently to supplement teachers’ salaries and the costs of materials in O-Class (Source 4). The downside of community contributions is, however, that they are often more forthcoming in areas that are better resourced, or among communities with greater motivation to extend and expand schooling options. Communities contribute substantially to financing schools, but this varies widely across regions and between woredas, with likely negative effects of relying on contributions for issues of equity (Oumer, 2009). Where resources are most stretched – and so where government policy aims to provide services first (MOE, 2015a) – there are scant resources to support the construction of a new classroom for O-Class, or to contribute to teacher salaries. 6.2. The distribution of School Grant finance across regions There is in general (i) insufficient resources via block grants for early learning, relative to goals; (ii) an unequal distribution of surplus resources, which communities and woredas can draw upon to provide access (e.g. teachers with spare periods or spare classrooms, all unevenly distributed); and (iii) targeted but relatively small-scale support from donors, NGOs and faith-based organisations to selected locations which makes up for some, but far from all, of the shortfall. In this context, School Grants provide much-needed finance to O-Class and guidelines stipulate that a payment of 60 Ethiopian Birr (approx. $2 USD) will be made to schools for each 6-year-old enrolled in O- Class. The decision to include only 6-year-olds in the allocation formula was taken on the basis that O-Class is designed as a reception year for 6-year-olds before they enter Grade 1 (see Figure 18 and Figure 19, discussed earlier in relation to scope of programs). Actively restricting payments to 6-year-olds could provide an incentive to regions, woredas and schools to control access to O-Class according to age and limit the additional school grant liability in GEQIP-II, which would arise from including another grade in annual disbursements (Source 1). This approach may also help to overcome issues related to multi-grade teaching of 4, 5 and 6-year-olds including overcrowded classrooms and a syllabus that was unprepared for children more than one year before school entry. There is a risk, however, that capitation grants based on enrolment are not the best mechanism to transfer finance in a rapidly expanding system: funds follow results, rather than supporting ‘new entrants’ to establish and expand O-Class. School Grant allocations to O-Class, as for all primary and secondary grades, are calculated based on official enrolment figures collected via the EMIS system. Using these same data (in this example for finance allocated in the 2016-17 academic year) it is possible to look at (A) the ‘actual’ per-child funding from school 40 grants based on total O-Class enrolment and (B) the ‘actual’ per-child funding from school grants based on total population. Figure 26 shows funding levels under (A) and (B), by region, which reveals surprising variation in funding per-child enrolled in O-Class and in funding per child in the population, across regions. This outcome is a result not of discrimination but of a uniform funding formula which combines with a process of service expansion that is far from equal across regions. Figure 26: estimated School Grant equivalent amounts per child in O-Class and per child in the population 2 1.5 1 $ USD 0.5 0 Tigray Afar Amhara Oromia BG SNNP Gambella Amount per child in O-Class $USD Amount per child in 4-6 pop. $USD So urc e: A ut ho rs ’ ca lcula ti on ba sed on Educa ti on Ma nagem en t In fo rma ti on Sy st em data , 20 14 - 15 . No te: Soma li data a r e n ot a va ilab l e s o ar e no t inc lude d . Of the seven regions shown, three are ‘emerging’, namely Afar, Benishangul-Gumuz and Gambella (data are not available for Somali). These are also 3 of the 4 regions with the lowest ‘funding per child’ in the population. As a result of its high and mostly ‘correct-age’ enrolment, Tigray receives almost $2 USD per child in O-Class. In contrast, Amhara receives less than half that rate. It should be stressed that these rates per child are not based on discrimination but are an automatic outcome of the allocation formula for school grants to O-Class, the differences in O-Class coverage across regions, and the status of correct-age enrolment in O-Class across regions. Inequalities in funding per target child (i.e. per child in the population) appear, however, to contradict the achievement of delegated objectives: i.e. provide at least one year of early learning for all children before they enter Grade 1 and reach a GER of 80 percent for 4-6-year-olds. Instead, a de facto system of ‘payment by results’ has emerged, which rewards the schools, woredas and regions that have provided access to O-Class and more so those that have restricted access to 6-year-olds only (Figure 27). A difficulty with this approach, however, is that those schools and woredas with better O-Class coverage tend also to be in more educationally developed locations (Figure 15) and wealthier regions (Box 3). The community contributions that allow access expansion and support to teacher payments are also more available in areas facing less severe resource constraints. In Figure 27 each region’s share of target population, share of O-Class enrolment and share of O-Class financing (through School Grants) is compared. For a region to receive the share of finance that represents its share of children in the population, the blue and grey columns would be expected to be equal. 41 Figure 27: shares of population, O-Class enrolment and financing, by region 0.50 0.40 0.30 0.20 0.10 0.00 Tigray Afar Amhara Oromia BG SNNP Gambella Share of population age 4-6 Share of O-Class enrolment Share of O-Class financing So urc e: A ut ho rs ’ ca lcula ti on ba sed on Educa ti on Ma nagem en t In fo rma ti on Sy st em data , 20 14 - 15 . No te: Soma li data a r e n ot a vaila bl e an d s o ar e no t inc lud ed . No te: ‘sha r es ’ he re a re c alcu lat ed ac r oss th e se ve n re gion s re pr es en te d i n t he cha r t . Under this financing model there are regions that gain and regions that suffer, in relative terms. Contrasting four regions: • SNNP: a large region required to manage and achieve the enrolment target for one in every five young children in the country. SNNP has made considerable progress compared to other regions, in enrolling children in O-Class, capturing almost 40 percent of nationwide enrolment. It also, as a result, captures around 40 percent of School Grant financing to O-Class. This means that it receives a share of total financing that is substantially higher than its share of target children in the population (ratio of funding share to population share = 1.8, which implies that it is receiving a lot more of the total budget than its population would command if all children were resourced equally). • Tigray: a small region with only 6 percent of the total target population of 4-6-year-olds. Tigray has also made considerable progress compared to other regions in enrolling children in O-class with access mostly focused on 6-years-old. As a result, Tigray captures 12 percent of total financing, also substantially higher than its share of target children in the population (ratio = 2.0). • Oromia: the largest region in Ethiopia, with the responsibility to provide for almost one in every two young children in the country. Oromia has enrolled fewer children in O-Class than Amhara and SNNP (two other very large regions) and this means that it captures only 25 percent of all financing to O- class (ratio = 0.57). • Afar: the pattern for Oromia is also found in Afar although on a different scale. A much smaller region, with only 2 percent of the target population, Afar has enrolled a moderate share of target children in O-Class. Afar captures only 0.3 percent of total O-Class financing, like Oromia much less than the share of the population that it needs to serve (ratio = 0.17). In striving to achieve enrolment targets, jointly negotiated between the Federal Ministry of Education and Regional Education Bureaus, regions and woredas have – in most circumstances – tried their best to establish O-Classes with whatever resource or facility is locally available. In certain circumstances, e.g. where a spare classroom exists, this is easier and, in some situations, e.g. where teachers are stretched to capacity and there is no community support to pay contract staff, this is more challenging. Early progress against targets is rewarded with school grant payments. These payments boost the resources available to continue to improve service provision and ‘quality’ in schools. School grants are relatively small sums, but they are the only core resource reported by regions. A problem with the capitation approach to funding established O-Classes is that those schools without an O-Class get nothing. The financing principle that prevails for O-Class means that those schools, woredas and regions with the most remaining to be achieved to provide equitable access, receive the least with which to achieve it. 42 6.3. Finance: a lever for increasing access to quality early learning, equitably This section covers research findings on the theme of ‘Finance’. The incoherence identified between delegation and finance within accountability relationships can be summarised as follows: • Ethiopia has ambitious goals for early learning coverage and service quality, but although there are multiple finance sources within the education system, national plans allocate only 3 percent of the education budget to early learning. Moreover, even though sector plans have a budget assignment, there is so far no finance dedicated to early learning via the core block grant. • Core finance is insufficient to achieve targets. Instead, regions cite the role of communities in financing, managing (through a committee) and taking ownership for schools – and more recently to supplement teachers’ salaries and the costs of materials in O-Class. Community support, and appetite for early learning, has been substantial and has underpinned rapid enrolment growth. • Recently introduced School Grants for O-Class provide much-needed additional finance, which allows schools to improve access to programs and to improve the quality of services offered. Although not large in absolute terms, Regional Education Bureaus consider School Grants to be the crucial source of finance for O-Class. • The funding model that has emerged, however, relies first on community supports and contributions, which are unequally distributed and favour resource-rich areas, and thereafter on capitation grants which follow enrolments. This returns major differences in per-child funding across regions which is incoherent with equity goals for early learning. • An enrolment capitation-grant model of financing might be appropriate in a scenario of full enrolment, allowing effective transfer of authority to schools. In a period of rapid expansion, however, a de facto ‘payment by results’ system emerges, which severely disadvantages those in the areas that are least able to establish services, exacerbating access inequalities. 6.3.1. Possible levers for reform In response to the incoherence identified, steps can be taken to improve access to early learning services, equitably. Policy options under this lever for reform might account for the following: • The most disadvantaged schools and areas will require more funds to initiate O-Class and will require substantially different amounts to reach full enrolment of 6-year-olds from their current positions. GEQIP-E aims, to an extent, to address this with some prioritisation for emerging regions. • Revised School Grant documentation can provide guidance to schools and incentivise the development of O-Class. There is no information available on how schools use School Grants for O- Class or whether they choose to cross-subsidise that service with resources from the general School Grant. While School Grants may not be used for construction of buildings, there are many opportunities to improve facilities for early learning and resources available in classrooms. • There is also some room for discussion in the GEQIP-E programme of School Grants which may allow federal and regional officials to adjust the amounts of funding to O-Class relative to other grades. It may also permit adjusted assignments to woredas/regions based on needs. This can help to overcome the short-term inequalities in funding per-child. • Ultimately, a costed roadmap of financing for early learning can support a framework for programmes. This can include assessments of basic resources required (including salaries) to deliver services to expected standards and, linked to indicators of coverage, can prioritise funding to regions and areas that are currently under-served. It can also consider the different resources required to deliver O-Class at school sites versus delivery at standalone ‘satellite’ sites. Regions could use this as part of a toolkit for equitable early learning expansion in regional strategy development. 43 7. Monitoring and Assuring Quality. Incoherence between delegation of targets and the information that is gathered to monitor progress, such that conditions in O-Class restrict quality and learning. A system can be incoherent when the design elements of a single relationship of accountability are internally incoherent. In this case, delegation – what the principals have as their objectives – is concerned with improving child cognitive and socio-emotional development, to improve readiness before entering Grade 1, but the information collected – what the principals use to monitor the performance of agents – is geared in favour of monitoring enrolment and resource availability (Figure 28). As a result, the information available does not work to improve understanding of early learning service quality, child development or school readiness with which to make evidence-based decisions for improvement. But this source incoherence can be overcome. This incoherence is shown through: (i) a review of the school readiness objectives of early learning and the government’s ongoing efforts for quality improvement; (ii) a description of the monitoring processes implemented at all levels of the early learning system; (iii) a discussion of the implementation experience from a field test of tools to assess early learning environments and child development, including some preliminary descriptive findings; and (iv) the proposal of a lever to take what has been learned from the field test and use this to inform quality assurance processes and the NEAEA’s suite of national survey tools. Figure 28: coherence within the accountability relationships: delegation flows, without appropriate associated information The system map represents the organisations, from Ministry of Education to schools, within which delegation and information flow, but are incoherent around national goals Key to relationships 44 7.1. School readiness and child development goals of early learning in Ethiopia The National Policy Framework for ECCE set out a combined package of services covering the prenatal period to age ‘6+’ which is intended to “Ensure all children the right to a healthy start in life, be nurtured in a safe, caring and stimulating environment and develop to their fullest potential” (MOE, 2010a). Within this package, objectives were established for pre-school programmes which, in principle, will dictate what is delegated from the Ministry of Education and Regional Education Bureaus to Woreda Education Offices and subsequently, from Woreda Education Offices to Schools: “…the program will cater for the acquisition of basic skills (pre-reading, pre-writing, counting and arithmetic) in preparation for the child’s formal schooling. Social-emotional competence, including self-regulation, intrinsic learning motivation and the ability to cooperate with other students are some of the benefits that children may gain from attending preschool.” (MOE, 2010a:26) Findings from consultations with Regional Education Bureaus are consistent with these objectives for early learning. Regional representatives explain that they have introduced programmes to increase participation and awareness among local communities about the importance of general education (Source 4) and to solve quality and efficiency problems in primary education which were reflected in high rates of absenteeism, repetition and dropout, particularly in Grade 1 (Source 4). These were linked to low levels of preparedness, which has historically tended to continue into primary grades: “Reading and writing skills were poor among students at primary level” (Source 4). With mass early learning expansion, regional representatives report that they are trying to improve quality in the early years so that “Children will be able to read/tell story and sing songs… [currently] many children can’t read before primary school” (Source 4). As these objectives are communicated within the education system, they also inform the accountability relationship that exists between woreda education offices and schools, their directors and teachers. Within this relationship, woreda education offices specify what they expect of schools – in the case of O-Class, that schools extend early learning services to improve child readiness for formal schooling. Young Lives reports that directors and parents share a view of O-Class content and purpose that is consistent with national policy objectives. It is captured here by a quotation from mothers in a focus group discussion: “First, they learn alphabets and numbers, which are the basis to construct words and statements and to do some mathematical problems. This can help them to make good preparation for Grade 1. Second, children learning in O-Class and kindergarten are able to improve their social interaction. Third, in the urban site, children learning in O-Class and kindergarten have access to indoor and outdoor play materials, which help them to learn through play. Fourth, children are able to improve their language skills as compared with children who have no access to any preschool provision” (Young Lives, 2017). Early learning to improve foundation skills and socio-emotional competence is, therefore, a well-established and commonly held point of view among officials from national to local levels and within communities accessing services. However, the information that is collected on service delivery relies (predominantly) on counts of enrolment and the audit of basic facilities and resources, containing very little on quality of service delivery – i.e. through an understanding of who teachers are and what they do in classes – or on outcomes in child development or readiness. 7.2. A monitoring process that prioritises enrolment and checklists of resources A well-established government system for education enrolment data collection exists and includes the early learning sub-sector. It relies on the collection, aggregation and upwards cascade of information from lowest (i.e. school) to highest (i.e. federal) levels (Figure 29). The data collection cascade consists of four layers in all 45 regions: school, woreda, region, ministry. Two additional layers exist in some circumstances: in some but not all regions, zones play a role, sitting between woreda and region; and in many cases cluster supervisors mediate the relationship between schools (in a cluster of five or six schools) and woredas. Figure 29: data collection content, frequency and process across seven regions So urc e: co nsu lta ti ons w it h Re gi ona l Educa ti on Bu r eaus , 201 8 46 Figure 29 captures a combination of data that are intended to be collected and data that are collected. It is not possible in Phase 1 to identify, across all regions, what is collected and reported regularly or what might be missed. However, Regional Education Bureaus report that data collection exercises that require more time and resources (e.g. school visits, supervision visits, observations at woreda and cluster levels) regularly suffer “shortage of budget”, “lack of logistics”, “transport problems”, “lack of manpower” and shortage of funds to cover subsistence payments (Source 4). One part of the data collection system that collects information regularly is the annual school census through EMIS, which has received financial support from GEQIP. As a national census, EMIS uses quantitative survey instruments to gather data at the school level. In all regions, schools are required to collect information on O-Class enrolment (which can be used to estimate attendance, repetition and dropout rates) and most regions also report that they collect information on the number of teachers and their training status as key human resource inputs (Source 4). The EMIS census is not designed to cover issues of child development or teaching and learning processes but the focus on enrolment does make it possible to monitor progress against access targets and illustrate relationships between increased coverage and indicators of dropout and repetition. Using EMIS data, enrolment growth in early learning can be contrasted with rates of repetition and dropout in Grade 1 (Figure 30). It is not possible to establish a causal association between enrolment growth and early grade efficiency using these data, but it does appear that during the period of rapid enrolment expansion, combined repetition and dropout rates in Grade 1 have declined from around 30-35 percent to around 20-25 percent, while remaining relatively high. Figure 30: enrolment in early learning and changes in dropout + repetition since introduction of 2010 National Policy Framework for ECCE 28 50% 40% 30% 20% 10% 0% 2009-10 2010-11 2011-12 2012-13 2013-14 2014-15 2015-16 ECCE GER G1 dropout + repetition So urc e: M ini st ry o f Ed ucat ion o f E thi op ia , E ducat io n Stat is tic s A nnua l A bs trac t s , 2 009 -10 t o 201 5 -1 6 . N ot t hat i t cha n ge s t he o ve ra ll i nt e rp re ta tio n o f th e gra ph b ut no te tha t d ro pou t /rep e ti ti o n rat es r e qui r e tw o y ear s o f d ata , so t ho se fi gu r es r ep or t ed in an y st at is tical a bs trac t r ef lec t r at es o f dr op ou t/r ep eti ti on that occu r re d on e y ea r ea rl ie r . For EMIS core data collection, a standard process is followed to pass data upwards to the Ministry of Education and on top of this, some regions and woredas collect other data for their own purposes. Data collected at school level are passed to woreda offices, which compile and forward to regions. In addition, 28If 100 children enrol in Grade 1, the sum of the children that progress to Grade 2, that dropout in Grade 1 and that repeat Grade 1 the following year must equal 100. The Ministry of Education determines progression rates from the new intake to Grade 2 in the following year and determines repetition rates from reports submitted by schools through the EMIS system (i.e. a count of the number of students that came back and repeated the same grade in the following year). The dropout rate is then calculated using the residual, after deducting those that progressed and those that repeated. There is some uncertainty around whether school reports capture the full number of repeaters for Grade 1. If they do not, and the repetition rate is underestimated then this inflates the dropout rate and can be misleading. As a result, for this chart, rather than focusing on dropouts alone, we have shown the combined dropout and repetition rate as an indicator of efficiency – i.e. the rate that captures all students that did not progress to Grade 2. 47 some woredas conduct school visits with self-prepared or regionally-standardised checklists covering issues such as infrastructure, budget utilisation, textbook availability and absenteeism. Outputs from checklists are used in preparing ‘mid-semester results’ and can feed into ‘performance evaluations’ (Sources 2, 4). This is consistent with the structure of the early learning system which places service delivery responsibility at the level of the woreda. Regional Education Bureaus and the Ministry of Education, on the whole, are recipients of data gathered at lower levels, with responsibility to validate and generate reports for their level of aggregation (Source 4). On an ad-hoc basis, Regional Education Bureaus and the Ministry of Education use checklists and questionnaires to conduct sample-based ‘supervisions’ of service delivery (Source 4). Within this ‘core’ data collection infrastructure, sitting between schools and woreda education offices, it is the cluster supervisor who plays a quality assurance role through monitoring and supporting teaching and learning processes (Source 4). Cluster supervisors in all regions contribute to the cascade of enrolment and teacher availability data. Cluster supervisors have other region-specific duties which include monitoring: teachers’ ‘gaps’ in the learning process (Amhara, Tigray), data for students with special educational needs (Gambella) and the proper use of curriculum, play and learning materials (Amhara, Oromia, Somali and Tigray). Some regions reported that this is conducted using checklists, whereas others mentioned that cluster supervisors undertake classroom observations. The most frequent information gathering is reported in Amhara where cluster supervisors will visit each of their schools once a week (Source 4). Data collected from O-Class teachers during MELQO implementation, from a small sample of 36 O-Class teachers across six regions reinforce an argument that supportive supervision is practised. Although observation and supervision of teaching and learning processes is not universal or standardised, most teachers report that they are observed once a month or more by a senior teacher or school director, for up to 30 minutes, and around half of these lead to feedback after each observation (Source 5). Major challenges faced by the network of cluster supervisors, however, relate to (i) content knowledge on what and how to supervise; and (ii) resource availability to fulfil data gathering, linked to budget, transport and materials, which in many cases means that these duties are unfulfilled and so do not contribute to the monitoring system (Source 4). Research on cluster supervision practices in Oromia and Benishangul-Gumuz finds that supervisors are generally ineffective at promoting teacher professional development and instructional practices, and that the benefits to teachers are small (Tadele & Bekele, 2017; Afework et al., 2017). In addition, lack of standardisation in information gathered prevents any sort of meaningful aggregation across woredas. This is not to say that quality assurance needs to be rigid in the tools and processes used, but an effective system does require that whatever processes are followed are able to assure quality according to common expectations. Moreover, without relevant and broadly comparable information on quality in early learning environments and on levels of child readiness at the end of programmes, the performance of frontline providers can be only partially assessed (i.e. in relation to enrolment and resource availability). There is a substantial incoherence between the mandate for early learning – to provide services that increase child readiness for school via the development of core foundation academic and socio-emotional skills – and the types and quantities of monitoring data gathered, which pay some attention to monitoring quality but much more to counting enrolment and available resources. If child progress on preparedness for formal schooling or social-emotional competence is going to be an important part of the delegation in the relationship between woreda education offices and frontline providers, then there must be adequate information about these – or the accountability relationship is likely to be incoherent. Experiences from a field test of tools that measure child development and the learning environment may provide opportunities to establish a culture of monitoring and supervision in the early learning system that is centred on quality and child readiness outcomes. 48 7.3. Implementation experience and preliminary findings from a MELQO field test During the 2017-18 academic year, ELP researchers collaborated with a team at the NEAEA to field test a suite of tools designed to measure early learning environments and child development and learning. As a field test, the objectives of MELQO implementation during ELP Phase 1 were: 1. Review modules and assess alignment with national standards and goals. 2. Modify items to improve alignment, pilot-test with NEAEA leadership, review item function. 3. Document key questions, process of adaptation and potential for modules to support monitoring and quality assurance in early learning. In this section we draw on implementation experiences to review opportunities that this suite of tools offers to improve monitoring and quality assurance of early learning service provision. First, we summarise the MELQO suite of tools, the survey approach and tool development in Ethiopia. Second, we provide illustrative preliminary findings from classroom observation and child direct assessment modules. Finally, we propose next steps if these tools are going to be taken to scale. A full MELQO validation exercise is ongoing. 7.3.1. The MELQO suite of tools The MELQO initiative began in 2014 and was designed to generate locally-relevant data on children’s learning and development at the start of school, and on early learning environments.29 MELQO modules are designed for use in low and middle-income countries and can be integrated into existing assessments following adaptation to national standards. MELQO comprises two modules which allow linkages to be made between a measure of the early learning environment and of child developmental outcomes: • MODEL: the measure of development of early learning module provides a measure of children’s broad development across domains indexing children’s pre-academic skills (literacy and numeracy), executive function, fine motor skills and recently added socio-emotional development. MODEL includes both a direct assessment sub-module and a teacher/caregiver report sub-module based on surveys with teachers and/or caregivers. • MELE: the measure of early learning environments module includes seven constructs: environment and physical setting; personnel; interactions (between teachers and students and among students); inclusiveness; family and community engagement; pedagogy and play. The state of evidence on quality in early learning environments, and the strong cultural influences on what defines “good quality” led to a conclusion to focus on constructs rather than specific items as there were a small number of items that were cross-culturally relevant. MELE includes questionnaire and facilities observation sub-modules along with a classroom observation to provide a snapshot of the quality of the environment and experiences of children within the early learning programme on the day observed. The classroom observation is not intended as an evaluation of the individual teacher or staff member delivering the programme. 7.3.2. The MELQO survey approach and sampling frame The MELQO field test was led by the National Assessment Directorate at the NEAEA, with technical support from ELP researchers via EDRI and guidance from the MELQO Global team. This group collaborated on tool development, adaptation and translation, the recruitment and training of fieldworkers, data collection, processing and preliminary analysis. The exercise was supported by GEQIP-II and ELP funds. 29A background of MELQO including a detailed description of all ‘core’ (generic) tools and their development can be found at http://ecdmeasure.org/. 49 The field test included six units. Four of these were administered as one-on-one interviews with respondents: (1) child direct assessment, (2) teacher interview, (3) school director interview, and (4) parent/caregiver interview. Two others were conducted as observations: (5) O-Class classroom observation; and (6) O-Class facilities observation. Following discussions with the Minister of Education it was decided that the field test should be implemented in six regions and so with six language groups, one for each region: Oromia (Afaan Oromo), Amhara (Amharic), Tigray (Tigrigna), SNNP (Sidaamu Afoo), Benishangul-Gumuz (Berta) and Somali (Af Somali). Six schools were selected in each region/language, with each school required to have an O-Class to enter the sampling frame. As a predominantly rural programme, O-Classes were sampled in rural areas. In each school, 32 children were to be selected, so that each language had close to 200 respondents for within- language analysis. Data collection took place half way through the 2017-18 academic year and so current O-Class attendees were sampled (target 50 percent of respondents) along with children in Grade 1 who had attended O-Class in the previous academic year (target 50 percent of respondents). This sampling strategy was chosen to provide an estimate of what might be the end-of-O-Class upper bound (from Grade 1 students) and lower bound (from O-Class students midway through the year) of performance on direct assessment exercises. The final sample included 1,144 students across 36 schools (Figure 31). Of these, 571 were in O-Class (average age 5.7 years, range 4 to 8 years) and 573 were in Grade 1 (average age 7.1 years, range 5 to 8 years). Figure 31: final MELQO sample for 2017-18 field test Benishangul Oromia Amhara Tigray SNNP Somali Total Gumuz Schools Total 6 6 6 6 6 6 36 Children Boy 47 49 48 51 50 51 296 O-Class Girl 46 47 48 43 46 45 275 Children Boy 47 48 48 48 47 49 287 Grade 1 Girl 50 48 48 45 49 46 286 Boy 94 97 96 99 97 100 583 Children Girl 96 95 96 88 95 91 561 Total Total 190 192 192 187 192 191 1144 7.3.3. Tool adaptation and development Tool adaptation occurred iteratively, over six phases: 1. Preliminary item adaptation: researchers from the ELP team reviewed survey modules and made suggested adaptations to items for the Ethiopian context (e.g. the adjustment of numeracy items for alignment with expectations in the O-Class syllabus). 2. Adaptation workshop: was convened with representatives of the education sector including Ministry of Education directorates, donor partners, early learning programme implementers, regional representatives and test item development experts from the NEAEA. During the workshop, survey modules were reviewed and participants assessed alignment with national standards and goals. Participants focused on child direct assessment and classroom observation items, making recommendations for adjustments to suit the local context and in respect of terminology used. For example, classroom observation items use a rubric which captures learning activities that take place. Among many other things, participants suggested the introduction of a zero ‘level’ in the rubric, to extend the scale and increase the chances of capturing differences at the lower levels; and participants suggested adaptations to record both whether a literacy or numeracy activity takes place, and what type of activity that is and how it relates to the curriculum. 50 3. Tool development: supported by regular discussions with the MELQO Global team, feedback from the adaptation workshop was used by the team in preparing final tools. As tools were being adapted for Ethiopia, so improvements to ‘core’ modules were being made by the MELQO Global team – as would be expected in a phase of tool development. The process of tool preparation was therefore iterative and somewhat cyclical, including changes to adapted versions, and then revisions based on modifications to the ‘core’ materials. 4. Translation and adaptation to national languages: once tools were finalised they were translated and adapted to national languages, first to Amharic so that two sources were available (with English) and then to the other five languages (Tigrigna, Afaan Oromo, Berta, Sidaamu Afoo and Af Somali). Translation was conducted first by a team at the NEAEA with modules translated by two experts per language and moderated by a third, with an aim to maintain item equivalence across languages (not just direct translation). EDRI then carried out a second phase of translation verification including inter-language comparison (note that there are very few individuals who speak more than two of the field test’s target languages and so any attempt to establish inter-language equivalence requires great care). A final phase of ‘translation’ included the transfer of paper-based modules to tablets for fieldwork. This was managed via EDRI, with tablets capable of reproducing all modules in any of the six national languages. 5. Pre-field test and collection of training videos: as an input to fieldworker training for the classroom observation module, a pre-field test was conducted in O-Classes around Addis Ababa, using the Amharic tool loaded onto a tablet. Video footage was captured by a videographer from EDRI. The pre-field test led to some changes to the tablet versions of tools, to improve ease of use. 6. Training and school-based trial: supervisor and fieldworker training took place over two weeks, based in Addis Ababa with participants drawn from each of the six regions and teams combining EDRI and NEAEA recruits. During the training a school visit was included to allow fieldworkers to work in small groups observing each other while tools were trialled with children attending O-Class in and around Addis Ababa (in Amharic). Supervisors were briefed on all survey modules in the first week and during the second week supervisors and enumerators were split with 24 receiving intensive training on the classroom observation and other MELE units (i.e. two persons per fieldwork team) and 24 receiving training on child direct assessment and other MODEL units (i.e. two persons per fieldwork team). During the training process tool improvements (for content and user friendliness) continued based on feedback from fieldwork teams – this represents one of the major benefits of working with tablets for administration and with a programmer who was with the team for the training duration and could make swift adjustments. 7. Field test: finally, the trained fieldworkers conducted the survey over the period 12-26 February 2018. 7.3.4. Preliminary findings for the quality of early learning environments (MELE) This section focuses on preliminary findings related to learning activities and teacher-child interactions from the classroom observation unit within MELE. The observation protocol relied on a rubric, with an observer scoring from 1 to 4 based on their observation across a full O-Class session (approximately 2.5 hours). Two observers worked together, taking notes throughout the O-Class for each observation item and then discussing at the end of the session to determine a combined judgment.30 Figure 32 provides an example of the rubric for a maths learning activity; a format which is common across all learning activity items. 30The average age of sampled O-Class teachers was 26-years-old and most (31 of 36) were female. Great care was taken to train observers and to put processes in place, led by the team supervisors, to reduce the threat of ‘personal reactivity’ among subjects during the observation. 51 Figure 32: example rubric item for ‘Maths Learning Activity’ We review scores for items on learning activities (seven items), teacher-child interactions (seven items) and engagement with materials (six items).31 Rather than providing item by item measures, a summary is presented for each category. As indicated in more detail in section 7.3.6, it should be noted that the results need to be interpreted with extreme caution. Not only was the approach not intended to be representative, but it was apparent from the training that observers understood very different things from the categories on the classroom observation tool. For example, ‘play-based learning’ means something very different to different people. In recognition of this, some common definitions of the terms were attempted (e.g. to refer to open-ended questions). However, this probably no longer related to play-based learning. Learning activities: Figure 33 suggests that, across seven learning activities (covering maths, literacy, expressive language, motor skills, free play, music and movement), in one quarter of instances, the learning activity does not occur; and in around one half of instances observers recorded that repetition only was used. In the remaining instances, learning activities are delivered with one or more elements of more interactive approaches for children. It appears that learning activities which relate to the development of pre-academic maths and literacy skills and the development of fine motor skills are much more likely to occur, but using repetition. Within the seven learning activities identified on the observation schedule, observers reported that three events are least likely to occur: (1) “Teacher reads an age-appropriate storybook with text and pictures”, (2) “Learning activities that promote free play or open choice”; and (3) “Learning opportunities that allow children to engage in Music/Movement activities”. No activity stood out as being more likely to be taught using elements of play, than others. 31 Other observation items covered facilities at the school level, including play areas, water and sanitation. Combined, these provide a snapshot, for the sample schools only, of resource and facility availability. These are not reported here as the section is concerned with discussing implementation challenges of innovative observation elements. 52 Figure 33: summary scores across all learning activity items in observation rubric 60% 56% 50% 40% 30% 27% 20% 12% 10% 5% 0% Does not occur Taught using repetition only Taught using one element of Taught using two or more 'play-based' learning [see elements of 'play-based' note] learning [see note] N ot e: da ta a r e fr om 3 6 O - Cla ss e s. E le me nt s of ‘p lay - ba sed le ar nin g’, acco rd in g to th e rubr ic , ar e ( i) “all ow in g c hil dr en s om e ch oic e i n h ow t o u se ma te ria ls or car ry out an ac ti vit y ”; o r (ii ) “teac he r en ga ges ch ild in disc us si on , u se s o pe n - e nd ed qu e st ion s ”. Teacher-child interactions: unlike learning activities, in the tool, teacher-child interactions do not have a common link between rubric number (i.e. 1-4) and what that means about the nature of teaching (i.e. taught using repetition or ‘play’ or other). Instead, each teacher-child interaction item retains the same 1 to 4 scoring but the number will take on a quite different meaning depending on the interaction in question (e.g. Figure 34). This raises one obvious challenge in reporting: even though items demonstrated smooth distributions of scores across options, a score of 2 on one item doesn’t necessarily bear any relation to a score of 4 on another item. In fact, the former could be a better example of teaching than the latter, even though it is numerically lower. This was a known challenge during item adaptation and development and relates to the foundation question of “What do quality teacher-child interactions look like in the Ethiopian O- Class context” and how can these be built into standards statements that exist in items. The field test, as a pilot, is one means of gathering information to improve the understanding and the ‘scoring’ of each teacher- child interaction item. Figure 34: example rubric item for ‘Grouping Types’ compared with ‘Individualised Instruction’ Rather than providing a summary of teacher-child interactions combined, Figure 35 illustrates measures for the two items shown above: ‘Grouping Types’ and ‘Individualised Instruction’. The modal rating for 53 ‘Grouping Types’ is a 1, suggesting that most instruction observed happened in whole group exercises, while the modal rating for ‘Individualised Instruction’ is a 2, suggesting that teachers oc casionally showed some awareness of the individual needs of children. These cannot be compared, leading to a conclusion that teachers are ‘better’ at individualised instruction than they are at ‘grouping’, but they may be useful indications of what an O-Class in our sample ‘looks like’, indicating that there is perhaps not enough small group or individual work supported by teachers, than international best-practice would suggest, and that teachers may benefit from support to increase the likelihood that they can adapt content and activities for different children. Of course, these two will go together, to a certain extent (i.e. if everything is done in a whole group, it is less likely that any child can be given a specific activity that targets their individual needs). Figure 35: example ratings for ‘Grouping Types’ and ‘Individualised Instruction’ items 90% 78% 80% 70% 60% 47% 50% 40% 28% 30% 17% 17% 20% 8% 10% 3% 3% 0% 1 2 3 4 Grouping Types Individualised Instruction Engagement with materials: overall – and although far from statistically representative – classrooms visited faced a scarcity of materials. For five of six categories observed (Figure 36), most classrooms were without materials identified in the observation schedule and in some cases although materials were present, children did not use them on the day observed. This was, however, less of a problem for basic writing utensils (including pencils, crayons and chalk), which were present and used by children in most O-Classes. It should be noted that, as the tool has been prepared for cross-country data collection, with only slight adaptation, it may currently miss opportunities to identify resources that might be of more relevance to particular contexts within Ethiopia. Figure 36: engagement with materials across categories Writing utensils (e.g. pencils, pens, crayons, chalk) Educational toys or maths materials (e.g. bottle caps, beads) Art materials (e.g. markers, paints, clay, sand, sticks) Storybooks (e.g. with pictures and text, including locally made) Blocks (wooden or plastic blocks, interlocking pieces) Fantasy play (e.g. dolls, masks, pots and spoons) 0 3 6 9 12 15 18 21 24 27 30 33 36 Children use materials Materials present but children do not use No materials present So urc e: ME LQO fi el d t es t i n Et hi opia . No te : in a ddi ti on , a co un t o f b ooks was tak en i n each of th e 36 O - Cla ss es . In 23 ca se s th er e w er e no book s p res en t , in n in e cas es t he r e we re b et we en 1 - 5 bo oks an d in f our ca s es t he r e we re mo re than f i ve bo oks in th e r el e van t local lan gua ge . 54 7.3.5. Preliminary findings for child developmental outcomes (MODEL) This section focuses on preliminary findings from the child direct assessment within MODEL based on analysis conducted by the MELQO Global and ELP research teams, which aimed to evaluate the psychometric properties of the measure. The direct assessment was a one-on-one assessment between fieldworker and child. Data were obtained from 1,144 children across six regions and language groups, with a 1:1 relationship between region and language (Figure 31). Region, school and student samples are not representative at any level; they were selected for convenience and to capture a diverse range of contexts for the purposes of evaluating items in a field test. While there are similar numbers of boys and girls in the sample, as the overall numbers are small, analysis by gender is problematic. Information collected to identify whether children had a disability was used to exclude them from the analysis rather than identify their inclusion. These issues will require further consideration if the tools are to be adopted on a larger scale, given the centrality of equity to the government’s motivation for expanding O-Class, and the focus of this within GEQIP-E. Children who were reported as having difficulty seeing, hearing, speaking, understanding, or walking/moving during the assessment,32 or for whom anything else affected the administration of the assessment, were omitted from analyses. The final analytic sample was based on 1,032 children. Multiple approaches to item analysis have been used and in this section we offer a summary of selected performance, some suggestions for further investigation and for revisions prior to future implementation. Overall, the preliminary results based on the field test indicated that the direct assessments were, overall, reliable and valid (Koziol, 2018). Factor analysis suggested the presence of four domains – early numeracy, early language/literacy, executive function, and fine motor skills – that can be combined to measure a single overarching early development and learning construct (see Annex 4). The early development and learning scores demonstrated associations with several key variables (child age and literacy interest, teacher experience, classroom book reading and presence of books), providing evidence of convergent validity (Koziol, 2018). For most items, the mean score was near the middle of the possible range of responses, suggesting that floor and ceiling effects were not present and that the difficulty of the items matched the ability level of the sample. The one exception was exercise 2 (Expressive Language) in which the average percent correct score across items was 93 percent. This exercise required children to identify five body parts (eye, ear, tooth, fingers, knee). Among the early numeracy exercises, there were no differences in performance across gender, for children of equivalent age. Exercise 16 (Mental Transformation) was the most difficult on average, and exercises 11 (Verbal Counting), 14 (Number Comparison), and 12 (Producing a Set) were the easiest. As an example, Figure 37 compares results for exercises 15 (Simple Addition and Subtraction) and 16 (Mental Transformation). The former required respondents to answer four items (three addition, followed by one subtraction) and item performance suggests that difficulty was suitably targeted for the O-Class children with regular steps in difficulty between items. In contrast, the five mental transformation items, which required children to identify which shape was formed when two others are combined were more difficult. Item Response Theory analysis shows weak fit for this exercise and items were only weakly related to the underlying numeracy domain, suggesting lower reliability of this item in the measurement of O-Class children’s numeracy skill. 32A World Health Organisation 10-point disability screen had been adapted for the Early Learning Outcomes Measure instruments in South Africa and was further adapted for the Ethiopian context. See http://elom.org.za/. 55 Figure 37: Simple Addition and Subtraction against Mental Transformation exercises, which contribute to the early numeracy domain Among the early language/literacy exercises,33 exercises 4 (Letter Identification), 5 (Letter Sound Identification), and 6 (Letter Sound Discrimination) were the most difficult and exercise 2 (Expressive Language) was the easiest. Across these exercises, large differences in student performance are seen across languages, which suggests that there may have been difficulties in maintaining item equivalence during translation and adaptation or that items were administered in such a way that they functioned differently according to language (note also that for literacy reasons, fieldwork teams worked in only one language). One good example of this issue is in exercise 4 (Letter Identification) which required respondents to identify 16 letters from their mother tongue, in turn. For all languages there was a positive difference between item scores for students in O-Class and Grade 1, as would be expected. In Amharic, Afaan Oromo, Sidaamu Afoo and Tigrigna, items in exercise 4 seem to have been of an acceptable and similar difficulty level for O-class students, and relatively easy for Grade 1 students, with good or acceptable item-rest correlation values. By contrast, in Somali, the items were very easy for both O-Class and Grade 1 students; and in Berta, the items were much more difficult for O-Class students (Figure 38). Notwithstanding the possibility that the sample in Berta was substantially less well prepared, the next step will be to investigate what it was about the items, the script used or the process for administration that led to such large differences across language groups. An iterative procedure, as has been learned through implementation and revision in early grade reading assessments for Ethiopia, will be required to improve this and similar exercises. Figure 38: large differences in Letter Identification across languages, suggesting areas for revision Finally, among the Executive Functioning exercises, exercise 18 (Backward Digit Span) was the most difficult on average, and exercise 17 (Forward Digit Span) was the easiest. These assessments require students to 33Comparison according to gender is less straightforward for language/literacy items because the sample is cut by language (into six pieces) and then between girls and boys of equivalent age. After the breakdown, there are no significant differences across gender. However, the sample size becomes very small. With a larger standardisation sample, it would remain relevant to investigate performance according to gender in language/literacy items. 56 listen to a sequence of numbers and repeat that back to the fieldworker either forwards (i.e. digits in the same order) or backwards (i.e. digits in reverse order). Student performance on the assessments is broadly as expected for the age cohort and Figure 39 shows the contrast between the forward and backward assessments, with a logical increase in item difficulty for both assessments as the number of digits rises.34 Overall, the executive function items were the most difficult to administer, owing in part to their unfamiliarity among fieldworkers and children. During training more time was allocated to improve consistency of implementation across these exercises, as would be required in any future implementation. Figure 39: Forward and Backward Digit Span exercises, which contribute to the executive function domain 7.3.6. Implementation challenges, next steps and recommendations for use at scale Concerning MELE: ensuring reliability at scale This diagnostic report does not intend to provide a full review of tool reliability and validity, but the field test can inform steps towards levers for reform. This can benefit from a summary of the major implementation challenges with tools, which can support their future refinement and use. The cautious write-up of MELE earlier in this section reflects a concern with reliability, which stems from weak inter-rater agreement observed during training. Measurement by observation involves subjective judgment of classroom activities and interactions and it proved incredibly hard to generate consensus among the 24 fieldworkers that would observe O-Class sessions. As a result, measurement error is expected and a comparison between codes from two schools is not necessarily a valid reflection of the ‘true’ differences, neither are aggregates for items where agreement is found necessarily a ‘true’ reflection of reality in O-Class (i.e. consensus between observers, does not necessarily mean that this agrees with the ‘correct’ coding). Opportunities to work towards a mutual understanding of the tool’s scoring requirements, and to review progress against this goal, were built in during the training. In training-based assessments of inter-rater agreement it was common for around one-half of observers to agree with the ‘correct’ code. Where observers disagreed, for learning activity items, they typically rated below the ‘correct’ code. For other items, ratings varied above and below the ‘correct’ code. Particularly problematic is that the scale of disagreement was often large: across all potential person-code combinations around one in five was rated two or more codes above or below the ‘correct’ code (e.g. a rating of 1 or 2 when the ‘correct’ code was 4 and vice versa). Again, this was most prevalent in learning activity items. 34Across the 55 items within the numeracy and executive function exercises, only three show statistically significant differences in performance according to gender, when comparing children of equivalent age. They are all within the executive function domain: two in ‘backward digit span’, in favour of boys, and one in ‘head, knees, shoulders, toes’, in favour of girls. The field test sample is not sufficient to conduct a detailed investigation of gender-based differential item functioning but can flag items for extra attention if a measure of child development and learning is implemented with a larger sample. 57 During training, areas of disagreement were identified and prioritised. Substantial time and effort across the MELQO Global and ELP teams was committed to improving mutual understanding by comparing scores and discussing variations after each trial. Nonetheless, reflecting on these challenges, persistent inter-rater disagreement seemed to relate to factors that include: 1. Definitional disagreements: in a multi-lingual and entirely non-English context, finding consensus on terms was not easy. Good examples are the terms “play-based” or “open-ended”, which caused hours of discussion and which, we discovered, carried quite different meanings between participants even after repeated attempts to fix a definition for the observation. 2. Observer bias: such that different observers approach observation from particular theoretical or pedagogical standpoints. As a first step towards implementation at scale within government systems, fieldworkers were recruited via NEAEA and EDRI systems, being eligible for selection based on superior performance in similar assessments (such as the Early Grade Reading Assessment). Different recruitment options were considered and it was decided that if there is an expectation that an observation instrument of this type will be used at scale then this group was the most appropriate: it represents the typical profile of individuals who are likely to be called to implement at scale. 35 In the end, many in the fieldwork team were trained as teachers – some with experience of early learning environments, others were graduate social scientists who regularly conduct surveys in households and at schools, others again were employees of federal or regional educational assessment bureaus. 3. Instrument/item structure: the observation rubric is structured insofar as it prompts fieldworkers to observe behaviour in the same way and follow the same coding rules, to minimise the effects of observer subjectivity – yet it is more than a calculator that requires a carefully defined ‘snapshot’ to be taken at regular intervals. The observation tool attempts to strike a balance between structure (i.e. not too rigid and quantitative but paying attention to the qualitative nature of interactions) and flexibility (i.e. acknowledging that interactions and pedagogical approaches are complex and cannot be described in a single statement of a rubric). Unfortunately, there were persistent inconsistencies in the way rules were applied by different observers and these may relate to ambiguities in the coding system or where coding requires inferences about behaviour observed. Ambiguity may arise from: (a) statements that are indeterminate; (b) item inconsistency; and (c) ‘overlapping’ items. a. statements that are indeterminate: phrases like “allowing children some choice” were particularly hard to find common ground on, particularly given language variations which affect the interpretation of English phrases. b. item consistency: items take different ‘positions’ with respect to accounting for frequency, intensity, duration and coverage of an activity. For example, the individualised instruction item is concerned with coverage of children in the class (e.g. for how many students the teacher demonstrated individualised instruction), while a maths learning activity item is not concerned with coverage, only that teachers do certain things once or more (e.g. a teacher can achieve a rating of 3 from a single open-ended question, to one child). The different rules in these items then allows different interpretations and items are likely to suffer different error in measurement. c. overlapping items: in some circumstances a single classroom activity will contribute to more than one learning activity. For example, a literacy activity that requires children to write letters can also be a fine motor activity; and a music/movement activity can also promote free play or open choice. Overlapping situations of this type were reviewed at length with fieldworkers but throughout the training it was apparent that ambiguity remained and scoring rules were applied inconsistently. 35There was no option to exclude trainees who did not achieve a reasonable level of inter-rater agreement, given long-standing and ongoing commitments between the NEAEA, EDRI and their colleagues that deliver assessments. 58 Reliability challenges are normal in the first stages of tool development and field testing, particularly for structured classroom observation in a multi-lingual context. They suggest, however, that the scale-up of classroom observation, using the same tool and training approach, is likely to be ineffective, with substantial risks to the reliability of data collected. On the other hand, field test results – along with experiences and inputs during module development – provide strong indications of how tools might be further adapted for the Ethiopian context and which elements may be taken forward as inputs to quality assurance processes for O-Class. Of course, reliability implies little about the validity of the instrument: the researchers’ consensus that the instrument is relevant to educational outcomes (Croll, 1986). Having adapted the instrument iteratively (including during fieldwork training) it will be necessary to step back and review tool validity. This can be achieved in collaboration with country stakeholders, to determine whether the spread of items is appropriate and covers the domains relevant to the expectations for early learning quality in Ethiopia. It will also be necessary to determine how ‘scores’ can be generated across items to provide an aggregate picture of the early learning environment. Is the appropriate ‘score’, for example, the simple mean across all items, or are weights required to emphasise certain activities; how are teacher-child interactions, which have different ‘scales’ under the 1 to 4 rubric, fairly compared or combined to contribute to an aggregate score? If reliability and validity next steps can be achieved then there is exciting potential for an instrument of this type to contribute to quality assurance processes in the early learning system, for example, as an input to an inspection framework or for use by cluster supervisors. Concerning MODEL: contextual validity, linguistic equivalence and standards setting Overall, the implementation of the child direct assessment module was considerably more satisfactory, with promising signs of instrument reliability and validity. Nonetheless, there remain a few areas for further investigation and consideration prior to any future use of the instrument in Ethiopia. These can be summarised as follows (in addition, Annex 5 includes exercise-language specific recommendations): • In some exercises there are opportunities to reduce the number of items (including Number Identification, Letter Identification, Mental Transformation and possibly Forward Digit Span and Backward Digit Span) or to remove the exercise altogether (Expressive Language). A careful reduction in number of items per exercise can increase administration efficiency, which will be particularly helpful if this assessment is taken to scale and implemented within 30 minutes per child. • Any decision about the removal or adjustment of an exercise will relate to a more general question of validity: how does the direct assessment relate to the readiness goals that Ethiopia holds for children before starting school? After adaptation, MODEL is a hybrid developmental assessment and curriculum-linked assessment. In literacy and numeracy exercises there is a strong link to the O-Class syllabus. In other exercises there can be no direct link to a stated learning standard, even if these are predictive of later outcomes (e.g. for higher-order skills and items that require conceptual reasoning). A mapping of exercises against syllabus was included during the adaptation and tool development process and indicated that there are many domains which are included in the O-Class syllabus but which are not covered by the tool (e.g. much on socio-emotional development, health and personal hygiene, understanding of the local environment).36 A preference to use well- established exercises for measuring child development – which have been shown in other contexts to be predictive of later outcomes – has certainly helped the team to return strong results from the field test. Next steps, however, should consider once again the mapping of assessment exercises to early learning programme expectations. This can seek to strike a balance between a developmental 36 A teacher assessment of O-Class children – completed by the teacher, reporting on each child in their class – was prepared but not implemented in the field test. This was due to stakeholder concerns about the burden on O-Class teachers in completing such a tool (multiple items, each for every child in the class). Nonetheless the development of a suitable teacher assessment can improve information on socio-emotional development, which is harder to gather through direct assessment. 59 assessment and a curriculum-linked measure and assure contextual validity among government and national stakeholders.37 • Differences according to language are large in some literacy/language exercises (Letter Identification, Letter Sound Identification, Letter Sound Discrimination). This may be the result of language-specific differential item functioning, sample differences (field test samples were not representative at any level) or administration procedures (which will be team specific and teams work in one language only). Further discussion with fieldwork teams on administration and scoring procedures is required (e.g. for Letter Sound Identification and Name Writing) and item analysis to evaluate language-specific differential item function (for all exercises, not only for language/literacy, because translation/adaptation differences can influence item function for all exercises). • Although age gradients are as expected, age validation has only been possible based on age in years, rather than months. In a next implementation, gathering information on age in months (if possible) can allow age validation according to month of birth. • Finally, although the preliminary results provided stronger validity and reliability evidence for the direct assessment scores – along with indications of an overarching ‘early learning and development’ construct38 – it is not yet possible to relate overall performance on this assessment to Ethiopia’s expectations for school readiness. For example, is high performance on a certain item a good indication of readiness, or is it simply a feature of the item – and how should we consider overall performance in school readiness terms? A next step will be to conduct the assessment with a ‘standardisation sample’ which is representative of the target population. From here, standards statements can be established which can guide monitoring efforts to ensure that all children are supported to move towards minimum school readiness expectations. This can be achieved with early learning stakeholders, to reinforce accountability around shared school readiness expectations and will give meaning to assessment performance, nationally or within sub-groups. 37 The MELQO tools are in a process of continual development and socio-emotional items were added in late 2017. These can be evaluated for Ethiopia and perhaps incorporated into later versions of the direct assessment tool. In addition, socio-emotional indicators are gathered from teacher and parent/caregiver modules as discussed in core MELQO documentation at http://ecdmeasure.org/. 38 ‘Underneath’ this construct are early numeracy, early literacy, executive function and fine motor skills domains, which could each have their own performance standard. 60 7.4. Monitoring and assuring quality: a lever to reframe the understanding of early learning progress in terms of child development This section covers research findings on the theme of ‘Monitoring and Assuring Quality’. The incoherence identified between delegation and information within accountability relationships can be summarised as follows: • National and regional policy goals, along with community- and school-level understanding of early learning services, seek programs that contribute to child cognitive and socio-emotional development, to improve readiness before entering Grade 1. • Information collected on early learning performance is, for the most part, based on enrolment retention and dropout including via the national EMIS systems and on infrastructure and resource availability. • If child progress on preparedness for formal schooling or social-emotional competence is going to be an important part of the delegation in the relationship between woreda offices, school leaders and O-Class teachers, then there must be some information about these. The MELQO field test was a first attempt at establishing whether a tool of this type could fill this information gap. • A team of cluster supervisors appears to play a local quality assurance role in some areas. This is supported by what appears to be reasonably regular, although unstructured, supervision at the classroom level. However, although systems are in place, regions report resource, content and logistical challenges for cluster supervisors. 7.4.1. Possible levers for reform In response to the incoherence identified, steps can be taken to improve quality of early learning services, equitably. Policy options under this lever for reform might account for the following: • Experience from a field test of MELQO suggests that the child development tool in MODEL could be revised for use in large-scale national assessments. Early indications of this type can be valuable for system improvements and to gauge progress against goals. Field test results indicate the tool demonstrates validity and reliability and it could complement the NEAEA suite for early grade measurement. Possible next steps have been outlined in this section and will begin from a review of the tool against recently updated expectations for O-Class (as reflected in the 32-week teachers’ guide) and with curriculum and teacher development experts to gauge assessment validity for the context. • Experience from a field test of MELQO also suggests that a classroom observation tool to measure the quality of early learning environments suffers substantial reliability issues and is probably not ready to be taken to scale. However, elements of the observation tool can inform nascent quality assurance processes, including: o The teacher-child interaction elements of the tool (e.g. interaction items on grouping choices and individualised instruction) could be revised for use by cluster supervisors and school leaders in local efforts to observe and supervise O-Class teachers. Information gathered at the cluster/school level can also – within an appropriate school or cluster reporting framework – strengthen the client power relationship among communities and schools. This type of information exchange is a way of linking measurement to accountability and expectation. o Resource availability and resource use elements of the tool can guide the establishment of minimum expectations in standards for O-Class. These can input to the preparation of an inspection framework for early learning which is due to be fully developed in GEQIP-E and will be linked to school supervision and guidance for school improvement planning. 61 • Collectively, these steps can strengthen a nascent culture of supportive supervision that is based on quality of the early learning environment rather than on enrolment and checklists of resources and facility standards. 62 8. References 8.1. Documents Afework, E., Frew, A. & Abeya, G. (2017). Cluster supervision practices in primary school of Jimma Zone. International Journal of Educational Administration and Policy Studies, Vol. 9, No. 5. Croll, P. (1986). Systematic Classroom Observation. London: Taylor and Francis Group. DFID. (2010). Adequacy and Effectiveness of Public Education Spending in Ethiopia. Addis Ababa: Department for International Development. DFID. (2012). Business Case for Promotion of Basic Services In Ethiopia 2012 – 2017. Addis Ababa: Department for International Development. Di Gropello, E. (2004). Education Decentralisation and Accountability Relationships in Latin America. World Bank Policy Research Working Paper 3453. Washington, D.C.: World Bank. Government of Ethiopia. (1994). Education and Training Policy of the Federal Democratic Republic Government of Ethiopia. Addis Ababa: Government of Ethiopia. Hoot, J., Szente, J. & Mebratu, B. (2004). Early Education in Ethiopia: Process and Prospects. Early Childhood Education Journal, Vol. 32, No. 1. Iyer, P. & Rossiter, J. (2018). Coherent for Equitable Learning? Understanding the Ethiopian Education System. RISE Insights. Oxford: Blavatnik School of Government. Khan, Q., Faguet, J-P., Gaukler, C. & Mekash, W. (2014) Improving Basic Services for the Bottom Forty Percent: Lessons from Ethiopia. World Bank Study 90430. Washington, D.C.: World Bank. Koziol, N. (2018). MELQO MODEL Results and Recommendations: Ethiopia, Nebraska Academy for Methodology, Analytics, & Psychometrics, for the MELQO Global Team. Ministry of Education. (2000). Education Statistics Annual Abstract 1992 E.C. (1999-2000). Addis Ababa: Government of Ethiopia. Ministry of Education. (2002a). The Ethiopia Education Sector Development Program II (ESDP II) 2002/03- 2004/05. Addis Ababa: Government of Ethiopia. Ministry of Education. (2002b). The Education and Training Policy and Its Implementation. Addis Ababa: Government of Ethiopia. Ministry of Education. (2005a). Education Sector Development Programme III (ESDP III) 2015/06-2010/11. Addis Ababa: Government of Ethiopia. Ministry of Education. (2005b). Education Statistics Annual Abstract 1997 E.C. (2004-05). Addis Ababa: Government of Ethiopia. Ministry of Education. (2010a). National Policy Framework for Early Childhood Care and Education. Addis Ababa: Government of Ethiopia. Ministry of Education. (2010b). Education Sector Development Programme IV (ESDP IV) 2010/11-2014/15. Addis Ababa: Government of Ethiopia. Ministry of Education. (2010c). Education Statistics Annual Abstract 2002 E.C. (2009-10). Addis Ababa: Government of Ethiopia. 63 Ministry of Education. (2013). Education Statistics Annual Abstract 2005 E.C. (2012-13). Addis Ababa: Government of Ethiopia. Ministry of Education. (2015a). Education Sector Development Programme V (ESDP V) 2015/16-2019/20. Addis Ababa: Government of Ethiopia. Ministry of Education. (2015b). Joint Review Mission 2015: School Grant Disbursement and Utilization in Selected Regions Of Ethiopia. Addis Ababa: Government of Ethiopia. Ministry of Education. (2017). Education Statistics Annual Abstract 2008 E.C. (2015-16). Addis Ababa: Government of Ethiopia. Orkin, K., Abebe, W., Yadete and Woodhead, M. (2012). Delivering quality early learning in low-resource settings: Progress and challenges in Ethiopia. Working Paper No. 59, Studies in Early Childhood Transitions. The Hague, The Netherlands: Bernard van Leer Foundation. Oumer, J. (2009). The Challenges of Free Primary Education in Ethiopia: Education Costs and Financing. Paris: UNESCO-IIEP. Pfaffe, J., Asefa, E., Garad, A. & Gebremariam, T. (2013). Comprehensive Evaluation of the General Education Quality Improvement Programme (GEQIP): Final Exit Survey Report. Stockholm: Hifab International. Pritchett, L. (2015). Creating education systems coherent for learning outcomes: making the transition from schooling to learning. RISE Working Paper 15/005. Oxford: Blavatnik School of Government. Rossiter, J. (2016). Scaling Up Access to Quality Early Education in Ethiopia: Guidance from International Experience. Policy Paper 8, Oxford: Young Lives. RTI. (2010). Ethiopia Early Grade Reading Assessment Data Analytic Report: Language and Early Learning. Addis Ababa: USAID. Tadele, K. & Bekele, M. (2017) Practices of Primary Schools Cluster Supervision: The Case of Jimma Town Primary Schools in Ethiopia. European Scientific Journal, Vol. 13, No. 17. Tafere, Y., & Pankhurst, A. (2015). Can Children in Ethiopian Communities Combine Schooling with Work? Working Paper 141, Oxford: Young Lives. Teferra, T. & Hagos, B. (2016). Assessment of the Status of O-Class in Four Regional States of Ethiopia. Addis Ababa: Institute of Education, Health and Development. UNESCO. (2000). The Dakar Framework for Action, Education for All: Meeting our Collective Commitments. Paris: UNESCO. UNICEF. (2016). Journeys to Scale: Accompanying the Finalists of the Innovations in Education Initiative. New York: UNICEF. Woodhead, M., Rossiter, J., Dawes, A. & Pankhurst, A. (2017). Scaling-up Early Learning as a Sustainable Development Priority: A Case Study of Ethiopia. London: SAGE Publications. World Bank. (2004). World Development Report 2004: Making Services Work for Poor People. Washington, D.C: World Bank & Oxford University Press. World Bank. (2010). Ethiopia Public Finance Review 2010. Washington, D.C.: World Bank. World Bank. (2016) Ethiopia Public Expenditure Review, Report No: ACS14541. Washington, D.C: World Bank. World Bank. (2017a) Program Appraisal Document, General Education Quality Improvement Program for Equity (GEQIP-E). Washington, D.C.: World Bank. 64 World Bank. (2017b). Program-For-Results Enhancing Shared Prosperity Through Equitable Services (ESPES): Program Paper. Washington D.C.: World Bank. World Bank. (2018) Investing in Ethiopia’s Future: Education Sector Financing Review, Report No: AUS15517. Washington, D.C: World Bank. Young Lives. (2015). Examining ECCE in Ethiopia: an Exploratory Study with the Ministry of Education and Regional Education Bureaus. Unpublished. Young Lives. (2017). Community Stakeholder Perspectives on what Children Need Prior to Starting School: an Exploratory Study with the Ministry of Education. Unpublished. 8.2. Fieldwork sources Fieldwork activities included actor mapping, review of policy documents, cost and finance to early learning, qualitative interviews with stakeholders in the early learning system, completion of SABER tables and implementation of MELQO in Ethiopia. Throughout this document there are references to ‘Sources’ which are identified in this table. For reasons of anonymity, sources have been grouped. Figure 40: fieldwork sources referenced in this report Interviews with ECCE experts and leaders in the School Improvement Programme Directorate, Curriculum Directorate, Special Support Directorate, Planning and Resource Source 1 Mobilisation Directorate and with Adviser to Minister of Education (former Planning and Resource Mobilisation Director) Interviews with UNICEF early learning programme specialist and consultant involved in Source 2 development of national policy framework for ECCE Source 3 Compiled SABER tables Consultations with ECCE experts and focal persons from Regional Education Bureaus in Source 4 Oromia, Amhara, Afar, Benishangul-Gumuz, Gambella, Tigray, and Somali Source 5 MELQO questionnaires, classroom observation and child direct assessment 65 9. Annexes Annex 1 Figure 41: population projections for 0-14-year-olds in Ethiopia, 2017 to 2100 So urc e: Uni te d N ati on s , D epa rt me nt o f Ec on omic an d Socia l Af fai rs , P opu lat ion Di vis io n (2 01 7) . W or ld Po pula ti on P r osp ec ts: Th e 20 17 R e vi si on 66 Annex 2 Figure 42: three models of O-Class expansion Three models of O-Class expansion are shown. The first (A) reflects an equality principle – i.e. that services are provided equally, which corresponds with many government priorities. The second (B) reflects the ‘equity’ principle elaborated for early learning in ESDP V (see Box 2). The third (C) provides a stylised version of the status of early learning service provision and a probable ‘business-as-usual’ future. Each model has two panels. The left panel reflects an average GER of 50 percent, equivalent to the current enrolment rate. The right panel shows an average GER of 80 percent, the government’s 2020 target. In each panel, ‘areas’ – be it woredas, communities or school sites – are divided according to wealth from poorest to richest, with their specific GER shown. Within an ‘area’, children are sorted according to age: 4-year-olds, 5-year-olds and 6-year-olds, all of whom are eligible for early learning services. Each cell can be taken to represent one child or any other equally-sized group of children. A: equal, but inequitable provision of O-Class; favours nobody; reflects government’s ‘equality’ principle 4 yo 5 yo 6 yo 4 yo 5 yo 6 yo 0 % 50 % 100 % 41 % 100 % 100 % 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Poores t 1 1 1 1 1 1 1 1 1 1 Poores t 1 1 1 1 50% 0.5 1 1 1 1 1 1 1 1 80% 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Poorer 1 1 1 1 1 1 1 1 1 1 Poorer 1 1 1 1 50% 0.5 1 1 1 1 1 1 1 1 80% 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Ri cher 1 1 1 1 1 1 1 1 1 1 Ri cher 1 1 1 1 50% 0.5 1 1 1 1 1 1 1 1 80% 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Ri ches t 1 1 1 1 1 1 1 1 1 1 Ri ches t 1 1 1 1 50% 0.5 1 1 1 1 1 1 1 1 80% 1 1 1 1 1 1 1 1 1 1 1 1 B: unequal, but equitable provision of O-Class; favours poorest areas; reflects vision for early learning established in ESDP V 4 yo 5 yo 6 yo 4 yo 5 yo 6 yo 9 % 41 % 100 % 41 % 100 % 100 % 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Poores t 1 1 1 1 1 1 1 1 1 1 1 1 Poores t 1 1 1 1 1 1 1 1 79% 1 1 1 1 1 1 1 1 1 1 1 1 100% 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Poorer 1 1 1 1 1 1 1 1 1 1 1 1 Poorer 1 1 1 1 1 1 54% 1 1 1 1 1 1 1 1 1 1 88% 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Ri cher 1 1 1 1 1 1 1 1 Ri cher 1 1 1 1 33% 1 1 1 1 1 1 1 1 67% 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Ri ches t 1 1 1 1 1 1 1 1 Ri ches t 1 1 1 1 33% 1 1 1 1 1 1 1 1 67% 1 1 1 1 1 1 1 1 1 1 1 1 C: unequal and inequitable provision of O-Class; favours resource-rich areas; reflects reality of expansion 4 yo 5 yo 6 yo 4 yo 5 yo 6 yo 28 % 56 % 66 % 72 % 81 % 88 % Poores t 1 1 Poores t 8% 1 1 1 1 1 1 1 1 1 1 50% 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Poorer 1 1 1 1 1 1 1 1 Poorer 1 1 25% 1 1 1 1 1 1 1 1 1 1 1 1 71% 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Ri cher 1 1 1 1 1 1 1 1 1 1 1 1 Ri cher 1 1 1 1 1 1 1 1 1 1 79% 1 1 1 1 1 1 1 1 1 1 1 1 100% 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Ri ches t 1 1 1 1 1 1 1 1 1 1 1 1 Ri ches t 1 1 1 1 1 1 1 1 1 1 1 1 88% 1 1 1 1 1 1 1 1 1 1 1 1 100% 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 67 Annex 3 Private kindergarten is an urban phenomenon and in cities and towns there is no government expectation of O-Class expansion (e.g. in Addis Ababa an abundance of kindergarten means that only 51 of the city’s 804 schools offers O-Class and these are planned to be converted to kindergartens (ESAA 2015-16)). It would not be appropriate to compare O-Class access between urban and rural areas. In sections 5.3 to 5.6 (excluding Figure 16), a sub-set of regions and woredas is therefore excluded from the analysis, based on two criteria: • Region: excluded if the region has a majority of schools classified as ‘urban’. This excludes Addis Ababa, Dire Dawa and Harari. • Woreda: excluded if the woreda has more than 30 percent of schools classified as ‘urban’. This excludes 110 of 735 woredas. The threshold was selected based on the distribution of ‘urban -share’ among woredas (Figure 43). Figure 43: histogram of urban-share for all woredas (after excluding Addis Ababa, Harari and Dire Dawa) 250 200 Number of woredas 150 100 50 0 0 .2 .4 .6 .8 1 Share of schools in woreda that are 'urban' Source: Authors’ calculation based on Education Management Information System data, 2016-17 68 Annex 4 Figure 44: the path diagram of the final hierarchical multi-domain direct assessment model (Koziol, 2018) In this, the four domains (Early Numeracy Skills, Early Language/Literacy Skills, Executive Functioning, and Fine Motor Skills) represent lower-order factors of a single overarching construct (Early Learning and Development). So urc e: ME LQO MOD EL R es ul ts a nd R ecom me nda ti ons: Eth io pia , p r epa re d by Nata li e Ko zio l , Ph .D . No te: O vals ind icate la t en t c on st ruc ts a nd r ec tan gle s i ndica t e man if es t (ob se r ved ) i ndica to rs . Fo r s imp lici ty , m ul tip le it em s wi th in a t es t a re r e pr es en te d in th e sam e re cta ngle but it em s we r e t re at ed s epa ra te ly in th e mo de ll in g pr oc ess . A bi fact or pa ram e te ri za ti on a llo we d f or t he i te ms to loa d o n b oth a do mai n fac to r a nd a se par at e , u nco rr ela t ed sub task fac to r , as n ec ess ary to ac cou nt fo r d ep en de ncy am on g i te ms wi thi n t he in di vi dual t es ts . Ite ms fo r tes ts 4 (L et t er Ide nt if icat ion ) , 5 ( L et t er So und Id en ti fica ti on ) , an d 6 ( Le t te r Sou nd Di scr im ina ti on) we re no t di rec tly co mpa rab le ac ro ss re gion s , so co mp osi t e sco r es w er e us ed i n pla ce o f i te m -l e ve l data fo r the s e t es ts . 69 Annex 5 A full list of the exercises included in MELQO modules implemented in the Ethiopia field test. Figure 45: MODEL exercises and items Exercise Items Materials needed* Language/Literacy 45 1a-f – Interest in Literacy 6 Literacy Interest Sheet* 2a-e– Expressive Language 5 None 3a-b – Expressive vocabulary 2 None 4a-p – Letter Identification 16 Letter Identification Sheet* and blank paper 5a-e – Letter sound identification 5 None 6a-c – Letter sound discrimination 5 None 7a-e – Story listening comprehension 5 None 8a– Name writing 1 Blank paper, writing utensil, hard surface Fine motor skills 4 9a-d – Copying shapes 4 Copying Images Sheet* Mathematics/Numeracy 31 10a-e – Receptive Spatial Vocabulary 5 Receptive Spatial Vocabulary Sheet* 11a – Verbal Counting 1 None 12a-c – Producing a set 3 20 uniform counters 13a-j – Number identification 10 Number Identification Sheet* and blank paper 14a-c – Number Comparison 3 None 15a-d – Simple addition and subtraction 4 10 uniform counters 16a-e – Mental transformation 5 Mental Transformation Images Sheets* Executive Function 25 17a-e– Forward Digit Span 5 None 18a-d – Backward Digit Span 5 None 20a-e, h-q – Head, Shoulders, Knees, Toes Task 15 None Total 105 *Th es e ma te ria ls ar e i nclu ded i n th e e num e rat or b ook le t . Figure 46: MELE domains Domain Items Items Learning activities 7 C10-16 Classroom interactions & approaches to Learning 7 C18-26 Child engagement with materials 6 D7-12 Total 29 Exercise-language specific recommendations • Exercise 1 (Literacy Interest): determine whether there are any confounding reasons for why item B was differentially easier to endorse for children in Tigray and item D was easier to endorse for children in Oromia • Exercise 9 (Copying): determine whether there are any confounding reasons for why item A was differentially easier for children in Amhara and item B was easier for children in Benishangul Gumuz • Exercise 15 (Simple Addition & Subtraction): determine whether there are any confounding reasons for why item A was differentially easier for children Benishangul Gumuz 70 • Exercise 16 (Mental Transformation): determine whether there are any confounding reasons for why item B was differentially harder for children in Somali and item E was differentially harder for children in Benishangul Gumuz • Exercise 17 (Forward Digit Span): determine whether there are any confounding reasons why the items functioned differently across the majority of regions • Exercise 18 (Backward Digit Span): determine whether there are any confounding reasons why the items functioned differently across the majority of regions • Exercise 20 (Head, Toes, Knees, Shoulders): determine whether there are any explanations for why the majority of items functioned differently for children in Somali 71