Oman SABER Country Report STUDENT ASSESSMENT 2013 Key Policy Areas for Student Assessment Status 1. Classroom Assessment In Oman, formal, system-level documents provide guidelines for classroom assessment. A variety of resources, such as guides describing expected learning outcomes for different subject areas and grades, are available to teachers system-wide to support them in carrying out their assessment activities. Classroom assessment activities tend to be aligned with the curriculum framework. However, they are generally considered to be weak due, for example, to the fact that the assessments tend to provide little useful feedback to students and are mainly used as administrative tools rather than pedagogical inputs. 2. Examinations The General Education Diploma Examination has been administered to grade 12 students since 1980. The examination results are used to certify student completion of secondary school and determine student selection to higher education institutions. Examination results are officially recognized by certification and selection systems in Oman and abroad. Regular funding is provided by the government and covers all core examination activities, including research and development. However, there are limited systematic mechanisms in place to ensure the quality of the examination and no up-to-date courses or workshops on examinations are available to teachers. 3. National Large-Scale Assessment (NLSA) The Cognitive Development Test was administered for the first time in 2007-2008. Since then, it has been administered every year, assessing all students in grade 7 in Mathematics, Science, and Environmental Geographic concepts. A written plan for future NLSA activities, specifying who will be tested and in which subject areas, is in place. Regular funding for the NLSA is allocated by the government and covers core activities. While there are some opportunities to prepare individuals to work on the NLSA, there are limited opportunities for teachers to learn about the assessment. 4. International Large-Scale Assessment (ILSA) In the last 10 years, Oman has participated in TIMSS 2007 and 2011, and PIRLS 2011. It also has taken concrete steps to participate in TIMSS 2015 and PIRLS 2016. Regular funding for ILSAs is provided by the government and covers all core activities. Oman offers a range of opportunities to learn about ILSAs that are available to ILSA team members, teachers, and specialists. These opportunities include workshops, online courses, and funding for attending international training. Oman-specific ILSA results are widely disseminated. However, it is not clear that decisions based on the results have had a positive impact on student achievement levels. OMAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 Introduction student assessment systems. The framework is structured around two main dimensions of assessment Oman has focused on increasing student learning systems: the types/purposes of assessment activities and outcomes by improving the quality of education in the the quality of those activities. country. An effective student assessment system is an important component of efforts to improve education Assessment types and purposes quality and learning outcomes because it provides the necessary information to meet stakeholders’ decision- Assessment systems tend to be comprised of three main making needs. In order to gain a better understanding of types of assessment activities, each of which serves a the strengths and weaknesses of its existing assessment different purpose and addresses different information system, Oman decided to benchmark this system using needs. These three main types are: classroom standardized tools developed under The World Bank’s assessment, examinations, and large-scale, system level Systems Approach for Better Education Results (SABER) assessments. program. SABER is an evidence-based program to help countries systematically examine and strengthen the Classroom assessment provides real-time information to performance of different aspects of their education support ongoing teaching and learning in individual systems. classrooms. Classroom assessments use a variety of formats, including observation, questioning, and paper- What is SABER-Student Assessment? and-pencil tests, to evaluate student learning, generally on a daily basis. SABER-Student Assessment is a component of the SABER Examinations provide a basis for selecting or certifying program that focuses specifically on benchmarking students as they move from one level of the education student assessment policies and systems. The goal of system to the next (or into the workforce). All eligible SABER-Student Assessment is to promote stronger students are tested on an annual basis (or more often if assessment systems that contribute to improved the system allows for repeat testing). Examinations cover education quality and learning for all. the main subject areas in the curriculum and usually involve essays and multiple-choice questions. National governments and international agencies are increasingly recognizing the key role that assessment of Large-scale, system-level assessments provide feedback student learning plays in an effective education system. on the overall performance of the education system at The importance of assessment is linked to its role in: particular grades or age levels. These assessments (i) providing information on levels of student typically cover a few subjects on a regular basis (such as learning and achievement in the system; every 3 to 5 years), are often sample based, and use (ii) monitoring trends in education quality over multiple-choice and short-answer formats. They may be time; national or international in scope. (iii) supporting educators and students with real- time information to improve teaching and Appendix 1 summarizes the key features of these main learning; and types of assessment activities. (iv) holding stakehol ders accountable for results. SABER-Student Assessment methodology The SABER-Student Assessment framework is built on the available evidence base for what an effective assessment system looks like. The framework provides guidance on how countries can build more effective 2 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 Quality drivers of an assessment system Table 1: Framework for building an effective assessment system, with indicator areas The key considerations when evaluating a student assessment system are the individual and combined quality of assessment activities in terms of the adequacy of the information generated to support decision making. There are three main drivers of information quality in an assessment system: enabling context, system alignment, and assessment quality. Enabling context refers to the broader context in which the assessment activity takes place and the extent to which that context is conducive to, or supportive of, the assessment. It covers such issues as the legislative or The indicators are identified based on a combination of policy framework for assessment activities; institutional criteria, including: and organizational structures for designing, carrying out, or using results from the assessment; the availability of sufficient and stable sources of funding; and the x professional standards for assessment; presence of trained assessment staff. x empirical research on the characteristics of effective assessment systems, including analysis of the characteristics that differentiate between the System alignment refers to the extent to which the assessment systems of low- versus high-performing assessment is aligned with the rest of the education nations; and system. This includes the degree of congruence between x theory — that is, general consensus among experts assessment activities and system learning goals, that it contributes to effective assessment. standards, curriculum, and pre- and in-service teacher training. Levels of development Assessment quality refers to the psychometric quality of The World Bank has developed a set of standardized the instruments, processes, and procedures for the questionnaires and rubrics for collecting and assessment activity. It covers such issues as design and evaluating data on the three assessment types and implementation of assessment activities, analysis and related quality drivers. interpretation of student responses to those activities, and the appropriateness of how assessment results are The questionnaires are used to collect data on the reported and used. characteristics of the assessment system in a particular country. The information from the questionnaires is then Crossing the quality drivers with the different applied to the rubrics in order to judge the development assessment types/purposes provides the framework and level of the country’s assessment system in different broad indicator areas shown in Table 1. This framework areas. is a starting point for identifying indicators that can be used to review assessment systems and plan for their The basic structure of the rubrics for evaluating data improvement. collected using the standardized questionnaires is summarized in Appendix 2. The goal of the rubrics is to provide a country with some sense of the development level of its assessment activities compared to best or recommended practice in each area. For each indicator, the rubric displays four development levels—Latent, 3 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 Emerging, Established, and Advanced. These levels are Oman has made major progress in primary and artificially constructed categories chosen to represent secondary enrolments, expanding considerably from key stages on the underlying continuum for each 1970, when there were only 3 schools in the country and indicator. Each level is accompanied by a description of a total of 909 students, to 600,000 students enrolled in what performance on the indicator looks like at that 1,300 schools in the country in 2008. In 2011, net level. enrollment rates were 97 percent and 94 percent at the primary and secondary levels respectively. x Latent is the lowest level of performance; it represents absence of, or deviation from, the desired The Government of Oman has focused on improving the attribute. quality, efficiency, and relevance of education. In x Emerging is the next level; it represents partial addition, due to concerns that the Omani economy is presence of the attribute. largely reliant on oil and gas revenues, the government x Established represents the acceptable minimum has prioritized education reform in order to facilitate the standard. development of other sectors. Oman’s ’Vision 2020’, the x Advanced represents the ideal or current best country’s long-term strategy for growth, called for practice. several reforms in the education sector including strengthening the teaching and learning of English in basic education, greater attention to science subjects, A summary of the development levels for each and greater alignment of secondary education with labor assessment type is presented in Appendix 3. market needs. Other reforms already introduced in the country include the implementation of continuous In reality, assessment systems are likely to be at assessment, a reduction in class size, and a school-based different levels of development in different areas. For management process. example, a system may be Established in the area of examinations, but Emerging in the area of large-scale, Recent figures on spending in the education sector system-level assessment, and vice versa. While intuition reflect the country’s commitment to its growth. In 2005 suggests that it is probably better to be further along in and 2007, total public expenditure on education as many areas as possible, the evidence is unclear as represented 4.6 percent of GDP, which further increased to whether it is necessary to be functioning at to 5.3 percent of GDP in 2009. Advanced levels in all areas. Therefore, one might view the Established level as a desirable minimum outcome to Detailed information was collected on Oman’s student achieve in all areas, but only aspire beyond that in those assessment system using the SABER-Student Assessment areas that most contribute to the national vision or questionnaires and rubrics. It is important to remember priorities for education. In line with these considerations, that these tools primarily focus on benchmarking a the ratings generated by the rubrics are not meant to be country’s policies and arrangements for assessment additive across assessment types (that is, they are not activities at the system or macro level. Additional data meant to be added to create an overall rating for an would need to be collected to determine actual, on-the- assessment system; they are only meant to produce an ground practices in Oman, particularly by teachers and overall rating for each assessment type). The students in schools. The following sections discuss the methodology for assigning development levels is findings by each assessment type, accompanied by summarized in Appendix 4. suggested policy options. The suggested policy options were determined in collaboration with key local Education in Oman stakeholders based on Oman’s immediate interests and needs. Detailed, completed rubrics for each assessment type are provided in Appendix 5. Oman is a high-income country in the Middle East & North Africa region. GDP per capita is $23,133 and GDP growth was 5.6 percent in 2010 and 0.3 percent in 2011. 4 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 Classroom Assessment little useful feedback to students and are mainly used as administrative tools rather than as pedagogical inputs. Level of development Systematic mechanisms are in place to monitor the quality of classroom assessment practices. For example, classroom assessment is a required component of a teacher's performance evaluation, and system-wide In Oman, formal, system-level documents, such as The reviews of the quality of education include a focus on General Document for Learning Assessment of Students classroom assessment. Testing and auditing teams, in Grades 1 to 12, authorized in 2012 by the General referred to as moderation teams, from regional Directorate of Educational Assessment in the Ministry of educational directorates visit schools to verify the Education, provide guidelines for classroom assessment. accuracy of classroom assessments conducted in grade Various resources are also available to teachers 12 classrooms. In addition, there is an external throughout the system to help them engage in classroom moderation system that reviews the difficulty of assessment activities. These include documents that classroom assessment activities and the appropriateness outline what students are expected to learn (and how of scoring criteria. well) in different subject areas at different grade levels. In addition, the Scope and Sequence Matrix document Classroom assessment information is required to be specifies the objectives that the students must achieve disseminated to all key stakeholders, including school at a particular grade level in order to pass to the next district and Ministry of Education officials, parents, and grade. There are also textbooks or workbooks that students. provide support for classroom assessment, scoring criteria for grading students' work, item banks with Suggested policy options examples of selection/multiple-choice and supply/open- ended questions, and online assessment resources. 1. Continue efforts to improve the Scope and Sequence Matrix, which specifies the objectives that students must Some mechanisms are in place to ensure that teachers achieve at each grade level in order to pass to the next develop skills and expertise in classroom assessment. For grade, to clarify the level of performance that students example, school inspection and teacher supervision must demonstrate for various learning objectives. include a component focused on classroom assessment. While there are no pre-service teacher training 2. In order to improve the quality of classroom opportunities, teachers and supervisors are trained each assessment practices, review existing in-service teacher year on implementing classroom assessment within training opportunities and adjust them so that they planned training programs as part of professional better support teachers in building knowledge and development programs, and on-going follow-up is capacity on the use of classroom assessment. Introduce conducted by regional supervisors from each regional policies for instituting yearly in-service teacher training educational directorate. opportunities on classroom assessment and make them available to all teachers on a regular basis. Classroom assessment activities take a variety of forms. For grades 1 to 4, activities include oral tests, quizzes, 3. Improve the availability of systematic mechanisms to and observation. For grades 5 to 12, activities include monitor the quality of classroom assessment practices. oral tests, quizzes, observation, periodic reports, and Make funding available for research on the quality of quarterly tests. Although it is uncommon for classroom classroom assessment and how to improve it. assessment activities to rely on multiple-choice or selection-type questions, they do tend to be about recalling information. In addition, it is commonly believed that classroom assessment activities provide 5 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 Examinations educational measurement and evaluation. In addition, there is funding for attending international programs and workshops on educational measurement and evaluation, Level of development as well as internships in the examination office. Compulsory courses and workshops on the examination are available to teachers, but are not regularly updated. The General Education Diploma Examination (previously Teachers are involved in tasks related to the General referred to as the General Secondary Certification Education Diploma Examination, including administering Examination) was first administered in 1980 with the and scoring the examination, acting as judges, and main purposes of student certification for school cycle supervising examination procedures. completion and student selection to higher-education institutions. Students in grade 12 are assessed in core and elective subjects from among the following: Islamic Internal and external reviews are in place to Education, Arabic Language, English Language, Physics, systematically ensure the quality of the examination. For Chemistry, Biology, Pure Mathematics, Applied example, a review of the examination by a committee Mathematics, Social Studies (History), Geography, external to the Ministry of Education is currently Science and the Environment, Musical Skills, Physical underway. The examination questions are also reviewed Education, and Fine Arts. A formal, system-level by specialized technical committees and then presented document, the Regulations of the Committees for to the assessment committee (composed of experienced General Diplomas Examinations, authorized the specialists) prior to the administration of the examination in 2010. examination. Regular funding for the examination is allocated by the There are some options for students who do not perform government, and covers all core examination activities as well on the examination, including retaking the well as research and development. examination (if they fail in three or fewer subjects), repeating the grade, or attending remedial or preparatory courses (offered in the private sector) The Department of Tests and Examinations before they retake the examination. Management, which is part of the Directorate General for Educational Assessment within the Ministry of Education, is responsible for running the General A permanent oversight committee is the only mechanism Education Diploma Examination. Although the in place to monitor the consequences of the Department of Tests and Examinations Management has examination. all of the required facilities to carry out the examination, it does not have enough full-time staff to meet the needs Suggested policy options of the examination. While specialized senior supervisors and teachers from directorates, such as the General 1. Introduce more mechanisms to ensure the positive Directorate for Curriculum Development and the impact of the examination on student learning, Directorate General for Human Resources Development, particularly in terms of the systematic use of are utilized for exam preparation, the examination is examination results by teachers, school level leaders, managed and administered by specialized committees. and regional subject supervisors. For example, conduct regular focus groups or surveys of key stakeholders to Oman offers a wide range of opportunities to prepare discuss exam results and how they can be used to further individuals for work on the examination. Options in place student learning. include university graduate programs at Sultan Qaboos University and non graduate training courses on 6 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 2. Provide teachers with regular opportunities, such as up-to-date courses or workshops, to learn about the examination. 3. Increase the number of qualified staff dedicated to carrying out the General Education Diploma Examination in the Department of Tests and Examinations Management. 7 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 National Large-Scale Assessment (NLSA) In addition, regular internal and external reviews by qualified experts are in place to ensure the alignment between the assessment instrument and what it is Level of development supposed to measure. Other mechanisms to ensure the quality of the NLSA include the training of all test administrators according to a protocol, and the preparation of a standardized manual for administrators. The Cognitive Development Tests have been administered on an annual basis since the 2007-2008 NLSA results are disseminated within 12 months of the school year as a way to monitor education quality at the assessment being administered. Reports are made system level, support schools and teachers, and inform available for all stakeholder groups and the results are policy design, evaluation, and decision making. The published on the Ministry’s Education Portal. Workshops assessment is administered to all students in grade 7 and and presentations are also used to share results with key covers Mathematics, Sciences, and Environmental stakeholders. However, the results are not featured in Geographic concepts. A formal, system-level document, the media—newspapers, magazines, radio, or the Cognitive Development Manual, authorized the television—and there is no media briefing. Cognitive Development Tests in 2009. In addition, there is a written plan for the NLSA for the coming years, specifying who will be tested and in which subject areas. There are some mechanisms in place to help monitor the impact of the NLSA in terms of its effects on policy dialog, design, and decision making. These include a permanent Regular funding for the Cognitive Development Tests is oversight committee, themed conferences that provide allocated by the government. Funding covers all core a forum to discuss research and other data on the NLSA activities, including assessment design and assessment, and expert review groups. However, administration, data analysis and reporting, and staff funding for independent research on the impact of the training. Funding also covers annual prizes for students NLSA, or regular focus groups and surveys of key who exhibit outstanding performance on the stakeholders are not in place. assessment. However, funding does not cover long- or medium-term planning of program milestones or research and development activities. Suggested policy options 1. Review the intended purposes and uses of the NLSA The Directorate General of Educational Evaluation is a and evaluate options to ensure that those purposes and permanent unit responsible for running the assessment uses are achieved, including the optimal frequency of and is adequately staffed to carry out the NLSA administering the NLSA, the subject matter to be effectively. At the same time, there have been some assessed, and the desired technical characteristics and issues related to poor training of test administrators, and quality of the NLSA. unclear instructions and guidelines for administering the assessment. 2. Utilize the funding that is made available by the Research Council (a public body that provides grants and Oman offers some opportunities to prepare individuals other support for research and innovation activities for work on the NLSA. These include university graduate across sectors) for research and development on the programs on educational measurement and evaluation NLSA. at the Sultan Qaboos University, internships or short- term employment in the large-scale assessment office, 3. Introduce a variety of mechanisms to ensure the and funding for attending international programs or technical quality of the NLSA. For example, introduce workshops on educational measurement and evaluation. double scoring of data or external certifications or audits. 8 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 national and international reports were distributed to International Large-Scale Assessment (ILSA) key stakeholders, and the country’s results were communicated through a press release. Products Level of development providing information to schools and educators were also made available. In addition, after the release of the country’s results for TIMSS and PIRLS 2011, a special seminar was held to discuss the results, which included participation by stakeholders from the Sultanate. Oman participated in Trends in International Mathematics and Science Studies (TIMSS) in 2007 and 2011, and Progress in International Reading Literacy ILSA results have been used to inform decision making in Study (PIRLS) in 2011. Oman has taken concrete steps to Oman. For example, results have been used to track the participate in PIRLS in 2016 and TIMSS in 2015. A formal impact of reforms on student achievement levels and to policy document, Ministerial Decision Number 140/2009, inform curriculum improvement, teacher training authorized by the Ministry of Education in 2009, programs, and other assessment activities in the system. addresses participation in ILSAs and the preparation and However, it is not clear that decisions based on ILSA publication of relevant national reports. results have had a positive impact on student achievement levels in the system, as there was no improvement in Oman’s scores between TIMSS 2007 and Regular funding for international assessment 2011. participation is allocated at the discretion of the government. Although funding covers all core activities, it does not cover research and development. Suggested policy options A national coordinator and team (called the International 1. Ensure that Oman’s ILSA results are used to inform Studies Program) carry out the ILSA activities. The ILSA decision making around school- and classroom-level office is adequately staffed to carry out the work. At the instructional support, assessment practices, and same time, although the team has previous experience curricular review. working on international assessments, and has attended some international workshops and meetings, it does not 2. Strengthen technical expertise on ILSAs by have all of the necessary training and experience to carry collaborating with universities to introduce university out required assessment activities effectively. courses on educational measurement and evaluation. Oman offers a wide range of opportunities to learn about ILSAs. For example, workshops and meetings are held on using international assessment databases, funding is made available for attending international workshops or training on international assessments, and online courses are available. However, there are no university courses on the topic of international assessments. Oman has met all technical standards required to have its data presented in the main displays of the international report. In addition, country-specific results are widely disseminated in the country. For example, after previous ILSA exercises, a national report as well as brochures and PowerPoint presentations with Oman's results were made available online, copies of the 9 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 Appendix 1: Assessment Types and Their Key Differences Classroom Large-scale assessment Examinations Surveys National International Exit Entrance Purpose To provide To provide To provide To certify To select immediate feedback on feedback on the students as they students for feedback to overall health of comparative move from one further inform the system at performance of level of the educational classroom particular the education education opportunities instruction grade/age system at system to the level(s), and to particular next (or into the monitor trends grade/age workforce) in learning level(s) Frequency Daily For individual For individual Annually and Annually and subjects offered subjects offered more often more often on a regular on a regular where the where the basis (such as basis (such as system allows system allows every 3-5 years) every 3-5 years) for repeats for repeats Who is All students Sample or A sample of All eligible All eligible tested? census of students at a students students students at a particular grade particular grade or age level(s) or age level(s) Format Varies from Usually multiple Usually multiple Usually essay Usually essay observation to choice and short choice and short and multiple and multiple questioning to answer answer choice choice paper-and-pencil tests to student performances Coverage of All subject areas Generally Generally Covers main Covers main curriculum confined to a confined to one subject areas subject areas few subjects or two subjects Additional Yes, as part of Frequently Yes Seldom Seldom information the teaching collected from process students? Scoring Usually informal Varies from Usually involves Varies from Varies from and simple simple to more statistically simple to more simple to more statistically sophisticated statistically statistically sophisticated techniques sophisticated sophisticated techniques techniques techniques 10 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 Appendix 2: Basic Structure of Rubrics for Evaluating Data Collected on a Student Assessment System Development Level LATENT ESTABLISHED (Absence of, or EMERGING (Acceptable deviation from, (On way to meeting minimum ADVANCED Dimension attribute) minimum standard) standard) (Best practice) Justification EC—ENABLING CONTEXT EC1—Policies EC2—Leadership, public engagement EC3—Funding EC4—Institutional arrangements EC5—Human resources SA—SYSTEM ALIGNMENT SA1—Learning/quality goals SA2—Curriculum SA3—Pre-, in-service teacher training AQ—ASSESSMENT QUALITY AQ1—Ensuring quality (design, administration, analysis) AQ2—Ensuring effective uses 11 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 Appendix 3: Summary of the Development Levels for Each Assessment Type Assessment Type LATENT EMERGING ESTABLISHED ADVANCED Absence of, or deviation On way to meeting Acceptable minimum Best practice from, the attribute minimum standard standard There is no system-wide There is weak system- There is sufficient There is strong system- institutional capacity to wide institutional system-wide wide institutional support and ensure the capacity to support and institutional capacity to capacity to support and quality of classroom ensure the quality of support and ensure the ensure the quality of assessment practices. classroom assessment quality of classroom classroom assessment CLASSROOM ASSESSMENT practices. assessment practices. practices. There is no standardized There is a partially There is a stable There is a stable examination in place for stable standardized standardized standardized key decisions. examination in place, examination in place. examination in place and a need to develop There is institutional and institutional institutional capacity to capacity and some capacity and strong run the examination. limited mechanisms to mechanisms to monitor EXAMINATIONS The examination monitor it. The it. The examination is of typically is of poor examination is of high quality and is quality and is perceived acceptable quality and is perceived as fair and as unfair or corrupt. perceived as fair for free from corruption. most students and free from corruption. There is no NLSA in There is an unstable There is a stable NLSA in There is a stable NLSA in place. NLSA in place and a place. There is place and institutional need to develop institutional capacity capacity and strong institutional capacity to and some limited mechanisms to monitor NATIONAL (OR SYSTEM- run the NLSA. mechanisms to monitor it. The NLSA is of high LEVEL) LARGE-SCALE Assessment quality and it. The NLSA is of quality and its ASSESSMENT impact are weak. moderate quality and its information is information is effectively used to disseminated, but not improve education. always used in effective ways. There is neither a Participation in an ILSA There is more or less There is stable history of participation has been initiated, but stable participation in an participation in an ILSA in an ILSA nor plans to there still is need to ILSA. There is and institutional participate in one. develop institutional institutional capacity to capacity to run the ILSA. capacity to carry out the carry out the ILSA. The The information from INTERNATIONAL LARGE- ILSA. information from the the ILSA is effectively SCALE ASSESSMENT ILSA is disseminated, but used to improve not always used in education. effective ways. 12 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 Appendix 4: Methodology for Assigning 6. Scores for certain key dimensions under ‘Enabling Development Levels Context’ (in the case of EXAM, NLSA, and ILSA) and under ‘System Alignment’ (in the case of CLASS) were set as ceiling scores, i.e., the overall mean score for the 1. The country team or consultant collects information particular assessment type cannot be greater than the about the assessment system in the country. score for these key dimensions. These key variables include formal policy, regular funding, having a 2. Based on the collected information, a level of permanent assessment unit, and the quality of development and score is assigned to each dimension in assessment practices. the rubrics: x Latent = 1 score point x Emerging = 2 score points x Established = 3 score points x Advanced = 4 score points 3. The score for each quality driver is computed by aggregating the scores for each of its constituent dimensions. For example: The quality driver, ‘Enabling Context,’ in the case of ILSA, has 3 dimensions on which a hypothetical country receives the following scores: Dimension A = 2 points; Dimension B = 2 points; Dimension C = 3 points. The hypothetical country’s overall score for this quality driver would be: (2+2+3)/3 = 2.33 4. A preliminary level of development is assigned to each quality driver. 5. The preliminary development level is validated using expert judgment in cooperation with the country team and The World Bank Task Team Leader. For scores that allow a margin of discretion (i.e., to choose between two levels of development), a final decision has to be made based on expert judgment. For example, the aforementioned hypothetical country has an ‘Enabling Context’ score of 2.33, corresponding to a preliminary level of development of ‘Emerging or Established.’ Based on qualitative information not captured in the rubric, along with expert judgment, the country team chooses ‘Emerging’ as the most appropriate level. 13 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 Appendix 5: SABER-Student Assessment Rubrics for Oman This appendix provides the completed SABER-Student Assessment rubrics for each type of assessment activity in Oman. In each row of the rubric, the relevant selection is indicated by a thick border and an asterisk. The selection may include a superscript number that refers to the justification or explanation for the selection. The explanation or justification text can be located in the “Development-level rating justifications” section at the end of each rubric. If a row includes a superscript, but not a thick border and an asterisk, this means that insufficient information was available to determine the relevant selection in the row. OMAN Classroom Assessment 14 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 ENABLING CONTEXT AND SYSTEM ALIGNMENT Overall policy and resource framework within which classroom assessment activity takes place in a country or system, and the degree to which classroom assessment activity is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT AND SYSTEM ALIGNMENT 1: Setting clear guidelines for classroom assessment There is no system-level document that There is an informal system-level There is a formal system-level document This option does not apply to this provides guidelines for classroom document that provides guidelines for that provides guidelines for classroom dimension. assessment. classroom assessment. assessment.1 * This option does not apply to this This option does not apply to this The availability of the document is The document is widely available. 2 dimension. dimension. restricted. * ENABLING CONTEXT AND SYSTEM ALIGNMENT 2: Aligning classroom assessment with system learning goals There are no system-wide resources for There are scarce system-wide resources There are some system-wide resources There are a variety of system-wide teachers for classroom assessment. for teachers for classroom assessment. for teachers for classroom assessment. resources available for teachers for classroom assessment. 3 There is no official curriculum or There is an official curriculum or There is an official curriculum or There is an official curriculum or * standards document. standards document, but it is not clear standards document that specifies what standards document that specifies what what students are expected to learn or to students are expected to learn, but the students are expected to learn and to what level of performance. level of performance required is not clear. what level of performance. 4 * ENABLING CONTEXT AND SYSTEM ALIGNMENT 3: Having effective human resources to carry out classroom assessment activities There are no system-level mechanisms to This option does not apply to this There are some system-level mechanisms There are a variety of system-level ensure that teachers develop skills and dimension. to ensure that teachers develop skills and mechanisms to ensure that teachers expertise in classroom assessment. expertise in classroom assessment.5 develop skills and expertise in classroom * assessment. 15 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 ASSESSMENT QUALITY Quality of classroom assessment design, administration, analysis, and use. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of classroom assessment Classroom assessment practices suffer Classroom assessment practices are Classroom assessment practices are Classroom assessment practices are from widespread weaknesses or there is known to be weak. 6 known to be of moderate quality. known to be generally of high quality. no information available on classroom assessment practices. * There are no mechanisms to monitor the There are ad hoc mechanisms to monitor There are limited systematic mechanisms There are varied and systematic quality of classroom assessment the quality of classroom assessment to monitor the quality of classroom mechanisms in place to monitor the practices. practices. assessment practices. quality of classroom assessment practices. 7 * ASSESSMENT QUALITY 2: Ensuring effective uses of classroom assessment Classroom assessment information is not This option does not apply to this Classroom assessment information is Classroom assessment information is required to be disseminated to key dimension. required to be disseminated to some key required to be disseminated to all key stakeholders. stakeholders. stakeholders. 8 * There are no required uses of classroom There are limited required uses of There are adequate required uses of There are adequate required uses of assessment to support student learning. classroom assessment to support student classroom assessment to support student classroom assessment to support student learning. learning, excluding its use as an input for learning, including its use as an input for external examination results. 9 * external examination results. 16 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 Classroom Assessment: Development-level rating justifications 1. In Oman, formal system-level documents such as The General Document for Learning Assessment of Students in Grades 1 to 12, authorized by the General Directorate of Educational Assessment in the Ministry of Education in 2012, provide guidelines for classroom assessment. 2. The General Document for Learning Assessment of Students in Grades 1 to 12 is widely available, including at the Sultanate of Oman Educational Portal, in schools, teacher training colleges, and in the educational assessment departments in the educational governorates, Directorate General of Curricula Development, Directorate General of Human Resources Development, and Directorate General of Private Schools. 3. There are a variety of system-wide resources available for teachers to engage in classroom assessment activities. Specifically, there are documents that outline what students are expected to learn in different subject areas at different grade/age levels and the level(s) of performance that students are expected to reach in different subject areas at different grade/age levels. In addition, the Scope and Sequence Matrix document specifies the objectives for each subject that the students must achieve in each grade in order to pass to the next. There are also textbooks or workbooks that provide support for classroom assessment, scoring criteria or rubrics for students' work, item banks or pools with examples of selection/multiple-choice or supply/open-ended questions, and online assessment resources available. However, there is no computer-based testing with instant reports on students' performance. 4. An official curriculum document, Scope and Sequence Matrix document for each education subject, specifies what students are expected to learn, but the level of performance required is not clear. 5. There are some system-level mechanisms in place to ensure that teachers develop skills and expertise in classroom assessment. For example, school inspection or teacher supervision includes a component focused on classroom assessment. Although there are no pre-service teacher training opportunities, as part of professional development programs, teachers and supervisors are trained each year on implementing classroom assessment within planned training programs, and ongoing follow-up is conducted by regional supervisors from each regional educational directorate. In addition, online resources on classroom assessment are made available, as well as opportunities to participate in conferences and workshops. Opportunities to participate in item development for, or scoring of, large-scale assessments or exams are also available. 6. While classroom assessment practices are generally considered to be weak, some classroom assessment activities are carried out in several forms. For grades 1 to 4, activities include oral tests, quizzes, and observation. For grades 5 to 12, activities include oral tests, quizzes, observation, and periodic reports as well as quarterly tests. Although it is uncommon for classroom assessment activities to rely mainly on multiple-choice, selection-type questions, it is very common that they are mainly about recalling information. It is common for teachers to not use explicit or a priori criteria for scoring or grading students' work, and the uneven application of standards for grading students' work and errors in the scoring or grading of students' work are serious problems. It is rare for parents to be poorly informed about students' grades or for classroom assessment activities to be misaligned with pedagogical or curricular work; yet classroom assessment activities commonly provide little useful feedback to students and are mainly used as an administrative or control tool rather than as a pedagogical resource. 17 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 7. There are varied and systematic mechanisms in place to monitor the quality of classroom assessment practices. Classroom assessment is a required component of a teacher's performance evaluation and school inspection or teacher supervision. In addition, national or other system-wide reviews of the quality of education include a focus on classroom assessment. There are also testing and auditing teams which visit schools to verify the accuracy of classroom assessments. However, there is no external moderation system that reviews the difficulty of classroom assessment activities or appropriateness of scoring criteria. There is also no government funding available for research on the quality of assessment activities or how to improve classroom assessment. 8. Classroom assessment information is required to be disseminated to all key stakeholders, including to school district and Ministry of Education officials, parents, and students. 9. There are adequate required uses of classroom assessment to support student learning, excluding its use as an input for external examination results. For example, classroom assessment activities are used in diagnosing student learning, providing feedback to students on their learning, informing parents about their child's learning, planning the next steps in instruction, and grading students for internal classroom uses. OMAN Examinations 18 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which assessment activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the assessment activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies No standardized examination has taken The standardized examination has been The examination is a stable program that This option does not apply to this place. operating on an irregular basis. has been operating regularly. 1 dimension There is no policy document that There is an informal or draft policy * There is a formal policy document that This option does not apply to this authorizes the examination. document that authorizes the authorizes the examination.2 dimension. examination. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this dimension. the public. public. 3 dimension. * This option does not apply to this This option does not apply to this The policy document addresses some key The policy document addresses all key dimension. dimension. aspects of the examination.4 aspects of the examination. * ENABLING CONTEXT 2: Having strong leadership All stakeholder groups strongly oppose Most stakeholder groups oppose the Most stakeholders groups support the All stakeholder groups support the the examination or are indifferent to it. examination. examination. 5 examination. * There are no attempts to improve the This option does not apply to this There are independent attempts to There are coordinated attempts to examination by stakeholder groups. dimension. improve the examination by stakeholder improve the examination by stakeholder groups.6 groups. * Efforts to improve the examination are This option does not apply to this Efforts to improve the examination are This option does not apply to this not welcomed by the leadership in charge dimension. generally welcomed by the leadership in dimension. of the examination charge of the examination.7 * 19 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 (CONTINUED) LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding There is no funding allocated for the There is irregular funding allocated for There is regular funding allocated for the This option does not apply to this examination. the examination. examination.8 dimension. * This option does not apply to this Funding covers some core examination Funding covers all core examination This option does not apply to this dimension. activities: design, administration, data activities: design, administration, data dimension. processing or reporting. processing and reporting.9 * This option does not apply to this Funding does not cover research and This option does not apply to this Funding covers research and dimension. development. dimension. 10 development. * ENABLING CONTEXT 4: Having strong organizational structures The examination office does not exist or is The examination office is newly The examination office is a stable This option does not apply to this newly established. established. organization. 11 dimension. * The examination office is not accountable This option does not apply to this The examination office is accountable to This option does not apply to this to an external board or agency. 12 dimension. an external board or agency. dimension. * Examination results are not recognized by Examination results are recognized by Examination results are recognized by Examination results are recognized by any certification or selection system. certification or selection system in the one certification or selection system in two or more certification or selection country. another country. systems in another country.13 * The examination office does not have the The examination office has some of the The examination office has all of the The examination office has state-of-the- required facilities to carry out the required facilities to carry out the required facilities to carry out the art facilities to carry out the examination. examination. examination. examination. 14 * (CONTINUED) 20 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources There is no staff to carry out the The examination office is inadequately The examination office is adequately The examination office is adequately examination. staffed to effectively carry out the staffed to carry out the examination staffed to carry out the assessment examination, issues are pervasive. effectively, with minimal issues.15 effectively, with no issues. The country does not offer opportunities This option does not apply to this * The country offers some opportunities The country offers a wide range of that prepare for work on the dimension. that prepare for work on the opportunities that prepare for work on examination. examination. the examination. 16 * 21 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 SYSTEM ALIGNMENT Degree to which the assessment is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning examinations with learning goals and opportunities to learn It is not clear what the examination This option does not apply to this There is a clear understanding of what the This option does not apply to this measures. dimension. examination measures.17 dimension. What the examination measures is This option does not apply to this What is measured by the examination is * This option does not apply to this questioned by some stakeholder groups. dimension. largely accepted by stakeholder groups.18 dimension. * Material to prepare for the examination is There is some material to prepare for the There is comprehensive material to There is comprehensive material to minimal and it is only accessible to very examination that is accessible to some prepare for the examination that is prepare for the examination that is few students. students. accessible to most students. accessible to all students. 19 * SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the examination There are no courses or workshops on There are no up-to-date courses or There are up-to-date voluntary courses or There are up-to-date compulsory courses examinations available to teachers. workshops on examinations available to workshops on examinations available to or workshops on examinations for teachers.20 teachers. teachers. * Teachers are excluded from all Teachers are involved in very few Teachers are involved in some Teachers are involved in most examination-related tasks. examination-related tasks. examination-related tasks. 21 examination-related tasks. * 22 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 ASSESSMENT QUALITY Degree to which the assessment meets quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring quality There is no technical report or other There is some documentation on the There is a comprehensive technical report There is a comprehensive, high quality documentation. examination, but it is not in a formal but with restricted circulation. technical report available to the general report format. public. 22 * There are no mechanisms in place to This option does not apply to this There are limited systematic mechanisms There are varied and systematic ensure the quality of the examination. dimension. in place to ensure the quality of the mechanisms in place to ensure the quality examination.23 of the examination. * ASSESSMENT QUALITY 2: Ensuring fairness Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the examination process is high. examination process is moderate. examination process is low.24 examination process is marginal. * The examination results lack credibility The examination results are credible for The examination results are credible for This option does not apply to this for all stakeholder groups. some stakeholder groups. all stakeholder groups.25 dimension. * The majority of the students (over 50 A significant proportion of students (10 A small proportion of students (less than All students can take the examination; percent) may not take the examination percent-50 percent) may not take the 10 percent) may not take the examination there are no language, gender or other because of language, gender, or other examination because of language, because of language, gender, or other equivalent barriers.26 equivalent barriers. gender, or other equivalent barriers. equivalent barriers. * (CONTINUED) 23 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 3: Using examination information in a fair way Examination results are not used in a Examination results are used by some Examination results are used by most Examination results are used by all proper way by all stakeholder groups. stakeholder groups in a proper way. stakeholder groups in a proper way. stakeholder groups in a proper way. 27 * Student names and results are public. This option does not apply to this Students’ results are confidential. 28 This option does not apply to this dimension. dimension. * ASSESSMENT QUALITY 4: Ensuring positive consequences of the examination There are no options for students who do There are very limited options for There are some options for students who There is a variety of options for students not perform well on the examination, or students who do not perform well on the do not perform well on the examination. who do not perform well on the 29 students must leave the education examination. examination. system. * There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There is a variety of mechanisms in place monitor the consequences of the dimension. monitor the consequences of the to monitor the consequences of the examination. examination.30 examination. * 24 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 Examinations: Development-level rating justifications 1. The General Education Diploma Examination was first administered in 1980 with the main purposes of student certification for grade or school cycle completion and student selection to university or other higher education institutions. Students in grade 12 are assessed in Islamic Education, Arabic Language, English Language, Physics, Chemistry, Biology, Pure Mathematics, Applied Mathematics, Social Studies (History), Geography, Science and the Environment, Musical Skills, Physical Education, and Fine Arts. 2. The Ministry of Education, through the Regulations of the Committees for General Diplomas Examinations document, authorized the examination in 2010. 3. The Regulations of the Committees for General Diplomas Examinations document is available to the public. 4. The Regulations of the Committees for General Diplomas Examinations document addresses many of the key aspects of the examination. For example, it outlines governance, distribution of power, and responsibilities among key entities, describes the purpose of the examination, states funding sources, outlines procedures to investigate and address security breaches, cheating, or other forms of inappropriate behavior, specifies who can sit for the examination, identifies rules about preparation, and explains the format of the examination questions. However, it does not describe the authorized uses of results, outline procedures for special/disadvantaged students, or explain alignment with curricula and standards. 5. Most stakeholder groups support the examination. Policy makers, universities, and employers all strongly support the examination, while educators, parents, and media support it. Although it is unclear whether teacher unions support or oppose the examination, students, think tanks, and NGOs are neutral to it. 6. Stakeholder groups have made independent attempts to improve the examination. 7. The leadership in charge of the examination generally welcomes efforts to improve it. 8. Regular funding is allocated for the examination by the government. 9. Funding covers all core examination activities including examination design and administration, data analysis and reporting, long- or medium-term planning of program milestones, and staff training. 10. Funding for the General Education Diploma Examination covers research and development. 11. The Department of Tests and Examinations Management is responsible for running the General Education Diploma Examination. The Department of Tests and Examinations Management is part of the Directorate General for Educational Assessment, a division of the Ministry of Education. 12. The Department of Tests and Examinations Management is not accountable to an external board or agency. 25 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 13. Examination results are officially recognized by certification or selection systems in Oman, as well as in countries worldwide. 14. The Department of Tests and Examinations Management has all of the required facilities to carry out the examination, including computers for all technical staff, a secure building and storage facilities, access to adequate computer servers, an ability to backup data, and adequate communication tools. 15. The Department of Tests and Examinations Management has permanent and full-time staff; however, the number of staff available is insufficient to meet the needs of the examination. While specialized senior supervisors and teachers from directorates such as the General Directorate for Curriculum Development and the Directorate General for Human Resources Development are utilized for exam preparation, execution and management of the examination is carried out by specialized committees. Although poor training of test administrators and unclear guidelines for administering the examination have been raised as issues with the performance of the human resources responsible for the examination, issues such as frequent errors in the examination questions and data processing have not been identified. 16. Oman offers a wide range of opportunities that prepare for work on the examination. University graduate programs at Sultan Qaboos University and non graduate training courses on educational measurement and evaluation are in place. In addition, funding for attending international programs, courses, or workshops on educational measurement and evaluation, as well as internships in the examination office, are available. 17. There is a clear understanding that the General Education Diploma Examination measures the national school curriculum guidelines or standards. 18. Stakeholder groups largely accept what is measured by the examination. 19. Comprehensive material to prepare for the examination is available and accessible to all students. Examples of the types of questions that are on the examination are accessible online at the Ministry of Education website and distributed to students during the school year. Additionally, information on how to prepare for the examination, the framework document explaining what is measured on the examination, and the report on the strengths and weaknesses in student performance are all publicly available. 20. Although there are compulsory courses or workshops on the examination available to teachers, these courses are not regularly updated. 21. Teachers are involved in some tasks related to the General Education Diploma Examination. Specifically, teachers are involved in administering and scoring the examination, acting as a judge, and supervising examination procedures. However, they are not involved in selecting or creating examination questions or scoring guides, or resolving inconsistencies between examination scores and school grades. 22. A comprehensive, high quality technical report is available to the general public. 23. Internal and external review or observers are in place to ensure the quality of the examination. Prior to administration, the General Education Diploma Examination questions are reviewed by specialized technical committees and presented to the assessment committee, which is composed of experienced 26 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 specialists. In addition, an assessment of the examination by committees external to the Ministry of Education is underway. However, there are no external certifications or audits, pilots or field testing, or translation verification. 24. Inappropriate behavior surrounding the examination process is low. While there have been occurrences of impersonation (when an individual other than the registered candidate takes the examination), copying from other candidates, using unauthorized materials such as prepared answers and notes, and collusion among candidates via mobile phones or passing of paper, certain mechanisms have been put in place to address such behaviors. For example, disciplinary measures are taken against impersonators, and their exam results are cancelled. Candidates are forbidden from bringing cell phones into examination halls, and if found in violation of this rule, their exam results are also subject to cancellation. Other behaviors, such as the leakage of the content of an examination paper prior to the examination, intimidation of examination supervisors, markers, or officials, the issuing of forged certificates or altering of results information, and provision of external assistance via the supervisor or mobile phone, have not been reported as typically occurring during the examination process. 25. All stakeholder groups perceive the examination results as credible. 26. All students can take the examination; there are no language, gender or other equivalent barriers. 27. All stakeholder groups use examination results in a proper way. 28. Student results are confidential. Only the student and persons with a legitimate, professional interest in the test taker can know the results. 29. There are some options for students who do not perform well on the examination, including retaking the examination (if they fail in three or fewer subjects), repeating the grade, or attending remedial or preparatory courses (offered in the private sector) prior to retaking the examination. 30. There is only one mechanism in place, a permanent oversight committee, to monitor the consequences of the examination. There are no studies that are updated regularly or funding for independent research on the impact of the examination. Additionally, regular focus groups or surveys of key stakeholders, or expert review groups are not in place. 27 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 OMAN National (or System-Level) Large-Scale Assessment (NLSA) 28 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which NLSA activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the NLSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for NLSA No NLSA exercise has taken place. The NLSA has been operating on an The NLSA is a stable program that has This option does not apply to this irregular basis. been operating regularly.1 dimension. There is no policy document pertaining to There is an informal or draft policy There is a formal policy document that * This option does not apply to this NLSA. document that authorizes the NLSA. authorizes the NLSA.2 dimension. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this dimension. the public. public.3 dimension. * There is no plan for NLSA activity. This option does not apply to this There is a general understanding that the There is a written NLSA plan for the dimension. NLSA will take place. coming years. 4 * ENABLING CONTEXT 2: Having strong public engagement for NLSA All stakeholder groups strongly oppose Some stakeholder groups oppose the Most stakeholders groups support the All stakeholder groups support the NLSA. the NLSA or are indifferent to it. NLSA. NLSA. 5 * (CONTINUED) 29 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding for NLSA There is no funding allocated to the NLSA. There is irregular funding allocated to the There is regular funding allocated to the This option does not apply to this NLSA. NLSA. 6 dimension. This option does not apply to this Funding covers some core NLSA activities: Funding covers all core NLSA activities: * This option does not apply to this dimension. design, administration, analysis and design, administration, analysis and dimension. reporting. reporting.7 * This option does not apply to this Funding does not cover research and This option does not apply to this Funding covers research and dimension. development activities. 8 dimension. development activities. * ENABLING CONTEXT 4: Having strong organizational structures for NLSA There is no NLSA office, ad hoc unit or The NLSA office is a temporary agency or The NLSA office is a permanent agency, This option does not apply to this team. group of people. institution or unit.9 dimension. This option does not apply to this Political considerations regularly hamper Political considerations sometimes * Political considerations never hamper dimension. technical considerations. hamper technical considerations. technical considerations. 10 * This option does not apply to this The NLSA office is not accountable to a The NLSA office is accountable to a clearly This option does not apply to this dimension. clearly recognized body.11 recognized body. dimension. * (CONTINUED) 30 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources for NLSA There is no staff allocated for running an The NLSA office is inadequately staffed to The NLSA office is adequately staffed to The NLSA office is adequately staffed to NLSA. effectively carry out the assessment. carry out the NLSA effectively, with carry out the NLSA effectively, with no minimal issues. 12 issues. * The country does not offer opportunities This option does not apply to this The country offers some opportunities to The country offers a wide range of that prepare individuals for work on dimension. prepare individuals for work on the NLSA. opportunities to prepare individuals for 13 NLSA. work on the NLSA. * 31 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 SYSTEM ALIGNMENT Degree to which the NLSA is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning the NLSA with learning goals It is not clear if the NLSA is based on This option does not apply to this The NLSA measures performance against This option does not apply to this curriculum or learning standards. dimension. curriculum or learning standards. 14 dimension. * What the NLSA measures is generally This option does not apply to this What the NLSA measures is questioned by What the NLSA measures is largely questioned by stakeholder groups. dimension. some stakeholder groups. accepted by stakeholder groups. 15 * There are no mechanisms in place to There are ad hoc reviews of the NLSA to There are regular internal reviews of the This option does not apply to this ensure that the NLSA accurately ensure that it measures what it is NLSA to ensure that it measures what it is dimension. measures what it is supposed to measure. intended to measure. intended to measure.16 * SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the NLSA There are no courses or workshops on the There are occasional courses or There are some courses or workshops on There are widely available high quality NLSA. workshops on the NLSA. 17 the NLSA offered on a regular basis. courses or workshops on the NLSA offered on a regular basis. * 32 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 ASSESSMENT QUALITY Degree to which the NLSA meets technical standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of the NLSA No options are offered to include all This option does not apply to this At least one option is offered to include all Different options are offered to include all groups of students in the NLSA. dimension. groups of students in the NLSA. groups of students in the NLSA. 18 There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in * ensure the quality of the NLSA. dimension. ensure the quality of the NLSA. 19 place to ensure the quality of the NLSA. There is no technical report or other There is some documentation about the There is a comprehensive technical report * There is a comprehensive, high quality documentation about the NLSA. technical aspects of the NLSA, but it is not but with restricted circulation. technical report available to the general in a formal report format. public. 20 * ASSESSMENT QUALITY 2: Ensuring effective uses of the NLSA NLSA results are not disseminated. NLSA results are poorly disseminated. NLSA results are disseminated in an This option does not apply to this effective way. 21 dimension. * NLSA information is not used or is used in This option does not apply to this NLSA results are used by some NLSA information is used by all ways inconsistent with the purposes or dimension. stakeholder groups in a way that is stakeholder groups in a way that is the technical characteristics of the consistent with the purposes and consistent with the purposes and assessment. technical characteristics of the technical characteristics of the assessment. assessment. 22 There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in * monitor the consequences of the NLSA. dimension. monitor the consequences of the NLSA. 23 place to monitor the consequences of the NLSA. * 33 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 National (of System-Level) Large Scale Assessment (NLSA): Development-level rating justifications 1. The Cognitive Development Tests have been administered to monitor education quality at the system level, support schools and teachers, and inform policy design, evaluation, and decision making. The assessment program was first carried out in 2007-2008 and is administered to all students in grade 7 to assess Mathematics, Sciences, and Environmental Geographic concepts. 2. The Cognitive Development Tests were authorized by the Ministry of Education's Cognitive Development Manual document in 2009. 3. The Cognitive Development Manual document is publicly available and accessible at the Ministry of Education's website. 4. There is a written NLSA plan for the coming years specifying who will be tested and in which subject areas. The plan is available to, and easily accessible by, the public. 5. Most stakeholder groups support the NLSA. While policy makers and universities strongly support it, educators, students, parents, media, and employers all support the NLSA. It is unclear whether teacher unions, think tanks, and NGOs support or oppose it. 6. There is regular funding for the Cognitive Development Tests allocated by the government. 7. Funding covers all core NLSA activities, including assessment design and administration, data analysis and reporting, and staff training. Funding also covers annual prizes for students exhibiting outstanding performance on the Cognitive Development Tests. However, funding does not cover long- or medium-term planning of program milestones. 8. Funding for the NLSA does not cover research and development activities. 9. The Directorate General of Educational Evaluation is a permanent unit responsible for running the assessment. 10. Political considerations never hamper technical considerations; large-scale assessment results have never been withheld from publication because of political reasons. 11. The Directorate General of Educational Evaluation is not accountable to a clearly recognized body. 12. The Directorate General of Educational Evaluation is adequately staffed with permanent and full-time staff to carry out the NLSA effectively, with minimal issues. However, there have been some issues related to the poor training of test administrators, and unclear instructions and guidelines for administering the assessment. 34 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 13. Some opportunities are offered in Oman to prepare individuals for work on the NLSA. University graduate programs at the Sultan Qaboos University on educational measurement and evaluation, internships or short-term employment in the large-scale assessment office, and funding for attending international programs, courses, or workshops on educational measurement and evaluation, are all available. However, there are no stand-alone university or non university courses or workshops on educational measurement and evaluation available in Oman. 14. The NLSA measures performance against national curriculum or learning standards. 15. Stakeholder groups largely accept what the NLSA measures. 16. Regular internal reviews and regular independent reviews by qualified experts are in place to ensure alignment between the assessment instrument and what it is supposed to measure. 17. Courses or workshops and presentations are occasionally offered on the NLSA. However, most teachers do not have access to live or online courses, and most courses do not provide teachers with resources relevant for use in their classrooms. 18. Different options are offered to include all groups of students in the NLSA. For example, special plans are made to ensure that the large-scale assessment is administered to students in hard-to-reach areas, and the large-scale assessment is offered in the language of instruction for almost all students. However, accommodations or alternative assessments are not provided for students with disabilities. 19. There are some mechanisms in place to ensure the quality of the NLSA. For example, all proctors or administrators are trained according to a protocol, there is a standardized manual for large-scale assessment administrators, all booklets are numbered, scorers are trained to ensure high inter-rater reliability, there is double processing of data, and there are internal reviewers or observers. However, there are no external reviewers or observers, external certification or audits, or double data scoring. In addition, there is no requirement for discrepancies to be recorded on a standard sheet and a pilot is not conducted before the main data collection takes place. 20. A comprehensive, high quality technical report on the NLSA is available to the general public. 21. NLSA results are disseminated within 12 months of the large-scale assessment being administered, and reports with results are made available for all stakeholder groups. The main reports on the results contain information on overall achievement levels and by sub-groups, and on trends over time overall and for sub-groups. However, the main reports on the results do not contain standard errors. There are workshops or presentations for key stakeholders on the results, and the results are published on the Ministry's Educational Portal. Results, however, are not featured in newspapers, magazines, radio, or television, and there is no media briefing organized to discuss results. 22. NLSA information is used by all stakeholder groups in a way that is consistent with the stated purposes and technical characteristics of the assessment. 35 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 23. There are some mechanisms in place to monitor the consequences of the NLSA. For example, a permanent oversight committee, themed conferences that provide a forum to discuss research and other data on the consequences of the large-scale assessment, and expert review groups are in place. However, funding for independent research on the impact of the large-scale assessment or regular focus groups and surveys of key stakeholders are not in place. OMAN International Large-Scale Assessment (ILSA) 36 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which ILSA takes place in a country or system and the extent to which that framework is conducive to, or supportive of, ILSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for ILSA The country/system has not participated This option does not apply to this The country/system has participated in at The country/system has participated in in an ILSA in the last 10 years. dimension. least one ILSA in the last 10 years. two or more ILSA in the last 10 years.1 * The country/system has not taken This option does not apply to this The country/system has taken concrete This option does not apply to this concrete steps to participate in an ILSA in dimension. steps to participate in at least one ILSA in dimension. the next 5 years. the next 5 years. 2 * There is no policy document that There is an informal or draft policy There is a formal policy document that This option does not apply to this addresses participation in ILSA. document that addresses participation in addresses participation in ILSA. 3 dimension. ILSA. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this dimension. the public. public.4 dimension. * ENABLING CONTEXT 2: Having regular funding for ILSA There is no funding for participation in There is funding from loans or external There is regular funding allocated at There is regular funding approved by law, ILSA. donors. discretion. 5 decree or norm. * This option does not apply to this Funding covers some core activities of the Funding covers all core activities of the This option does not apply to this dimension. ILSA. ILSA. 6 dimension. * Funding does not cover research and This option does not apply to this This option does not apply to this Funding covers research and development activities. 7 dimension. dimension. development activities. * 37 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 (CONTINUED) LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having effective human resources for ILSA There is no team or national/system There is a team or national/system There is a team and national/system This option does not apply to this coordinator to carry out the ILSA coordinator to carry out the ILSA coordinator to carry out the ILSA dimension. activities. activities. activities. 8 * This option does not apply to this The national/system coordinator or other The national/system coordinator is fluent This option does not apply to this dimension. designated team member may not be in the language of the assessment. 9 dimension. fluent in the language of the assessment. * This option does not apply to this The ILSA office is inadequately staffed or The ILSA office is adequately staffed or The ILSA office is adequately staffed and dimension. trained to carry out the assessment trained to carry out the ILSA effectively, trained to carry out the ILSA effectively, effectively. with minimal issues. 10 with no issues. * 38 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 SYSTEM ALIGNMENT Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Providing opportunities to learn about ILSA The ILSA team has not attended The ILSA team attended some The ILSA team attended all international This option does not apply to this international workshops or meetings. international workshops or meetings. 11 workshops or meetings. dimension. The country/system offers no This option does not apply to this * The country/system offers some The country/system offers a wide range opportunities to learn about ILSA. dimension. opportunities to learn about ILSA. of opportunities to learn about ILSA. 12 This option does not apply to this This option does not apply to this Opportunities to learn about ILSA are Opportunities to learn about ILSA are * dimension. dimension. available to the country's/system's ILSA available to a wide audience, in addition team members only. to the country's/system's ILSA team members. 13 * 39 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 ASSESSMENT QUALITY Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of ILSA Data from the ILSA has not been The country/system met sufficient The country/system met all technical This option does not apply to this published. standards to have its data presented standards required to have its data dimension. beneath the main display of the presented in the main displays of the international report or in an annex. international report. 14 * The country/system has not contributed This option does not apply to this This option does not apply to this The country/system has contributed new new knowledge on ILSA. dimension. dimension. knowledge on ILSA. 15 * ASSESSMENT QUALITY 2: Ensuring effective uses of ILSA If any, country/system-specific results Country/system-specific results and Country/system-specific results and Country/system-specific results and and information are not disseminated in information are disseminated irregularly information are regularly disseminated in information are regularly and widely the country/system. in the country/system. the country/system. disseminated in the country/system. 16 Products to provide feedback to schools This option does not apply to this Products to provide feedback to schools Products to provide feedback to schools * and educators about the ILSA results are dimension. and educators about the ILSA results are and educators about ILSA results are not made available. sometimes made available. systematically made available. 17 * There is no media coverage of the ILSA There is limited media coverage of the There is some media coverage of the ILSA There is wide media coverage of the ILSA results. ILSA results. results.18 results. * If any, country/system-specific results Results from the ILSA are used in a limited Results from the ILSA are used in some Results from the ILSA are used in a variety and information from the ILSA are not way to inform decision making in the ways to inform decision making in the of ways to inform decision making in the used to inform decision making in the country/system. country/system.19 country/system. country/system. * It is not clear that decisions based on ILSA This option does not apply to this This option does not apply to this Decisions based on the ILSA results have results have had a positive impact on dimension. dimension. had a positive impact on students' students' achievement levels. 20 achievement levels. * 40 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 International Large Scale Assessment (ILSA): Development-level rating justifications 1. Oman participated in TIMSS in 2007 and 2011, and PIRLS in 2011. 2. Oman has taken concrete steps to participate in PIRLS in 2016 and TIMSS in 2015. 3. The formal policy document, Ministerial Decision Number 140/2009, authorized by the Ministry of Education in 2009, addresses participation in ILSAs and the preparation and publication of relevant national reports. 4. The document, Ministerial Decision Number 140/2009, is available to the public. 5. Regular funding for international assessment participation is allocated at the discretion of the government. 6. Funding covers all core activities of the ILSAs, including international participation fees, administration of the assessment exercise in Oman, processing and analyzing data collected from implementation of the assessment exercise, reporting and disseminating the assessment results in Oman, and attendance at international expert meetings for the assessment exercise. 7. Funding for the ILSAs does not cover research and development activities. 8. There is a team, the International Studies Program, and a national coordinator to carry out the ILSA activities. 9. The ILSA national coordinator is fluent in the language of the assessment. 10. The ILSA office is adequately staffed to carry out the ILSAs effectively, with minimal issues. Although the team is sufficiently staffed, has previous experience working on international assessments, and has attended some international workshops or meetings, it does not have the necessary training or experience to carry out the required assessment activities effectively. 11. The ILSA team has attended some international workshops or meetings. 12. Oman offers a wide range of opportunities to learn about ILSAs. There are workshops or meetings on using international assessment databases, funding for attending international workshops or training on international assessments, and online courses on international assessments. However, there are no university courses on the topic of international assessments. 41 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 13. Opportunities to learn about ILSAs are available to a wide audience besides Oman's ILSA team members. Educational Supervisors in the Educational Governorates, teachers, and specialists from the Curricula and Educational Assessment Departments can benefit from these opportunities; however, they are not available to university students studying assessment, or to professionals and university staff interested in assessment. 14. Oman has met all technical standards required to have its data presented in the main displays of the international report. 15. Oman has contributed new knowledge on ILSAs. 16. Country specific results and information are regularly and widely disseminated in Oman. For example, a national report was made available online, copies of the national and international reports were distributed to key stakeholders, and Oman's results were communicated through a press release. Brochures and PowerPoint presentations with Oman's results were made available online, and products providing feedback to the schools or educators about the results were also made available. In addition, a seminar was held to present and discuss the results of participating in TIMSS and PIRLS 2011 and included stakeholders from the Sultanate. 17. Products providing feedback to schools and educators about ILSA results are systematically made available. 18. In addition to editorials or columns commenting on the international assessment results, the Ministry organized presentations on the radio and television, and interviews, to provide information on the results in TIMSS and PIRLS in 2011. 19. ILSA results are used in some ways to inform decision making in Oman. For example, results have been used in tracking the impact of reforms on student achievement levels, and in informing curriculum improvement, teacher training programs, and other assessment activities in the system. However, results have not been used to inform resource allocation. 20. It is not clear that decisions based on ILSA results have had a positive impact on students' achievement levels, as there was no improvement in student achievement between TIMSS 2007 and 2011. 42 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 Acknowledgements References This report, part of a 16-country benchmarking exercise Clarke, M. 2012. “What Matters Most for Student in the Middle East & North Africa and Africa regions, was Assessment Systems: A Framework Paper.” READ/SABER prepared by the World Bank SABER-Student Assessment Working Paper Series. Washington, DC: World Bank. team, in partnership with the Arab League Educational, Cultural, and Scientific Organization (ALECSO) and the Rassekh, Shapour. 2004. “Education as a Motor for United Nations Educational, Scientific, and Cultural Development: Recent Education Reforms in Oman with Organization (UNESCO), which led data collection efforts. Particular Reference to the Status of Women and Girls.” This effort is part of the Arab Regional Agenda for Geneva: UNESCO-IBE. Improving Education Quality (ARAIEQ), led by ALECSO in partnership with the World Bank. It benefited from Sultanate of Oman Ministry of Education and the World feedback and review from Shahram Paksima, Education Bank. Education in Oman: The Drive for Quality. 2012. Specialist and Task Team Leader for education projects Muscat: Sultanate of Oman. in Oman in the World Bank’s Education Global Practice, as well as comments received during a national United Nations Educational, Scientific, and Cultural validation workshop held in Oman. Organization – Institute for Statistics. 2011. UIS Statistics in Brief. “Education Profile: Oman.” Montreal: UNESCO- UIS. World Bank. 2012. Oman Country Indicator Data. Washington, DC: World Bank. Data retrieved from http://data.worldbank.org/ on October 15, 2013. 43 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS OMAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2013 www.worldbank.org/education/saber The Systems Approach for Better Education Results (SABER) initiative produces comparative data and knowledge on education policies and institutions, with the aim of helping countries systematically strengthen their education systems. SABER evaluates the quality of education policies against evidence-based global standards, using new diagnostic tools and detailed policy data. The SABER country reports give all parties with a stake in educational results—from administrators, teachers, and parents to policymakers and business people—an accessible, objective snapshot showing how well the policies of their country's education system are oriented toward ensuring that all children and youth learn. This report focuses specifically on policies in the area of student assessment. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. 2 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS