SABER IN ACTION SABER TOOLS AS A FRAMEWORK FOR EDUCATION SYSTEM ASSESSMENT JUNE | 2016 The World Bank Group’s Systems Approach for Better Education Results (SABER) initiative produces data on education system policies and institutions, analyzes and evaluates their quality, and provides decision makers and stakeholders with an assessment that fosters structured and informed policy dialogue on how to most effectively strengthen education systems. During its tenure, SABER has grown rapidly to become a key component in the design of education programming across the globe, providing countries with quality data and information. By generating a broader knowledge base on education systems worldwide, these tools are helping reform education at the country level while also building a high-quality knowledge base for education reforms at the global level. SABER materials, including framework and other background documents, user manuals, questionnaires, rubrics, as well as country reports and analytical papers, are made publicly available to encourage widespread use of SABER tools and publications, helping to strengthen education policy development and reform. The openness of SABER methodology and data also enables stakeholders to adapt SABER tools in a way that is relevant to their individual needs. The World Bank Group’s education team in China, in collaboration with researchers at the Shanghai Normal University, recently published a report “How Shanghai Does It: Insights and Lessons from the Highest-Ranking Education System in the World”, which uses SABER as a framework to organize and benchmark education policies across several areas of the education system. The findings of the report and the related analyses were highlighted during the Global Conference on Equity and Excellence in Basic Education in Shanghai in May 2016. Systems Approach: From Policy to Learning The China education team collected information and data on policies and institutions across multiple SABER domains, including: teachers, school finance, school autonomy and accountability, and student assessment. Too often education systems research examines individual inputs without understanding how resources, accountability, governance, information, and delivery actually function together as a system. The application of multiple SABER domains to gather policy intent information enabled the team to develop a more comprehensive analysis of the education system in Shanghai. Saber.worldbank.org P1 SABER IN ACTION: JUNE SABER TOOLS AS A FRAMEWORK FOR EDUCATION SYSTEM ASSESSMENT 2016 The China education team’s work recognizes the reality that policy intent, which is measured by the SABER tools, is not always implemented as expected on the ground level due to various constraints, including financial constraints, political climate, shifting priorities, etc. The China education team also collected implementation data to determine how effectively written policies were actually implemented on the ground in Shanghai. The team designed questionnaires and worked with Shanghai Normal University to develop and administer a survey to 153 school principals to assess how education practices are aligned with policy goals. The 153 schools chosen were those that participated in both PISA 2009 and PISA 2012. Given that PISA uses a random, representative sampling of schools to administer its test, the schools chosen can be said to be reflective of education practices in Shanghai. PHOTO: LIANG QIANG / WORLD BANK To connect policy intent and implementation with student outcomes, the China education team draws on PISA 2009 and PISA 2012 cross-sectional data to investigate which school-level characteristics (policy and practice) are associated with specific learning outcomes. The report finds that Shanghai has a high degree of coherence between policy intent and implementation, and that the policy environment in Shanghai is constantly evolving. The report’s analysis of PISA scores also offers insight into how specific policies are associated with learning outcomes. saber.worldbank.org P2 SABER IN ACTION: JUNE SABER TOOLS AS A FRAMEWORK FOR EDUCATION SYSTEM ASSESSMENT 2016 Benefits and Risks of Adapting SABER Tools As previously mentioned, SABER tools were intentionally developed to allow researchers and policy-makers to use them for their own analyses and studies. The China education team’s use of SABER tools across four domains as a framework for their analysis proved to be an effective method for evaluating Shanghai’s education system to better understand its successes in ensuring student learning. The China education team did not implement fully each of the four SABER domain questionnaires, instead choosing to adapt each of the questionnaires based on their expert knowledge of Shanghai’s education system and the general policy- making process in China, and Shanghai specifically. The benefits of allowing for tool and scoring adaptations based on local knowledge and expertise is best evidenced in the report’s analysis of Shanghai’s school autonomy and accountability. The analytical framework for SABER-School Autonomy and Accountability emphasizes the importance of localized stakeholder input in the management of schools and the formalized involvement of parents via school councils, etc. The Shanghai report found that community and parent involvement in schools is not very common, and where parent councils do exist, their roles are largely limited to nonteaching aspects of schools. However, the report finds that in places where educational institutions are strong—like Shanghai—trust can replace direct involvement as the main element of accountability. Parents trust schools in Shanghai because the empirical evidence, e.g., PISA scores, indicates strong results. The China education team’s understanding of the on-the-ground realities in Shanghai help provide context for SABER results and analysis. Of course, there are risks associated with adapting tools and analysis. Most notably, comparability of data across countries is no longer valid when tools are adapted. Because the Shanghai report only examines Shanghai’s education system, this was not a concern in this particular instance. Another concern is that when tools are adjusted based on a researcher’s knowledge of a particular system there is an inherent possibility of bias. The benefit of utilizing the SABER tools as they were originally created is that they are designed to provide an objective assessment of education policy; when the tools are changed, that objectivity can be compromised. Next Up: SABER Implementation Tools In their analysis, the China education team recognized the importance of looking beyond policy intent to evaluating policy implementation. This has long been a recognized next step for SABER. The SABER-School Autonomy and Accountability team has produced a second, more in-depth version of their questionnaire that looks beyond policy intent to begin gauging implementation. Similarly, the SABER-Engaging the Private Sector team has developed an “in-depth” tool that looks at implementation as well as intent. Other domain teams are in the process of developing updated tools that assess policy at the implementation level. Additionally, efforts are being made to explore the links between SABER policy intent findings and policy implementation, service delivery, and/or learning data through a pilot that is reviewing the Service Delivery Indicator (SDI) survey saber.worldbank.org P3 SABER IN ACTION: JUNE SABER TOOLS AS A FRAMEWORK FOR EDUCATION SYSTEM ASSESSMENT 2016 methodology to include indicators at district and school level focused on the implementation of policy. This pilot is intended to utilize both this SDI ‘Plus’ survey simultaneously with SABER domain analyses, thus resulting in datasets at both ends, from policy intent to classroom reality. “How Shanghai Does It: Insights and Lessons from the Highest-Ranking Education System in the World” can be found at: https://openknowledge.worldbank.org/bitstream/handle/10986/24000/9781464807909.pdf. PHOTO: LIANG QIANG / WORLD BANK saber.worldbank.org P4